Image detection method and device, storage medium, and electronic device

Abstract

The present invention discloses an image detection method and device, a storage medium, and an electronic device, wherein the method comprises: acquiring a target image, and inputting the target image into a first network model, wherein the first network model is a network model obtained by cropping the channels of the second network model according to the scaling coefficient of the second batch normalization layer of the second network model during the training of the second network model; detecting the target image through the first network model to determine the target object in the target image, that is, detecting the target image through the cropped network model. The above technical solution is adopted to solve the problem of low efficiency of image detection due to the large number of channels of the network model in the related art.

Description

Technical Field

[0001] The present invention relates to the field of communications, and in particular to an image detection method and device, a storage medium, and an electronic device. Background Art

[0002] Currently, there are many excellent human-machine non-detection algorithms developed based on deep learning, but they ultimately need to be deployed in actual scenarios and run on embedded and other terminal devices. The CPU processing power and memory resources on embedded and other terminal devices are weak, which requires real-time processing capabilities for human-machine non-detection algorithms. Currently, there are many excellent large-scale human-machine non-detection algorithms applied to cameras, but the network model cannot be successfully implemented due to the large amount of calculation and large size of the detection algorithm of the network model. In addition, due to the excessive number of channels in the network model, the efficiency of image detection is low.

[0003] In the related art, in order to reduce the computational complexity and volume of the detection algorithm of the network model, the network model is usually compressed. For example, the network model to be compressed is obtained, and the network model includes at least two filters. The central filter and the non-central filter are determined by calculating the vector distance between any two filters, and the non-central filter is cropped. Since the filter is selected arbitrarily, this method may crop all important filter parameters; or, the cropping order of the network is determined according to the size of the convolution kernel parameters of the network model. This cropping strategy is not suitable for general target detection algorithms, and the cropping effect is poor. Moreover, the cropping effect can generally only be reflected when the network is deep enough.

[0004] In related technologies, there is still a problem that the efficiency of image detection is low due to the large number of channels in the network model, and no effective solution has been proposed yet. Summary of the invention

[0005] Embodiments of the present invention provide an image detection method and device, a storage medium, and an electronic device to at least solve the problem in the related art that the efficiency of image detection is low due to the large number of channels in the network model.

[0006] According to one embodiment of the present invention, there is provided an image detection method, comprising: acquiring a target image, and inputting the target image into a first network model, wherein the first network model is a network model obtained after cropping the channels of the second network model according to the scaling coefficient of the second batch normalization layer of the second network model during the training process of the second network model; and detecting the target image through the first network model to determine the target object in the target image.

[0007] In an exemplary embodiment, before acquiring a target image and inputting the target image into the first network model, the method further includes: determining the loss function of the second network model based on an original training loss function and a penalty term of the second network model, wherein the original training loss function is used to indicate a function determined by a discriminator of the second network model, and the penalty term includes a scaling factor of the second batch of normalized layers; adjusting the scaling factor of the second batch of normalized layers according to the loss function; and when it is determined that the absolute value of the adjusted scaling factor is less than a preset threshold, cropping the channel corresponding to the adjusted scaling factor in the second network model to obtain the first network model.

[0008] In an exemplary embodiment, before determining the loss function of the second network model based on the original training loss function and the penalty term of the second network model, the method also includes: obtaining a balance factor of the penalty term and a set of scaling coefficients of the second batch of normalization layers; respectively calculating the norm of any scaling coefficient in the set of scaling coefficients of the second batch of normalization layers to obtain multiple norms; and determining the penalty term based on the balance factor of the penalty term and the multiple norms.

[0009] In an exemplary embodiment, according to the original training loss function and penalty term of the second network model, determining the loss function of the second network model includes: determining the loss function of the second network model by the following formula: L = L loss +λ p ∑ γ∈Γ ||γ||1, where L loss is the original training loss function of the second network model, λ p ∑ γ∈Γ ||γ||1 is the penalty term of the second network model, λ p is the balancing factor of the penalty term, Γ is the set of scaling coefficients of the second batch of normalized layers, and ||γ||1 is the norm of the scaling coefficients of the second batch of normalized layers.

[0010] In an exemplary embodiment, when it is determined that the absolute value of the adjusted scaling factor is less than a preset threshold, after cropping the channels corresponding to the adjusted scaling factor in the second network model to obtain the first network model, the method further includes: combining the operations of the first batch of normalization layers and the first convolutional layer of the first network model; calculating the target image input into the first network model through the combined first batch of normalization layers and the first convolutional layer to obtain a feature image corresponding to the target image.

[0011] In an exemplary embodiment, before combining the operations of the first batch of normalized layers and the first convolutional layer of the first network model, the method further includes: converting the first batch of normalized layers into target convolutional layers by the following formula: Wherein, F is the feature vector of the target image, is the result of normalization of the feature vector F, b is used to indicate the first normalization layer, c is used to indicate the number of channels, and W BN is the convolution kernel of the target convolution layer, b BN is the bias term of the target convolutional layer, i is the horizontal coordinate of the pixel corresponding to the feature vector, and j is the vertical coordinate of the pixel corresponding to the feature vector.

[0012] In an exemplary embodiment, after converting the batch normalization layer into a target convolution layer, the method further includes: obtaining the convolution kernel of the target convolution layer, the bias item of the target convolution layer, the convolution kernel of the first convolution layer, the bias item of the first convolution layer, and the feature vector of the pixel points of the target image; based on the convolution kernel of the target convolution layer, the bias item of the target convolution layer, the convolution kernel of the first convolution layer, the bias item of the first convolution layer, and the feature vector of the pixel points of the target image, combining the operations of the first batch normalization layer and the first convolution layer of the first network model.

[0013] In an exemplary embodiment, obtaining the convolution kernel of the target convolution layer, the bias item of the target convolution layer, the convolution kernel of the first convolution layer, the bias item of the first convolution layer, and the feature vector of the pixel point of the target image includes: combining the operations of the first batch of normalization layers and the first convolution layer through the following formula: Among them, W BN W conv and W BN b conv +b BN is the parameters and bias of the combined first batch of normalization layers and the first convolutional layer, W BN is the convolution kernel of the target convolution layer, b BN is the bias term of the target convolutional layer, W conv is the convolution kernel of the first convolutional layer and b conv is the bias term of the first convolutional layer, f ij The feature vector of the pixel point of the target image, The result of normalizing and convolving the feature vectors of the pixels of the target image.

[0014] According to another embodiment of the present invention, an image detection device is also provided, including: an acquisition module, used to acquire a target image and input the target image into a first network model, wherein the first network model is a network model obtained after cropping the channels of the second network model according to the scaling coefficient of the second batch normalization layer of the second network model during the training process of the second network model; and a detection module, used to detect the target image through the first network model to determine the target object in the target image.

[0015] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0016] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0017] Through the present invention, a target image is acquired and input into a first network model, wherein the first network model is a network model obtained after the channels of the second network model are cropped according to the scaling coefficient of the second batch normalization layer of the second network model during the training process of the second network model; the target image is detected by the first network model to determine the target object in the target image, that is, the target image is detected by the cropped network model. The above technical solution solves the problem in the related art that the efficiency of image detection is low due to the large number of channels of the network model. When the network model is trained, the network model can automatically select the channels that need to be cropped, thereby realizing a compressed network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 is a hardware structure block diagram of a mobile terminal of an image detection method according to an embodiment of the present invention;

[0020] Figure 2 is a flow chart of an image detection method according to an embodiment of the present invention;

[0021] Figure 3 is a flow chart of an image detection method according to an optional embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of an image detection method according to an optional embodiment of the present invention;

[0023] Figure 5 is a structural block diagram of an image detection device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0026] The method embodiment provided in the embodiment of the present application can be executed in a mobile terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of an image detection method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. In an exemplary embodiment, the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Equivalent functions or comparisons shown Figure 1 A different configuration with more features is shown.

[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image detection method of the mobile terminal in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] In this embodiment, an image detection method is provided, which is applied to the above mobile terminal. Figure 2 : is a flow chart of an image detection method according to an embodiment of the present invention, and the flow chart includes the following steps:

[0030] Step S202, obtaining a target image, and inputting the target image into a first network model, wherein the first network model is a network model obtained by clipping channels of the second network model according to a scaling factor of a second batch of normalization layers of the second network model during a training process of the second network model;

[0031] Step S204: Detect the target image using the first network model to determine the target object in the target image.

[0032] Through the above steps, a target image is obtained, and the target image is input into the first network model, wherein the first network model is a network model obtained after cropping the channels of the second network model according to the scaling coefficient of the second batch normalization layer of the second network model during the training process of the second network model; the target image is detected by the first network model to determine the target object in the target image, that is, the target image is detected by the cropped network model, which solves the problem in the related art that the efficiency of image detection is low due to the large number of channels of the network model. When the network model is trained, the network model can automatically select the channels that need to be cropped, thereby realizing a compressed network model.

[0033] Through the above steps, it can be known that the scaling coefficients of the second batch of normalized layers of the second network model are obtained, the channels in the second network model are cropped according to the scaling coefficients, and the network model after the channel cropping is the first network model. It can be understood that the first network model is the network model after the channel cropping of the second network model. The target image is detected through the first network model to obtain the target object in the target image at a faster speed.

[0034] In an exemplary embodiment, the first network model is obtained in the following manner: based on the original training loss function and penalty term of the second network model, the loss function of the second network model is determined, wherein the original training loss function is used to indicate the function determined by the discriminator of the network model, and the penalty term includes the scaling coefficient of the second batch of normalized layers; the scaling coefficient of the second batch of normalized layers is adjusted according to the loss function; when it is determined that the absolute value of the adjusted scaling coefficient is less than a preset threshold, the channel corresponding to the adjusted scaling coefficient in the network model is cropped to obtain the first network model.

[0035] That is to say, the original training loss function and penalty term of the second network model are determined, and the loss function of the second network model and the scaling coefficient of the second batch of normalization layers are determined according to the determined original training loss function and penalty term. By determining the scaling coefficient that is less than the preset threshold, the channels corresponding to the adjusted scaling coefficients in the second network model are pruned. The above technical solution is used to solve the problems in the related technology, such as the large amount of calculation and large size of the detection algorithm of the network model, which leads to a low landing rate of the network model. When the network model is trained, the second network model can automatically select the channels that need to be pruned.

[0036] In an exemplary embodiment, based on the original training loss function and penalty term of the second network model, before determining the loss function of the second network model, a balance factor of the penalty term and a set of scaling coefficients of the second batch of normalized layers are obtained; the norm of any scaling coefficient in the set of scaling coefficients of the second batch of normalized layers is calculated respectively to obtain multiple norms; and the penalty term is determined based on the balance factor of the penalty term and the multiple norms.

[0037] That is to say, the backbone network of the second network model is structurally optimized. Specifically, the scaling coefficient γ of the second batch of normalized layers is first placed behind all channels of each network layer of the second network model, and the coefficient is directly multiplied by the output of the channel to obtain a set of manually set balance factors and scaling coefficients of the second batch of normalized layers; the norm of each scaling coefficient in the set of scaling coefficients is calculated, multiple norms are added, and multiplied with the balance factor to determine the penalty term, wherein the norm is preferably the L1 norm, which is not limited in the embodiment of the present invention.

[0038] In an exemplary embodiment, according to the original training loss function and penalty term of the second network model, determining the loss function of the second network model includes: determining the loss function of the second network model by the following formula: L = L loss +λ p Σ γ∈Γ ||γ||1, where L loss is the original training loss function of the second network model, λ p ∑ γ∈Г ||γ||1 is the penalty term of the second network model, λ p is the balancing factor of the penalty term, Γ is the set of scaling coefficients of the second batch of normalized layers, and ||γ||1 is the norm of the scaling coefficients of the second batch of normalized layers.

[0039] Specifically, an algorithm is trained to calculate the loss function of the second network model as follows: p ,Γ,||γ||1,L loss Calculate the loss function of the second network model: L = L loss +λ p ∑ γ∈Г ||γ||1, where L loss is the original training loss function of the second network model, λ p is the balance factor of the penalty term, which is a fixed value set by humans; Γgamma is the set of scaling coefficients of the second batch of normalized layers participating in channel clipping, ||γ||1 is the calculation of the L1 norm of the scaling system of the second batch of normalized layers, where λ p This is a fixed value designed according to specific scenarios.

[0040] Furthermore, it is necessary to fine-tune the first network model after the second network model is pruned, that is, the above steps S202 to S204 are executed in a loop to determine the first network model with the best performance among the multiple first network models obtained.

[0041] In an exemplary embodiment, a target scaling factor whose absolute value of the scaling factor is less than a preset threshold is determined, and after cropping the channel corresponding to the target scaling factor in the network model, the operations of the first batch of normalization layers and the first convolutional layer of the first network model are combined; the target image input into the first network model is calculated through the combined first batch of normalization layers and the first convolutional layer to obtain a feature image corresponding to the target image.

[0042] That is, the first batch of normalization layers and the first convolutional layers in the backbone network of the first network model are fused, and the target image input into the first network model is normalized and convolved through the fused first batch of normalization layers and the first convolutional layers to obtain a feature image corresponding to the normalized and convolved target image.

[0043] It should be noted that before the target image is input into the first network model, the resolution of the target image is reduced in a step-by-step manner, and then the accuracy and recall rate of the target detection algorithm of the target images of each resolution are statistically analyzed, and the optimal image resolution that meets the accuracy requirement is selected as the target resolution of the target image input into the first network model.

[0044] In an exemplary embodiment, before combining the operations of the first batch of normalized layers and the first convolutional layer of the first network model, the first batch of normalized layers are converted into target convolutional layers by the following formula: Wherein, F is the feature vector of the target image, is the result of normalization of the feature vector F, b is used to indicate the normalization layer, c is used to indicate the number of channels, and W BN is the convolution kernel of the target convolution layer, b BN is the bias term of the target convolutional layer, i is the horizontal coordinate of the pixel corresponding to the feature vector, and j is the vertical coordinate of the pixel corresponding to the feature vector.

[0045] Specifically, a sample of the first normalization layer is normalized; for a certain layer feature x1, x2, x3…, x n , which is calculated as follows:

[0046]

[0047] Among them, μ represents the sample mean of the first batch of normalization layers, σ 2represents the sample variance on the current normalization layer, and the mean and means of different channels are calculated separately. ε is a new minimum value used to solve the problem of zero division. β and γ are trainable parameters, representing the scaling system and offset coefficient respectively. During the training phase, μ and σ 2 It can be calculated by the current first batch of normalization layers; in the test phase, μ and σ 2 The exponential sliding average calculated during the training process is used. In the test phase, the parameters in the first network model, such as β and γ, are not iteratively updated, and the operations of the first batch of normalization layers are combined with the operations of the first convolutional layer before it; here, the parameters of the first batch of normalization layers are represented as μ c , γ c , β c , where c represents the cth channel.

[0048] The first batch of normalized layers are converted to target convolutional layers. For the coordinates (i, j) of the pixel points corresponding to the feature vector, the operation of converting the first batch of normalized layers to the target convolutional layers can be expressed by the following formula:

[0049]

[0050]

[0051]

[0052] in, is the result of normalization of feature image F. Formula The operation process can be seen as a 1x1 convolution, where W BN is the convolution kernel of the target convolution layer, b BN is the bias term of the target convolutional layer.

[0053] In an exemplary embodiment, after converting the first batch of normalized layers into target convolutional layers, the method further includes: obtaining the convolution kernel of the target convolutional layer, the bias item of the target convolutional layer, the convolution kernel of the first convolutional layer, the bias item of the first convolutional layer, and the feature vector of the pixel points of the target image; based on the convolution kernel of the target convolutional layer, the bias item of the target convolutional layer, the convolution kernel of the first convolutional layer, the bias item of the first convolutional layer, and the feature vector of the pixel points of the target image, combining the operations of the first batch of normalized layers and the first convolutional layer.

[0054] That is, based on the convolution kernel of the target convolution layer, the bias item of the target convolution layer, the convolution kernel of the first convolution layer, the bias item of the first convolution layer and the feature vector of the pixel points of the target image, the operations of the first batch of normalization layers and the first convolution layer are combined.

[0055] Specifically, the operations of the first normalization layer and the first convolutional layer are combined using the following formula: Among them, W BN W conv and W BN b conv +b BN is the parameters and bias of the combined first batch of normalization layers and the first convolutional layer, W BN is the convolution kernel of the target convolution layer, b BN is the bias term of the target convolutional layer, W conv is the convolution kernel of the first convolutional layer and b conv is the bias term of the first convolutional layer, f ij The feature vector of the pixel point of the target image, The result of normalizing and convolving the feature vectors of the pixels of the target image.

[0056] The combined formula is The normalized and convolved feature images of the target image are calculated through the combined calculation formula, thereby reducing the amount of calculation of the algorithm in the first network model.

[0057] In order to better understand the process of the above-mentioned image detection method, the process of the above-mentioned image detection method is described below in combination with optional embodiments, but it is not used to limit the technical solution of the embodiment of the present invention.

[0058] Figure 3 is a flow chart of an image detection method according to an optional embodiment of the present invention, such as Figure 3 As shown, the process of the image detection method can be implemented by the following steps:

[0059] Step S301: obtaining a full-image human-machine non-image (equivalent to the target image in the above embodiment) (which can be understood as images including pedestrians, motor vehicles, and non-motor vehicles) from an image acquisition device, reducing the image resolution of the target image in a step-by-step manner, and then counting the accuracy and recall rate of the human-machine non-target detection algorithm when the target images of each resolution are input into a neural network model (equivalent to the second network model in the above embodiment), and selecting the optimal input image resolution that meets the accuracy requirement;

[0060] Step S302: Structural optimization is performed on the backbone network of the neural network model. Specifically, the scaling coefficient γ of the batch normalization layer is first placed after all channels of each layer of the network, and the coefficient is directly multiplied by the output of the channel;

[0061] Step S303: Calculate the loss function of the neural network model as follows:

[0062] L=L loss +λ p ∑ γ∈Γ ||γ||1; among them, the first term L loss is the original training loss function of the network, and the second term is the newly added penalty term, where λ p is the balancing factor of the penalty term, a fixed value set manually; Γgamma is the set of scaling coefficients of the batch normalization layer involved in channel pruning, and ||γ||1 is the calculation of the L1 norm of the scaling system of the batch normalization layer;

[0063] Step S304: jointly training the network weights and all scaling factors;

[0064] Step S305: Crop the corresponding channels whose absolute values ​​of the scaling coefficients are less than the set threshold, such as Figure 4 As shown, Figure 4 for Figure 4 is a schematic diagram of an image detection method according to an optional embodiment of the present invention;

[0065] Step S306: fine-tune the pruned network again. Repeat steps S302-S305 to obtain a pruned network model with optimal performance;

[0066] Step S307: Fuse the batch normalization layer and the convolution layer in the backbone network.

[0067] Specifically, the samples of a batch normalization layer are normalized;

[0068] Step S3071: For a certain layer of features x1, x2, x3…, x n , which is calculated as follows:

[0069]

[0070] Among them, μ represents the sample mean on the current batch normalization layer, σ 2 Represents the sample variance on the current normalization layer, and the mean and method of different channels are calculated separately. ε is a minimum value added to solve the problem of zero division. β and γ are trainable parameters, representing the scaling system and offset coefficient respectively.

[0071] Among them, during the training phase, μ and σ 2It can be calculated by the current batch normalization layer; in the test phase, μ and σ 2 The exponential sliding average calculated during the training process is used. In the test phase, the parameters in the network are not iteratively updated, and the operations of the batch normalization layer are combined with the operations of the previous convolutional layer; here the batch normalization layer parameters are represented as μ c , γ c , β c , where c represents the cth channel.

[0072] Step S3072: Convert the batch normalization layer to a convolutional layer. For the coordinate (i, j), the operation of the batch normalization layer can be expressed by the following formula:

[0073]

[0074]

[0075]

[0076] in is the result of normalization of feature map F. Formula The operation process can be seen as a 1x1 convolution, where W BN is the convolution kernel, b BN is the bias term;

[0077] Step S3073: Combine the batch normalization layer operation and the convolution layer operation, here denoted by W conv and b conv is the parameter of the convolution layer, the point (i, j) vector f ij The corresponding fused input is:

[0078]

[0079] Among them, W BN W conv and W BN b conv +b BN are the parameters and bias items of the convolutional layer after fusion.

[0080] The present invention finds the optimal image resolution by reducing the image resolution of the target image in a step-by-step manner; the backbone network is structurally optimized by fine-tuning the network channels. The batch normalization layer and the convolution layer are fused in the forward reasoning stage. Through the above three methods, the above problems in the related technology are solved, such as the large amount of computation of the network model detection algorithm and the low network model landing rate due to the large volume. When training the network model, the network model can automatically select the channels that need to be pruned, and the computation of the network model detection algorithm is reduced.

[0081] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0082] In the present embodiment, an image detection device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0083] Figure 5 is a structural block diagram of an image detection device according to an embodiment of the present invention; Figure 5 As shown, including:

[0084] An acquisition module 52 is used to acquire a target image and input the target image into a first network model, wherein the first network model is a network model obtained by clipping the channels of the second network model according to the scaling coefficients of the second batch normalization layers of the second network model during the training process of the second network model;

[0085] The detection module 54 is used to detect the target image through the first network model to determine the target object in the target image.

[0086] Through the above module, a target image is obtained, and the target image is input into the first network model, wherein the first network model is a network model obtained after cropping the channels of the second network model according to the scaling coefficient of the second batch normalization layer of the second network model during the training process of the second network model; the target image is detected by the first network model to determine the target object in the target image, that is, the target image is detected by the cropped network model. The above technical solution is adopted to solve the problem of low efficiency of image detection due to the large number of channels of the network model in the related art. When the network model is trained, the network model can automatically select the channels that need to be cropped, thereby realizing a compressed network model.

[0087] It can be seen from the above embodiments that the scaling coefficients of the second batch of normalized layers of the second network model are obtained, the channels in the second network model are pruned according to the scaling coefficients, and the network model after the channel pruned is the first network model. It can be understood that the first network model is the network model after the channel of the second network model is pruned.

[0088] In an exemplary embodiment, an acquisition module is used to acquire a first network model in the following manner: determining a loss function of the second network model based on an original training loss function and a penalty term of the second network model, wherein the original training loss function is used to indicate a function determined by a discriminator of the network model, and the penalty term includes a scaling factor of the second batch of normalized layers; adjusting the scaling factor of the second batch of normalized layers according to the loss function; and when it is determined that the absolute value of the adjusted scaling factor is less than a preset threshold, cropping the channel corresponding to the adjusted scaling factor in the network model to obtain the first network model.

[0089] That is to say, the original training loss function and penalty term of the second network model are determined, and the loss function of the second network model and the scaling coefficient of the second batch of normalization layers are determined according to the determined original training loss function and penalty term. By determining the scaling coefficient that is less than the preset threshold, the channels corresponding to the adjusted scaling coefficients in the second network model are pruned. The above technical solution is used to solve the problems in the related technology, such as the large amount of calculation and large size of the detection algorithm of the network model, which leads to a low landing rate of the network model. When the network model is trained, the second network model can automatically select the channels that need to be pruned.

[0090] In an exemplary embodiment, the determination module is also used to obtain a balance factor of the penalty item and a set of scaling coefficients of the second batch of normalized layers; calculate the norm of any scaling coefficient in the set of scaling coefficients of the second batch of normalized layers respectively to obtain multiple norms; and determine the penalty item based on the balance factor of the penalty item and the multiple norms.

[0091] That is to say, the backbone network of the second network model is structurally optimized. Specifically, the scaling coefficient γ of the second batch of normalized layers is first placed behind all channels of each network layer of the second network model, and the coefficient is directly multiplied by the output of the channel to obtain a set of manually set balance factors and scaling coefficients of the second batch of normalized layers; the norm of each scaling coefficient in the set of scaling coefficients is calculated, multiple norms are added, and multiplied with the balance factor to determine the penalty term, wherein the norm is preferably the L1 norm, which is not limited in the embodiment of the present invention.

[0092] In an exemplary embodiment, the determination module is further configured to determine the loss function of the second network model by the following formula: L = L loss +λ p ∑ γ∈Γ ||γ||1, where L loss is the original training loss function of the second network model, λ p ∑ γ∈Γ ||γ||1 is the penalty term of the second network model, λ p is the balancing factor of the penalty term, Γ is the set of scaling coefficients of the second batch of normalized layers, and ||γ||1 is the norm of the scaling coefficients of the second batch of normalized layers.

[0093] Specifically, an algorithm is trained to calculate the loss function of the second network model as follows: p ,Γ,||γ||1,L loss Calculate the loss function of the second network model: L = L loss +λ p ∑ γ∈Γ ||γ||1, where L loss is the original training loss function of the second network model, λ p is the balance factor of the penalty term, which is a fixed value set by humans; Γgamma is the set of scaling coefficients of the second batch of normalized layers participating in channel clipping, ||γ||1 is the calculation of the L1 norm of the scaling system of the second batch of normalized layers, where λ p This is a fixed value designed according to specific scenarios.

[0094] In an exemplary embodiment, the above-mentioned device also includes: a combination module, used to combine the operations of the first batch of normalization layers and the first convolutional layer of the first network model; the target image input into the first network model is calculated through the combined first batch of normalization layers and the first convolutional layer to obtain a feature image corresponding to the target image.

[0095] That is, the first batch of normalization layers and the first convolutional layers in the backbone network of the first network model are fused, and the target image input into the first network model is normalized and convolved through the fused first batch of normalization layers and the first convolutional layers to obtain a feature image corresponding to the normalized and convolved target image.

[0096] It should be noted that before the target image is input into the first network model, the resolution of the target image is reduced in a step-by-step manner, and then the accuracy and recall rate of the target detection algorithm of the target images of each resolution are statistically analyzed, and the optimal image resolution that meets the accuracy requirement is selected as the target resolution of the target image input into the first network model.

[0097] In an exemplary embodiment, the combination module is further used to convert the first batch of normalized layers into target convolutional layers by the following formula: Wherein, F is the feature vector of the target image, is the result of normalization of the feature vector F, b is used to indicate the normalization layer, c is used to indicate the number of channels, and W BN is the convolution kernel of the target convolution layer, b BN is the bias term of the target convolutional layer, i is the horizontal coordinate of the pixel corresponding to the feature vector, and j is the vertical coordinate of the pixel corresponding to the feature vector.

[0098] Specifically, normalize the samples of a normalization layer; for a certain layer feature x1, x2, x3…, x n , which is calculated as follows:

[0099]

[0100] Among them, μ represents the sample mean of the first batch of normalization layers, σ 2 represents the sample variance on the current normalization layer, and the mean and means of different channels are calculated separately. ε is a new minimum value used to solve the problem of zero division. β and γ are trainable parameters, representing the scaling system and offset coefficient respectively. During the training phase, μ and σ 2 It can be calculated by the current first batch of normalization layers; in the test phase, μ and σ 2 The exponential sliding average calculated during the training process is used. In the test phase, the parameters in the first network model, such as β and γ, are not iteratively updated, and the operations of the first batch of normalization layers are combined with the operations of the first convolutional layer before it; here, the parameters of the first batch of normalization layers are represented as μ c , γ c , β c , where c represents the cth channel.

[0101] The first batch of normalized layers are converted to target convolutional layers. For the coordinates (i, j) of the pixel points corresponding to the feature vector, the operation of converting the first batch of normalized layers to the target convolutional layers can be expressed by the following formula:

[0102]

[0103]

[0104]

[0105] in, is the result of normalization of feature image F. Formula The operation process can be seen as a 1x1 convolution, where W BN is the convolution kernel of the target convolution layer, b BN is the bias term of the target convolutional layer.

[0106] In an exemplary embodiment, the combination module is also used to obtain the convolution kernel of the target convolution layer, the bias item of the target convolution layer, the convolution kernel of the first convolution layer, the bias item of the first convolution layer and the feature vector of the pixel points of the target image; based on the convolution kernel of the target convolution layer, the bias item of the target convolution layer, the convolution kernel of the first convolution layer, the bias item of the first convolution layer and the feature vector of the pixel points of the target image, the operations of the first batch of normalization layers and the first convolution layer of the first network model are combined.

[0107] That is, based on the convolution kernel of the target convolution layer, the bias item of the target convolution layer, the convolution kernel of the first convolution layer, the bias item of the first convolution layer and the feature vector of the pixel points of the target image, the operations of the first batch of normalization layers and the first convolution layer are combined.

[0108] Specifically, the operations of the first normalization layer and the first convolutional layer are combined using the following formula: Among them, W BN W conv and W BN b conv +b BN is the parameters and bias of the combined first batch of normalization layers and the first convolutional layer, W BN is the convolution kernel of the target convolution layer, b BN is the bias term of the target convolutional layer, W conv is the convolution kernel of the first convolutional layer and b conv is the bias term of the first convolutional layer, f ij The feature vector of the pixel point of the target image, The result of normalizing and convolving the feature vectors of the pixels of the target image.

[0109] The combined formula is The normalized and convolved feature images of the target image are calculated through the combined calculation formula, thereby reducing the amount of calculation of the algorithm in the first network model.

[0110] An embodiment of the present invention further provides a storage medium, which includes a stored program, wherein the program executes any of the above methods when it is run.

[0111] In an exemplary embodiment, in this embodiment, the above storage medium may be configured to store a computer program for performing the following steps:

[0112] S1, obtaining a target image, and inputting the target image into a first network model, wherein the first network model is a network model obtained by clipping the channels of the second network model according to the scaling coefficient of the second batch normalization layer of the second network model during the training process of the second network model;

[0113] S2: Detect the target image through the first network model to determine the target object in the target image.

[0114] In an exemplary embodiment, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0115] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0116] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0117] In an exemplary embodiment, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0118] S1, obtaining a target image, and inputting the target image into a first network model, wherein the first network model is a network model obtained by clipping the channels of the second network model according to the scaling coefficient of the second batch normalization layer of the second network model during the training process of the second network model;

[0119] S2: Detect the target image through the first network model to determine the target object in the target image.

[0120] In an exemplary embodiment, the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0121] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices. In an exemplary embodiment, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present invention is not limited to any specific combination of hardware and software.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image detection method, characterized in that: include: Acquire a target image, and input the target image into a first network model, wherein the first network model is a network model obtained by clipping channels of the second network model according to a scaling factor of a second batch of normalization layers of the second network model during training of the second network model; Detecting the target image by using the first network model to determine the target object in the target image; Wherein, before obtaining the target image and inputting the target image into the first network model, the method further includes: Determining a loss function of the second network model according to an original training loss function and a penalty term of the second network model, wherein the original training loss function is used to indicate a function determined by a discriminator of the second network model, and the penalty term includes a scaling coefficient of a second batch normalization layer of the second network model; adjusting scaling factors of the second batch of normalization layers according to the loss function; When it is determined that the absolute value of the adjusted scaling factor is less than a preset threshold, clipping the channel corresponding to the adjusted scaling factor in the second network model to obtain the first network model; Wherein, when it is determined that the absolute value of the adjusted scaling factor is less than a preset threshold, after clipping the channel corresponding to the adjusted scaling factor in the second network model to obtain the first network model, the method further includes: The operations of the first normalization layer and the first convolutional layer are combined using the following formula: Among them, W BN W conv and W BN b conv +b BN are the parameters and bias terms of the combined first batch of normalization layers and the first convolutional layer, W BN is the convolution kernel of the target convolution layer, b BN is the bias term of the target convolutional layer, W conv is the convolution kernel of the first convolutional layer and b conv is the bias term of the first convolutional layer, f ij The feature vector of the pixel point of the target image, The result of normalizing and convolving the feature vector of the pixel points of the target image.

2. The image detection method according to claim 1, characterized in that: Before acquiring the target image and inputting the target image into the first network model, the method further includes: Determining a loss function of the second network model according to an original training loss function and a penalty term of the second network model, wherein the original training loss function is used to indicate a function determined by a discriminator of the second network model, and the penalty term includes a scaling coefficient of a second batch normalization layer of the second network model; adjusting scaling factors of the second batch of normalization layers according to the loss function; When it is determined that the absolute value of the adjusted scaling factor is less than a preset threshold, the channel corresponding to the adjusted scaling factor in the second network model is cropped to obtain the first network model.

3. The image detection method according to claim 2, characterized in that: Before determining the loss function of the second network model according to the original training loss function and the penalty term of the second network model, the method further includes: Obtaining a balance factor of the penalty term and a set of scaling coefficients of the second batch of normalization layers; respectively calculating the norm of any scaling coefficient in the set of scaling coefficients of the second batch of normalization layers to obtain a plurality of norms; The penalty term is determined according to a balance factor of the penalty term and the plurality of norms.

4. The image detection method according to claim 2, characterized in that: Determining the loss function of the second network model according to the original training loss function and the penalty term of the second network model includes: The loss function of the second network model is determined by the following formula: L=L loss +λ p ∑ γ∈Γ ||γ||1, where L loss is the original training loss function of the second network model, λ p ∑ γ∈Γ ||γ||1 is the penalty term of the second network model, λ p is a balancing factor of the penalty term, Γ is a set of scaling coefficients of the second batch of normalized layers, and ||γ||1 is the norm of the scaling coefficients of the second batch of normalized layers.

5. The image detection method according to claim 2, characterized in that: When it is determined that the absolute value of the adjusted scaling factor is less than a preset threshold, after clipping the channel corresponding to the adjusted scaling factor in the second network model to obtain the first network model, the method further includes: Combining the operations of the first normalization layer and the first convolutional layer of the first network model; The target image input into the first network model is calculated through the combined first batch of normalization layers and the first convolutional layer to obtain a feature image corresponding to the target image.

6. The image detection method according to claim 5, characterized in that: Before combining the operations of the first normalization layers and the first convolutional layer of the first network model, the method further includes: The first batch of normalization layers are converted into target convolutional layers by the following formula: Wherein, F is the feature vector of the target image, is the result of normalization of the feature vector F, b is used to indicate the first batch of normalization layers, c is used to indicate the number of channels, and W BN is the convolution kernel of the target convolution layer, b BN is the bias term of the target convolutional layer, i is the horizontal coordinate of the pixel corresponding to the feature vector, and j is the vertical coordinate of the pixel corresponding to the feature vector.

7. The image detection method according to claim 6, characterized in that: After converting the first batch of normalized layers into target convolutional layers, the method further includes: Acquire the convolution kernel of the target convolution layer, the bias item of the target convolution layer, the convolution kernel of the first convolution layer, the bias item of the first convolution layer, and the feature vector of the pixel point of the target image; Based on the convolution kernel of the target convolution layer, the bias item of the target convolution layer, the convolution kernel of the first convolution layer, the bias item of the first convolution layer and the feature vector of the pixel point of the target image, the operation of the first batch of normalization layers and the first convolution layer of the first network model is combined.

8. An image detection device, characterized in that: include: An acquisition module, used for acquiring a target image and inputting the target image into a first network model, wherein the first network model is a network model obtained by clipping the channels of the second network model according to the scaling coefficient of the second batch normalization layer of the second network model during the training process of the second network model; A detection module, configured to detect the target image through the first network model to determine a target object in the target image; The acquisition module is used to acquire the first network model in the following manner: determine the loss function of the second network model according to the original training loss function and penalty term of the second network model, wherein the original training loss function is used to indicate the function determined by the discriminator of the network model, and the penalty term includes the scaling coefficient of the second batch of normalized layers; adjust the scaling coefficient of the second batch of normalized layers according to the loss function; when it is determined that the absolute value of the adjusted scaling coefficient is less than a preset threshold, crop the channel corresponding to the adjusted scaling coefficient in the network model to acquire the first network model; The device further includes: a combining module, configured to combine the operations of the first normalization layer and the first convolution layer by using the following formula: Among them, W BN W conv and W BN b conv +b BN are the parameters and bias terms of the combined first batch of normalization layers and the first convolutional layer, W BN is the convolution kernel of the target convolution layer, b BN is the bias term of the target convolutional layer, W conv is the convolution kernel of the first convolutional layer and b conv is the bias term of the first convolutional layer, f ij The feature vector of the pixel point of the target image, The result of normalizing and convolving the feature vector of the pixel points of the target image.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images