Image key feature area detection method, electronic device and storage medium

Abstract

The embodiments of the present application provide a method, electronic device and storage medium for detecting key feature areas of an image, and relate to the field of image detection technology. The method includes: inputting a target image into a trained image classification model based on a convolutional neural network, obtaining the input vector of the last classification layer and the output feature map of the last convolution layer when the image classification model processes the target image; calculating the rate of change measure of each neuron activation value in each channel of the output feature map for the maximum component of the input vector, and obtaining the mean of the rate of change measure of each channel; generating a reconstructed image with the same pixel distribution as the target image according to the neuron activation value at the same position of each channel and the mean of the rate of change measure of each channel; and using the position set determined by the reconstructed image as the key feature area of ​​the target image. The implementation provided by the present application improves the positioning accuracy and detection granularity of the key feature area of ​​the input image.

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and in particular to a method for detecting key feature areas of an image, an electronic device, and a corresponding storage medium. Background Art

[0002] At present, with the rapid development of artificial intelligence technology, the application of artificial intelligence technology is becoming more and more extensive. It has been applied to many fields such as image analysis, speech recognition, text processing, intelligent recommendation, security detection, etc. However, deep neural network models have always been regarded as "black box" models, and their interpretability has been questioned, which has hindered the widespread application of deep learning technology in some mission scenarios (such as autonomous driving, smart medical care and other high-risk mission scenarios). A scientific and reasonable explanation of the input-output relationship of the deep neural network model can enhance users' trust in the deep learning model and system, help discover the defects and deficiencies of the deep learning model, and facilitate the application and promotion of deep learning technology.

[0003] In the prior art, there is already a method of using gradient activation mapping to establish a model that focuses on the relationship between features and model decisions, which has a similar application effect to the key technology of this application. However, this scheme uses the gradient mean of each feature information at all positions in the image as the weight of the feature information, and the evaluation result is relatively rough and fuzzy. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, electronic device and storage medium for detecting key feature areas of an image, by analyzing the influence of each pixel of the input image data on the intermediate output results of the model, evaluating the importance of each pixel of the input image data, detecting and determining the key feature areas of the input image data, so as to at least solve some of the problems in the background technology.

[0005] In order to achieve the above-mentioned purpose, a method for detecting key feature areas of an image is provided in the present application, and the method includes: inputting a target image into a trained image classification model based on a convolutional neural network, obtaining the input vector of the last classification layer and the output feature map of the last convolution layer when the image classification model processes the target image; calculating the rate of change measure of each neuron activation value in each channel of the output feature map for the maximum component of the input vector, and obtaining the mean of the rate of change measure of each channel; generating a reconstructed image with the same pixel distribution as the target image according to the neuron activation value at the same position of each channel and the mean of the rate of change measure of each channel; determining a position set corresponding to a set of pixels greater than a preset threshold in the reconstructed image, and taking the position set in the target image as the key feature area of ​​the target image.

[0006] Optionally, the method also includes: determining that the value of the maximum component of the input vector when it is acquired is the current value; calculating the minimum value of the maximum component corresponding to the change process when the domain of each pixel in the target area of ​​the target image changes; and obtaining an importance evaluation value of each pixel based on the current value and the minimum value of each pixel.

[0007] Optionally, the target area is the key feature area or the entire area.

[0008] Optionally, the last classification layer is a softmax layer.

[0009] Optionally, the maximum component of the input vector is a maximum component among components distinguished based on classification categories.

[0010] Optionally, the rate of change measure is a derivative; calculating the rate of change measure of the maximum component of the input vector for each neuron activation value in each channel of the output feature map includes: obtaining a mapping relationship in the image classification model with the neuron activation value as the independent variable and the maximum component of the input vector as the dependent variable; calculating the derivative of the mapping relationship to obtain the rate of change measure corresponding to the neuron activation value.

[0011] Optionally, a reconstructed image having the same pixel distribution as the target image is generated based on the neuron activation value at the same position of each channel and the mean of the rate of change measurement of each channel, including: calculating and accumulating the product of the neuron activation value at the same position and the mean of the rate of change measurement of each channel, and using the accumulated result as the value of the position corresponding to the same position in the first intermediate image; processing the negative values ​​in the first intermediate image to obtain a second intermediate image; and upsampling the second intermediate image to obtain a reconstructed image having the same pixel distribution as the target image.

[0012] Optionally, the upsampling process uses a bilinear interpolation method.

[0013] Optionally, the minimum value of the maximum component corresponding to the change process when the domain of each pixel of the target image changes is calculated, including: if each pixel of the target image is a vector, the minimum value of the maximum component is solved by the gradient descent method; if each pixel of the target image is a scalar, the minimum value of the maximum component is solved by the derivative.

[0014] Optionally, obtaining the importance evaluation value of each pixel according to the current value and the minimum value of each pixel includes: taking the difference between the current value and the minimum value of each pixel as the importance evaluation value of each pixel.

[0015] The present application also provides an image key feature area detection device, which includes: a data acquisition module, used to input a target image into a trained image classification model based on a convolutional neural network, and obtain the input vector of the last classification layer and the output feature map of the last convolution layer when the image classification model processes the target image; a channel mean module, used to calculate the change rate measurement of the maximum component of the input vector for each neuron activation value in each channel of the output feature map, and obtain the mean of the change rate measurement of each channel; an image reconstruction module, used to generate a reconstructed image with the same pixel distribution as the target image based on the neuron activation value at the same position of each channel and the mean of the change rate measurement of each channel; and a region determination module, used to determine the position set corresponding to the pixel set greater than a preset threshold in the reconstructed image, and use the position set in the target image as the key feature area of ​​the target image.

[0016] The present application also provides an electronic device, comprising: at least one processor; a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the aforementioned image key feature area detection method by executing the instructions stored in the memory.

[0017] The present application also provides a machine-readable storage medium on which instructions are stored. When the instructions are executed by a processor, the processor is configured to execute the aforementioned method for detecting key feature areas of an image.

[0018] The present application also provides a computer program product, including a computer program, which implements the aforementioned image key feature area detection method when executed by a processor.

[0019] The above technical solution has the following beneficial effects:

[0020] (1) The existing deep learning model input-output relationship interpretation technology is improved, and a method for detecting key feature areas of images is proposed. This method can realize pixel-level importance evaluation of input image data and improve the model's positioning accuracy and detection granularity of key feature areas of input images.

[0021] (2) Some implementation methods use the intermediate output results of the model to evaluate pixel importance. Compared with using the final output results of the model to evaluate pixel importance, this can eliminate the mutual interference between the outputs of the model for different categories and improve the accuracy of the pixel importance evaluation results.

[0022] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0024] Figure 1 The following is a schematic diagram showing the steps of a method for detecting key feature areas of an image according to an embodiment of the present application;

[0025] Figure 2 The following is a schematic diagram showing the implementation process of the method for detecting key feature areas of an image according to an embodiment of the present application;

[0026] Figure 3 The structure diagram of the key feature area detection device of the image according to the embodiment of the present application is schematically shown;

[0027] Figure 4 The internal structure of an electronic device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0028] The specific implementation of the embodiment of the present application is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present application, and is not used to limit the embodiment of the present application.

[0029] Figure 1 The following is a schematic diagram of the steps of the method for detecting key feature areas of an image according to an embodiment of the present application. Figure 1 As shown, a method for detecting key feature areas of an image comprises:

[0030] S01. Input a target image to a trained image classification model based on a convolutional neural network, and obtain an input vector of the last classification layer and an output feature map of the last convolution layer when the image classification model processes the target image;

[0031] S02, calculating the change rate metric of the maximum component of the input vector for each neuron activation value in each channel of the output feature map, and obtaining the mean of the change rate metric of each channel;

[0032] S03, generating a reconstructed image having the same pixel distribution as the target image according to the neuron activation value at the same position of each channel and the mean of the rate of change measurement of each channel;

[0033] S04. Determine a position set corresponding to a pixel set greater than a preset threshold in the reconstructed image, and use the position set in the target image as a key feature area of ​​the target image.

[0034] In step S01, the trained image classification model based on convolutional neural network can be assumed to be M, where M is a trained image classification model based on CNN (Convolutional Neural Networks). All activation functions of model M use ReLU activation function, and the last layer of model M is a softmax layer. The input vector dimension of the softmax layer of model M is , is the total number of image categories that can be recognized by model M. The target image is the source image for extracting key feature areas, and is also the input image of model M. The input image is an RGB image or a grayscale image. The input image contains pixels, and are the number of pixels per row and column in the input image, respectively. When the input image is an RGB image, each pixel is a three-dimensional vector; when the input image is a grayscale image, each pixel is a scalar. H, W and All are positive integers.

[0035] Assume that the target image to be detected is X , target image X The pixel value of the hth row and wth column is denoted by .Will Input model M and get the softmax layer of model M for The input vector is represented by Z, and its largest component is The value of The last convolutional layer targets X Output multiple channel feature maps, using Indicates The feature map of each channel, is a matrix consisting of the activation values ​​of the neurons in the convolutional layer. , is the total number of different channel feature maps output by the last convolutional layer of model M. Representation feature map No. Line The neuron activation values ​​of the columns, , , and Respectively represent matrices The total number of rows and columns, , , , , All are positive integers.

[0036] In step S02, represents the maximum component of the input vector, , k is a positive integer, k represents the target image predicted by model M X The most likely category number. First calculate for A measure of the rate of change of Then calculate the first The mean of all derivatives of channels ,For example:

[0037] .

[0038] In step S03, due to is based on each channel, so you need to Channel After synthesis, based on the subscript i and subscript j The result is recorded as , which represents the model M for image data In the feature map Line The evaluation value of the kth category at the column position. For the Line The pixel values ​​of the columns are processed into an image. X The pixel distribution of the images is the same, and a reconstructed image is obtained. The same pixel distribution here includes the same number of pixels in each row and the same number of pixels in each column.

[0039] In step S04, it is assumed that the reconstructed image is , reconstruct the image The pixel value of the hth row and wth column is denoted by , the preset threshold is 0, for image data X Each pixel ,if , then the pixel Put it into the set R. The positions of all pixels in the set R on the image data X together constitute the image data X key feature areas.

[0040] Through the above implementation methods, the existing deep learning model input-output relationship interpretation technology is improved, and a method for detecting key feature areas of an image is proposed to improve the model's positioning accuracy and detection granularity of key feature areas of the input image.

[0041] In some embodiments of the present application, the method further includes: determining that the value of the maximum component of the input vector at the time of acquisition is the current value; that is, the input vector acquired in step S01 Z The maximum value of The value of , calculate the minimum value of the maximum component corresponding to the change process when the domain of each pixel in the target area of ​​the target image changes, recorded as ; Obtain the importance evaluation value of each pixel according to the current value and the minimum value of each pixel; and Construct importance assessment values, such as and The input parameter and the importance evaluation value are output parameters, and a common function such as a mathematical function, a trigonometric function or a combination of the two is selected to construct a mapping relationship between the two. Through the above implementation, the pixel-level importance evaluation of the input image data can be realized according to the influence of the pixel on the classification vector.

[0042] The target area in the previous embodiment is not limited, so it can be any area or ROI (region of interest) of the target image. In the implementation of the method, it is often selected as the key feature area or the entire area. When the target area is the entire area, the implementation time of this method is after step S01, but at this time there is no regional restriction such as the key feature area, and more irrelevant pixels will be evaluated for importance, which has the disadvantages of large amount of calculation. When the target area is the key feature area, its implementation time is after the key feature area is obtained, that is, after the aforementioned step S04. At this time, it is equivalent to fine-grained detection after determining the key feature area, and accurately determining which pixel is important in the key feature area. At this time, it can make up for the defects that the criticality that can be represented by a key feature area is vague, unclear, too large in range, lack of details, etc.

[0043] In some embodiments of the present application, the rate of change measure is a derivative; calculating the rate of change measure of the maximum component of the input vector for each neuron activation value in each channel of the output feature map includes: obtaining a mapping relationship in the image classification model with the neuron activation value as the independent variable and the maximum component of the input vector as the dependent variable; calculating the derivative of the mapping relationship to obtain the rate of change measure corresponding to the neuron activation value. , first calculate for The derivative of ;in, for The changes are mapped based on the image classification model M.

[0044] In some embodiments of the present application, a reconstructed image with the same pixel distribution as the target image is generated based on the neuron activation value at the same position of each channel and the mean of the rate of change measure of each channel, including: calculating and accumulating the product of the neuron activation value at the same position and the mean of the rate of change measure of each channel, using the accumulated result as the value of the position corresponding to the same position in the first intermediate image; processing the negative value in the first intermediate image to obtain the second intermediate image; upsampling the second intermediate image to obtain a reconstructed image with the same pixel distribution as the target image. For example, the process can be expressed as:

[0045] .

[0046] in, Indicates l Channel No. Line The neuron activation values ​​of the columns, for l The mean of the derivatives of the channels; L The total number of different channel feature maps output by the last convolutional layer of model M. Function , which can treat negative values ​​as 0 and keep positive values ​​unchanged. The final result is Represents image data In the feature map Line Column position k The evaluation value of the class. Indicated by indivual The image composed of For this image Line The pixel value of the column. Upsample to generate a reconstructed image with W pixels per row and H pixels per column .

[0047] The above-mentioned upsampling method is not limited, and upsampling methods such as nearest neighbor difference, deconvolution, bed of nails method, etc. can be used. In some embodiments of the present application, the upsampling process uses bilinear interpolation, which has the advantage of simple calculation.

[0048] In some implementations of the importance evaluation value calculation process, calculating the minimum value of the maximum component corresponding to the change process when the domain of each pixel of the target image changes includes: if each pixel of the target image is a vector, solving the minimum value of the maximum component by gradient descent method. If each pixel of the target image is a scalar, solving the minimum value of the maximum component by derivative. For example, for the target image X Each pixel in ,by is the independent variable, is the dependent variable, and the mapping relationship between the two is the mapping relationship determined by model M. is a vector, then calculate For vector The gradient of The minimum value of ;if is a scalar, then calculate For scalar The derivative of Solution The minimum value of , when the derivative is less than 0, by increasing Solution The minimum value of .

[0049] In some embodiments of the present application, obtaining the importance evaluation value of each pixel according to the current value and the minimum value of each pixel includes: taking the difference between the current value and the minimum value of each pixel as the importance evaluation value of each pixel. Z The maximum value of The value of , importance assessment value It can be calculated as follows:

[0050] .

[0051] The larger the value, the greater the influence of the pixel on the prediction result of the model, so it can be considered that the pixel is more important. On the contrary, if If it is very small or close to zero, the pixel has little impact on the prediction result of the model, and it can be considered that the pixel is not so important.

[0052] Figure 2 The following is a schematic diagram showing the implementation process of the method for detecting key feature areas of an image according to the embodiment of the present application. Figure 2 As shown, it includes the following implementation steps: input the target image to the modelX , get the model's softmax layer for the input image X The maximum component of the input vector ; Calculate the target image X Each pixel Importance assessment value ; According to the target image X All channel feature maps generate corresponding images ; Use bilinear interpolation method to image Upsample to generate image ; According to the image , determine the image X key feature areas.

[0053] The following is an explanation of the implementation of the present application in a specific scenario. The implementation is applied to a malware detection system based on grayscale images and convolutional neural networks. According to the method described in detail in the present application, the specific process is as follows:

[0054] (1) Based on a large amount of grayscale image data of malware and normal software with labeled malware family categories, a malware detection model based on convolutional neural network is trained. This model is a multi-classification model with the last layer being the Softmax layer. The model can determine the malware family category to which the software belongs based on the grayscale image data of the malware.

[0055] (2) Read the binary file of the software to be tested as an unsigned integer every 8 bits, whose range is [0, 255]. Use each integer as a pixel of a grayscale image, and construct a grayscale image of fixed width W for the software. The length of the grayscale image varies depending on the size of the software. If the total number of pixels in the grayscale image is not an integer multiple of the fixed width, it is padded with an integer 0.

[0056] (3) Input the grayscale image of the software into the above malware detection model. If the model determines that the software is malware, obtain the input vector of the model's softmax layer for the grayscale image of the software , compare and determine the largest component in vector Z , at this time The value of , , m is the total number of malware family categories in the training data.

[0057] (4) For each pixel in the grayscale image of the software , , , i is a positive integer, is the total number of pixels of the grayscale image of the software, calculated For scalar The derivative of The minimum value of , calculate the pixels of the grayscale image of the software The importance assessment value.

[0058] (5) Obtain the feature maps of multiple channels output by the last convolutional layer of the malware detection model for the grayscale image, and obtain the feature maps of each channel output by the last convolutional layer of the malware detection model. , calculate the malware detection model for the grayscale image in the feature map Line The evaluation value of the kth category at the column position .

[0059] (6) Use bilinear interpolation to image Upsample to generate an image with W pixels per row and H pixels per column ,image The pixel value of the hth row and wth column is denoted by .

[0060] (7) For each pixel of the grayscale image ,if , then the pixel Put them into the set R. The positions of all pixels in the set R on the grayscale image together constitute the key feature area of ​​the grayscale image, and the importance evaluation value corresponding to each pixel represents the importance of the pixel in the key feature area.

[0061] In the above implementation, security technicians can assist in analyzing and determining the correctness and credibility of the output results of the malware detection model based on the specific software binary code corresponding to the key feature area.

[0062] Based on the same inventive concept, the present application also provides an image key feature area detection device, Figure 3 The schematic diagram of the structure of the image key feature area detection device according to the embodiment of the present application is shown. Figure 3As shown, the device includes: a full selection determination module, which is used to determine that the client nodes in the federated learning system will not be fully selected; a probability calculation module, which is used to obtain the evaluation value corresponding to the client node, weight the number of local training samples of the client node with the absolute value of the evaluation value as the weight, and obtain the probability of the client node being selected by the proportion of the weighted number of local training samples in the total number of weighted local training samples in the federated learning system; a set determination module, which is used to select a number of client nodes according to the selection probability of each client node to obtain a current round of training sets; and a parameter sending module, which is used to send model parameters to the client nodes in the current round of training sets, and the model parameters are used by the client nodes to obtain updated gradients after local training based on the sent model parameters.

[0063] In some optional embodiments of the present application, the device also includes an importance assessment module, which is used to: determine that the value of the maximum component of the input vector when it is acquired is the current value; calculate the minimum value of the maximum component corresponding to the change process when the domain of each pixel in the target area of ​​the target image changes; and obtain the importance assessment value of each pixel based on the current value and the minimum value of each pixel.

[0064] In some optional embodiments of the present application, the target area is the key feature area or the entire area.

[0065] In some optional implementations of the present application, the last classification layer is a softmax layer.

[0066] In some optional embodiments of the present application, the maximum component of the input vector is the maximum component among the components distinguished based on the classification category.

[0067] In some optional embodiments of the present application, the rate of change measure is a derivative; calculating the rate of change measure of the maximum component of the input vector for each neuron activation value in each channel of the output feature map includes: obtaining a mapping relationship in the image classification model with the neuron activation value as the independent variable and the maximum component of the input vector as the dependent variable; calculating the derivative of the mapping relationship to obtain the rate of change measure corresponding to the neuron activation value.

[0068] In some optional embodiments of the present application, a reconstructed image having the same pixel distribution as the target image is generated based on the neuron activation value at the same position of each channel and the mean of the rate of change measurement of each channel, including: calculating and accumulating the product of the neuron activation value at the same position and the mean of the rate of change measurement of each channel, and using the accumulated result as the value of the position corresponding to the same position in the first intermediate image; processing the negative values ​​in the first intermediate image to obtain a second intermediate image; and upsampling the second intermediate image to obtain a reconstructed image having the same pixel distribution as the target image.

[0069] In some optional implementations of the present application, the upsampling process uses a bilinear interpolation method.

[0070] In some optional embodiments of the present application, the minimum value of the maximum component corresponding to the change process of each pixel of the target image when its definition domain changes is calculated, including: if each pixel of the target image is a vector, the minimum value of the maximum component is solved by the gradient descent method; if each pixel of the target image is a scalar, the minimum value of the maximum component is solved by the derivative.

[0071] In some optional embodiments of the present application, the importance evaluation value of each pixel is obtained according to the current value and the minimum value of each pixel, including: taking the difference between the current value and the minimum value of each pixel as the importance evaluation value of each pixel.

[0072] The specific definition of each functional module in the above-mentioned image key feature area detection device can refer to the definition of the image key feature area detection method above, which will not be repeated here. Each module in the above-mentioned system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. It also achieves the advantages of improving the positioning accuracy and detection granularity of the model for the key feature area of ​​the input image. Its step of pixel importance evaluation also eliminates the mutual interference between the outputs of the model for different categories, which has the advantage of improving the accuracy of the pixel importance evaluation results.

[0073] In some embodiments of the present application, an electronic device is also provided, comprising: at least one processor; a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the aforementioned image key feature area detection method. Its internal structure diagram can be as follows Figure 4 shown. Figure 4The internal structure diagram of an electronic device according to an embodiment of the present application is schematically shown. The electronic device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected via a system bus. Among them, the processor A01 of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The network interface A02 of the electronic device is used to communicate with an external terminal through a network connection. When the computer program B02 is executed by the processor A01, a method for detecting key feature areas of an image is implemented.

[0074] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0075] In one embodiment provided in the present application, a machine-readable storage medium is provided, on which instructions are stored. When the instructions are executed by a processor, the processor is configured to execute the aforementioned image key feature area detection method.

[0076] In one embodiment provided in the present application, a computer program product is provided, including a computer program, which implements the aforementioned image key feature area detection method when executed by a processor.

[0077] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0078] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0079] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0081] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0082] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0083] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0084] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0085] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for detecting key feature areas of an image, characterized in that: The method includes: Inputting a target image into a trained image classification model based on a convolutional neural network, and obtaining an input vector of a last classification layer and an output feature map of a last convolutional layer when the image classification model processes the target image; Calculate the change rate metric of the maximum component of the input vector for each neuron activation value in each channel of the output feature map to obtain a mean value of the change rate metric of each channel; Generate a reconstructed image with the same pixel distribution as the target image according to the neuron activation value at the same position of each channel and the mean of the change rate measurement of each channel; A position set corresponding to a pixel set composed of pixels greater than a preset threshold in the reconstructed image is determined, and the position set in the target image is used as a key feature area of ​​the target image.

2. The method according to claim 1, characterized in that The method further comprises: Determine that the value of the maximum component of the input vector at the time of acquisition is the current value; Calculate the minimum value of the maximum component corresponding to the change process when the definition domain of each pixel in the target area of ​​the target image changes; An importance evaluation value of each pixel is obtained according to the current value and the minimum value of each pixel.

3. The method according to claim 2, characterized in that The target area is the key feature area or the entire area.

4. The method according to claim 1, characterized in that The last classification layer is a softmax layer.

5. The method according to claim 1, characterized in that The maximum component of the input vector is the maximum component among the components distinguished based on the classification category.

6. The method according to claim 1, characterized in that The rate of change measure is a derivative; Calculating a change rate measure of the maximum component of the input vector for each neuron activation value in each channel of the output feature map, comprising: Acquire a mapping relationship in the image classification model with the neuron activation value as the independent variable and the maximum component of the input vector as the dependent variable; The derivative of the mapping relationship is calculated to obtain a change rate metric corresponding to the neuron activation value.

7. The method according to claim 1, characterized in that Generating a reconstructed image having the same pixel distribution as the target image according to the neuron activation value at the same position of each channel and the mean of the change rate measurement of each channel, comprising: Calculate and accumulate the product of the neuron activation value at the same position and the mean value of the change rate measure of each channel, and use the accumulated result as the value of the position corresponding to the same position in the first intermediate image; Processing negative values ​​in the first intermediate image to obtain a second intermediate image; The second intermediate image is up-sampled to obtain a reconstructed image having the same pixel distribution as the target image.

8. The method according to claim 7, characterized in that The up-sampling process adopts a bilinear interpolation method.

9. The method according to claim 2, characterized in that: Calculating the minimum value of the maximum component in the corresponding change process when the definition domain of each pixel of the target image changes, including: If each pixel of the target image is a vector, the minimum value of the maximum component is solved by the gradient descent method; If each pixel of the target image is a scalar, the minimum value of the maximum component is solved by the derivative.

10. The method according to claim 2, characterized in that Obtaining an importance evaluation value of each pixel according to the current value and the minimum value of each pixel, including: The difference between the current value and the minimum value of each pixel is used as the importance evaluation value of each pixel.

11. An electronic device, characterized in that: include: at least one processor; a memory connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the steps of the image key feature area detection method according to any one of claims 1 to 10 by executing the instructions stored in the memory.

12. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the method for detecting key feature areas of an image as described in any one of claims 1 to 10 are implemented.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the method for detecting key feature areas of an image as described in any one of claims 1 to 10 are implemented.

Images