Cycling target detection method and electronic device based on attention map

Abstract

The present application provides a method and electronic device for detecting a cycling target based on an attention map. Based on the present application, by extracting and fusing features of an original image containing a target to be detected and its attention map, a feature map containing more information is obtained, which can improve the feature discrimination capability and thus improve the cycling target detection performance.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and electronic device for detecting a cycling target based on an attention map. Background Art

[0002] In current computer vision (CV), object detection is one of the main tasks. The task of object detection is to find all the targets (objects) of interest in the image and determine their categories and locations. For example, to detect cycling targets in an image. Since various objects have different appearances, shapes and postures, and are interfered by factors such as illumination and occlusion during imaging, object detection has always been the most challenging problem in the field of computer vision. At present, the performance of cycling target detection still needs to be improved. Summary of the invention

[0003] The embodiments of the present application provide a cycling target detection method and an electronic device based on an attention map.

[0004] In a first aspect, an embodiment of the present application provides a cycling target detection method based on an attention map, the method comprising:

[0005] Based on the residual neural network-18 model and the gradient-weighted class activation mapping algorithm, an attention map of the original image is obtained, which includes the target to be detected;

[0006] Based on the first convolutional neural network CNN, feature extraction is performed on the original image to obtain a feature map of the original image;

[0007] Based on the first spatial feature transform SFT network, feature extraction is performed on the attention map to obtain a feature map of the attention map;

[0008] Multiplicatively fuse the feature map of the original image and the feature map of the attention map to obtain a fused feature map;

[0009] Make target detection decisions based on the fused feature map;

[0010] Among them, the residual neural network-18 model is used to extract features of the original image to obtain the feature map of the original image, the gradient weighted class activation mapping algorithm is used to generate a heat map corresponding to the feature map of the original image based on the last convolutional layer of the residual neural network-18 model, and the attention map is obtained based on the superposition of the feature map of the original image and the heat map.

[0011] Optionally, the target detection process is back-propagation trained and optimized based on a first center loss function, the first center loss function is calculated based on fused features, and the first center loss function includes a cross entropy loss function and a center loss function.

[0012] Optionally, the first center loss function is determined based on the following formula:

[0013] ,

[0014] in, represents the first center loss function, represents the cross entropy loss function, represents the center loss function, Is a balancing coefficient used to control the weights of the cross entropy loss function and the center loss function;

[0015] Determined based on the following formula:

[0016] ,

[0017] Determined based on the following formula:

[0018]

[0019] in, Indicates The feature vector of the samples, Indicates The labels of samples, represents the weight matrix, represents the bias vector, represents the number of categories, represents the batch size, Indicates The center of the corresponding category of samples, Represents the square of the L2 norm.

[0020] Optionally, the first CNN is a fine-tuned CNN.

[0021] In a second aspect, an embodiment of the present application provides an electronic device, including:

[0022] An attention map generation module is used to obtain the attention map of the original image including the target to be detected based on the residual neural network-18 model and the gradient-weighted class activation mapping algorithm;

[0023] A first extraction module is used to extract features from the original image based on a first convolutional neural network (CNN) to obtain a feature map of the original image;

[0024] A second extraction module is used to extract features of the attention map based on the first spatial feature transform SFT network to obtain a feature map of the attention map;

[0025] A fusion module is used to multiply fuse the feature map of the original image and the feature map of the attention map to obtain a fused feature map;

[0026] The decision module is used to make target detection decisions based on the fused feature map;

[0027] Among them, the residual neural network-18 model is used to extract features of the original image to obtain the feature map of the original image, the gradient weighted class activation mapping algorithm is used to generate a heat map corresponding to the feature map of the original image based on the last convolutional layer of the residual neural network-18 model, and the attention map is obtained based on the superposition of the feature map of the original image and the heat map.

[0028] Optionally, the target detection process is back-propagation trained and optimized based on a first center loss function, the first center loss function is calculated based on fused features, and the first center loss function includes a cross entropy loss function and a center loss function.

[0029] Optionally, the first center loss function is determined based on the following formula:

[0030] ,

[0031] in, represents the first center loss function, represents the cross entropy loss function, represents the center loss function, Is a balancing coefficient used to control the weights of the cross entropy loss function and the center loss function;

[0032] Determined based on the following formula:

[0033] ,

[0034] Determined based on the following formula:

[0035]

[0036] in, Indicates The feature vector of the samples, Indicates The labels of samples, represents the weight matrix, represents the bias vector, represents the number of categories, represents the batch size, Indicates The center of the corresponding category of samples, Represents the square of the L2 norm.

[0037] Optionally, the first CNN is a fine-tuned CNN.

[0038] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, at least one processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a method as described in any one of the above-mentioned first aspects is implemented.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described in any one of the above-mentioned first aspects is implemented.

[0040] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes any one of the methods described in the first aspect.

[0041] Based on the present application, by performing feature extraction and feature fusion on the original image containing the target to be detected and its attention map, a feature map containing more information is obtained, which can improve the feature discrimination ability and thus improve the cycling target detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for describing the embodiments are briefly introduced below.

[0043] Figure 1 It is a flow chart of a cycling target detection method based on an attention map provided in an embodiment of the present application.

[0044] Figure 2 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application.

[0045] Figure 3 It is a structural schematic diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solution of the embodiment of the present invention will be described below in conjunction with the accompanying drawings in the embodiment of the present invention.

[0047] In current computer vision, target detection is one of the main tasks. The task of target detection is to find all the targets (objects) of interest in the image and determine their categories and locations. For example, to detect the cycling target in the image. However, due to the different appearances, shapes and postures of various objects, coupled with the interference of factors such as illumination and occlusion during imaging, target detection has always been the most challenging problem in the field of computer vision.

[0048] At present, the performance of object detection still needs to be improved. Taking cycling object detection as an example, the demand for cycling object detection has become more and more common, but cycling object detection still faces the following challenges: (1) There are large differences within categories. For example, even if the person is riding a bicycle, different samples will have huge differences due to factors such as different people, different bicycles, and different shooting angles. (2) There are small differences between categories. For example, the images of electric vehicle riding and motorcycle riding are very similar. (3) There are many types of objects in the scene, such as bicycles, electric vehicles, and motorcycles. The brands and models are very rich.

[0049] Based on this, Figure 1 The embodiment of the present application provides a method for detecting a cycling target based on an attention map, the method comprising steps S110 to S150. The method is suitable for detecting a cycling target in an image.

[0050] In step S110, based on a residual neural network (ResNet)-18 model and a gradient-weighted class activation mapping algorithm, an attention map of the original image is obtained, where the original image includes the target to be detected.

[0051] The ResNet-18 model can be used to extract features from the original image to obtain a feature map of the original image, that is, the ResNet-18 model can be used as a feature extractor for the original image. The ResNet-18 model can be pre-trained, and the high-level semantic features of the image can be automatically learned by inputting the original image into it. As an example, the original image can be a three-primary color RGB image.

[0052] The gradient-weighted class activation mapping algorithm can be used to generate a heat map corresponding to the feature map of the original image based on the last convolutional layer of the ResNet-18 model, and the attention map is obtained based on the superposition of the feature map and the heat map of the original image. As an example, the process of generating a heat map of the original image by the gradient-weighted class activation mapping algorithm can be described as follows.

[0053] Select layer: First, select the last convolutional layer in the ResNet-18 model network, because this layer usually contains rich semantic information and can better represent the high-level features in the image.

[0054] Compute gradients: For the category of interest, calculate the gradients between the feature maps output by this layer and the final classification score. These gradients represent the importance of each feature map for category prediction.

[0055] Generate weights: For each feature map, calculate the average value of its gradient over the entire feature map. These average values ​​are the weights of the feature map.

[0056] Weighted feature map: Weight each feature map using the weights obtained in the previous step.

[0057] Generate a heat map: Add the weighted feature maps element by element, and then pass them through the rectified linear unit (ReLU) function to retain only positive values, so that a heat map representing the strength of the model's response to a specific category is obtained.

[0058] Visualization: Overlaying the generated heatmap onto the original image allows you to visually see which parts of the image the model focused on when making its decision. This step is not required if manual observation is not required.

[0059] The advantage of the gradient-weighted class activation mapping algorithm is that it does not depend on a specific network architecture, can be applied to any convolutional neural network (CNN)-based classification model, and can provide an intuitive explanation to help understand how the model views and interprets images.

[0060] In step S120, feature extraction is performed on the original image based on the first CNN model to obtain a feature map of the original image. Optionally, the first CNN model can be a fine-tuned CNN model, and the fine-tuning method is not limited, which helps to more effectively extract features from the original image.

[0061] In step S130, feature extraction is performed on the attention map based on a first spatial feature transformation (SFT) network to obtain a feature map of the attention map.

[0062] The SFT network is a network designed specifically for processing grayscale images, which can effectively extract valuable information from attention maps. The SFT network can usually be used to process tasks such as semantic segmentation, object detection, and image classification, especially in scenarios where it is necessary to understand the complex spatial relationship between objects in the image. The SFT network has features such as spatial fusion module, Transformer structure, multi-scale feature fusion, and end-to-end training.

[0063] Spatial fusion module: This module can effectively integrate the spatial information in the image and help the network better understand the relationship between objects.

[0064] Transformer structure: The self-attention mechanism of the Transformer architecture enables the network to capture long-range dependencies, which is very important for understanding contextual information in images.

[0065] Multi-scale feature fusion: SFT networks are usually designed with a multi-scale feature fusion mechanism, which enables the network to process feature information of different scales simultaneously, thereby improving the understanding of image details and the overall situation.

[0066] End-to-end training: SFT networks can usually be trained end-to-end, which means that all components of the network can be trained at once through the back-propagation algorithm, simplifying the training process.

[0067] In step S140, the feature map of the original image and the feature map of the attention map are multiplicatively fused to obtain a fused feature map.

[0068] Multiplicative fusion assumes that elements at the same position in two feature maps can enhance each other, thereby improving the discriminative ability of features. For example, if a pixel has a high brightness in the original image and a high response value in the attention map, then this pixel still has a high response value in the fused feature map, indicating that it is an important feature. In addition, the original image and the attention map can provide complementary information and can describe the scene content more comprehensively. Therefore, the fused feature map can contain more useful information, which can improve the discriminative ability of features and thus improve the performance of cycling target detection. At the same time, the fused feature map is more robust to interference factors such as noise and occlusion.

[0069] In step S150, a decision of target detection is made based on the fused feature map. As an example, the decision of target detection may determine a category label of the target.

[0070] Optionally, the target detection process can be back-propagated and optimized based on the first center loss function, the first center loss function can be calculated based on the fused features, and the first center loss function can include a cross entropy loss function and a center loss function. As an example, the first center loss function here can be used for training and optimization of the first CNN model and / or the first SFT network, thereby improving the performance of the model.

[0071] Optionally, the first center loss function can be determined based on the following formula:

[0072] ,

[0073] in, represents the first center loss function, represents the cross entropy loss function, represents the center loss function, Is a balancing coefficient used to control the weights of the cross entropy loss function and the center loss function.

[0074] Cross Entropy Loss Function It can be determined based on the following formula:

[0075] ,

[0076] in, Indicates The feature vector of the samples, Indicates The labels of samples, represents the weight matrix, represents the bias vector, represents the number of categories, Indicates the batch size.

[0077] Center loss function It can be used to reduce intra-class differences and make feature vectors of the same category closer. Center loss function It can be determined based on the following formula:

[0078]

[0079] in, Indicates The center of the corresponding category of samples, Represents the square of the L2 norm.

[0080] Based on the first center loss function, the above model must not only minimize the cross entropy loss function but also minimize the center loss function during the training process, thereby improving the discriminative ability of features and intra-class similarity.

[0081] Based on the present application, by performing feature extraction and feature fusion on the original image containing the target to be detected and its attention map, a feature map containing more information is obtained, which can improve the feature discrimination ability and thus improve the cycling target detection performance.

[0082] The method provided in the embodiment of the present application is described in detail above. The device provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0083] See also Figure 2 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 2The electronic device 200 shown can be used for cycling target detection, and the electronic device 200 may include:

[0084] An attention map generation module 210 is used to obtain an attention map of an original image based on a ResNet-18 model and a gradient-weighted class activation mapping algorithm, where the original image includes a target to be detected;

[0085] A first extraction module 220, configured to extract features from the original image based on a first convolutional neural network (CNN) model to obtain a feature map of the original image;

[0086] A second extraction module 230 is used to extract features from the attention map based on the first spatial feature transform SFT network to obtain a feature map of the attention map;

[0087] A fusion module 240 is used to perform multiplicative fusion on the feature map of the original image and the feature map of the attention map to obtain a fused feature map;

[0088] A decision module 250, used for making a decision on target detection based on the fused feature map;

[0089] Among them, the ResNet-18 model is used to extract features from the original image to obtain the feature map of the original image. The gradient-weighted class activation mapping algorithm is used to generate a heat map corresponding to the feature map of the original image based on the last convolutional layer of the ResNet-18 model. The attention map is obtained based on the superposition of the feature map of the original image and the heat map.

[0090] Optionally, the target detection process can be back-propagation trained and optimized based on a first center loss function, the first center loss function can be calculated based on the fused features, and the first center loss function can include a cross entropy loss function and a center loss function.

[0091] Optionally, the first center loss function can be determined based on the following formula:

[0092] ,

[0093] in, represents the first center loss function, represents the cross entropy loss function, represents the center loss function, Is a balancing coefficient used to control the weights of the cross entropy loss function and the center loss function;

[0094] It can be determined based on the following formula:

[0095] ,

[0096] It can be determined based on the following formula:

[0097]

[0098] in, Indicates The feature vector of the samples, Indicates The labels of samples, represents the weight matrix, represents the bias vector, represents the number of categories, represents the batch size, Indicates The center of the corresponding category of samples, Represents the square of the L2 norm.

[0099] Optionally, the first CNN model can be a fine-tuned CNN model, and the fine-tuning method is not limited, which helps to more effectively extract features from the original image.

[0100] The present application also provides an electronic device. Figure 3 As shown, the electronic device 300 includes: at least one processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the at least one processor 301. When the processor 301 executes the computer program 303, the method provided in the present application is implemented.

[0101] Exemplarily, the computer program 303 may be divided into one or more modules / units, one or more modules / units are stored in the memory 302, and are executed by the processor 301 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 300.

[0102] Those skilled in the art will understand that Figure 3 It is only an example of the electronic device 300 and does not constitute a limitation of the electronic device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 300 may also include input and output devices, network access devices, buses, etc.

[0103] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0104] The memory 302 may be an internal storage unit of the electronic device 300, such as a hard disk or memory of the electronic device 300. The memory 302 may also be an external storage device of the electronic device 300, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 300. Further, the memory 302 may also include both an internal storage unit of the electronic device 300 and an external storage device. The memory 302 is used to store computer programs and other programs and data required by the electronic device 300. The memory 302 may also be used to temporarily store data that has been output or is to be output.

[0105] The electronic device 300 provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0106] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0107] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above method embodiments, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, ROM (read-only memory), RAM (random access memory), CD-ROM (compact disc read-only memory), magnetic tape, floppy disk and optical data storage device. The computer-readable storage medium mentioned in the present application can be a non-volatile storage medium, in other words, it can be a non-transient storage medium.

[0109] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0110] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0111] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0112] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0113] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0114] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0115] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A cycling target detection method based on attention graph, characterized in that: include: Based on the residual neural network-18 model and the gradient-weighted category activation mapping algorithm, an attention map of the original image is obtained, wherein the original image includes a target to be detected, the residual neural network-18 model is used to extract features from the original image to obtain a feature map extracted by the residual neural network-18 model, the gradient-weighted category activation mapping algorithm is used to generate a heat map corresponding to the feature map extracted by the residual neural network-18 model based on the last convolutional layer of the residual neural network-18 model, and the attention map is obtained based on the superposition of the feature map extracted by the residual neural network-18 model and the heat map; Performing feature extraction on the original image based on a first convolutional neural network (CNN) model to obtain a feature map of the original image; Performing feature extraction on the attention map based on a first spatial feature transform SFT network to obtain a feature map of the attention map; Performing multiplicative fusion on the feature map of the original image and the feature map of the attention map to obtain a fused feature map; The target detection decision is made based on the fused feature map.

2. The method according to claim 1, characterized in that The target detection process is back-propagation trained and optimized based on a first center loss function, wherein the first center loss function is calculated based on the fused features, and the first center loss function includes a cross entropy loss function and a center loss function.

3. The method according to claim 2, characterized in that The first center loss function is determined based on the following formula: ， Among them, the represents the first center loss function, the represents the cross entropy loss function, the represents the central loss function, the Is a balancing coefficient used to control the weights of the cross entropy loss function and the center loss function; Said Determined based on the following formula: ， Said Determined based on the following formula: Among them, the Indicates The feature vector of samples, Indicates The labels of samples, represents the weight matrix, represents the bias vector, Represents the number of categories, Represents the batch size, Indicates The samples correspond to the center of the category, Represents the square of the L2 norm.

4. The method according to claim 3, characterized in that The first CNN is a fine-tuned CNN.

5. An electronic device, the electronic device is used for cycling target detection, characterized in that: include: An attention map generation module, which is used to obtain an attention map of an original image based on a residual neural network-18 model and a gradient-weighted category activation mapping algorithm, wherein the original image includes a target to be detected, the residual neural network-18 model is used to perform feature extraction on the original image to obtain a feature map extracted by the residual neural network-18 model, the gradient-weighted category activation mapping algorithm is used to generate a heat map corresponding to the feature map extracted by the residual neural network-18 model based on the last convolutional layer of the residual neural network-18 model, and the attention map is obtained based on the superposition of the feature map extracted by the residual neural network-18 model and the heat map; A first extraction module, used for performing feature extraction on the original image based on a first convolutional neural network (CNN) model to obtain a feature map of the original image; A second extraction module is used to extract features from the attention map based on a first spatial feature transform SFT network to obtain a feature map of the attention map; A fusion module, used for multiplicatively fusing the feature map of the original image and the feature map of the attention map to obtain a fused feature map; A decision module is used to make a decision on the target detection based on the fused feature map.

6. The electronic device according to claim 5, characterized in that: The target detection process is back-propagation trained and optimized based on a first center loss function, wherein the first center loss function is calculated based on the fused features, and the first center loss function includes a cross entropy loss function and a center loss function.

7. The electronic device according to claim 6, characterized in that: The first center loss function is determined based on the following formula: ， Among them, the represents the first center loss function, the represents the cross entropy loss function, the represents the central loss function, the Is a balancing coefficient used to control the weights of the cross entropy loss function and the center loss function; Said Determined based on the following formula: ， Said Determined based on the following formula: Among them, the Indicates The feature vector of samples, Indicates The labels of samples, represents the weight matrix, represents the bias vector, Represents the number of categories, Represents the batch size, Indicates The samples correspond to the center of the category, Represents the square of the L2 norm.

8. The electronic device according to claim 7, characterized in that: The first CNN is a fine-tuned CNN.

9. An electronic device comprising a memory, at least one processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images