Image target detection method and device

Abstract

The application discloses an image target detection method, which comprises the following steps: using a Transform network as a backbone network to extract a multi-resolution feature map from an image; inputting the multi-resolution feature map into a cross-scale attention feature pyramid network; in the cross-scale attention feature pyramid network, starting from a small-resolution feature input, gradually performing feature fusion and reorganization to a large resolution by using a cross-scale attention module, and accumulating and fusing the features from a small resolution to a large resolution; further sending the fused features into a subsequent processing and prediction module to perform regression of a detection frame and prediction of a category; training the cross-scale attention feature pyramid network based on a target data set until convergence; and using the trained cross-scale attention feature pyramid network to perform regression of a detection frame and prediction of a category on a picture to be detected, so that the performance of final target detection is improved. The application also provides a corresponding image target detection device.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning and computer vision, and more particularly relates to an image target detection method and device. BACKGROUND

[0002] Target detection is one of the most basic and important tasks in the field of computer vision, which usually uses deep neural networks to extract and model features of visual data and predict the corresponding positions and categories of target objects. Current deep learning methods attempt to solve the target detection task as a classification problem or a regression problem or a combination of the two.

[0003] Visual images have high complexity and diversity, and target detection usually needs to capture object information of multiple scales. In order to better model target features of various scales, a feature pyramid network (FPN) is widely used in the framework of target detection. FPN takes features extracted by backbone networks at various resolutions as input and performs fusion. Low-resolution features have stronger semantics but lack details, while high-resolution features are detailed but have weaker semantics. FPN interpolates low-resolution features and superimposes them on high-resolution features for fusion. This approach enriches semantic information from multiple levels and makes visual representation more sensitive to objects of multiple scales.

[0004] The Transformer network was first proposed and used for various scenarios and tasks of natural language processing (NLP), and has achieved great success. In recent years, Transformer has also been widely used in image classification, semantic segmentation, target detection and other visual tasks, and has achieved very strong performance, to some extent, surpassing the use of more convolutional neural networks (CNN) in previous visual tasks. Among them, the self-attention mechanism is the core component of the Transformer network, which measures the response between features and features and reorganizes the features according to the response value, thereby automatically establishing the relationship between the features. Existing detection methods based on traditional FPN technology usually directly implement cross-scale feature fusion by interpolating and adding features, which still has limited modeling capability for objects with complex scales. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides an image target detection method and device, which introduces an FPN with a cross-scale attention mechanism to model object features that are more robust to scale and have stronger expression capability, thereby improving the performance of final target detection.

[0006] To achieve the above object, according to one aspect of the present application, an image target detection method is provided, comprising the following steps:

[0007] Step one: using a Transformer network as a backbone network to extract a multi-resolution feature map from an image;

[0008] Step two: inputting the multi-resolution feature map in step one into a cross-scale attention feature pyramid network;

[0009] Step three: in the feature pyramid network, starting from the small-resolution feature input in step two, using a cross-scale attention module to gradually perform feature fusion and reorganization toward large resolution, and the features are accumulated and fused from small resolution to large resolution;

[0010] Step four: further sending the features fused in step three to a subsequent processing and prediction module to perform bounding box regression and class prediction, and training the above cross-scale attention feature pyramid network based on a target data set until convergence;

[0011] Step five: using the trained cross-scale attention feature pyramid network to perform bounding box regression and class prediction on a picture to be detected.

[0012] In one embodiment of the present application, the cross-scale attention module in step three is implemented by the following steps:

[0013] (3.1) the features of the nth level and the features of the (n+1)th level are first converted into 1-dimensional token sequences, i.e. and where H n ,W n are the spatial dimensions of the features F n in the height and width dimensions, respectively, and C is the channel dimension size of the features; n+1 ,W n+1 are the spatial dimensions of the features F n+1 in the height and width dimensions, respectively;

[0014] (3.2) mapping the two feature sequences obtained in (3.1) to query Query, key Key and value Value three spaces to obtain three spatial feature matrices Q, K, V;

[0015] (3.3) performing attention mechanism operation on the three matrices Q, K, and V obtained in (3.2).

[0016] In an embodiment of the present application, the Query matrix is obtained by linear mapping the feature F n of the nth level, that is,

[0017] Q=F n x W,

[0018] where W is the matrix parameter of linear mapping, and the mapped Query matrix is Q.

[0019] In an embodiment of the present application, the Key matrix and the Value matrix are both obtained by directly concatenating the two groups of features F n and F n+1 , that is,

[0020] K=V=[F n , F n+1 ],

[0021] where [·] represents the concatenation operation, K, and V represent the obtained Key matrix and Value matrix.

[0022] In an embodiment of the present application, the step (3.3) multiplies the Query matrix and the Key matrix to obtain an attention response graph, and the response graph is further applied to the Value matrix to obtain a new token sequence.

[0023] In an embodiment of the present application, in the step (3.3), the token sequence F n of the nth level will be added again to the new token sequence in the form of residual connection, and the whole process is represented as F attn =softmax(QK T )V+F n , where F attn is the obtained output feature matrix, K T is the transpose of the key matrix K, and softmax is a normalized exponential function.

[0024] In an embodiment of the present application, the cross-scale attention module performs operation within each local feature window.

[0025] In an embodiment of the present application, the window size is determined according to specific requirements.

[0026] In an embodiment of the present application, in the step one, the Transformer network is Swin-Transformer.

[0027] According to another aspect of the present application, there is also provided an image target detection device comprising at least one processor and a memory connected through a data bus between the at least one processor and the memory, the memory storing instructions executable by the at least one processor, the instructions, when executed by the processor, being used to complete the image target detection method described above.

[0028] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0029] The present application provides a novel target detection method based on a cross-scale self-attention feature pyramid to solve the problem that the current detection method has limited modeling capability for complex scale objects, which can significantly improve the modeling capability of multi-scale objects at a small computational cost and improve the performance of the final target detection. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a flowchart of the image target detection method in the embodiment of the present application;

[0031] Figure 2 is a structural diagram of the feature pyramid network based on the cross-scale attention mechanism in the embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0033] The present application relates to the field of deep learning and computer vision, and in particular to a general target detection algorithm based on cross-scale feature relationship modeling, visual feature attention mechanism and feature pyramid network (FPN, Feature Pyramid Networks).

[0034] The purpose of the present application is achieved by the following technical solutions: cross-resolution features are modeled and reorganized through a self-attention mechanism, Figure 1 For a target detection method based on a Transformer backbone network and a cross-scale attention feature pyramid network in the embodiment of the present application, as shown in Figure 1 The present application provides an image target detection method, comprising the following steps:

[0035] Step 1: Use a Transformer network (such as Swin-Transformer) as the backbone network to extract multi-resolution feature maps from the image.

[0036] Step 2: Input the multi-resolution feature map from Step 1 into the feature pyramid network for cross-scale attention.

[0037] Step 3: In the feature pyramid network, starting from the small-resolution features input in Step 2, the proposed cross-scale attention module is used to gradually fuse and reorganize features towards larger resolutions. Features are accumulated and fused from small resolution to large resolution.

[0038] Step 4: The features fused in Step 3 will be further fed into the subsequent processing and prediction modules for bounding box regression and category prediction. The above network module is trained on the target dataset until convergence.

[0039] Furthermore, such as Figure 2 As shown, the scale-attention block in step three will be implemented through the following steps:

[0040] (3.1) Features of the nth level and features of the (n+1)th level First, it is converted into a 1-dimensional sequence of tokens, that is... and Where H n W n Features F n In the spatial dimensions of height and width, C represents the channel dimension of the feature; H n+1 W n+1 Features F n+1 Spatial dimensions in both height and width.

[0041] (3.2) Map the two feature sequences obtained in (3.1) to three spaces: Query, Key, and Value. Specifically, the feature matrices for the three spaces are obtained as follows: the Query matrix is ​​obtained by mapping the feature F at the nth level. n This is obtained by performing a linear projection, i.e.

[0042] Q = F n ×W,

[0043] Where W is the matrix parameter of the linear mapping, This is the mapped Query matrix; both the Key and Value matrices are directly concatenated using F... n and Fn+1 Two sets of features are obtained, namely

[0044] K = V = [F n F n+1 ],

[0045] Where [·] denotes a cascading operation, K, This represents the obtained Key matrix and Value matrix.

[0046] (3.3) The attention mechanism is applied to the three matrices Q, K, and V obtained in (3.2). Specifically, the Query matrix and the Key matrix are multiplied to obtain an attention response map, which is then applied to the Value matrix to obtain a new token sequence. In particular, the token sequence F of the nth level... n Ultimately, it will be added back to the new token sequence in the form of a residual concatenation.

[0047] The entire process can be represented as

[0048] F attn =softmax(QK) T V+F n ,

[0049] Among them, F attn For the obtained output feature matrix, K T Let K be the transpose of the key matrix. The softmax function, a normalized exponential function, is calculated as follows:

[0050] Where j = 1, ..., K

[0051] In particular, to save computational costs, the cross-scale attention modules described above are typically operated within local feature windows, and the window size can be customized according to specific needs.

[0052] Furthermore, the present invention also provides an image target detection device, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, wherein the instructions, after being executed by the processor, are used to complete the above-described image target detection method.

[0053] The test was conducted on environmental image data of relevant power inspection channels, and the results are shown in Table 1 below:

[0054] Table 1 Comparison of Test Results

[0055]

[0056]

[0057] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image target detection method, characterized by, Comprising the following steps: Step one: using a Transformer network as the backbone network to extract multi-resolution feature maps from the image; Step two: input the multi-resolution feature maps in step one into the cross-scale attention feature pyramid network; Step 3: In the feature pyramid network, starting from the small-resolution features input in Step 2, the cross-scale attention module is used to gradually fuse and reorganize the features towards larger resolutions. The features are accumulated and fused from small resolution to large resolution. The cross-scale attention module in Step 3 is implemented through the following steps: (3.1) Features at each level and the Features at each level First, it is converted into a 1-dimensional sequence of terms, that is... and ,in , Features In terms of spatial dimensions in both height and width The channel dimension size of the feature; , Features In terms of spatial dimensions in height and width; (3.2) map the two feature sequences obtained in (3.1) to three spaces: query, key, and value, to obtain the feature matrix of the three spaces. ; (3.3) performing attention mechanism operation on the three matrices obtained in (3.2); the Query matrix is obtained by performing linear mapping on the features of the first level, that is: wherein is the matrix parameter of linear mapping, is the mapped Query matrix; the Key matrix and the Value matrix are both obtained by directly concatenating the features of the second and the third level, that is wherein represents the concatenation operation, and the obtained Key matrix and Value matrix; the step (3.3) multiplies the Query matrix and the Key matrix to obtain an attention response graph, and the response graph is further applied to the Value matrix to obtain a new token sequence; in the step (3.3), the token sequence of the first level will finally be added to the new token sequence in the form of a residual connection, and the whole process is represented as wherein, is the obtained output feature matrix, is the transpose of the key matrix , and softmax is a normalized exponential function.​​ Step four: the fused features in step three are further sent to the subsequent processing and prediction module for bounding box regression and class prediction, and the cross-scale attention feature pyramid network is trained based on the target data set until convergence; Step five: using the trained cross-scale attention feature pyramid network to perform bounding box regression and class prediction on the image to be detected.

2. The image target detection method of claim 1, wherein, The cross-scale attention module operates within each local feature window.

3. The image target detection method of claim 2, wherein, The window size is determined according to specific requirements.

4. The image target detection method of claim 1, wherein, In the step one, the Transformer network is Swin-Transformer.

5. An image target detection device, characterized in that: It comprises at least one processor and a memory, which are connected through a data bus, the memory stores instructions executable by the at least one processor, and the instructions are used to complete the image target detection method in any one of claims 1-4 after being executed by the processor.

Images