Tiny object detection method, device and equipment based on attention mechanism and storage medium

Abstract

The invention discloses a micro object detection method based on an attention mechanism, and the method comprises the steps: S1, obtaining a to-be-detected image, and extracting a feature vector of the to-be-detected image based on a convolutional neural network; s2, on the basis of a multi-head attention mechanism and softmax function processing, key information of feature vectors is extracted, probability distribution of each target object is obtained, and the target object with the highest probability is reserved; s3, splicing the feature vectors of the target object with the highest probability to form an image, comparing the image with the to-be-detected image, obtaining a region with the highest target probability in the to-be-detected image, and cutting and amplifying the region with the highest target object probability in the to-be-detected image; and S4, obtaining a probability value of existence of the tiny object in the region. The invention further discloses a tiny object detection device based on the attention mechanism, corresponding equipment and a storage medium. The method optimizes feature extraction, positioning and classification of tiny objects, and is suitable for high-precision detection tasks.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and more specifically, to a method, device, equipment and storage medium for detecting tiny objects based on an attention mechanism. Background Art

[0002] Deep learning technology in the field of AI is a fruitful way of current image detection. However, the existing methods of obtaining location information through target detection queries are not designed for detecting tiny objects. When it is necessary to detect the location information of tiny objects, because their size is much smaller than that of general objects, the algorithm obtains many candidate areas to implement exhaustive search, resulting in a large consumption of computing resources and reduced detection speed. At the same time, the existing methods of obtaining location information through target detection queries are not suitable for detecting samples containing tiny objects, such as subject aviation datasets and cell datasets, because the number of instances between different images is unbalanced, resulting in large differences in parameter magnitudes and reduced detection accuracy. Summary of the invention

[0003] In response to at least one defect or improvement need in the prior art, the present invention provides a tiny object detection method, apparatus, device and storage medium based on an attention mechanism, which can solve at least one of the technical problems existing in the above-mentioned background technology.

[0004] To achieve the above object, according to a first aspect of the present invention, a method for detecting tiny objects based on an attention mechanism is provided, the method comprising:

[0005] S1 obtains an image to be detected, and extracts a feature vector of the image to be detected based on a convolutional neural network;

[0006] S2 extracts the key information of the feature vector based on the multi-head attention mechanism and softmax function processing, obtains the probability distribution of each target object, and retains the target object with the highest probability;

[0007] S3 concatenates the feature vectors of the target object with the highest probability to form an image, compares the image with the image to be detected, obtains the area with the highest probability of the target in the image to be detected, and crops and enlarges the area with the highest probability of the target object in the image to be detected;

[0008] S4 obtains the probability value of the existence of a tiny object in the area.

[0009] Furthermore, the above-mentioned tiny object detection method based on the attention mechanism, before step S4, further includes:

[0010] Repeat steps S1-S3 until the size of the region with the highest probability of the target object in the cropped and enlarged image to be detected meets the first preset size.

[0011] Furthermore, the above-mentioned tiny object detection method based on the attention mechanism, based on the convolutional neural network, extracting the feature vector of the image to be detected, specifically includes:

[0012] The image to be detected Y0 is input into the convolutional neural network, and Y1 is obtained after convolution, batch normalization, ReLU activation function and maximum pooling;

[0013] Perform multi-layer downsampling on Y1 to obtain Y2, Y3 and Y4 respectively, and extract the feature vector of the image to be detected from Y4.

[0014] Furthermore, the above-mentioned tiny object detection method based on the attention mechanism, the multi-head attention mechanism and softmax function processing are used to extract the key information of the feature vector, specifically including:

[0015] The feature vectors are concatenated and expanded to obtain an out vector, and multiple sets of key matrices, value matrices, and query matrices are obtained through full connection;

[0016] Transpose each key matrix to get a new key matrix, multiply each new key matrix by the query matrix, and process the result with the softmax function to get multiple new matrices.

[0017] Multiply each new matrix by the value matrix, and concatenate each vector of the result to obtain multiple attention matrices;

[0018] The multiple attention matrices obtained are concatenated through the multi-head attention mechanism to obtain a mutli-head matrix.

[0019] According to a second aspect of the present invention, there is also provided a tiny object detection device based on an attention mechanism, comprising:

[0020] A feature extraction module is used to obtain an image to be detected and extract a feature vector of the image to be detected based on a convolutional neural network;

[0021] A feature attention module is used to extract key information of the feature vector based on a multi-head attention mechanism and softmax function processing, obtain the probability distribution of each target object, and retain the target object with the highest probability;

[0022] A feature magnification module is used to splice the feature vectors of the target object with the highest probability to form an image, compare the image with the image to be detected, obtain the area with the highest probability of the target in the image to be detected, and crop and magnify the area with the highest probability of the target object in the image to be detected;

[0023] The acquisition module is used to obtain the probability value of the existence of a tiny object in the area.

[0024] According to the third aspect of the present invention, there is also provided a tiny object detection device based on an attention mechanism, which comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit performs the steps of any one of the above-mentioned methods.

[0025] According to the fourth aspect of the present invention, a storage medium is also provided, which stores a computer program that can be executed by a tiny object detection device based on an attention mechanism. When the computer program runs on the tiny object detection device based on the attention mechanism, the tiny object detection device based on the attention mechanism executes the steps of any one of the above-mentioned methods.

[0026] According to a fifth aspect of the present invention, there is also provided a computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the steps of the small object detection method based on the attention mechanism as described in any one of the above claims are implemented.

[0027] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0028] The tiny object detection method based on the attention mechanism provided by the present invention, for tiny object detection scenarios, significantly improves the detection accuracy, robustness and stability through refined feature processing and enhanced expression of tiny objects. Especially in complex backgrounds and environments with uneven samples, the synergy of convolution calculation and multi-head attention mechanism optimizes the feature extraction, positioning and classification of tiny objects, and is suitable for high-precision detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 A flowchart of a tiny object detection method based on an attention mechanism is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0032] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0033] Figure 1 A flowchart of a method for detecting small objects based on an attention mechanism is provided in an embodiment of the present application, such as Figure 1 As shown, a method for detecting small objects based on an attention mechanism is provided in an embodiment of the present application, comprising:

[0034] S1 obtains an image to be detected, and extracts a feature vector of the image to be detected based on a convolutional neural network.

[0035] Specifically, the image to be detected can select a specific size according to the actual application situation. For example, when training and testing on the ImageNet dataset, the size of the input image is usually 224*224 pixels.

[0036] In one embodiment, the convolutional neural network can select the ResNet-50 network as a feature extractor to extract high-level feature representations of the image. Based on the detection of small target objects, in the feature extraction stage, it is chosen to capture more feature information by increasing the network depth, and the ResNet network architecture just meets the above requirements. When the convergence degree decreases due to increasing the network depth, the network performance does not degrade with the increase of depth. The output of ResNet-50 will be the feature vector of the input image, which can be used for subsequent deep learning tasks.

[0037] This step maps the image sample set from a high-dimensional space to a low-dimensional feature space, reducing the data dimension while extracting effective features for subsequent use.

[0038] S2 extracts the key information of the feature vector based on the multi-head attention mechanism and softmax function processing, obtains the probability distribution of each target object, and retains the target object with the highest probability.

[0039] Specifically, the obtained pixel values ​​of the feature map are first concatenated and expanded to obtain the out vector, and multiple sets of key, value and query are obtained through full connection. Then the obtained key matrix is ​​transposed to obtain multiple new key matrices, and then multiplied with the query matrix. After the softmax operation is performed on the result, multiple new matrices are obtained, and the new matrix is ​​multiplied with the value matrix. Each vector of the result is concatenated to obtain multiple attention matrices. Finally, the multiple attention matrices obtained through multi-head attention are concatenated to obtain the mutli-head matrix, which obtains the key information extracted by the multi-head attention, and then the result is obtained through full connection operation. The obtained information is optimized to generate a weight map that can emphasize the foreground object, enhance the features of important areas in the image, and further improve the feature expression ability of tiny objects.

[0040] S3 concatenates the feature vectors of the target object with the highest probability to form an image, compares the image with the image to be detected, obtains the area with the highest target probability in the image to be detected, and crops and enlarges the area with the highest probability of the target object in the image to be detected.

[0041] Specifically, firstly, the features are extracted through the convolution layer, and then the pixel values ​​of the object modules in the picture are increased through attention. Then, the obtained feature map is compared with the original image, and the cropping operation is performed on the original image to obtain the enlargement of the important target area. After these operations, the input of the next scale is obtained, and this process can be iterated. The most discriminative area in the image is located, and the probability value of the existence of a small object at a certain position. The objects to be classified are mainly similar in shape, contour and other features. Starting from the detail texture, high-resolution and refined operations are used to enlarge the features, thereby more efficiently improving the accuracy of refined object classification. Considering the fact that there are many similar targets in the recognized image, weighted feature enlargement is performed on the fine-grained image to accurately capture the small changes of the object.

[0042] S4 obtains the probability value of the existence of a tiny object in the area.

[0043] The tiny object detection method based on the attention mechanism provided in the embodiment of the present application, for tiny object detection scenarios, significantly improves the detection accuracy, robustness and stability through refined feature processing and enhanced expression of tiny objects, especially in complex backgrounds and environments with uneven samples. Through the synergy of convolution calculation and multi-head attention mechanism, it optimizes the feature extraction, positioning and classification of tiny objects, and is suitable for high-precision detection tasks.

[0044] Optionally, the tiny object detection method based on the attention mechanism provided in the embodiment of the present application further includes, before step S4:

[0045] Repeat steps S1-S3 until the size of the region with the highest probability of the target object in the cropped and enlarged image to be detected meets the first preset size.

[0046] Specifically, based on actual needs, a size can be estimated in advance as the first preset size, as the target size for enlarging tiny objects, at which the existence and position of tiny objects can be easily determined. The image spliced ​​from the features of the image to be detected can be cropped and enlarged, and the generated image can be further cropped and enlarged, and this can be iterated several times so that the size of the area with the highest probability of the target object in the cropped and enlarged image to be detected meets the first preset size.

[0047] Optionally, the method for detecting small objects based on the attention mechanism provided in the embodiment of the present application, wherein the feature vector of the image to be detected is extracted based on a convolutional neural network, specifically includes:

[0048] The image to be detected Y0 is input into the convolutional neural network, and Y1 is obtained after convolution, batch normalization, ReLU activation function and maximum pooling;

[0049] Perform multi-layer downsampling on Y1 to obtain Y2, Y3 and Y4 respectively, and extract the feature vector of the image to be detected from Y4.

[0050] Specifically, in the ResNet-50 network, the Y1 part extracts more spatial information than the Y0 part, and after multi-layer downsampling, the Y4 part extracts more semantically relevant information, and the target feature vector is obtained from the Y1-Y4 process through the forward propagation convolution operation.

[0051] In one embodiment, the specific process is to first input the image Y0 of size 3*224*224 into the network, and then obtain Y′1 of size 64*56*56 after processing based on convolution operation, Batch Normalization (BN) layer, ReLU activation function, and MaxPooling (MPL) layer.

[0052] Among them, Y′1 is a residual structure, that is, firstly, a 1*1 convolution operation is used to reduce the dimension of the feature map to complete the sampling, and then a 3*3 convolution kernel is used to continue the feature extraction operation, and finally a 1*1 convolution is used to adjust the dimension. At the same time, the convolution layer on the right uses a 1*1 convolution to match the difference in input and output dimensions to obtain the first layer of Y″1, with a size of 255*56*56. The subsequent operations of Y″1 are the same as most of the operations of Y′1, except that this part directly adds the input to the final output. The operation steps of each subsequent part are similar to those of Y′1 and Y″1, and finally the corresponding features are obtained through average pooling and Full Connection (FC) operations.

[0053] Optionally, the method for detecting small objects based on the attention mechanism provided in the embodiment of the present application, wherein the key information of the feature vector is extracted based on the multi-head attention mechanism and the softmax function processing, specifically includes:

[0054] The feature vectors are concatenated and expanded to obtain an out vector, and multiple sets of key matrices, value matrices, and query matrices are obtained through full connection;

[0055] Transpose each key matrix to get a new key matrix, multiply each new key matrix by the query matrix, and process the result with the softmax function to get multiple new matrices.

[0056] Multiply each new matrix by the value matrix, and concatenate each vector of the result to obtain multiple attention matrices;

[0057] The multiple attention matrices obtained are concatenated through the multi-head attention mechanism to obtain a mutli-head matrix.

[0058] The present application also provides a tiny object detection device based on an attention mechanism, comprising:

[0059] A feature extraction module is used to obtain an image to be detected and extract a feature vector of the image to be detected based on a convolutional neural network;

[0060] A feature attention module is used to extract key information of the feature vector based on a multi-head attention mechanism and softmax function processing, obtain the probability distribution of each target object, and retain the target object with the highest probability;

[0061] A feature magnification module is used to splice the feature vectors of the target object with the highest probability to form an image, compare the image with the image to be detected, obtain the area with the highest probability of the target in the image to be detected, and crop and magnify the area with the highest probability of the target object in the image to be detected;

[0062] The acquisition module is used to obtain the probability value of the existence of a tiny object in the area.

[0063] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0064] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0065] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0066] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0067] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0068] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0069] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program code.

[0070] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0071] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0072] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0073] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A small object detection method based on attention mechanism, characterized in that: include: S1 obtains an image to be detected, and extracts a feature vector of the image to be detected based on a convolutional neural network; S2 extracts the key information of the feature vector based on the multi-head attention mechanism and softmax function processing, obtains the probability distribution of each target object, and retains the target object with the highest probability; S3 concatenates the feature vectors of the target object with the highest probability to form an image, compares the image with the image to be detected, obtains the area with the highest probability of the target in the image to be detected, and crops and enlarges the area with the highest probability of the target object in the image to be detected; S4 obtains the probability value of the existence of a tiny object in the area.

2. The method for detecting small objects based on the attention mechanism according to claim 1, characterized in that: Before step S4, the method further includes: Repeat steps S1-S3 until the size of the region with the highest probability of the target object in the cropped and enlarged image to be detected meets the first preset size.

3. The method for detecting small objects based on the attention mechanism as claimed in claim 1, characterized in that: The extracting the feature vector of the image to be detected based on the convolutional neural network specifically includes: The image to be detected Y0 is input into the convolutional neural network, and Y1 is obtained after convolution, batch normalization, ReLU activation function and maximum pooling; Perform multi-layer downsampling on Y1 to obtain Y2, Y3 and Y4 respectively, and extract the feature vector of the image to be detected from Y4.

4. The method for detecting small objects based on the attention mechanism as claimed in claim 1, characterized in that: The key information of the feature vector is extracted based on the multi-head attention mechanism and softmax function processing, specifically including: The feature vectors are concatenated and expanded to obtain an out vector, and multiple sets of key matrices, value matrices, and query matrices are obtained through full connection; Transpose each key matrix to get a new key matrix, multiply each new key matrix by the query matrix, and process the result with the softmax function to get multiple new matrices. Multiply each new matrix by the value matrix, and concatenate each vector of the result to obtain multiple attention matrices; The multiple attention matrices obtained are concatenated through the multi-head attention mechanism to obtain a mutli-head matrix.

5. A tiny object detection device based on attention mechanism, characterized in that: include: A feature extraction module is used to obtain an image to be detected and extract a feature vector of the image to be detected based on a convolutional neural network; A feature attention module is used to extract key information of the feature vector based on a multi-head attention mechanism and softmax function processing, obtain the probability distribution of each target object, and retain the target object with the highest probability; A feature magnification module is used to stitch the feature vectors of the target object with the highest probability to form an image, compare the image with the image to be detected, obtain the area with the highest probability of the target in the image to be detected, and crop and magnify the area with the highest probability of the target object in the image to be detected; The acquisition module is used to obtain the probability value of the existence of a tiny object in the area.

6. A tiny object detection device based on an attention mechanism, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method according to any one of claims 1 to 4.

7. A storage medium, characterized in that: It stores a computer program that can be executed by a tiny object detection device based on an attention mechanism. When the computer program runs on the tiny object detection device based on the attention mechanism, the tiny object detection device based on the attention mechanism executes the steps of the method described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the small object detection method based on the attention mechanism as described in any one of claims 1 to 4 are implemented.

Images