Object detection method basedon attention mechanism and meta-learning

Abstract

An object detection method utilizing an attention mechanism and meta-learning is disclosed. The disclosed object detection method comprises: a step of generating a feature vector from an input image; a step of inputting the feature vector into a pre-trained attention network to determine a region of interest in the input image; and a step of classifying the class of an object included in the region of interest, wherein the attention network comprises: a first convolution network that performs a convolution operation on the feature vector using a kernel of a first size; and a second convolution network that performs a convolution operation on the output vector of the first convolution network using a kernel of a second size larger than the first size.

Description

Technology Field

[0001] The present invention relates to an object detection method, and more specifically, to an object detection method utilizing an attention mechanism and meta-learning. Background Technology

[0003] Technology for detecting objects in images is being researched in various fields, such as autonomous driving, tracking people on surveillance cameras, drones, and industrial site surveillance, and active research is being conducted in the relevant research, academic, and industrial sectors. Along with the advancement of artificial neural network technology, research utilizing artificial neural networks for object tracking is also actively underway, and algorithms such as Faster R-CNN and YOLO have been disclosed.

[0004] Furthermore, given that existing object detection algorithms are known to require a large amount of data, a major drawback, few-shot-based object detection networks are currently being researched. These networks offer advantages in terms of time, cost, and manpower for collecting, processing, and handling the large amount of data required for training. In addition, as meta-learning-based methods are widely used, methods for detecting objects using an attention mechanism are being developed to maximize the performance of meta-learning-based few-shot object detection networks.

[0005] An attention mechanism is a mechanism that predicts the location of features that require focus for object tracking. According to an attention mechanism-based object detection method, object detection performance can be improved by emphasizing and extracting features related to a specific area where an object is located.

[0006] Related prior art includes the patent documents Korean Patent Publication No. 2022-0129821 and No. 2022-0067931, and the non-patent document "Saliency Attention method for detecting protruding objects based on deep learning, Kim Hoe-jun et al., Journal of the Korea Convergence Society Vol. 11 No. 12, 2020". The problem to be solved

[0008] The present invention aims to provide an object detection method with improved object detection performance by utilizing an improved attention mechanism.

[0009] Furthermore, the present invention aims to provide a method for efficiently detecting objects even in environments where training data is scarce. means of solving the problem

[0011] According to one embodiment of the present invention for achieving the above-mentioned purpose, an object detection method is provided, comprising: a step of generating a feature vector from an input image; a step of inputting the feature vector into a pre-trained attention network to determine a region of interest in the input image; and a step of classifying the class of an object included in the region of interest, wherein the attention network comprises: a first convolution network that performs a convolution operation on the feature vector using a kernel of a first size; and a second convolution network that performs a convolution operation on the output vector of the first convolution network using a kernel of a second size larger than the first size.

[0012] Additionally, according to another embodiment of the present invention for achieving the above-mentioned purpose, an object detection method is provided comprising: a step of determining a region of interest in a query image and a support image using a pre-trained region of interest extraction network; and a step of classifying the class of an object included in the region of interest of the query image according to the class of an object included in the region of interest of the support image using a pre-trained classification network, wherein the weights of the region of interest extraction network and the classification network are updated through the training of a correlation feature extraction network that generates a correlation feature vector representing the similarity between the query image and the support image. Effects of the invention

[0014] According to one embodiment of the present invention, object detection performance can be improved by using an attention network in which the reception area is gradually expanded from local context information to global context information for an input image for object detection.

[0015] In addition, according to one embodiment of the present invention, through a meta-contrast learning method, objects can be efficiently detected even in an environment where training data is scarce. Brief explanation of the drawing

[0017] FIG. 1 is a drawing for explaining an object detection method according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an object detection method according to another embodiment of the present invention. FIG. 3 is a diagram illustrating a meta-contrast learning method according to an embodiment of the present invention. FIG. 4 is a diagram illustrating an artificial neural network for object detection according to an embodiment of the present invention. FIG. 5 is a diagram illustrating an attention network according to an embodiment of the present invention. FIG. 6 is a diagram illustrating first and second convolution networks according to an embodiment of the present invention. FIG. 7 is a diagram illustrating a correlation feature extraction network according to an embodiment of the present invention. Specific details for implementing the invention

[0018] The present invention is susceptible to various modifications and may have various embodiments; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention. Similar reference numerals have been used for similar components in the description of each drawing.

[0020] The attention mechanism first introduced in Seq2Seq models extracts feature information of the image to be emphasized from global contextual information regarding the image being detected as an object, and from this perspective, it can be viewed as a global attention mechanism. However, detecting objects using only global contextual information presents a problem in environments with insufficient training data, where it is difficult to learn which parts of the image features should be emphasized.

[0021] Based on this, the present invention proposes an object detection method based on an attention mechanism that utilizes both local context information and global context information for an image. An embodiment of the present invention sequentially extracts features regarding local context information and global context information for an image. An embodiment of the present invention extracts features regarding local context information for an image using a convolutional network and extracts global context information for an image using a global attention mechanism. Through this, unnecessary degrees of freedom associated with existing global attention mechanisms can be reduced, and object detection performance can be improved by directly utilizing the relationship with neighboring pixels in the image for object detection.

[0022] In addition, one embodiment of the present invention detects objects using a meta-contrast learning method improved from few-shot meta-learning. The meta-contrast learning method is a method of learning to extract a correlation feature vector representing the similarity between a query image and a support image, and is a learning method performed based on the similarity between a query image and a support image regardless of the class of the support image.

[0023] An object detection method according to one embodiment of the present invention can be performed on a computing device including memory and a processor.

[0024] Hereinafter, embodiments according to the present invention will be described in detail with reference to the attached drawings.

[0026] FIG. 1 is a drawing for explaining an object detection method according to an embodiment of the present invention.

[0027] Referring to FIG. 1, a computing device according to an embodiment of the present invention generates a feature vector from an input image (S110). For generating the feature vector, a convolution network may be used as an embodiment.

[0028] Then, the computing device inputs the feature vector generated in step S110 into a pre-trained attention network to determine a region of interest in the input image (S120). A Region Proposal Network (RPN) may be used to determine the region of interest, and the feature vector output from the attention network may be input into the RPN. Among the multiple regions of interest extracted through the RPN, the upper region of interest with a high class score may be determined as the region of interest for object detection.

[0029] The attention network includes a first and a second convolution network, and, depending on the embodiment, may further include a global attention network based on a global attention mechanism. The first convolution network performs a convolution operation on the feature vector generated in step S110 using a kernel of a first size. The first convolution may perform a depth-wise convolution operation.

[0030] Then, the second convolution network performs a convolution operation on the output vector of the first convolution network using a kernel of size 2, which is larger than the first size. The second convolution network can perform depth-wise dilated convolution operations. The output vector of the second convolution network can be input to the global attention network. The global attention network receives the output vector of the second convolution network and calculates an attention score for the input image. The global attention network calculates an attention score from the output vector of the second convolution network, just as a general attention network calculates an attention score from a feature vector for an image.

[0031] Since the size of the second kernel is larger than the size of the first kernel, the receptive field is gradually expanded, allowing local context information of the input image to be extracted. As an example, the size of the first kernel may be 5x5, and the size of the second kernel may be 7x7.

[0032] More specifically, the first convolution network may include first to third layers. The first layer performs 1x1 convolution, and the second layer performs depthwise convolution on the output vector of the first layer. The third layer performs 1x1 convolution on the output vector of the second layer.

[0033] The second convolution network may also include first to third layers. The first layer performs 1x1 convolution, and the second layer performs depthwise dilated convolution on the output vector of the first layer. The third layer performs 1x1 convolution on the output vector of the second layer. Depthwise dilated convolution refers to a convolution that combines depthwise convolution and dilated convolution.

[0034] A computing device according to an embodiment of the present invention classifies the class of objects included in the region of interest determined in step S120 (S130).

[0035] According to one embodiment of the present invention, object detection performance can be improved by using an attention network in which the reception area is gradually expanded from local context information to global context information for an input image for object detection.

[0037] FIG. 2 is a drawing for explaining an object detection method according to another embodiment of the present invention, and FIG. 3 is a drawing for explaining a meta-contrast learning method according to one embodiment of the present invention.

[0038] FIG. 2 is a diagram illustrating an object detection method combined with the aforementioned meta-contrast learning method, wherein, according to an embodiment, the aforementioned attention network may be used together for object detection.

[0039] Referring to FIG. 2, a computing device according to an embodiment of the present invention determines a region of interest in a query image and a support image (S210) using a pre-trained region of interest extraction network. As described above, the region of interest can be determined through the RPN.

[0040] Then, the computing device uses a pre-trained classification network to classify the class of an object included in the region of interest of the query image according to the class of an object included in the region of interest of the support image (S220). The computing device can determine the class of an object included in the region of interest of the support image as the class of an object included in the region of interest of the query image.

[0041] The weights of the region of interest extraction network and the classification network are updated through the training of a correlation feature extraction network that generates correlation feature vectors representing the similarity between the query image and the support image. The correlation feature extraction network that generates correlation feature vectors is not utilized in the object detection process but is a network used in the training process of the region of interest extraction network and the classification network, and is trained according to the aforementioned meta-contrast learning method.

[0042] The correlation feature extraction network is trained such that when the object class of the training query image and the object class of the training support image are the same, the similarity between the first correlation feature vector of the training query image and the training support image and the second correlation feature vector of the positive sample increases, and the similarity between the first correlation feature vector and the third correlation feature vector of the negative sample decreases. Positive samples include a pair of images containing objects of the same class, and the second correlation feature vector is a feature vector representing the similarity between the pair of images included in the positive samples. Negative samples include a pair of images containing objects of different classes, and the third correlation feature vector is a feature vector representing the similarity between the pair of images included in the negative samples.

[0043] The first correlation feature vector can be applied even when the object classes of the training query image and the training support image are different, in which case the positive samples include a pair of images containing objects of different classes, and the negative samples include a pair of images containing objects of the same class.

[0044] As illustrated in FIG. 3, when the class of the object (310) in the training query image and the object (320) in the training support image are the same as cat, the correlation feature extraction network can be trained so that the similarity between the second correlation feature vector and the first correlation feature vector of positive samples containing objects (330, 340) of the same class as dog is increased. And the correlation feature extraction network can be trained so that the similarity between the third correlation feature vector and the first correlation feature vector of negative samples containing objects (350, 360) of different classes as cat and bird is decreased. Similarity can be determined by the distance (metric) between the correlation feature vectors, such as cosine distance, L1 distance, L2 distance, etc.

[0045] That is, unlike the few-shot meta-learning method, which is based on the similarity between the feature vector of the query image and the feature vector of the support image, the meta-contrast learning method according to one embodiment of the present invention is based on the similarity between the correlation feature vector of the query image and the support image and the correlation feature vector of the positive sample and the negative sample.

[0046] Returning to FIG. 2, the computing device can generate feature vectors for the query image and support image, respectively, in step S220, and input the generated feature vectors into the attention network of the region of interest extraction network to determine regions of interest in the query image and support image. The attention network may include first and second convolution networks and a global attention network as described above.

[0047] According to one embodiment of the present invention, objects can be efficiently detected even in environments where training data is scarce through a meta-contrast learning method. For example, even in environments where it is difficult to obtain training data, such as with rare lesions, and where there is a very small amount of data, lesion regions can be efficiently detected and the class of lesions can be classified.

[0049] FIG. 4 is a diagram illustrating an artificial neural network for object detection according to an embodiment of the present invention.

[0050] Referring to FIG. 4, an artificial neural network according to an embodiment of the present invention includes a backbone network (410, 420), a HAM (430, 460, Hierarchical Attention Module), an RPN (440), a region of interest size adjustment unit (450, ROI Align), a classifier (470), a position adjustment unit (480), and a Meta-CLM (490, Meta-Contrastive Learning Module). Here, the HAM (430, 460) corresponds to the aforementioned attention network, and the Meta-CLM (490) corresponds to a correlation feature extraction network.

[0051] The backbone network (410, 420) generates feature vectors for the query image and support image, and the feature vectors are input to the HAM (430). The HAM (430) generates feature vectors for each of the query image and support image, with features for object detection emphasized, and then concatenates the feature vectors and transmits them to the RPN (440). The RPN (440) determines a region of interest for each of the query image and support image from the concatenated feature vectors, and the size of the determined region of interest is adjusted by the region of interest size adjustment unit (450).

[0052] HAM (460) generates feature vectors for regions of interest in the query image and support image and transmits them to the classifier (460) and the position control unit (470).

[0053] The classifier (460) determines the class of an object included in the region of interest of the query image through binary classification. That is, the classifier (460) can classify the class of an object included in the region of interest of the query image by determining whether the class of an object included in the region of interest of the query image is the same as the class of an object included in the region of interest of the support image.

[0054] The position adjustment unit (470) modifies the position of the region of interest in the query image and support image.

[0055] Meta-CLM (480) receives feature vectors for regions of interest of the training query images and training support images from HAM (460) during the training process, and outputs correlation feature vectors of objects included in each region of interest.

[0057] FIG. 5 is a drawing for explaining an attention network according to an embodiment of the present invention, and FIG. 6 is a drawing for explaining a first and second convolution network according to an embodiment of the present invention.

[0058] As described above, an attention network according to an embodiment of the present invention includes a first convolution network (511, 521), a second convolution network (512, 522), and a global attention network (530). Feature vectors (510, 520) for a query image and a support image, respectively, are input to the first convolution network (511, 521), and the global attention network (530) concatenates the feature vectors for the query image and the support image to produce an output (540). The first and second convolution networks (511, 512) to which feature vectors for the query image are input, and the first and second convolution networks (521, 522) to which feature vectors for the support image are input, share weights with each other.

[0059] Referring to FIG. 6, feature vectors (F) for the query image and support image q,s) is input to the first layer (610) that performs 1x1 convolution of the first convolution network (511, 521), passes through the activation function layer (620), and is input to the second layer (630) that performs depth-wise convolution. Then, the result of the tensor product operation between the output vector of the second layer (630) and the output vector of the activation function layer (620) is input to the third layer (640) that performs 1x1 convolution. Then, the output vector of the third layer (640) and the feature vector (F) for the query image and support image q,s The tensor sum operation of ) is performed.

[0060] Output vector of the first convolutional network ( ) is input to the first layer (650) that performs 1x1 convolution of the second convolution network (512, 522), passes through the activation function layer (660), and is input to the second layer (670) that performs depthwise expanded convolution. Then, the output vector of the second layer (670) passes through the 1x1 convolution layer (680), is tensor-multiplied with the output vector of the activation function layer (660), and is input to the third layer (690) that performs 1x1 convolution. Then, the output vector of the third layer (690) and the output vector of the first convolution network ( The tensor sum operation of ) is performed.

[0061] Output vector of the second convolutional network ( ) is input into the global attention network.

[0063] FIG. 7 is a diagram illustrating a correlation feature extraction network according to an embodiment of the present invention.

[0064] As shown in Fig. 7, the correlation feature extraction network provides feature vectors for the region of interest of the query image ( A first fully connected neural network (710) into which ) is input, and a feature vector for the region of interest of the support image ( It includes a second fully connected neural network (720) into which ) is input. The first and second fully connected neural networks (710, 720) share weights with each other.

[0065] The correlation feature extraction network connects the output vectors of the first and second fully connected neural networks (710) to output a correlation feature vector (730).

[0066] During the learning process, the first fully connected neural network (710) outputs a feature vector for an object included in the region of interest of the training query image, and the second fully connected neural network (720) outputs a feature vector for an object included in the region of interest of the training support image. Then, the correlation feature extraction network outputs a first correlation feature vector (730) in the form of the output vectors of the first and second fully connected neural networks (710) connected.

[0067] As previously described, the weights of the correlation feature extraction network and the weights of other networks are updated according to the similarity between the second correlation feature vector and the first correlation feature vector, and the similarity between the third correlation feature vector and the first correlation feature vector.

[0069] The technical details described above may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiments, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0071] As described above, the present invention has been explained by specific details such as specific components, limited embodiments, and drawings; however, this is provided merely to aid in a more comprehensive understanding of the invention, and the invention is not limited to the above embodiments. A person skilled in the art to which the invention pertains can make various modifications and variations from this description. Therefore, the scope of the invention should not be limited to the described embodiments, and all things equivalent to or having equivalent variations to the claims set forth below, as well as the claims themselves, shall be considered to fall within the scope of the concept of the invention.

Claims

Claim 1 An object detection method performed in a computing device including memory and a processor, comprising: a step of generating a feature vector from an input image; a step of inputting the feature vector into a pre-trained attention network to determine a region of interest in the input image; and a step of classifying the class of an object included in the region of interest, wherein the attention network comprises: a first convolution network that performs a convolution operation on the feature vector using a kernel of a first size; and a second convolution network that performs a convolution operation on the output vector of the first convolution network using a kernel of a second size larger than the first size, wherein the first convolution network performs a depth-wise convolution operation and the second convolution network performs a depth-wise dilated convolution operation, and the attention network further comprises a global attention network that receives the output vector of the second convolution network. Claim 2 delete Claim 3 An object detection method according to claim 1, wherein the first convolution network comprises: a first layer performing 1x1 convolution; a second layer performing depth-wise convolution on the output vector of the first layer; and a third layer performing 1x1 convolution on the output vector of the second layer. Claim 4 An object detection method according to claim 1, wherein the second convolution network comprises: a first layer performing a 1x1 convolution; a second layer performing a depth-wise expanded convolution on the output vector of the first layer; and a third layer performing a 1x1 convolution on the output vector of the second layer. Claim 5 delete Claim 6 An object detection method performed on a computing device including memory and a processor, comprising the step of determining a region of interest in a query image and a support image using a pre-trained region of interest extraction network; The object detection method includes the step of classifying the class of an object included in the region of interest of the query image according to the class of an object included in the region of interest of the support image using a pre-trained classification network, wherein the weights of the region of interest extraction network and the classification network are updated through the training of a correlation feature extraction network that generates a correlation feature vector representing the similarity between the query image and the support image, and wherein the correlation feature extraction network is trained such that when the class of an object in the training query image and the class of an object in the training support image are the same, the similarity between the first correlation feature vector of the training query image and the training support image and the second correlation feature vector of the positive sample increases, and the similarity between the first correlation feature vector and the third correlation feature vector of the negative sample decreases, wherein the positive sample includes a pair of images containing objects of the same class, and the negative sample includes a pair of images containing objects of different classes. Claim 7 delete Claim 8 In claim 6, the correlation feature extraction network comprises a first fully connected neural network that outputs a feature vector for an object in the training query image; and a first fully connected neural network that outputs a feature vector for an object in the training support image, wherein the first correlation feature vector is a vector in which the feature vectors for the object in the training query and the support image are connected. Claim 9 In claim 6, the step of determining the region of interest comprises: generating a feature vector for each of the query image and the support image; and inputting the feature vector into an attention network of the region of interest extraction network to determine a region of interest in the query image and the support image, wherein the attention network comprises: a first convolution network that performs a convolution operation on the feature vector using a kernel of a first size; and a second convolution network that performs a convolution operation on the output vector of the first convolution network using a kernel of a second size larger than the first size. Claim 10 An object detection method according to claim 9, wherein the first convolution network performs a depth-wise convolution operation and the second convolution network performs a depth-wise dilated convolution operation. Claim 11 An object detection method according to claim 10, wherein the first convolution network comprises: a first layer that performs 1x1 convolution; a second layer that performs depth-wise convolution on the output vector of the first layer; and a third layer that performs 1x1 convolution on the output vector of the second layer. Claim 12 In claim 10, the object detection method comprises: a first layer that performs a 1x1 convolution; a second layer that performs a depth-wise expanded convolution on the output vector of the first layer; and a third layer that performs a 1x1 convolution on the output vector of the second layer. Claim 13 In claim 9, the object detection method further includes a global attention network that receives the output vector of the second convolution network.

Images