Method and apparatus for context-embedded region-based object detection

Abstract

A method for detecting an object in an image using a convolutional neural network (CNN) includes: generating a plurality of reference feature maps and a corresponding feature pyramid including a plurality of final feature maps based on the image; obtaining a proposed region of interest (ROI); generating at least a first context ROI, the first context ROI having an area greater than an area of ​​the proposed ROI; assigning the proposed ROI and the first context ROI to a first final feature map and a second final feature map having different sizes; extracting a first set of features from the first final feature map using the proposed ROI and extracting a second set of features from the second final feature map using the first context ROI by performing ROI pooling; and determining at least one of a location of an object and a category of the object based on the extracted first set of features and the extracted second set of features.

Description

Technical Field

[0001] Various example embodiments generally relate to methods and apparatus for performing region-based object detection. Background Art

[0002] Object detection is a task in the field of computer vision, which aims to locate and identify object instances using bounding boxes. Object detection based on convolutional neural networks (CNNs) can be used in the fields of visual surveillance, advanced driver assistance systems (ADAS), and human-machine interaction (HMI).

[0003] Current object detection frameworks can be divided into two main streams: region-based methods and region-free methods. Examples of region-based detectors are discussed, for example, in YS Cao, X. Niu, and Y. Dou, "Region-based convolutional neural networks for object detection in very high resolution remote sensing images", In International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery, 2016, R. Girshick, "Fast r-cnn", Computer Science, 2015; and S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: toward real-time object detection with region proposal networks", International Conference on Neural Information Processing Systems, 2015, pp. 91-99. Typically, region-based methods divide object detection into two steps. In the first step, a region proposal network (RPN) generates high-quality proposals. Then, in the second step, the proposals are further classified and regressed by a region-by-region subnetwork. Typically, region-free methods detect objects by regular and dense sampling in locations, scales, and aspect ratios. Summary of the invention

[0004] According to at least some example embodiments, a method for detecting an object in an image using a convolutional neural network (CNN) includes: generating, by the CNN, a plurality of reference feature maps based on the image; generating a feature pyramid, the feature pyramid including a plurality of final feature maps respectively corresponding to the plurality of reference feature maps; obtaining a proposed region of interest (ROI); generating at least a first context ROI based on the proposed ROI, such that an area of ​​the first context ROI is greater than an area of ​​the proposed ROI; assigning the proposed ROI to a first final feature map among the plurality of final feature maps; assigning the first context ROI to a second final feature map among the plurality of final feature maps, the size of the first final feature map being different from that of the second final feature map; extracting a first set of features from the first final feature map by performing an ROI pooling operation on the first final feature map using the proposed ROI; extracting a second set of features from the second final feature map by performing an ROI pooling operation on the second final feature map using the first context ROI; and determining at least one of a position of the object relative to the image and a category of the object based on the extracted first set of features and the extracted second set of features.

[0005] A feature pyramid may be generated based on a plurality of reference feature maps according to a feature pyramid network (FPN) architecture.

[0006] The area of ​​the first context ROI can be twice the area of ​​the proposed ROI. 2 times.

[0007] The method may further include concatenating the extracted first set of features and the extracted second set of features, wherein determining includes determining at least one of a position of the object relative to the image and a category of the object based on the concatenated set of extracted features.

[0008] The method may further include applying the concatenated set of extracted features to an extrusion and excitation block (SEB), wherein at least one of a position of the object relative to the image and a class of the object is determined based on an output of the SEB.

[0009] The method may also include: generating a second context ROI based on the proposed ROI, so that the area of ​​the second context ROI is larger than the area of ​​the first context ROI; assigning the second context ROI to a third final feature map among the multiple final feature maps, the size of the third final feature map being different from the size of the first final feature map and the size of the second final feature map; and extracting a third set of features from the first final feature map by performing ROI pooling on the first final feature map using the second context ROI, wherein the determination includes determining at least one of a position of the object relative to the image and a category of the object based on the extracted first set of features, the extracted second set of features, and the extracted third set of features.

[0010] A feature pyramid may be generated based on a plurality of reference feature maps according to a feature pyramid network (FPN) architecture.

[0011] The area of ​​the first context ROI can be twice the area of ​​the proposed ROI. 2 times, and the area of ​​the second context ROI can be 4 times the area of ​​the proposed ROI 2 times.

[0012] The method may further include concatenating the extracted first set of features, the extracted second set of features, and the extracted third set of features, wherein determining includes determining at least one of a position of the object relative to the image and a category of the object based on the concatenated set of extracted features.

[0013] The method may further include applying the concatenated set of extracted features to an extrusion and excitation block (SEB), wherein at least one of a position of the object relative to the image and a class of the object is determined based on an output of the SEB.

[0014] According to at least some example embodiments, a computer-readable medium includes program instructions for causing an apparatus to at least perform the following: generate, by a convolutional neural network (CNN), a plurality of reference feature maps based on an image including an object; generate a feature pyramid including a plurality of final feature maps respectively corresponding to the plurality of reference feature maps; obtain a proposed region of interest (ROI); generate at least a first context ROI based on the proposed ROI, such that an area of ​​the first context ROI is greater than an area of ​​the proposed ROI; assign the proposed ROI to a first final feature map among the plurality of final feature maps; assign the first context ROI to a second final feature map among the plurality of final feature maps, a size of the first final feature map being different from a size of the second final feature map; extract a first set of features from the first final feature map by performing an ROI pooling operation on the first final feature map using the proposed ROI; extract a second set of features from the second final feature map by performing an ROI pooling operation on the second final feature map using the first context ROI; and determine at least one of a position of the object relative to the image and a category of the object based on the extracted first set of features and the extracted second set of features.

[0015] A feature pyramid may be generated based on a plurality of reference feature maps according to a feature pyramid network (FPN) architecture.

[0016] The area of ​​the first context ROI can be twice the area of ​​the proposed ROI. 2 times.

[0017] The computer-readable medium may also include program instructions for causing the apparatus to at least perform the following: cascading the extracted first set of features and the extracted second set of features, wherein determining includes determining at least one of a position of the object relative to the image and a category of the object based on the cascaded set of extracted features.

[0018] The computer-readable medium of claim 14 may further include program instructions for causing the apparatus to at least perform the following: applying the cascaded set of extracted features to an extrusion and excitation block (SEB), wherein at least one of a position of an object relative to the image and a class of the object is determined based on an output of the SEB.

[0019] According to at least some example embodiments, an apparatus includes: at least one processor; and at least one memory including computer program code, the at least one memory and the computer program code being configured to, together with the at least one processor, cause the apparatus to at least perform: generating, by a convolutional neural network (CNN), a plurality of reference feature maps based on an image including an object; generating a feature pyramid, the feature pyramid including a plurality of final feature maps respectively corresponding to the plurality of reference feature maps; acquiring a proposed region of interest (ROI); generating at least a first context ROI based on the proposed ROI, such that an area of ​​the first context ROI is greater than an area of ​​the proposed ROI; assigning the proposed ROI to a first final feature map among the plurality of final feature maps; assigning the first context ROI to a second final feature map among the plurality of final feature maps, the size of the first final feature map being different from that of the second final feature map; extracting a first set of features from the first final feature map by performing an ROI pooling operation on the first final feature map using the proposed ROI; extracting a second set of features from the second final feature map by performing an ROI pooling operation on the second final feature map using the first context ROI; and determining at least one of a position of the object relative to the image and a category of the object based on the extracted first set of features and the extracted second set of features.

[0020] A feature pyramid may be generated based on a plurality of reference feature maps according to a feature pyramid network (FPN) architecture.

[0021] The area of ​​the first context ROI may be twice the area of ​​the proposed ROI.

[0022] The at least one memory and the computer program code may also be configured to, together with the at least one processor, cause the apparatus to at least perform: cascading the extracted first set of features and the extracted second set of features, wherein determining comprises determining at least one of a position of the object relative to the image and a category of the object based on the cascaded set of extracted features.

[0023] The at least one memory and the computer program code may also be configured to, with the at least one processor, cause the apparatus to at least perform: applying the concatenated set of extracted features to an extrusion and excitation block (SEB), wherein at least one of a position of the object relative to the image and a class of the object is determined based on an output of the SEB. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] At least some example embodiments will become more fully understood through the detailed description and accompanying drawings provided below, in which like elements are represented by like reference numerals, which are given by way of example only and thus do not limit the example embodiments, and in which:

[0025] Figure 1 is a diagram of a monitoring network 10 according to at least some example embodiments.

[0026] Figure 2 is a diagram illustrating an example structure of an object detection apparatus according to at least some example embodiments.

[0027] Figure 3 The object detection subnetwork of the Multi-Scale Convolutional Neural Network (MS-CNN) detector is shown.

[0028] Figure 4 A portion of a backbone convolutional neural network (CNN) is shown, according to at least some example embodiments.

[0029] Figure 5 A Feature Pyramid Network (FPN) is shown according to at least some example embodiments.

[0030] Figure 6 A diagram illustrating a portion of a context-embedded, region-based object detection network 600 according to at least some example embodiments.

[0031] Figure 7 is a flow chart illustrating an example algorithm for performing a context-embedded, region-based object detection method according to at least some example embodiments. DETAILED DESCRIPTION

[0032] Various example embodiments will now be described more fully with reference to the accompanying drawings, in which some example embodiments are shown.

[0033] Detailed illustrative embodiments are disclosed herein. However, for the purpose of describing at least some example embodiments, the specific structural details and functional details disclosed herein are only representative. However, the example embodiments may be embodied in many alternative forms and should not be construed as being limited to the embodiments set forth herein.

[0034] Therefore, although the example embodiments are capable of various modifications and alternative forms, their embodiments are shown by way of example in the drawings and will be described in detail herein. However, it should be understood that it is not intended to limit the example embodiments to the specific forms disclosed, but on the contrary, the example embodiments will cover all modifications, equivalents, and alternative forms that fall within the scope of the example embodiments. Throughout the description of the drawings, like reference numerals refer to like elements. As used herein, the term "and / or" includes any and all combinations of one or more of the items listed in the relevant list.

[0035] It should be understood that when an element is referred to as being "connected" or "coupled" to another element, the element may be directly connected or coupled to the other element, or there may be intervening elements. Conversely, when an element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements. Other words used to describe the relationship between elements (e.g., "between" versus "directly between," "adjacent" versus "directly adjacent," etc.) should be interpreted in a similar manner.

[0036] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that when used herein, the terms "include" and / or "comprise" specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0037] It should also be noted that in some alternative implementations, the functions / actions mentioned may not occur in the order mentioned in the figures. For example, depending on the functions / actions involved, two figures shown in succession may actually be executed substantially simultaneously, or may sometimes be executed in the reverse order.

[0038] The exemplary embodiments are discussed herein as being implemented in a suitable computing environment. Although not required, the exemplary embodiments will be described in the general context of computer-executable instructions (e.g., program code) (such as program modules or functional processes) executed by one or more computer processors or CPUs. Typically, program modules or functional processes include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types.

[0039] In the following description, unless otherwise indicated, the illustrative embodiments will be described with reference to actions and symbolic representations (e.g., in the form of flow charts) of operations performed by one or more processors. It should be understood, therefore, that such actions and operations, sometimes referred to as computer-executed, involve manipulation by a processor of electrical signals representing data in a structured form. The manipulation transforms the data or saves it at a location in the memory system of the computer, thereby reconfiguring or otherwise changing the operation of the computer in a manner well known to those skilled in the art.

[0040] I. Overview

[0041] As discussed in more detail below, a context-embedded, region-based object detection method according to at least some example embodiments is based on a region-based object detection method and includes embedding a context branch to obtain rich context information to improve object detection. According to at least some example embodiments, the context information is beneficial for detecting small, blurred, and occluded objects. In addition, as discussed in more detail below, a context-embedded, region-based object detection method according to at least some example embodiments employs extrusion and excitation blocks in conjunction with context branches to reduce noise information, or alternatively avoid noise information. A context-embedded, region-based object detection method according to at least some example embodiments can be applied in several different ways, including, for example, visual surveillance.

[0042] An example structure of a monitoring network and an object detection device 100 that can utilize a context-embedded region-based object detection method according to at least some example embodiments will be discussed below in Section II of the present disclosure. Then, an example of performing object detection using a feature pyramid and context embedding will be discussed in Section III of the present disclosure. Then, an example of a convolutional neural network (CNN) architecture and algorithm for performing a context-embedded region-based object detection method according to at least some example embodiments will be discussed in Section IV of the present disclosure. In addition, a method for training a CNN architecture will be discussed in Section V of the present disclosure.

[0043] II. Region-Based Object Detection Methods for Implementing Contextual Embedding According to At Least Some Example Embodiments Example structure of

[0044] For example, Figure 1 1 shows a diagram of a monitoring network 10 according to at least some example embodiments. Figure 1 As shown, the monitoring network 10 may include an object detection device 100 and a monitoring system 150 .

[0045] Monitoring system 150 may include one or more cameras, each of which captures image data representing a scene near the camera location. Figure 1As shown, monitoring system 150 includes camera 152, which captures monitoring scene 154. Camera 152 can capture monitoring scene 154 by, for example, continuously capturing multiple time-adjacent images of monitoring scene 154 (i.e., capturing video or motion image data). According to at least some example embodiments, camera 152 transmits image data 120 corresponding to the captured monitoring scene 154 to object detection device 100. Now, reference will be made below to Figure 2 An example structure of the object detection device 100 is discussed in more detail.

[0046] Figure 2 is a diagram illustrating an example structure of an object detection apparatus 100 according to at least some example embodiments.

[0047] refer to Figure 2 , the object detection device 100 may include, for example, a data bus 259 , a transmission unit 252 , a reception unit 254 , a memory unit 256 , and a processing unit 258 .

[0048] The transmission unit 252 , the reception unit 254 , the memory unit 256 , and the processing unit 258 may send data to and / or receive data from each other using the data bus 259 .

[0049] The transmission unit 252 is a device including hardware and any necessary software for transmitting signals (including, for example, control signals or data signals) via one or more wired and / or wireless connections with one or more other network elements in a wireless communication network.

[0050] The receiving unit 254 is a device including hardware and any necessary software for receiving wireless signals (including, for example, control signals or data signals) via one or more wired and / or wireless connections with one or more other network elements in a wireless communication network.

[0051] The memory unit 256 may be any device capable of storing data, including magnetic storage, flash storage, etc. In addition, although not shown, the memory unit 256 may also include a port, a docking station, a drive (e.g., an optical drive), or an opening for receiving and / or installing a removable storage medium (e.g., one or more of a USB flash drive, an SD card, an embedded multimedia card (eMMC), a CD, a DVD, and a Blu-ray disc).

[0052] Processing unit 258 may be any device capable of processing data, including, for example, a processor.

[0053] According to at least one example embodiment, the present invention is described in detail with reference to Figures 1 to 7 Any operation described as being performed by the object detection device in any of the figures may be performed by a Figure 2 The structure of the object detection device 100 shown in the figure can be performed by an electronic device. For example, according to at least one example embodiment, the object detection device 100 can be programmed in software and / or hardware to perform any or all functions described herein as being performed by the object detection device. Therefore, the object detection device 100 can be implemented as a special-purpose computer through software and / or hardware programming.

[0054] An example of programming the object detection device 100 in software to perform any or all of the functions described herein as being performed by the object detection device will now be discussed below. For example, the memory unit 256 may store a program that includes executable instructions (e.g., program code) corresponding to any or all of the operations described herein as being performed by the object detection device. According to at least one example embodiment, in addition to or instead of being stored in the memory unit 256, the executable instructions (e.g., program code) may be stored in a computer-readable medium, the computer-readable medium including, for example, an optical disk, a flash drive, an SD card, etc., and the object detection device 100 may include hardware for reading data stored on the computer-readable medium. In addition, the processing unit 258 may be a processor that is configured to perform the operations described herein with reference to Figures 1 to 4 Any or all operations described as being performed by the object detection device, for example, by reading and executing executable instructions (e.g., program code) stored in at least one of the memory unit 256 and a computer-readable storage medium that is loaded into hardware included in the object detection device 100 to read the computer-readable medium.

[0055] Examples of programming the object detection device 100 in hardware to perform any or all of the functions described herein as being performed by the object detection device will now be discussed below. Figures 1 to 7 The processing unit 258 may include executable instructions (eg, program codes) corresponding to the functions performed by the object detection device, and may include a processor specifically configured to perform the functions described herein. Figures 1 to 6 A circuit (e.g., an integrated circuit) described as a structured design for any or all operations performed by an object detection device. For example, the above circuit included in the processing unit 258 may be an FPGA or an ASIC that is physically programmed to perform the referenced Figures 1 to 7 Describes any or all operations performed by an object detection device.

[0056] According to at least some example embodiments, the object detection device 100 uses context embedding to perform region-based object detection, which results in improved object detection performance for small, blurred, and occluded objects relative to other object detection methods, while also being able to detect objects at multiple scales. Two features, feature pyramids, and embedding contexts used by some other object detection methods will now be discussed in detail in Section III below.

[0057] III. Feature Pyramid and Embedding Context

[0058] For example, some object detection methods utilize feature pyramids, which include feature maps at multiple levels (i.e., multiple scales). For example, a region-based detector, multi-scale CNN (MS-CNN), uses convolutional layers of different spatial resolutions to generate region proposals of different scales. However, different layers of the MS-CNN detector may have inconsistent semantics. Examples of MS-CNN are discussed, for example, in Z. Cai, Q. Fan, RS Feris, and N. Vasconcelos "A unified multi-scale deep convolutional neural network for fast object detection", European Conference on Computer Vision. Springer, Cham, 2016.

[0059] Furthermore, in addition to using feature pyramids to generate region proposals, the MS-CNN detector also includes an object detection subnetwork that exploits contextual embeddings. Figure 3 The object detection subnetwork 300 of the MS-CNN detector is shown. Figure 3 As shown, the MS-CNN object detection subnetwork 300 includes a backbone CNN layer 310, a first feature map 320 corresponding to the conv4-3 convolution layer, and a second feature map 330 corresponding to the conv4-3-2x convolution layer, wherein the second feature map 330 is obtained by performing a deconvolution operation on the first feature map 320, so that the second feature map 330 is an enlarged version of the first feature map 320. Figure 3 In the example shown, the size of the first feature map 320 is H / 8×W / 8×512, and the size of the second feature map 330 is H / 4×W / 4×512, where H is the height of the input image originally input to the MS-CNN detector, and W is the width of the input image.

[0060] like Figure 3As shown, within the second feature map 330, there is a first region 334A (i.e., the innermost cube shown within the second feature map 330) and a second region 332A (i.e., the cube shown within the second feature map 330 as surrounding the first region 334A). The second region 332A is an enlarged version of the first region 334A and is 1.5 times larger than the first region 334A. Figure 3 As shown, through ROI pooling, the features corresponding to the first region 334A of the second feature map 330 are reduced to a first fixed-size feature map 334B having a size of 7×7×512. In addition, through ROI pooling, the features corresponding to the second region 332A of the second feature map 330 are reduced to a second fixed-size feature map 332B also having a size of 7×7×512. Figure 3 As shown, the MS-CNN object detection subnetwork 300 cascades the first fixed-size feature map 334B and the second fixed-size feature map 332B, reduces the resulting feature map to a third fixed-size feature map 340B having a size of 5×5×512, and feeds the features of the third fixed-size feature map 340B to the fully connected layer 350 to determine the class probability 370 and the bounding box 360. By using the enlarged second region 332A in conjunction with the first region 334A, the MS-CNN detector attempts to embed high-level context information of the feature pyramid included in the MS-CNN detector. However, since both the enlarged second region 332A and the first region 334A are mapped to the same level of the feature pyramid (i.e., the conv4-3-2x layer), the richness of the context information corresponding to the enlarged second region 332A may be limited.

[0061] On the contrary, as shown below Figures 4 to 6 As explained, the context-embedded region-based object detection method according to at least some example embodiments disclosed herein includes an embedded context branch that enables extraction of features corresponding to a proposed region of interest (RoI) and context information corresponding to one or more enlarged RoIs from multiple levels of a feature pyramid. Therefore, relative to the context information of the MS-CNN detector, the richness of the extracted context information can be improved, and therefore, the object detection performance of the context-embedded region-based object detection method according to at least some example embodiments can be improved.

[0062] Examples of convolutional neural network (CNN) architectures and algorithms for performing context-embedded region-based object detection methods according to at least some example embodiments will now be discussed in Section IV of the present disclosure.

[0063] IV. Region-Based Object Detection Methods for Implementing Contextual Embedding According to At Least Some Example Embodiments Example CNN architecture and algorithm

[0064] According to at least some example embodiments, reference is made below to Figures 4 to 7 The discussed CNN structure and algorithm can be found in the above references Figure 1 and Figure 2 Therefore, the following reference is made to the object detection device 100 discussed above. Figures 4 to 7 Any or all of the operations discussed may be performed or controlled by the object detection device 100 (ie, the processing unit 258 ).

[0065] According to at least some example embodiments, a CNN architecture for implementing a context-embedded region-based object detection method may include a backbone CNN and a feature pyramid network (FPN), which may be used together to implement one or both of a region proposal network (RPN) and a context-embedded region-based object detection network.

[0066] For example, Figure 4 A portion of a backbone CNN 400 according to at least some example embodiments is shown. In addition, one type of CNN that can be used as the backbone CNN 400 is a residual network CNN (i.e., ResNet), examples of which (including ResNet36 and ResNet50) are discussed in, for example, K He, X Zhang, S Ren, and J Sun, "Deep Residual Learning for Image Recognition", Proc. IEEE Computer Vision and Pattern Recognition, 2016. For simplicity, Figure 4 The structure of the backbone CNN 400 shown is the structure of the ResNet36 CNN. However, according to at least some example embodiments, the backbone CNN 400 is implemented by the ResNet50 CNN. In addition, the backbone CNN 400 is not limited to the ResNet36 CNN and the ResNet50 CNN. According to at least some example embodiments, the backbone CNN 400 can be implemented by any CNN that generates multiple feature maps with different scales.

[0067] like Figure 4 As shown, when the backbone CNN 400 is implemented by ResNet, the backbone CNN 400 may include a plurality of convolutional layers that respectively output a plurality of reference feature maps. Figure 4 The backbone CNN 400 shown includes a first convolutional layer conv1_x (not shown), an output second reference feature map C 2 The second convolutional layer conv2_x outputs the third reference feature map C 3 The third convolutional layer conv3_x outputs the fourth reference feature map C4 The fourth convolutional layer conv4_x, and the output fifth reference feature map C 5 The fifth convolutional layer conv5_x. As will be referenced below Figure 5 For a more detailed discussion, see Feature Figure C 2 , C 3 , C 4 and C 5 Can form the basis of FPN.

[0068] Figure 5 FIG. 5 shows an FPN 500 according to at least some example embodiments. The FPN 500 may be based on the reference feature maps (eg, the second to fifth reference feature maps C ) of the backbone CNN 400. 2 -C 5 ) to construct. For example, examples of FPN are discussed in: T. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature Pyramid Networks for Object Detection", Proc. IEEE Computer Vision and Pattern Recognition, 2017; T. Kong, F. Sun, A. Yao, H. Liu, M. Lu, and Y. Chen, "Ron: Reverse connection with objectness prior networks for object detection", Proc. IEEE Computer Vision and Pattern Recognition, 2017; and Lin TY, Goyal P, Girshick R et al., "Focal Loss for Dense Object Detection", Proc. IEEE Computer Vision and Pattern Recognition. Similar to the above references Figure 4 In contrast to the multi-scale feature maps of the MS-CNN detector discussed above, FPN 500 adopts a top-down architecture to create a feature pyramid that includes high-level semantic feature maps at all scales. For example, FPN 500 creates feature maps C and C respectively. k0+2 , C k0+1 , C k0 , C k0-1 , C k0-2 The corresponding final feature map P k0+2 , P k0+1 , P k0, P k0-1 , P k0-2 , where k 0 is a constant, k 0 The value of may be set, for example, according to the preferences of the designer and / or user of the object detection device 100. The constant k 0 We will refer to Equation 1 and Figure 6 as well as Figure 7 In addition, as shown below, Figure 6 and Figure 7 As discussed in more detail, the final feature map P generated by the FPN 500 may be used for one or both of region proposal and context-embedded region-based object detection.

[0069] Figure 6 A diagram illustrating a portion of a context-embedded region-based object detection network 600 according to at least some example embodiments. Figure 7 is a flow chart illustrating an example algorithm for performing a context-embedded region-based object detection method according to at least some example embodiments. Figures 4 to 7 An example algorithm for performing a context-embedded region-based object detection method according to at least some example embodiments is discussed with respect to the following example scenario, in which the algorithm is executed by an object detection device 100, and the object detection device 100 implements (i.e., embodies) the backbone CNN 400, the FPN 500, and the object detection network 600. Therefore, with reference to Figures 4 to 7 Operations described as being performed by the backbone CNN 400, FPN 500, or object detection network 600, or elements thereof, may be performed by the object detection device 100 (e.g., by executing computer-readable program code corresponding to the operations of the backbone CNN 400, FPN 500, and object detection network 600 by the processing unit 258 of the object detection device 100).

[0070] Furthermore, for the purpose of simplicity and ease of description, reference will be made to detecting a single object included in an input image. Figure 7 However, the algorithm for performing the context-embedded region-based object detection method according to at least some example embodiments is not limited to receiving an image including only one object, nor is it limited to detecting only one object. The input image may include multiple objects, and the algorithm is capable of simultaneously detecting multiple objects with different categories, locations, and scales.

[0071] refer to Figure 7 In step S710, the object detection device 100 receives an input image including an object. According to at least one example embodiment of the present inventive concepts, the object detection device 100 may receive the input image as part of the image data 120 received from the monitoring system 150, as described above with reference to Figure 1 After receiving the input image, the object detection apparatus 100 may apply the received image as input to the backbone CNN 400. After step S710, the object detection apparatus 100 proceeds to step S720.

[0072] In step S720, the object detection apparatus 100 may generate a reference feature map. For example, the object detection apparatus 100 may use the backbone CNN 400 to generate a plurality of reference feature maps based on the input image received in step S710.

[0073] For example, in step S720, the second to fifth convolutional layers {conv2_x, conv3_x, conv4_x, conv5_x} of the backbone CNN 400 may generate second to fifth reference feature maps {C 2 , C 3 , C 4 , C 5}. Reference feature graph {C 2 , C 3 , C 4 , C 5} may each have different sizes / scales, which are derived from the second reference feature map C 2 To the fifth reference feature map C 5 After step S720 , the object detection apparatus 100 proceeds to step S730 .

[0074] In step S730, the object detection apparatus 100 may generate a feature pyramid including a final feature map using FPN. For example, the object detection apparatus 100 may generate a feature pyramid including a plurality of final feature maps respectively corresponding to the plurality of reference feature maps generated in step S720.

[0075] For example, as referenced above Figure 5 As discussed in the FPN 500 shown in FIG. 5 , in step S720, the FPN 500 may generate the first final feature map to the fifth final feature map and an optionally additional sixth final feature map {P 2 , P 3 , P 4 , P 5 , P 6}. The first final feature map to the fifth final feature map {P 2 , P 3 , P 4 , P 5} may correspond to the first reference feature map to the fifth reference feature map generated in step S720, respectively. 2 , C 3 , C4 , C 5}. The sixth final feature map P 6 The FPN 500 can be based on the fifth final feature map P 5 To generate, for example, the fifth final feature map P 5 The stride 2 subsampling is discussed, for example, in T. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature Pyramid Networks for Object Detection”, Proc. IEEE Computer Vision and Pattern Recognition, 2017. The final feature map {P 2 , P 3 , P 4 , P 5 , P 6} may each have different sizes / scales, which are obtained from the second final feature map P 2 To the sixth final feature map P 6 After step S730 , the object detection apparatus 100 proceeds to step S740 .

[0076] In step S740 , the object detection apparatus 100 acquires the proposed region of interest (RoI or ROI) and generates one or more context RoIs.

[0077] For example, according to at least some example embodiments, the object detection apparatus 100 may acquire the proposed RoI from an external source. Alternatively, the object detection apparatus 100 may acquire the proposed RoI by implementing a region proposal network (RPN) based on the FPN 500 and generating the proposed RoI using the RPN based on the FPN.

[0078] For example, according to at least some example embodiments, Figure 5 The final feature map P generated by FPN 500 is shown k0+2 , P k0+1 , P k0 , P k0-1 , P k0-2It can be used to implement an RPN based on FPN. One of ordinary skill in the art will recognize that an example method for implementing an RPN based on FPN is discussed in T. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature Pyramid Networks for Object Detection”, Proc. IEEE Computer Vision and Pattern Recognition, 2017. For example, when k 0 =4, FPN 500 generates the second final feature map to the sixth final feature map P 2 , P 3 , P 4 , P 5 and P 6 . The sixth final feature map P 6 The fifth final feature map P may be generated in the same manner as discussed above with reference to step S730. 5 In addition, in order to generate region proposals, the FPN-based RPN can be used for the second final feature map to the sixth final feature map P 2 -P 6 Each final feature map in uses three anchors with different aspect ratios {1:2, 1:1, 2:1}, so that in the five different final feature maps {P 2 , P 3 , P 4 , P 5 , P 6 The anchor points used on} have 5 different areas {32 2 , 64 2 , 128 2 , 256 2 , 512 2}.

[0079] Therefore, in step S740 , the object detection apparatus 100 may acquire the proposed RoI by any one of receiving the proposed RoI and generating the proposed RoI.

[0080] In addition, in step S740, based on the acquired proposed RoI, the object detection device 100 may acquire one or more context RoIs by enlarging the proposed RoI. For example, Figure 6An input image 605, a proposed RoI 610, and a first context RoI 615A and a second context RoI 615B are shown. According to at least some example embodiments of the present inventive concepts, the object detection network 600 generates the first context RoI 615A by magnifying the area (i.e., w×h) of the proposed RoI 610 by a factor s1, and the object detection network 600 generates the second context RoI 615B by magnifying the area (i.e., w×h) of the proposed RoI 610 by a factor s2, where "w" is the width of the input image 605, "h" is the height of the input image 605, and both s1 and s2 are positive numbers greater than 1. Figure 6 In the example shown, s1 = 2 2 and s2 = 4 2 Furthermore, according to at least some example embodiments, the object detection network 600 may determine the coordinates of the context RoI generated by enlarging the proposed RoI in such a manner that the context RoI is concentric with the proposed RoI.

[0081] In addition, for the purpose of simplicity and ease of description, step S740 is described as obtaining the "proposed RoI". However, the algorithm for performing the context-embedded region-based object detection method according to at least some example embodiments is not limited to obtaining only one RoI or only one RoI at a time. For example, in step S740, the object detection device 100 can simultaneously obtain multiple RoIs with different positions, scales, and aspect ratios.

[0082] In addition, although step S740 is described above with reference to an example scenario in which, according to at least some example embodiments, two context RoIs (i.e., two enlarged versions of the proposed RoI 610) are generated, any number of context RoIs (e.g., 1, 3, 5, etc.) may be generated by enlarging the proposed RoI 610. After step S740, the object detection apparatus 100 proceeds to step S750.

[0083] In step S750, the object detection device 100 assigns the proposed RoI and one or more context RoIs to the final feature map. For example, in step S750, the object detection device may assign the proposed RoI 610, the first context RoI 615A, and the second context RoI 615B to the final feature map {P 2 , P 3 , P 4 , P 5 , P 6}The final feature map in .

[0084] For example, to perform the above reference assignment, the object detection apparatus 100 may use the following equation:

[0085]

[0086] In Equation 1, “w” represents width, “h” represents height, and k 0 is a constant, k 0 The value of k may be set, for example, according to the preference of the designer and / or user of the object detection device 100. 0 Additional details are discussed in document [6]. Figure 6 In the example scenario shown, k 0 =4. This means k 0 Corresponds to 224 2 The area (i.e., w×h=224 2 ). Equation 1 is discussed, for example, in T. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature Pyramid Networks for Object Detection,” Proc. IEEE Computer Vision and Pattern Recognition, 2017.

[0087] For each of the proposed ROI 610, the first context ROI 615A, and the second context ROI 615B, the object detection apparatus 100 may apply the width “w” and height “h” of the RoI to the above equation 1 to obtain an output k, and assign the RoI to the kth final feature map P k For example, in Figure 6 In the example scenario shown, when the width w and height h of the proposed RoI 610 are applied to Equation 1, k=3. Therefore, the object detection network 600 assigns the proposed RoI 610 to the third final feature map P 3 ,like Figure 6 Similarly, when the width w and height h of the first context RoI 615A and the second context RoI 615B are applied to Equation 1, respectively, k=4 and 5. Therefore, the object detection network 600 assigns the first context RoI 615A and the second context RoI 615B to the fourth final feature map P, respectively. 4 and the fifth final feature map P 5 ,like Figure 6 After step S750, the object detection apparatus 100 proceeds to step S760.

[0088] In step S760, the object detection apparatus 100 extracts a set of features from each final feature map assigned with one of the RoIs using RoI pooling. For example, in step S760, the object detection network 600 embodied by the object detection apparatus 100 may perform RoI pooling on the final feature maps assigned with the proposed RoI 610 and the proposed RoI 610. Specifically, with respect to the proposed RoI 610, the object detection network 600 performs RoI pooling on the final feature map assigned with the proposed RoI 610 (i.e., the third final feature map P 3 ) performs RoI pooling so that the third final feature map P 3 The features falling within the proposed RoI 610 are pooled through the RoI pooling operation to generate a fixed-size original feature map 620. Therefore, the fixed-size original feature map 620 is based on the initially proposed RoI (the proposed RoI 610) from the third final feature map P 3 A set of features extracted.

[0089] In addition, in step S760, the object detection network 600 forms a context branch 630 by performing RoI pooling on the first context RoI 615A and the second context RoI 615B and the final feature map to which the first context RoI 615A and the second context RoI 615B are assigned. Specifically, with respect to the first context RoI 615A and the second context RoI 615B, the object detection network 600 performs RoI pooling on the final feature map to which the first context RoI 615A and the second context RoI 615B are respectively assigned (i.e., the fourth final feature map P 4 and the fifth final feature map P 5 ) performs RoI pooling so that the fourth final feature map P 4 The features falling within the first context RoI 615A are pooled by the RoI pooling operation to generate a first fixed-size context feature map 632, and the fifth final feature map P 5 The features falling within the second context RoI 615B are pooled through the RoI pooling operation to generate a second fixed-size context feature map 634. Therefore, the first fixed-size context feature map 632 is based on the first context RoI 615A from the fourth final feature map P 4 A set of features extracted from the fifth final feature map P is obtained by extracting a set of features from the fifth final feature map P, and the second fixed size context feature map 634 is obtained based on the second context RoI 615B 5 A set of features extracted.

[0090] According to at least some example embodiments, the RoI pooling operation discussed above with reference to step S750 may be performed by using the operation of the RoI pooling layer discussed in document R. Girshick, “Fast r-cnn”, Computer Science, 2015. Alternatively, according to at least some example embodiments, the RoI pooling operation discussed above with reference to step S750 may be performed by using the operation of the RoIAlign layer. Examples of the RoIAlign layer are discussed, for example, in K. He, G. Gkioxari, P. Dollar, and R. Girshick, “Mask R-CNN”, In ICCV 2018. After step S760, the object detection device 100 proceeds to step S770.

[0091] In step S770, the object detection apparatus 100 determines the category and / or position of the object included in the image. For example, in step S770, the object detection network 600 may perform context embedding by cascading the first fixed-size context feature map 632 and the second fixed-size context feature map 634 to the fixed-size original feature map 620, thereby forming a cascade feature map 625, as shown in FIG. Figure 6 shown.

[0092] In addition, in relation to the above Figure 3 Compared to the discussed MS-CNN object detection subnetwork 300, the object detection network 600 can obtain richer context features and improved object detection results because the features included in the cascade feature map 625 are not all from the same convolutional layer or feature pyramid {P 2 , P 3 , P 4 , P 5 , P 6} extracted from the same layer.

[0093] Also like Figure 6As shown, the object detection network 600 includes a squeeze and excitation (SE) block 640, and the cascaded feature map 625 can be applied to the SE block 640 to reduce noise information, or alternatively eliminate noise information, for example, by recalibrating the channel-by-channel feature response. The SE block 640 includes two steps: squeezing and excitation. The first step is to compress the global spatial information into the channel descriptor. This is achieved by generating channel-by-channel statistics using global average pooling. The second step is adaptive recalibration. For example, the SE block 640 may include a fully connected layer fc1, followed by a rectified linear unit (ReLU), and the output of the rectified linear unit (ReLU) has a size of 1×1×C'. In addition, the SE block 640 may include another fully connected layer fc2, followed by a sigmoid, the output of the sigmoid has a size of 1×1×C (where typically C'=C / 16), and the sigmoid is used to rescale the initial features of the cascaded feature map 625, for example, via channel-by-channel multiplication, such as Figure 6 Example structures and methods for building and using SE blocks are described, for example, in Hu, Jie, Li Shen, and Gang Sun, “Squeeze-and-excitation networks”, arXiv:1709.01507, 2017.

[0094] Then, the category and bounding box (i.e., position) of the object included in the input image 605 are determined by using the output of the SE block 640 to generate a category probability value 660 and a bounding box value 670. For example, the output of the SE block 640 may be applied to another fully connected layer 650 to generate a category probability value (or category label) 660 and a bounding box value 670.

[0095] Object detection uses bounding boxes to accurately locate the location of an object and assign the correct category label to the object. When an image block or a video frame of a video is used as an input image in step S710, the category probability value 660 and the bounding box value 670 are the same as those in the above reference. Figures 4 to 7 Object detection results of region-based object detection methods with contextual embedding are discussed.

[0096] References Figures 4 to 7 At least some example embodiments of the discussed context-embedded region-based object detection methods may be applied to a variety of functions, including autonomous driving systems and video surveillance. Figure 1 When the camera 152 of the monitoring network 10 is placed at the entrance of the subway station, the above reference Figures 4 to 7The discussed context-embedded region-based object detection method object detection device 100 can help count pedestrian traffic through a subway. In addition, when the camera 152 of the monitoring network 10 is placed in a market, the object detection device 100 implementing the context-embedded region-based object detection method according to at least some example embodiments can help count the number of customers in the market, so that the owner or operator of the market can control the number of customers, for example, for safety reasons.

[0097] In addition, the context-embedded region-based object detection method according to at least some example embodiments includes: enlarging the size of the original RoI (e.g., the proposed RoI 610) so as to obtain more context information using the enlarged RoI (e.g., the first context RoI 615A and the second context RoI 615B). In addition, the enlarged RoI is mapped to a feature map different from the original RoI, thereby improving the representation capability of the context information obtained via the enlarged RoI. Therefore, the obtained context information is beneficial for the task of detecting small and occluded objects in the input image.

[0098] We will now discuss training a CNN architecture to perform the above reference in Section V below. Figures 4 to 7 Example approaches for contextually embedded region-based object detection methods are discussed.

[0099] V. Example Training Method

[0100] To execute the above reference Figures 4 to 7 The CNN architecture of the discussed context-embedded region-based object detection method can, for example, be trained according to known CNN training techniques to set various values ​​of filters used in various convolutional layers (e.g., Figure 4 Filters of the first to fifth convolutional layers conv1_x-conv5_x of the backbone CNN 400 shown).

[0101] To start the training phase, an appropriate loss function is designed. For the task of object detection, a multi-task loss function can be used. For example, examples of multi-task loss functions are discussed in Lin TY, Goyal P, Girshick R et al., "FocalLoss for Dense Object Detection", Proc. IEEE Computer Vision and Pattern Recognition, 2017. In addition, according to at least some example embodiments, training can be performed by using the Common Objects in Context (COCO) training and val-minus-minival datasets as training data. Using the back-propagation technique, the parameters of the above-mentioned filter can be iteratively updated until convergence through the stochastic gradient descent (SGD) algorithm.

[0102] Example embodiments being thus described, it will be apparent that the same may be varied in many ways. Such variations are not to be regarded as a departure from example embodiments, and all such modifications are intended to be included within the scope of example embodiments.

Claims

1. A method for detecting an object in an image using a convolutional neural network (CNN), the method comprising: Generating, by the CNN, a plurality of reference feature maps based on the image; generating a feature pyramid, the feature pyramid comprising a plurality of final feature maps respectively corresponding to the plurality of reference feature maps, wherein each of the plurality of reference feature maps is respectively associated with a different level of the feature pyramid; Get the proposed region of interest ROI; generating at least a first context ROI based on the proposed ROI, such that an area of ​​the first context ROI is larger than an area of ​​the proposed ROI; assigning the proposed ROI to a first final feature map of the plurality of final feature maps associated with one of the different levels of the feature pyramid; assigning the first context ROI to a second final feature map of the plurality of final feature maps associated with another level of the different levels of the feature pyramid, the size of the first final feature map being different from the size of the second final feature map; extracting a first set of features from the first final feature map by performing a ROI pooling operation on the first final feature map associated with the one of the different levels of the feature pyramid using the proposed ROI; extracting a second set of features from the second final feature map by performing an ROI pooling operation on the second final feature map associated with the other level in the different levels of the feature pyramid using the first context ROI; applying a concatenation of the first set of features and the second set of features to an extrusion and actuation block SEB; as well as Based on the output of the SEB, one or more probability values ​​associated with the location of the object relative to the image and the category of the object are determined.

2. The method according to claim 1, wherein the feature pyramid is generated based on the multiple reference feature maps according to a feature pyramid network (FPN) architecture.

3. The method according to claim 1, wherein the area of ​​the first context ROI is twice the area of ​​the proposed ROI. 2 times.

4. The method according to claim 1, The determination includes: At least one of a position of the object relative to the image and a category of the object is determined based on the concatenated set of extracted features.

5. The method according to claim 1, further comprising: generating a second context ROI based on the proposed ROI, such that an area of ​​the second context ROI is larger than an area of ​​the first context ROI; assigning the second context ROI to a third final feature map among the plurality of final feature maps, wherein a size of the third final feature map is different from the size of the first final feature map and the size of the second final feature map; as well as extracting a third set of features from the first final feature map by performing ROI pooling on the first final feature map using the second context ROI, The determining comprises: determining at least one of the position of the object relative to the image and the category of the object based on the extracted first set of features, the extracted second set of features, and the extracted third set of features.

6. The method of claim 5, wherein the feature pyramid is generated based on the plurality of reference feature maps according to a Feature Pyramid Network (FPN) architecture.

7. The method of claim 5, wherein the area of ​​the first context ROI is twice the area of ​​the proposed ROI. 2 times, and the area of ​​the second context ROI is 4 times the area of ​​the proposed ROI 2 times.

8. The method according to claim 5, further comprising: Cascading the extracted first set of features, the extracted second set of features, and the extracted third set of features, The determining comprises determining at least one of the position of the object relative to the image and the category of the object based on a concatenated set of extracted features.

9. The method according to claim 8, further comprising: The cascaded set of extracted features is applied to the extrusion and excitation block SEB, Wherein the position of the object relative to the image and the category of the object are determined based on the output of the SEB.

10. A computer readable medium comprising program instructions, wherein the program instructions are used to cause an apparatus to at least perform the following: Generate a plurality of reference feature maps based on an image using a convolutional neural network (CNN), wherein the image includes an object; generating a feature pyramid, the feature pyramid comprising a plurality of final feature maps respectively corresponding to the plurality of reference feature maps, wherein each of the plurality of reference feature maps is respectively associated with a different level of the feature pyramid; Get the proposed region of interest ROI; generating at least a first context ROI based on the proposed ROI, such that an area of ​​the first context ROI is larger than an area of ​​the proposed ROI; assigning the proposed ROI to a first final feature map of the plurality of final feature maps associated with one of the different levels of the feature pyramid; assigning the first context ROI to a second final feature map of the plurality of final feature maps associated with another level of the different levels of the feature pyramid, the size of the first final feature map being different from the size of the second final feature map; extracting a first set of features from the first final feature map by performing a ROI pooling operation on the first final feature map associated with the one of the different levels of the feature pyramid using the proposed ROI; extracting a second set of features from the second final feature map by performing an ROI pooling operation on the second final feature map associated with the other level in the different levels of the feature pyramid using the first context ROI; applying a concatenation of the first set of features and the second set of features to an extrusion and actuation block SEB; as well as Based on the output of the SEB, one or more probability values ​​associated with the location of the object relative to the image and the category of the object are determined.

11. The computer-readable medium of claim 10, wherein the feature pyramid is generated based on the plurality of reference feature maps according to a Feature Pyramid Network (FPN) architecture.

12. The computer-readable medium of claim 10, wherein the area of ​​the first context ROI is twice the area of ​​the proposed ROI. 2 times.

13. The computer readable medium of claim 10, The determination includes: At least one of the position of the object relative to the image and the category of the object is determined based on the concatenated set of extracted features.

14. An apparatus for performing object recognition, comprising: at least one processor; as well as at least one memory including computer program code, The at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform: Generate a plurality of reference feature maps based on an image using a convolutional neural network (CNN), wherein the image includes an object; generating a feature pyramid, the feature pyramid comprising a plurality of final feature maps respectively corresponding to the plurality of reference feature maps, wherein each of the plurality of reference feature maps is respectively associated with a different level of the feature pyramid; Get the proposed region of interest ROI; generating at least a first context ROI based on the proposed ROI, such that an area of ​​the first context ROI is larger than an area of ​​the proposed ROI; assigning the proposed ROI to a first final feature map of the plurality of final feature maps associated with one of the different levels of the feature pyramid; assigning the first context ROI to a second final feature map of the plurality of final feature maps associated with another level of the different levels of the feature pyramid, the size of the first final feature map being different from the size of the second final feature map; extracting a first set of features from the first final feature map by performing a ROI pooling operation on the first final feature map associated with the one of the different levels of the feature pyramid using the proposed ROI; extracting a second set of features from the second final feature map by performing an ROI pooling operation on the second final feature map associated with the other level in the different levels of the feature pyramid using the first context ROI; applying a concatenation of the first set of features and the second set of features to an extrusion and actuation block SEB; as well as Based on the output of the SEB, one or more probability values ​​associated with the location of the object relative to the image and the category of the object are determined.

15. The apparatus according to claim 14, wherein the feature pyramid is generated based on the plurality of reference feature maps according to a Feature Pyramid Network (FPN) architecture. The apparatus of claim 14 , wherein an area of ​​the first context ROI is twice an area of ​​the proposed ROI.

17. The device according to claim 14, The determination includes: At least one of the position of the object relative to the image and the category of the object is determined based on the concatenated set of extracted features.

Images