Convolutional neural network for object detection

Abstract

A convolutional neural network for object detection is disclosed. A computer-readable medium including program code is also disclosed, which, when executed by a processing circuitry system, causes the system to generate a feature map from an input image, extract a region of interest from the feature map, and generate a prediction mask based on the region of interest. The processing circuitry system can use the prediction mask and the true mask to learn the convolutional neural network system. The true mask includes a first pixel corresponding to a true boundary and a second pixel corresponding to a pseudo boundary adjacent to the true boundary.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2018-0123183 filed on October 16, 2018 and Korean Patent Application No. 10-2019-0036153 filed on March 28, 2019, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] Some exemplary embodiments of the inventive concepts described herein relate to convolutional neural network systems, and more specifically, to a convolutional neural network system and its operation method that can improve learning efficiency. Background Technology

[0004] In the context of object detection in images or videos, deep learning-based convolutional neural networks (CNNs) are being actively researched and developed. CNNs consist of multiple layers that repeatedly perform convolution and pooling operations. However, because there is a trade-off between the object detection performance and the depth of CNNs, many attempts have been made to improve object detection performance while simplifying the neural network algorithm.

[0005] In some examples of object detection, in addition to the problem of identifying objects from the background, instance segmentation may also involve the problem of distinguishing objects from one another. Typically, significant computation and learning can be utilized to improve the efficiency of instance segmentation. Furthermore, instance segmentation can be directly related to the performance of object detection. Therefore, a relatively simple algorithm that can improve the performance of instance segmentation might be desirable. Summary of the Invention

[0006] Some exemplary embodiments of some inventive concepts provide a convolutional neural network system that can improve learning efficiency.

[0007] In detail, some exemplary embodiments of some inventive concepts can provide a way to improve learning efficiency by changing the pixel thickness of the true mask used for learning.

[0008] According to some example embodiments, a computer-readable medium includes program code that, when executed by a processing circuitry system, causes the processing circuitry system to perform a convolution operation on an input image to generate a feature map, extract a region of interest (ROI) from the feature map based on an object state score associated with the presence of an object, align the extracted ROI with an ROI having a reference size, determine the object category and the object's position information on the input image based on the aligned ROI, form a boundary surrounding the object on the input image based on the determined result, and learn a convolutional neural network system based on a predicted mask based on the category, position information, and / or the boundary, and / or a true mask based on the true boundary of the object in the input image, and / or the true mask includes a first pixel corresponding to the true boundary and / or a second pixel corresponding to a pseudo boundary adjacent to the true boundary.

[0009] According to some example embodiments, a computer-readable medium includes program code that, when executed by a processing circuitry system, causes the processing circuitry system to perform a convolution operation on an input image to generate a feature map, extract a region of interest (ROI) from the feature map based on an object state score associated with the presence of an object, align the extracted ROI with an ROI having a reference size, determine the object category and the object's location information on the input image based on the aligned ROI, form a boundary surrounding the object on the input image based on the determined result, and learn a convolutional neural network system based on a predicted mask based on the category, location information, and / or boundary, and a true mask including true bounding boxes surrounding the object on the input image, and / or the true mask including a first pixel corresponding to the true bounding box and a second pixel corresponding to a pseudo bounding box adjacent to the true bounding box.

[0010] According to some example embodiments, a convolutional neural network system includes: a processing circuit system configured to perform a convolution operation on an input image to generate a feature map, extract a region of interest (ROI) from the feature map based on an object state score associated with the presence of an object, and align the extracted ROI with a ROI having a reference size; a classifier that determines the category of an object based on the aligned ROI, determines the location information of the object on the input image based on the aligned ROI, and forms a boundary surrounding the object on the input image. The convolutional neural network system may use a predictive mask based on category, location information, and boundary, and a true mask based on the true boundary of the object in the input image, wherein the true mask may include a first pixel corresponding to the true boundary and a second pixel corresponding to a pseudo-boundary adjacent to the true boundary. Attached Figure Description

[0011] The above and other objects and features of the present invention will become apparent from the detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings.

[0012] Figure 1 Example configurations of convolutional neural network systems according to some example embodiments of this disclosure are shown.

[0013] Figure 2 Illustrations based on some example embodiments Figure 1 Example configuration of the feature extractor.

[0014] Figure 3 Illustrations based on some example embodiments Figure 1 Example configuration of the feature extractor.

[0015] Figure 4 Conceptually illustrated according to some example embodiments Figure 1 Example operation of the RoI extractor.

[0016] Figure 5 Conceptually illustrated according to some example embodiments Figure 1 Example alignment of the RoI aligner.

[0017] Figure 6 Illustrations based on some example embodiments Figure 1 Example configuration of the splitter.

[0018] Figure 7 An example general true mask is shown that can be used with a predictive mask output from a segmenter, according to some example embodiments.

[0019] Figure 8 Example true masks that can be used with predictive masks are shown according to some example embodiments of the present disclosure.

[0020] Figure 9 and Figure 10 Examples of thick boundary shape masks that can be used with predictive masks are shown according to some example embodiments of the present disclosure.

[0021] Figure 11 Example true masks that can be used with predictive masks are shown according to some example embodiments of the present disclosure.

[0022] Figure 12 Examples of thick boundary shape masks that can be used with predictive masks are shown according to some example embodiments of the present disclosure.

[0023] Figure 13 An example score boundary shape mask that can be used with a prediction mask is shown according to some example embodiments of the present disclosure.

[0024] Figure 14 An example of a general true mask is shown for use with a prediction mask output from a segmenter.

[0025] Figure 15Example true masks that can be used with predictive masks are shown according to some example embodiments of the present disclosure.

[0026] Figure 16 Example true masks that can be used with predictive masks are shown according to some example embodiments of the present disclosure.

[0027] Figure 17 The present invention conceptually illustrates examples of how a convolutional neural network system according to some example embodiments of the present disclosure may use a prediction mask and a score boundary shape mask.

[0028] Figure 18 An example learning processor of a convolutional neural network system according to some example embodiments of the present disclosure is illustrated schematically.

[0029] Figure 19 and Figure 20 Conceptually, this illustrates example processing for learning backpropagation in a convolutional neural network system, based on some example embodiments.

[0030] Figure 21 Example configurations of convolutional neural network systems according to some example embodiments of this disclosure are shown.

[0031] Figure 22 Example configurations of convolutional neural network systems according to some example embodiments of this disclosure are shown.

[0032] Figure 23 Example configurations of convolutional neural network systems according to some example embodiments of this disclosure are shown.

[0033] Figure 24 Example configurations of convolutional neural network systems according to some example embodiments of this disclosure are shown.

[0034] Figure 25 Examples of object detection by a convolutional neural network system capable of learning bounding box masks, according to some exemplary embodiments of the present disclosure, are shown.

[0035] Figure 26 Examples of object detection based on boundary shape mask learning convolutional neural network systems according to some exemplary embodiments of the present disclosure are shown. Detailed Implementation

[0036] Below, some exemplary embodiments of inventive concepts are described in detail and clearly, enabling those skilled in the art to easily implement some inventive concepts.

[0037] The components described in the specific embodiments, as well as the functional blocks shown in the accompanying drawings, can be implemented using software, hardware, or a combination thereof. In some example embodiments, the software may be machine code, firmware, embedded code, and application software. For example, the hardware may include electrical circuits, electronic circuits, processors, computers, integrated circuits, integrated circuit cores, pressure sensors, inertial sensors, microelectromechanical systems (MEMS), passive components, or combinations thereof.

[0038] Figure 1 Example configurations of convolutional neural network systems according to some exemplary embodiments of this disclosure are shown. Figure 1 In the example, the convolutional neural network system 100 is configured to process an input image and generate an output image. For example, the output image includes bounding shapes of the outlines of objects included in the image and / or bounding boxes that define the boundaries of the objects as rectangular boxes. In other words, the convolutional neural network system 100 can be configured to search for objects included in an image and / or generate bounding shapes and / or bounding boxes corresponding to the detection results.

[0039] In some example embodiments, one or more components of the convolutional neural network system may be included in and / or implemented by a processing circuitry system. For example, the processing circuitry system may include: hardware, such as logic circuitry; hardware / software combinations, such as a processor executing software; or combinations thereof. For example, a processor may include (but is not limited to) a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc.

[0040] Some example embodiments may include computer-readable media (CRM). Such computer-readable media may include various types of computer-readable media. As a first example, a first subset of computer-readable media may include physical tangible media, such as hard disk drives, solid-state storage devices, flash memory devices, magnetic and / or electronic circuitry (e.g., memory chips), and / or optical media (e.g., CD-ROMs or DVD-ROMs); a second subset of computer-readable media may include intangible media such as propagated electromagnetic signals; and a third subset of computer-readable media may include a combination of one or more physical tangible media and one or more intangible media. As a second example, a first subset of computer-readable media may include volatile memory, such as periodically refreshed system RAM; a second subset of computer-readable media may include non-volatile and / or non-transitory memory, such as hard disk drives; and a third subset of computer-readable media may include a combination of volatile memory and non-volatile or non-transitory memory, such as persistent memory. As a first set of example embodiments, computer-readable media may encode program code (e.g., a set of instructions) that, when read and processed by a processing circuitry system, causes the processing circuitry system to perform operations including at least a portion of this disclosure. As a second set of example embodiments, a computer-readable medium may include data generated by another example embodiment (e.g., a processing circuitry system) that generates a neural network system according to the present disclosure. As a third set of example embodiments, a computer-readable medium may include data that induces or causes the generation of another example embodiment, such as a serialized version of a neural network system according to the present disclosure and / or a mapping of a field-programmable gate array (FPGA) that, when programmed using that mapping, operates as a neural network system according to the present disclosure. Some computer-readable media may include combinations of one or more such example embodiments as described herein.

[0041] In some example embodiments, the convolutional neural network system 100 may include a convolutional neural network (CNN) that repeatedly performs convolution operations between input data or kernels that indicate specific features and specific parameters (e.g., weights or biases). For example, the convolutional neural network may be based on (but not limited to) R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, and / or various other types of convolutional neural networks similar to these. Below, in this specification, a mask R-CNN-based convolutional neural network system will be described in some examples.

[0042] In some example embodiments, the convolutional neural network system 100 may include a feature extractor 110, a region of interest (RoI) extractor 120, a RoI aligner 130, a classifier 140, a bounding box regressor 150, and / or a segmenter 160.

[0043] In some example embodiments, the processing circuitry may include a feature extractor 110, which may include multiple layers configured to apply multiple feature maps from an input image. For example, the feature extractor 110 may include at least one convolutional layer configured to perform convolution operations and / or at least one pooling layer configured to perform computational operations (e.g., subsampling). The feature extractor 110 may be configured to determine patterns and / or extract features from the input image, for example, by repeatedly performing convolution and / or pooling operations. In some example embodiments, the feature extractor 110 may include a Residual Network (ResNet) 101, a Residual Network 50, or a similar network.

[0044] In some example embodiments, the processing circuitry may include a RoI extractor 120 configured to search for at least one region from a feature map that may contain an object. This region is referred to as a “Region of Interest (RoI)”. The RoI extractor 120 may include a Region Proposal Network (RPN) configured to determine regions on the feature map where an object is likely (e.g., very likely) to be present. The RPN may be configured to scan the feature map in a sliding manner (e.g., by using multiple anchor points) and / or search for RoIs. For example, the RPN may add an “objectness score” to one or more proposals. The “objectness score” may represent a score (probability) associated with whether an object is visible in the RoI and / or whether a set of pixels presented in the image contains an object. For example, the RPN may be implemented using a fully convolutional network (but not limited to this).

[0045] In some example embodiments, the processing circuitry may include a RoI aligner 130, which may be configured to adjust the size of each RoI output from the RoI extractor 120 and / or to generate feature maps. For example, the RoI aligner 130 may be configured to convert RoIs of various sizes to a reference size. For example, RoIs of various sizes may be extracted by the RoI extractor 120. In some example embodiments, the ground truth coordinates of the feature map may be quantized to adjust the various sizes of the RoIs. However, in some examples, this adjustment may result in misalignment due to the nature of quantization. Therefore, in some example embodiments, the RoI aligner 130 may be configured to use bilinear interpolation to adjust the size of each RoI.

[0046] In some example embodiments, the processing circuitry may include a classifier 140, which may be configured to determine information relating to the category of each aligned RoI and / or the category of objects visible therein. For example, a category represents the specific class of objects in each aligned RoI (e.g., person, chair, or car). In some example embodiments, the classifier 140 may be configured to determine a score associated with whether an object in each aligned RoI belongs to a particular category and / or, for example, determine the category of the RoI based on the determined score.

[0047] In some example embodiments, the processing circuitry may include a bounding box regressor 150, which may be configured to determine information relating to the location of each aligned RoI. For example, the information relating to the location of the RoI may relate to the position and / or size of the rectangular bounding box surrounding each aligned RoI. For example, the information relating to the location of the RoI may include the coordinates of the four vertices of the rectangular bounding box surrounding an object on the input image.

[0048] In some example embodiments, the classifier 140 and / or bounding box regressor 150 described above can be implemented using a Region Classification Network (RCN). For example, in Figure 1 In the example embodiments, classifier 140 and / or bounding box regressor 150 are shown as independent components, but in some other example embodiments, these two components may be implemented as a functional block of RCN. In this case, RCN may include multiple fully connected networks, and operations using fully connected networks may be referred to as "fully connected operations". Aligned RoIs input to the fully connected network may be fully connected data, and / or all nodes of the input data may maintain connections with all nodes of the output data.

[0049] In some example embodiments, the processing circuitry may include a segmenter 160, which may be configured to generate a boundary shape mask for object detection. For example, since the segmenter 160 can perform segmentation based on each aligned RoI, it may be configured to identify multiple objects belonging to the same category. Furthermore, since the segmenter 160 may be configured to generate a boundary shape mask for each category, conflicts between categories may not occur or may be reduced. In some example embodiments, the segmenter 160 may include a fully convolutional network (FCN). In some example embodiments, when implementing the segmenter 160 using an FCN, various networks such as AlexNet, VGGNet, GoogLeNet, or similar networks may be used. In some example embodiments, to prevent the loss of object location information due to the FCN, the final layer of the FCN may be implemented using a 1×1 convolutional layer. As a result, spatial information for each aligned RoI may be maintained and / or scores for objects with high values ​​may be output.

[0050] In some example embodiments, the bounding shape mask output from segmenter 160 may be an inaccurate mask that does not match the contour of the real object. Therefore, convolutional neural network system 100 may be configured to perform learning on segmenter 160 via backpropagation. That is, processing can be performed to reduce errors by adjusting the bias and / or weight values ​​of the network constituting segmenter 160. Additionally, since the object category and bounding box may be inaccurate, a neural network system including classifier 140 and / or bounding box regressor 150 may be used to perform the learning.

[0051] As used herein, “learning” can refer to the training of a neural network system or a portion thereof, wherein such training may include various training models such as supervised training, unsupervised training, and / or reinforcement-based training. Additionally, as used herein, “learning” can refer to the evaluation of the performance of a previously and / or currently trained neural network system, such as verifying the continued adequacy of the neural network system's performance, and / or determining whether further training may be desirable to maintain performance and / or to adjust based on new data (e.g., new categories of images or objects).

[0052] In some example embodiments, the neural network system can be learned based on a boundary shape mask output from the segmenter 160 and / or a real mask that includes the true boundaries of objects in the input image (hereinafter referred to as a "real mask"). For example, the neural network system can be learned based on the value of each pixel of the real mask and / or the value of each pixel of the boundary shape mask. The difference between the two values ​​corresponds to an error, and the convolutional neural network system 100 can be configured to learn using the segmenter 160 via backpropagation.

[0053] In some exemplary embodiments of this disclosure, the boundaries of objects in the true mask used for learning may be thicker than the true boundaries of the objects. For example, the boundaries of objects in the true mask may include not only the true boundaries but also pseudo-boundaries and / or false boundaries surrounding the inner, outer, or inner and outer sides of the true boundaries. For example, the pixel values ​​of the true boundaries of the true mask may be the same as the pixel values ​​of the pseudo-boundaries. Alternatively, the pixel values ​​of the pseudo-boundaries of the true mask may be smaller than the pixel values ​​of the true boundaries.

[0054] In some example embodiments, the performance of object detection (e.g., instance segmentation) can be experimentally observed by making the boundary thickness of the object used for learning the true mask greater than the true boundary thickness of the object. This will be described more fully below.

[0055] Figure 2 Illustrations based on some example embodiments Figure 1Example configuration of feature extractor 110. Feature extractor 110 may include multiple layers L1 to L4, which are configured to repeatedly perform convolution operations and / or pooling operations.

[0056] exist Figure 2 In the example, the first layer L1 can be configured to perform a convolution operation on the input image 10 using a first kernel K1 and output a first feature map FM1. For example, data from the input image 10 that overlaps with the first kernel K1 can be multiplied by weights defined in the first kernel K1. The sum of the multiplication values ​​is output as a feature value. In the case of sequential shifting of the first kernel K1, the multiplication and summation operations can be performed repeatedly. As a result, as the first kernel K1 shifts, a feature value corresponding to each position can be determined. The set of feature values ​​determined by shifting the first kernel K1 can form the first feature map FM1. For example, in the case where the first kernel K1 includes multiple kernels, a first feature map FM1 composed of arrays can be generated, the number of arrays corresponding to the number of kernels. The second layer L2 can be configured to perform a pooling operation on the first feature map FM1 using a second kernel K2 and output a second feature map FM2. In some example embodiments, in the second layer L2, subsampling (or downsampling) for reducing the size of the first feature map FM1 can be performed. Pooling, as an example of subsampling, can be performed. Pooling may include adjusting the width and / or height of the channels while uniformly maintaining the number of channels in the spatial domain of the first feature map FM1. Subsequently, a third layer L3 may be configured to perform a convolution operation on the second feature map FM2 using a third kernel K3 and output a third feature map FM3, and / or a fourth layer L4 may be configured to perform a pooling operation on the third feature map FM3 using a fourth kernel K4 and output a fourth feature map FM4. Figure 2 Four layers are shown as an example, but the number of layers is not limited to this. That is, the number of layers can vary depending on the type of network used as the backbone of feature extractor 110. The final feature map (i.e., FM4) can be input into RoI extractor 120 (see reference). Figure 1 ) and RoI Aligner 130 (refer to) Figure 1 ).

[0057] Figure 3 An example configuration of the feature extractor 110 according to some example embodiments is shown. For the sake of simplicity, the multiple layers L1 to L4 (refer to) used to perform convolution and pooling operations are not shown. Figure 2 ), and only feature diagrams FM1 to FM4 and FM1' to FM4' indicating the results of the operation are shown. Figure 3In the examples, feature extractor 110 can be implemented as a Feature Pyramid Network (FPN). In some example embodiments, instead of using only the final feature map in the feature maps, the FPN can be used to extract RoIs more finely, for example, based on a set of feature maps from high-order feature maps to low-order feature maps. In some examples, information may be lost from the feature maps due to multiple convolution operations and / or multiple pooling operations. Therefore, instead of using only low-order feature maps, feature maps containing a larger amount of information can be used through all or at least some feature maps in each order. An element-wise product can be performed between the result of two resizing of the low-order feature maps and the result of performing a 1×1 convolution operation on the high-order feature maps. This operation can be repeated, for example, until the highest-order feature map becomes the target of the element-wise product. As a result, the feature map FM2' obtained at the highest order can be input into RoI extractor 120 (see reference). Figure 1 ) and RoI Aligner 130 (refer to) Figure 1 ).

[0058] Figure 4 Conceptually illustrated according to some example embodiments Figure 1 Example operation of RoI extractor 120. (e.g.) Figure 4 As shown, the RoI extractor 120 can be configured to determine regions on a feature map where objects are likely (e.g., very likely) to exist. In some example embodiments, the RoI extractor 120 may include a classifier 122. The RoI extractor 120 can be configured, for example, to use a sliding window (e.g., as shown in the diagram). Figure 4 The feature map is scanned using a 3×3 size (as shown), and / or can be configured to map the scanned regions at a specific size (e.g., a lower specification). As a result of the mapping, the RoI extractor 120 can be configured to perform as an intermediate layer. At each sliding window position, the classifier 122 can be configured to generate regions where objects may exist (e.g., regions where objects are likely to exist, i.e., region proposals) based on multiple anchor points. That is, the classifier 122 can be configured to determine an “object state score” that indicates whether the scanned regions contain objects, and / or can be configured to output at least one RoI in the scanned regions based on a given reference. At least one RoI output from the classifier 122 can be input to the RoI aligner 130. In some example embodiments, the classifier 122 can be implemented using (but not limited to) 1×1 convolutional layers.

[0059] Figure 5 Conceptually illustrated according to some example embodiments Figure 1 Example alignment of RoI aligner 130. (See example) Figure 5As shown, when using RoI pooling to extract RoIs from a feature map, the bounding boxes of the ground truth can be predicted. In this example, information loss may occur in the process of changing the real values ​​to approximate integer values ​​for each pixel. For example, when pooling a 5×5 feature map into a 2×2 feature map, information loss may occur due to the different pooling sizes. In this case, linear interpolation can be used to prevent or reduce information loss in the RoI pooling process. In some example embodiments, when predicting the RoI of the ground truth, for example, the RoI aligner 130 can be configured to use some or all of the values ​​of the corresponding pixels without changing the real values ​​to approximate integer values. As a result, some or all of the pixel values ​​of the feature map are applied to the RoI, thus improving the accuracy of the RoI.

[0060] Figure 6 Illustrations based on some example embodiments Figure 1 Example configuration of splitter 160. Figure 6 In the example, segmenter 160 is configured to generate a mask for predicting the contours of objects. Segmenter 160 may include a neural network configured to perform convolutional and / or deconvolutional operations on RoIs that can be cropped and / or resized by RoI aligner 130. Figure 6 In the example embodiment, the segmenter 160 performs a 2×2 deconvolution operation and a 3×3 convolution operation.

[0061] In some example embodiments, the segmenter 160 may be configured to adjust the size of aligned RoIs, which facilitates further processing by the neural network system based on predicted masks and true masks. In some example embodiments, the segmenter 160 may be configured to generate feature maps with a size of 28×28×512 by performing deconvolution operations on aligned RoIs, each with a size of 14×14×512. In some example embodiments, the segmenter 160 may be configured to generate multiple predicted masks by performing 3×3 convolution operations on the feature maps, each with a size of 28×28×512.

[0062] In some example embodiments, multiple predictive masks may include information relating to the probability that each object belongs to each category. For example, each predictive mask may include 28×28 pixel values ​​and / or values ​​indicating the probability that each pixel belongs to a particular category. The convolutional neural network system may be configured to learn the segmenter 160 based on the pixel values ​​of the predictive masks, the pixel values ​​of the true masks, and the differences between the pixel values ​​(e.g., error information). This will be described below.

[0063] Figure 7Examples of general true masks (e.g., boundary shape masks) that can be used with the predictive mask output from segmenter 160 are shown in some example embodiments. Figure 7 In the example, the boundary shape mask consists of 10×18 pixels. Pixels marked with "1" correspond to the true boundary of the object, and pixels marked with "0" correspond to areas other than the contour. In this example, the predicted mask output from segmenter 160 may not match the boundary shape mask exactly. Therefore, segmenter 160 can learn based on the pixel values ​​of the two masks. However, in some examples, the true boundary marked with "1" may be very thin, and the values ​​may change drastically or significantly between pixels marked with "1" and pixels marked with "0". This can increase the learning difficulty.

[0064] Figure 8 Example true masks that can be used with predictive masks are shown according to some example embodiments of the present disclosure. Figure 8 The true mask shown includes not only the true boundary of the object but also the inner and outer pseudo boundaries. As a result, the outline of the object in the true mask can be thicker than the outline of the object in the predicted mask. In this respect, Figure 8 The true mask shown can be called a "thick-bounded shape mask". Because... Figure 8 The thickness of the outline of the thick boundary shape mask is thicker than that of the predicted mask, so the segmenter 160 can be learned more effectively even if there is a small error between the predicted mask and the thick boundary shape mask.

[0065] Figure 9 and Figure 10 Examples of thick-bounded shape masks that can be used with predictive masks are shown according to some exemplary embodiments of this disclosure. First, Figure 9 The thick-bounded shape mask includes an inner pseudo-boundary adjacent to the inside of the object's true boundary. Conversely, Figure 10 The thick boundary shape mask includes an outer pseudo boundary adjacent to the outside of the true boundary of the object.

[0066] Figure 11 Example true masks that can be used with predictive masks are shown according to some example embodiments of this disclosure. Figure 11 In the example, the thick boundary shape mask and Figure 8 Similar to the thick-boundary shape mask, because the thick-boundary shape mask can include inner and outer pseudo-boundaries. However, only pixels corresponding to the true boundary of the object can have a value of 1, while pixels at the inner and outer pseudo-boundaries can have values ​​less than 1. In this respect, Figure 11 The true mask shown can be called a "score boundary shape mask". Figure 11 An example is shown where the pixels of the inner and outer pseudo-boundaries have a value of 0.9, but some example embodiments of some inventive concepts may not be limited to this.

[0067] In some examples, the pixel values ​​of the true mask and the pixel values ​​of the pseudo-boundary of an object can be set differently. For example, even if the true boundary of the object in the true mask may not perfectly match the contour of the object in the predicted mask, similar to... Figure 7 In some embodiments, where pixel values ​​are uniformly set to "1", errors may not be properly applied to learning. Therefore, allowing small differences between the true boundary of an object in a thick-bounded shape mask and the contour of an object in a predicted mask can facilitate learning in examples where a penalty is applied based on that difference.

[0068] Figure 12 Example thick-bounded shape masks for use with prediction masks are shown, according to some example embodiments of the present disclosure. Figure 12 In the example, the mask and Figure 8 The mask in the example is similar. However, the thickness of the inner pseudo-boundary can correspond to multiple pixels. Similarly, the thickness of the outer pseudo-boundary can correspond to multiple pixels. The example embodiment is shown where the thickness of the pseudo-boundary corresponds to a thickness defined by two pixels.

[0069] Figure 13 Examples of scored bounding shape masks that can be used with prediction masks are shown, according to some exemplary embodiments of the present disclosure. Figure 13 In the example, the mask and Figure 11 The mask is similar. However, the first inner pseudo-boundary and the second inner pseudo-boundary are located inside the true boundary of the object. Additionally, the first outer pseudo-boundary and the second outer pseudo-boundary are located outside the true boundary of the object. In some example embodiments, the pixels of the true boundary of the object may have a value of 1. In some example embodiments, the pixels of the first inner pseudo-boundary and the first outer pseudo-boundary may have a value of 0.9. In some example embodiments, the pixels of the first outer pseudo-boundary and / or the second inner pseudo-boundary may have a value of 0.8. That is, the pixel value may become smaller as the distance from the true boundary increases. However, the number of pseudo-boundaries provided and / or the values ​​of the pixels constituting each boundary are not limited to, for example... Figure 13 Examples.

[0070] Figure 14 Example general true masks (e.g., bounding box masks) that can be used with prediction masks output from a segmenter are shown. In some example embodiments, the bounding box mask may consist of 10 × 18 pixels. Pixels corresponding to the box that encloses the true boundary of the object in a rectangular shape may be marked with a "1", and pixels corresponding to the area outside the box may be marked with a "0".

[0071] Figure 15 Example true masks that can be used with predictive masks are shown according to some example embodiments of this disclosure. Figure 15 In the example, the true mask includes not only the true boundary of the object but also the inner and outer pseudo-boundaries. As a result, the bounding box surrounding the object in the true mask can be thicker than the bounding box surrounding the object in the predicted mask. In this respect, Figure 15 The true mask shown can be called a "thick bounding box mask". Because... Figure 15 In the example, the thickness of the thick bounding box mask is greater than the thickness of the object's box in the prediction mask. Therefore, even if there is a small error between the prediction mask and the thick true mask, the segmenter 160 can be learned more effectively.

[0072] Figure 16 Example true masks that can be used with predictive masks are shown according to some example embodiments of this disclosure. Figure 16 In the example, the true mask includes not only the true boundary composed of pixels with a value of 1, but also an inner pseudo boundary and an outer pseudo boundary composed of pixels with a value of 0.9. In this respect, Figure 16 The mask in the example can be called a "score bounding box mask". Figure 16 The mask in the example and Figure 12 The mask is similar, except that the boundary is in the form of a rectangular box; therefore, additional descriptions will be omitted to avoid redundancy. Furthermore, although not shown in the figures, bounding box masks of various shapes can be used. In some example embodiments, such as... Figure 9 Similar to thick bounding shape masks, bounding box masks including inner pseudo-boundaries can be used for learning. In some example embodiments, such as Figure 10 Similar to the thick bounding shape mask, the bounding box mask, including the outer pseudo-boundary, can be used for learning.

[0073] In some example embodiments, such as Figure 12 Similar to a thick bounding shape mask, a bounding box mask including a pseudo-boundary whose thickness is defined by multiple pixels can be learned. In some example embodiments, such as Figure 13 Similar to the scoring bounding shape mask, a bounding box mask including multiple pseudo-boundaries can be used for learning, and the pixel values ​​of these multiple pseudo-boundaries can decrease as the distance between the pixel and the true boundary of the object increases.

[0074] Figure 17 This conceptually illustrates examples of how a convolutional neural network system, according to some example embodiments of this disclosure, can use prediction masks and score boundary shape masks. Figure 17 In the example, convolutional neural network system 100 (e.g., Figure 1(Example embodiment) The boundaries of objects in an image can be determined and / or learned based on the values ​​of each pixel in the prediction mask and the values ​​of each pixel in the score-bound shape mask. If the value of any pixel in the prediction mask is the same as or similar to the value of the corresponding pixel in the score-bound shape mask (e.g., the difference in values ​​is within an error threshold), the corresponding pixel in the prediction mask is determined to be an accurate prediction. Conversely, if the value of any pixel in the prediction mask differs significantly from the value of the corresponding pixel in the score-bound shape mask (e.g., the difference in values ​​exceeds an error threshold), the prediction is determined to be incorrect.

[0075] like Figure 17 As further shown in the example, the convolutional neural network system 100 can be configured to generate error information based on the difference between two pixels. For example, the convolutional neural network system 100 can be configured to determine the difference between all pixels of the prediction mask and all corresponding pixels of the score bounding shape mask. Specifically, Figure 17 The difference D1 between the pixel values ​​in the second row and third column and the difference D2 between the pixel values ​​in the fourth row and eighth column are shown. In some example embodiments, the convolutional neural network system 100 may be configured to determine the difference based on the mask resolution (i.e., 10×18), and / or may be configured to sum the differences. The final value of the sum of the differences may be used as error information and / or may be used to learn one or more components of the convolutional neural network system, such as the RoI aligner 130 (see reference). Figure 1 ), Classifier 140 (refer to) Figure 1 ), bounding box regressor 150 (reference) Figure 1 ), Divider 160 (refer to) Figure 1 In some example embodiments where the final value of the sum of differences is very large (e.g., exceeding the error and threshold), in order to prevent or reduce large deviations in the output values ​​during learning, learning can be based on the value obtained by dividing the final value of the sum by the number of pixels.

[0076] Figure 18 Example learning processes of a convolutional neural network system 100 according to some example embodiments of the present disclosure are illustrated schematically. Figure 18 In the examples, using references, etc. Figure 17 The error information obtained from the prediction mask and the boundary shape mask can be provided to the previous stage of the convolutional neural network system 100. Some example embodiments can utilize the error information for learning, such as backpropagation. For example, the loss during training may include four losses, and the convolutional neural network system 100 can be learned by the sum of these four losses. For example, in Figure 18In this process, "Loss_O" can be associated with the "object state score" output by RoI extractor 120, "Loss_C" can be associated with the category output by RoI classifier 140, "Loss_B" can be associated with the object location information output by bounding box regressor 150, and "Loss_M" can be associated with the bounding shape direction output by segmenter 160. The convolutional neural network system 100 can be learned by summing the above four losses.

[0077] Figure 19 and Figure 20 Conceptually, an example of backpropagation for learning a convolutional neural network system 100 is shown. See also... Figure 18 and Figure 19 Learning the convolutional neural network system 100 can include feedforward prediction and backpropagation. For example, feedforward prediction might correspond to the process of feeding an input image into the convolutional neural network system 100 and generating a prediction mask (e.g., a boundary shape mask output from the segmenter 160). Figure 19 In the examples, the convolutional neural network system 100 can be configured to determine an error δ corresponding to the difference between the pixel value y of the prediction mask and the pixel value z of the score bounding shape mask. In some examples, an insufficiently learned convolutional neural network system 100 may result in a large value of error δ (e.g., exceeding an error threshold). Conversely, in some examples, a fully learned convolutional neural network system 100 may result in a small value of error δ (e.g., within an error threshold). The convolutional neural network system 100 can be configured, for example, to obtain δ4 by using the obtained error δ, and can proceed as follows: Figure 20 δ5 is obtained in a similar manner as shown. The convolutional neural network system 100 can be configured to update the weights and biases simultaneously with the error δ obtained from backpropagation. As the convolutional neural network system 100 repeatedly performs the above processing, the weights and / or biases can be changed to values ​​optimized for the training data (X1, X2). However, in some example embodiments, different ways of learning the convolutional neural network system 100 can be utilized; for example, various methods of updating the weights and / or biases can be employed, such as using values ​​obtained by performing partial differentiation on the sigmoid function.

[0078] Figure 21 Example configurations of a convolutional neural network system 200 according to some example embodiments of the present disclosure are shown. Figure 21 In the example, refer to Figures 1 to 20The convolutional neural network system 100 described may be based on a masked R-CNN, and / or the convolutional neural network system 200 may be based on OverFeat. The OverFeat architecture can be implemented using a network that scans the input image in a sliding window manner and extracts feature maps. Additional details regarding the configuration and operation of the OverFeat architecture are omitted. The convolutional neural network system 200 may be configured to extract object category and / or location information, for example, by using a classifier 240 and / or a bounding box regressor 250, and / or may be configured to (e.g., simultaneously, concurrently, and / or continuously) generate bounding shape masks that surround the contours of objects determined by the segmenter 260. The true mask used to learn the convolutional neural network system 200 may include reference... Figures 8 to 13 , Figure 15 and Figure 16 The described masks and / or modified versions of these masks.

[0079] Figure 22 Example configurations of a convolutional neural network system 300 according to some example embodiments of the present disclosure are shown. Figure 22 In the example, the convolutional neural network system 300 may be based on a faster R-CNN. The mask R-CNN described in the specification is based on a faster R-CNN. Details of the configuration and operation of the faster R-CNN are omitted. The convolutional neural network system 300 may be configured to perform classification using a classifier 340, and / or perform regression using a bounding box regressor 350, and / or may (e.g., simultaneously, concurrently, and / or continuously) generate masks for learning the bounding shape mask determined by the segmenter 360 (e.g., refer to...). Figures 8 to 13 , Figure 15 and Figure 16 The described mask or a modified version of these masks).

[0080] Figure 23 Example configurations of a convolutional neural network system 400 according to some example embodiments of the present disclosure are shown. Figure 23 In the example, the convolutional neural network system 400 may be based on YOLO (You Only Look Once). YOLO can be configured to scan an image in a sliding window manner using multiple anchors, and / or generate feature maps. In some example embodiments, YOLO may not include the RoI extractor 120 (see reference). Figure 1 Details of YOLO's configuration and operation are omitted. The convolutional neural network system 400 can be configured to perform classification and / or regression and / or generate bounding shape masks (e.g., simultaneously, concurrently, and / or continuously).

[0081] Figure 24 Example configurations of convolutional neural network systems 500 and / or 600 according to some example embodiments of this disclosure are shown. Figure 24 In some examples, the first convolutional neural network system 500 includes a computationally intensive (or fine-grained) network such as a residual network, and / or the second convolutional neural network system 600 may include a computationally conservative (or less fine-grained) network such as a mobile network (Mobi leNet). Therefore, the learning and object detection of the second convolutional neural network system 600 may be less fine-grained and / or computationally intensive than that of the first convolutional neural network system 500. In some example embodiments, the first convolutional neural network system 500 and the second convolutional neural network system 600 may constitute model compression. That is, the learning results determined and / or provided by the first convolutional neural network system 500 can be used to learn the second convolutional neural network system 600. For example, the first convolutional neural network system 500 may be configured to provide the second convolutional neural network system 600 with various information determined by the segmenter 560 for learning. In some example embodiments, the learning results of the first convolutional neural network system 500 can be used for supervised learning of the second convolutional neural network system 600. In some example embodiments, a second convolutional neural network system 600, generated through supervised learning using the learning results of the first convolutional neural network system 500, can be ported to mobile devices, etc.

[0082] Figure 25 This illustration depicts object detection using a convolutional neural network system based on bounding box mask learning, according to some example embodiments of the present disclosure. In the example embodiment, residual network 101 is used as feature extractor 110 (see reference). Figure 1 The framework of this system performs object detection on the MS COCO val dataset. In many object recognition systems, detecting very small objects in an image can be difficult or inaccurate, but some example embodiments of this disclosure are able to detect small objects in images, such as... Figure 25 As shown.

[0083] Table 1 below shows the boundary shape mask of this disclosure based on... Figure 25 The example convolutional neural networks discussed here are test results against the MSCOCO minval. In Table 1, “R” represents ResNet, “X” represents ResNeXt, and “Inst.mask” represents bounding box masks according to some example embodiments of this disclosure. It can be observed from Table 1 that the highest scores are achieved in all domains when performing object detection using the bounding shape boxes of this disclosure. Specifically, the scores obtained in all domains are higher than those of ResNeXt neural networks using the same or similar backbones. That is, it is observed that the performance of object detection can be improved by performing learning while adjusting the boundary thickness of objects in the true mask, without changing the algorithm of existing convolutional neural network systems.

[0084] [Table 1]

[0085]

[0086] Figure 26 This illustration shows object detection using a convolutional neural network system based on bounded shape mask learning, according to exemplary embodiments of the present disclosure. In an example embodiment, residual network 101 is used as feature extractor 110 (see reference 110). Figure 1 The framework is based on the MS COCO val dataset, and object detection is performed on it. Typically, detecting very small objects in an image may not be difficult. However, as... Figure 26 As shown, some convolutional neural network systems according to some example embodiments of this disclosure are able to detect small objects in images.

[0087] Table 2 below shows the data based on... Figure 26 The proposed models of convolutional neural networks discussed are for object detection (e.g., instance segmentation). "Inst.mask" refers to the bounded shape mask of this disclosure. Even if the same or similar network (residual network 50) is used as feature extractor 110 (see reference...) Figure 1 As can be seen from the backbone of the convolutional neural network system according to some example embodiments of the present disclosure, it achieves the highest scores in most domains.

[0088] [Table 2]

[0089]

[0090] In some example embodiments, the neural network system 100 trained according to this disclosure can be used in various scenarios. As a first example, such a neural network system 100 can use object detection, localization, and / or classification as part of an image processing system, such as scanning printed media (e.g., optical character recognition) and / or processing still images from a stationary camera or moving images from a motion camera within one or more detectable wavelength ranges. Examples of such uses include: identifying individuals and personal biometrics in biometric authentication systems; focusing objects in cameras; object recognition in extended reality presentations (e.g., augmented reality and virtual reality applications); and 3D modeling, such as for digital animation and manufacturing via 3D printing. As a second example, such a neural network system 100 can use object detection, localization, and / or classification as part of a machine vision system for autonomous vehicles, for example, to identify, localize, and / or classify objects such as other vehicles, bicycles, pedestrians, traffic lights and signs, and obstacles. As a third example, such a neural network system 100 can use object detection, localization, and / or classification as part of a machine vision system for robots that can be used in public, residential, commercial, industrial, scientific, and / or military environments. Many such scenarios may include object detection, localization, and / or classification using convolutional neural network systems disclosed herein as standalone systems and / or as components of broader systems (such as image processing, machine vision, statistical classification, and / or deep learning components (such as restricted Boltzmann machines (RBMs), deep belief networks, tree-based classifiers (e.g., random forests), Bayesian classifiers, recurrent or reentrant neural networks, initial neural networks, capsule networks, etc.) as a whole).

[0091] According to some example embodiments of this disclosure, the learning efficiency of a convolutional neural network system can be improved by changing the pixel thickness of the true mask used for learning.

[0092] Although some inventive concepts have been described with reference to exemplary embodiments thereof, it will be apparent to those skilled in the art that various changes and modifications may be made thereto without departing from the spirit and scope of some exemplary embodiments of the inventive concepts set forth in the following claims. For example, some exemplary embodiments presented herein may demonstrate exemplary organization of components, such as... Figure 1 The examples provided are a collection of sample components. Other example embodiments may include different organization of components (such as renaming, rearranging, adding, splitting, copying, merging, and / or removing components), collections of components, and relationships between them, without departing from the scope of this disclosure. All such variations that are technically reasonable and logically possible and do not contradict other statements are intended to be included in this disclosure, the scope of which shall be understood to be limited only by the claims.

Claims

1. A computer-readable medium comprising program code, said program code, when executed by a processing circuitry system, causing the processing circuitry system to: Perform a convolution operation on the input image to generate a feature map; Extract regions of interest from the feature map based on object state scores associated with the existence of the object; Align the extracted region of interest with the region of interest having a reference size; The category of the object and the position information of the object on the input image are determined based on the aligned region of interest; Based on the determined results, a boundary is formed that surrounds the object on the input image; and A convolutional neural network system is learned based on a predicted mask based on the category, the location information, and the boundary, and a true mask based on the true boundary of the object in the input image. The true mask includes a first pixel corresponding to the true boundary and a second pixel corresponding to the pseudo boundary adjacent to the true boundary.

2. The computer-readable medium according to claim 1, wherein, The pixel value of the first pixel is greater than the pixel value of the second pixel.

3. The computer-readable medium according to claim 1, wherein, The pixel value of the first pixel is the same as the pixel value of the second pixel.

4. The computer-readable medium according to claim 1, wherein, The second pixel is adjacent to the outside of the true boundary formed by the first pixel. Wherein, the pseudo-boundary is the first pseudo-boundary, and The true mask further includes a third pixel corresponding to the first pseudo boundary adjacent to the inside of the true boundary.

5. The computer-readable medium according to claim 1, wherein, The second pixel is adjacent to the inside of the true boundary formed by the first pixel. Wherein, the pseudo-boundary is the first pseudo-boundary, and The true mask further includes a third pixel corresponding to the first pseudo boundary adjacent to the outside of the true boundary.

6. The computer-readable medium according to claim 1, wherein, When the program code is executed by the processing circuit system, the processing circuit system uses multiple anchor points to further search the feature map in a window sliding manner when extracting the region of interest.

7. The computer-readable medium according to claim 1, wherein, When the program code is executed by the processing circuitry system, it causes the processing circuitry system to further perform a fully connected operation on the aligned region of interest when forming the boundary, and The object's category and location information are generated based on the result of the fully connected operation.

8. The computer-readable medium according to claim 1, wherein, The thickness of the pseudo-boundary is greater than the thickness of the true boundary.

9. The computer-readable medium according to claim 1, wherein, When the program code is executed by the processing circuit system, the processing circuit system learns a convolutional neural network system through backpropagation based on error information based on the predicted mask and the true mask.

10. The computer-readable medium according to claim 1, wherein, When the program code is executed by the processing circuit system, the processing circuit system performs the determination of the object's category, the determination of the object's location information, and the formation of the boundary in parallel.

11. A computer-readable medium comprising program code, said program code, when executed by a processing circuitry system, causing the processing circuitry system to: Perform a convolution operation on the input image to generate a feature map; Extract regions of interest from the feature map based on object state scores associated with the existence of the object; Align the extracted region of interest with the region of interest having a reference size; The category of the object and the position information of the object on the input image are determined based on the aligned region of interest; Based on the determined results, a boundary is formed that surrounds the object on the input image; and A convolutional neural network system is learned based on a predicted mask based on the category, the location information, and the boundary, as well as a true mask including true bounding boxes surrounding the objects in the input image. The true mask includes a first pixel corresponding to the true bounding box and a second pixel corresponding to a pseudo bounding box adjacent to the true bounding box.

12. The computer-readable medium of claim 11, wherein, The second pixel is adjacent to the outside of the true bounding box formed by the first pixel. Wherein, the pseudo-boundary box is the first pseudo-boundary box, and The true mask further includes a third pixel corresponding to the first pseudo bounding box adjacent to the inside of the true bounding box.

13. The computer-readable medium according to claim 12, wherein, The pixel values ​​of the first pixel, the second pixel, and the third pixel are the same.

14. The computer-readable medium of claim 12, wherein, The pixel value of the first pixel is greater than the pixel value of the second pixel and the pixel value of the third pixel.

15. The computer-readable medium according to claim 12, wherein, The second pixel is adjacent to the inside of the true bounding box formed by the first pixel. Wherein, the pseudo-boundary box is the first pseudo-boundary box, and The true mask further includes a third pixel corresponding to the first pseudo bounding box adjacent to the outside of the true bounding box.

16. A convolutional neural network system, comprising: A processing circuit system configured to perform a convolution operation on an input image to generate a feature map; Extract regions of interest from the feature map based on object state scores associated with the existence of the object; Align the extracted region of interest with the region of interest having a reference size; The category of the object is determined based on the aligned region of interest; The position information of the object on the input image is determined based on the aligned region of interest; and Forming a boundary surrounding the object on the input image, and Learning is performed based on a predicted mask based on the category, the location information, and the boundary, and a true mask based on the true boundary of the object in the input image. The true mask includes a first pixel corresponding to the true boundary and a second pixel corresponding to the pseudo boundary adjacent to the true boundary.

17. The convolutional neural network system according to claim 16, wherein, The processing circuitry is configured to use a region proposal network to extract the region of interest.

18. The convolutional neural network system according to claim 16, wherein, The processing circuitry is configured to use multiple fully connected networks to determine the category of the object and the object's location information on the input image.

19. The convolutional neural network system according to claim 16, wherein, The second pixel is adjacent to the outside of the true boundary formed by the first pixel. Wherein, the pseudo-boundary is the first pseudo-boundary, and The true mask further includes a third pixel corresponding to the first pseudo boundary adjacent to the inside of the true boundary.

20. The convolutional neural network system according to claim 19, wherein, The pixel value of the first pixel, the pixel value of the second pixel, and the pixel value of the third pixel are the same, or The pixel value of the first pixel is greater than the pixel value of the second pixel and the pixel value of the third pixel.

Images