Object detection system and learning method

Abstract

This invention provides an object detection system that can train a model to accurately detect small objects. [Solution] The object detection system comprises a feature extraction network, a candidate region extraction network, an object detection network, and a learning control unit. The candidate region extraction network outputs an object-likeness map and a displacement map. The learning control unit calculates the degree of overlap between the anchor region and the correct object with respect to the object-likeness map, and sets the correct value of the probability that the anchor region is an object to 1 if the degree of overlap > t1. The learning control unit calculates the correct value of the displacement amount with respect to the displacement map based on the anchor region with an overlap degree > t1 and the correct object.

Description

[Technical Field]

[0001] This invention relates to an object detection system and a learning method. [Background technology]

[0002] Research has been conducted on technologies for detecting objects from images. Object detection technology is an important image recognition technique that detects bounding boxes surrounding each object in an input image and identifies the class of that object. Object detection makes it possible to identify the location, type, and number of specific objects in images and videos. For this reason, it is a technology that is often used in fields such as visual inspection in manufacturing, and in medicine and construction. Familiar examples include the detection of people in smartphone cameras and pedestrian detection in autonomous driving vehicles.

[0003] Non-patent document 1 proposes Faster R-CNN as a representative method for object detection. Faster R-CNN consists of the following three networks: The first feature extraction network is a network for extracting feature maps from the input image. For example, deep learning models such as VGG16 and ResNet are used as feature extraction networks. The second candidate region extraction network (Region Proposal Network, RPN) is a network for extracting regions that are highly likely to contain objects (hereinafter referred to as candidate regions). For multiple anchor boxes centered on each position in the feature map, the RPN calculates the probability that each anchor box is an object and the amount of deviation from the correct object region. Anchor boxes with a high probability are extracted as candidate regions. The third object detection network is a network for detecting objects from the candidate regions. This network estimates the object class and position for each candidate region.

[0004] Non-patent document 2 describes Mask R-CNN, a multi-task learning model for object detection.

[0005] Non-patent document 3 describes YOLO (You Only Look Once), an object detection method that detects objects in an image with high accuracy. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Shaoqing Ren, Kaiming He, Ross B. Girshick, Jian Sun: “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.” Conference on Neural Information Processing Systems (NeurIPS) 2015: 91-99. [Non-Patent Document 2] Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross B. Girshick: “Mask R-CNN.” IEEE International Conference on Computer Vision (ICCV) 2017: 2980-2988. [Non-Patent Document 3] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, Ali Farhadi: “You Only Look Once: Unified, Real-Time Object Detection.” Computer Vision and Pattern Recognition (CVPR) 2016: 779-788. [Overview of the project] [Problems that the invention aims to solve]

[0007] However, conventional technologies also have challenges that need to be addressed. For example, while Faster R-CNN achieves outstanding performance, it has the problem of being unable to accurately detect small objects that are small in size, have low resolution, and are difficult to distinguish from the background. The main reason for this is that training images containing small objects are difficult to collect and therefore few in number, and as a result, Faster R-CNN's ability to detect small objects is not sufficiently learned during the training phase compared to normal objects.

[0008] Similar issues exist with Mask R-CNN and YOLO.

[0009] This invention was made based on the above-mentioned problem recognition, and aims to provide an object detection system that can train a model to accurately detect small objects, and a learning method for the object detection system. [Means for solving the problem]

[0010] [1] To solve the above problems, an object detection system according to one aspect of the present invention includes: a feature extraction network that extracts features for each position in an input image and outputs a feature map based on the input image; a candidate region extraction network configured to be machine learning capable, which outputs, based on the feature map, an object-likeness map that holds the probability that each anchor region including each position in the feature map is an object, and a deviation map that holds a deviation amount representing the degree to which the anchor region deviates from the correct object region, as information for candidate regions; an object detection network configured to be machine learning capable, which outputs, based on the feature map, the classification of the object for each candidate region output by the candidate region extraction network, and the position and size of the candidate region; a training data storage unit that stores training data which is a set of pairs of training images and correct data consisting of the classification of the object included in the training image and the position and size of the object; and (1) the training images included in the training data are supplied as images to be input to the feature extraction network, and based on the training images The object detection system comprises: (2) a learning control unit which, with respect to the object detection system, causes the feature extraction network, the candidate region extraction network, and the object detection network to perform processing, calculates a loss function value for the object characteristics map and the displacement map based on the object characteristics map and the displacement map output by the candidate region extraction network, and updates the internal parameters of the candidate region extraction network based on the loss function value for the object characteristics map and the displacement map, and (3) calculates a loss function value for the classification and the amount of deviation from the correct object region based on the classification and the amount of deviation from the correct object region output by the object detection network, and updates the internal parameters of the object detection network based on the loss function value for the classification and the amount of deviation from the correct object region, wherein the learning control unit calculates the degree of overlap between the anchor region and the object included in the correct data with respect to the object characteristics map, and uses a predetermined threshold t1,If the degree of overlap > t1, the correct value of the probability that the anchor region is an object is set to 1. The learning control unit calculates the correct value of the displacement amount based on the anchor region and the objects included in the correct data when the degree of overlap > t1. For objects included in the image, the learning control unit calculates the degree of overlap between the correct object and all anchor regions, and determines the threshold t1 based on the calculated statistical values ​​of the degree of overlap with all anchor regions.

[0011] [2] Another aspect of the present invention is the object detection system described in [1] above, further comprising a data expansion unit that divides a training image read from the training data storage unit into a plurality of patches and generates a patch-shuffled image by shuffling the plurality of patches, generates an additional image by pasting an additional image containing an additional object onto an image containing at least the patch-shuffled image, and stores pairs of correct data consisting of the classification of the additional object contained in the additional image and the position and size of the additional object, and the additional image, as additional training data in the training data storage unit.

[0012] [3] Another aspect of the present invention is a learning method for learning the candidate region extraction network and the object detection network of an object detection device, comprising: a feature extraction network that extracts features for each position in an input image and outputs a feature map based on the input image; a candidate region extraction network configured to be machine learning capable, which outputs, based on the feature map, an object-likeness map that holds the probability that each anchor region including each position in the feature map is an object, and a deviation map that holds a deviation amount representing the degree to which the anchor region deviates from the correct object region, as information for candidate regions; and an object detection network configured to be machine learning capable, which outputs, based on the feature map, the classification of the object for each candidate region output by the candidate region extraction network, and the position and size of the candidate region, wherein the training data storage unit stores training data which is a set of pairs of training images and correct data consisting of the classification of the object included in the training image and the position and size of the object, and the learning control unit (1) the training images included in the training data (1) The training image is supplied as input to the feature extraction network, and the feature extraction network, the candidate region extraction network, and the object detection network are made to process the training image. (2) Based on the object-likeness map and the displacement map output by the candidate region extraction network, and the correct values ​​of the object-likeness map and the displacement map, a loss function value is calculated for the object-likeness map and the displacement map, and the internal parameters of the candidate region extraction network are updated based on the loss function value for the object-likeness map and the displacement map. (3) Based on the class classification and the amount of deviation from the correct object region output by the object detection network, and the correct values ​​of the class classification and the amount of deviation from the correct object region, a loss function value is calculated for the class classification and the amount of deviation from the correct object region, and the internal parameters of the object detection network are updated based on the loss function value for the class classification and the amount of deviation from the correct object region. The learning control unit calculates the degree of overlap between the anchor region and the object included in the correct data with respect to the object-likeness map, and uses a predetermined threshold t1.If the degree of overlap > t1, the correct value of the probability that the anchor region is an object is set to 1. The learning control unit calculates a correct value of the shift amount for the shift map based on the anchor region and the object included in the correct data when the degree of overlap > t1. The learning control unit calculates the degree of overlap between the correct answer of the object and all anchor regions for the object included in the image, and obtains the threshold value t1 based on a statistical value of the calculated degrees of overlap with all anchor regions. This is a learning method.,

[0013] [4] In one aspect of the present invention, in the learning method according to [3] above, a data expansion unit divides a training image read from the training data storage unit into a plurality of patches, generates a patch-shuffled image by a process of shuffling the plurality of patches, pastes an additional image including an additional object onto an image including at least the patch-shuffled image to generate an additional image, and stores a pair of the additional image and correct data including the classification of the additional object included in the additional image and the position and size of the additional object in the training data storage unit as additional training data.

Effects of the Invention

[0014] According to the present invention, an object detection system excellent in detecting small objects and a learning method for the object detection system can be realized.

Brief Description of Drawings

[0015] [Figure 1] It is a block diagram showing a schematic functional configuration of an object detection system according to an embodiment of the present invention. [Figure 2] It is a schematic diagram showing an example of processing by a feature extraction network in the same embodiment. [Figure 3] It is a schematic diagram for explaining an outline of processing by a candidate region extraction network (RPN) in the same embodiment. [Figure 4] It is a schematic diagram showing an example of patch shuffling with different patch sizes in the same embodiment. [Figure 5] This is a schematic diagram showing an example of how to shuffle patches according to the same embodiment, specifically a general shuffling process. [Figure 6] This is a schematic diagram illustrating a hierarchical shuffling process, which is another example of how patches are shuffled according to the same embodiment. [Figure 7] This is a schematic diagram showing an example of the training data configuration in the same embodiment. [Figure 8] This embodiment is a schematic diagram showing the processing flow of the data augmentation unit performing training data augmentation. [Figure 9] This is a block diagram showing an example of the internal configuration (computer) of an object detection system consisting of an object detection device and a learning device in the same embodiment. [Modes for carrying out the invention]

[0016] Next, one embodiment of the present invention will be described with reference to the drawings. There is a technique that uses machine learning methods to detect objects contained in (imaged in) an image. However, as mentioned above, there is a problem that there are few training images containing small objects that are small in size, have low resolution and are difficult to distinguish from the background, and therefore sufficient learning to detect such small objects cannot be performed. In the embodiment described below, as the first feature, the original training image is divided into a plurality of square patches (small pieces of the image) and a transformation is performed in which the positions of these patches are randomly shuffled. Furthermore, the transformed image obtained by shuffling the patches (patch shuffle image) and the original training image are combined to generate a final background image. Furthermore, images of small objects (for example, small objects cut out from the training image) are pasted into arbitrary (predetermined) locations on the final background image to generate a new training image. This makes it possible to increase the number of training images, and the increased number of training images enables learning that is suitable for detecting small objects. In addition, as the second feature of the embodiment, with respect to small objects, the anchor boxes used for training are automatically selected based on the statistical characteristics of the anchor boxes used for candidate region extraction learning. This allows for the selection of anchor boxes necessary for learning small object detection capabilities, adapting to the position and size of the small objects. As a result, more anchor boxes containing small objects can be used for training. Each of these two features enables an improvement in the accuracy of small object detection. However, the shape of the anchor box is not limited to a specific shape (e.g., a rectangle). Any region of any shape that serves a similar role to the anchor box described below may be called an "anchor region."

[0017] More specifically, in this embodiment, to address the aforementioned problems, we propose a data augmentation method based on the spatial structure destruction of images (particularly described in Section 2.1 below). This method increases the training data for small objects and concentrates the training of the object detection model on small objects. We also propose a method for adaptively learning the candidate region extraction capability (particularly described in Section 2.4 below). This method improves the diversity of training data available during the training phase, i.e., the balance between normal objects and small objects, thereby improving the small object detection performance of the model.

[0018] The object detection device of this embodiment constructs an object detection model and applies it to object detection. This involves two stages of processing: a training stage and an identification stage. In the training stage, a large amount of training data is loaded into the object detection model, and the model parameters are iteratively updated and determined to create an optimally tuned object detection model. Then, in the identification stage, an image in which an object to be detected is input to the trained object detection model, and the bounding box surrounding the object and the class of that object are obtained as output.

[0019] The method proposed in this embodiment is applied to the training phase. Below, we will first describe the structure of the object detection model and then describe the processing in the classification phase. Next, we will describe the processing in the training phase and explain the method specific to this embodiment. In the following description, CNN refers to "Convolutional Neural Network".

[0020] Figure 1 is a block diagram illustrating the schematic functional configuration of the object detection system according to this embodiment. As shown in the figure, the object detection system 1 comprises an object detection device 10 and a learning device 30.

[0021] The object detection device 10 is a device that detects objects in an input image and estimates the class to which the detected objects belong. The object class represents the type of object, such as a person, dog, car, building, desk, chair, telephone, scissors, or cup. The object classes are not limited to those exemplified here. The object detection device 10 is implemented with an internal machine learning model.

[0022] The learning device 30 is a device that has the function of performing machine learning on the model of the object detection device 10 described above. The learning device 30 executes the learning method described below. The learning device 30 supplies training data for machine learning to the object detection device 10 and calculates values ​​to update the model parameters based on the difference (loss) between the result output by the object detection device 10 and the correct answer in order to perform machine learning.

[0023] The object detection device 10 and the learning device 30 can each be implemented, for example, by a computer and a program. Furthermore, each of the object detection device 10 and the learning device 30 has internal storage means. These storage means may be, for example, variables in the program or semiconductor memory allocated during program execution. Alternatively, non-volatile storage means such as magnetic hard disk drives or solid-state drives (SSDs) may be used as needed. Additionally, at least a portion of the functions of each of the object detection device 10 and the learning device 30 may be implemented as dedicated electronic circuits rather than programs.

[0024] The detailed functional configurations of the object detection device 10 and the learning device 30 are as follows:

[0025] The object detection device 10 comprises an image input unit 12, a feature extraction network 14, a candidate region extraction network 16, an object detection network 18, a result output unit 20, and a parameter storage unit 22. The functions of each unit are as follows.

[0026] The image input unit 12 acquires an image to be used for object detection. The image input unit 12 may also acquire an unknown image to be used for object detection. Furthermore, when model training is performed, the image input unit 12 acquires training images passed from the training control unit 32. The image input unit 12 passes the acquired image to the feature extraction network 14.

[0027] The feature extraction network 14 extracts location-specific features from the image input by the image input unit 12, generates a feature map, and outputs it. The feature extraction network 14 is implemented using a neural network. The feature extraction network 14 has internal parameters and is configured to be machine-learnable. The values ​​of these internal parameters are updated and optimized through machine learning.

[0028] The candidate region extraction network 16 extracts and outputs information about candidate regions based on the feature map output by the feature extraction network 14. Specifically, the candidate region extraction network 16 generates two maps, an object-likeness map and a shift map, for each anchor box (anchor region) centered on each position in the input feature map. However, the anchor boxes (anchor regions) do not necessarily have to be centered on each position in the input feature map. The anchor boxes (anchor regions) only need to include each position in the input feature map. The same applies to the following explanation. The object-likeness map holds the probability that each anchor box (anchor region) is an object and the probability that it is background. However, the object-likeness map does not necessarily need to hold the probability that each anchor box (anchor region) is background. It is sufficient that the object-likeness map holds at least the probability that each anchor box (anchor region) is an object. The same applies to the following explanation. The shift map holds the amount of shift, which represents the degree to which each anchor box deviates from the correct object region. The candidate region extraction network 16 is implemented using a neural network. The candidate region extraction network 16 has internal parameters and is configured to be machine-learnable. The values ​​of these internal parameters are updated and optimized through machine learning.

[0029] The object detection network 18 outputs the object classification for each candidate region output by the candidate region extraction network 16, along with the location and size of the candidate region, based on the feature map output by the feature extraction network 14. The object detection network 18 is implemented using a neural network. The object detection network 18 has internal parameters and is configured to be machine-learnable. The values ​​of these internal parameters are updated and optimized through machine learning.

[0030] The result output unit 20 outputs information about the results detected by the object detection network 18. Specifically, the result output unit 20 outputs the classification (label) of the objects depicted in the input image and information about the region of those objects within the image.

[0031] The parameter storage unit 22 stores the values ​​of the internal parameters of each of the feature extraction network 14, the candidate region extraction network 16, and the object detection network 18. The parameter values ​​stored in the parameter storage unit 22 are referenced by each of the feature extraction network 14, the candidate region extraction network 16, and the object detection network 18 when they perform their respective processing. In addition, the parameter values ​​stored in the parameter storage unit 22 are updated when machine learning is performed by each of the feature extraction network 14, the candidate region extraction network 16, and the object detection network 18.

[0032] The learning device 30 comprises a learning control unit 32, a data expansion unit 34, and a training data storage unit 36. The functions of each unit are as follows:

[0033] The learning control unit 32 controls the machine learning of each model in the object detection device 10. Specifically, when training a model, the learning control unit 32 supplies training data, processes the training images contained in the training data, calculates a loss based on the processing results and the ground truth data, calculates the values ​​of the model's internal parameters using backpropagation based on the calculated loss, and updates the values ​​of those internal parameters.

[0034] Details of the functions of the learning control unit 32 are as follows. (1) The learning control unit 32 supplies a training image included in training data as an image to be input to the feature extraction network 14, and causes the feature extraction network 14, the candidate region extraction network 16, and the object detection network 18 to perform processing based on the training image. (2) The learning control unit 32 also calculates loss function values for the object-likelihood map and the deviation map based on the object-likelihood map and deviation map output by the candidate region extraction network 16, and the correct value of the object-likelihood map and the correct value of the deviation map, and updates the internal parameters of the candidate region extraction network based on the loss function values for the object-likelihood map and the deviation map. (3) The learning control unit 32 also calculates a loss function value for the classification and the amount of deviation from the correct object region based on the classification and the amount of deviation from the correct object region output by the object detection network 18, and the correct classification and the correct amount of deviation from the correct object region, and updates the internal parameters of the object detection network 18 based on the loss function value for the classification and the amount of deviation from the correct object region.

[0035] Note that, regarding the object-likelihood map, the learning control unit 32 calculates an IOU (Intersection over Union) value between the anchor box and the bounding box of the object included in the correct data, and uses predetermined thresholds t1 and t2 (t2 < t1): if IOU value > t1, sets the correct value for the probability that the anchor box is an object to 1, and sets the correct value for the probability that the anchor box is the background to 0; if IOU value < t2, sets the correct value for the probability that the anchor box is an object to 0, and sets the correct value for the probability that the anchor box is the background to 1; and excludes anchor boxes satisfying t2 ≦ IOU value ≦ t1 from training targets. Note that the above IOU value is a numerical value representing the degree of overlap between regions. It is not always necessary to use the IOU value as the numerical value representing the degree of overlap. Processing may be performed using another numerical value for the degree of overlap calculated instead of the IOU value. The same applies hereinafter.

[0036] The learning control unit 32 calculates the correct value of the displacement amount based on the anchor box and the bounding box of the object included in the correct data when the IOU value > t1 with respect to the displacement map.

[0037] The learning control unit 32 calculates the IOU value between the correct bounding box of a small object (an object whose area in the image is smaller than a predetermined standard) and all of its anchor boxes, and determines the threshold t1 based on the statistical values ​​of the calculated IOU values ​​with all of its anchor boxes. It also calculates the IOU value between all of its anchor boxes and the correct bounding box for all of its anchor boxes in the image containing the correct bounding box of the small object, and determines the threshold t2 based on the statistical values ​​of the calculated IOU values ​​with all of its anchor boxes. Note that when calculating the statistical values ​​of the IOU value (degree of overlap), anchor boxes with an IOU value of 0 with the correct bounding box of the small object may be excluded. In other words, when determining the threshold t1, the statistical values ​​of the IOU value are calculated for all bounding boxes that have even a small overlap with the correct bounding box of the small object.

[0038] Furthermore, the objects processed by the learning control unit 32 are not necessarily limited to the "small objects" described above. The learning control unit 32 may also calculate the IOU value (degree of overlap) between the correct object and all anchor boxes (anchor regions) for general objects included in an image, regardless of the area of ​​the object in the image, and perform similar processing.

[0039] The data augmentation unit 34 augments the training data held by the training data storage unit 36. Specifically, the data augmentation unit 34 generates additional images by attaching additional objects (small objects) based on existing training images, and generates correct data for the additional objects (small objects) in those additional images.

[0040] The procedure by which the data augmentation unit 34 performs data augmentation is as follows: The data augmentation unit 34 divides the training image read from the training data storage unit 36 ​​into multiple patches and generates a patch-shuffled image by shuffling these multiple patches. The data augmentation unit 34 also generates an additional image by pasting an additional image containing additional objects onto an image that includes at least the patch-shuffled image. Furthermore, the data augmentation unit 34 stores pairs of the additional image and the ground truth data, consisting of the classification of the additional objects included in the additional image and the position and size of the bounding boxes of the additional objects, as additional training data in the training data storage unit. The ground truth classification of these additional objects is known to the data augmentation unit 34.

[0041] The training data storage unit 36 ​​stores training data for training the model of the object detection device 10. The training data is a set of pairs of training images 361 and ground truth data 362. The ground truth data 362 consists of the classification of objects contained in the training image 361, along with the position and size of the bounding box of each object. A single training image 361 may have ground truth data associated with multiple objects.

[0042] Training image 361 is an image used for training. Training image 361 is supplied as an input image to the object detection device 10 during model training.

[0043] The ground truth data 362 is the ground truth data corresponding to the training image 361. Here, "ground truth" includes the ground truth information about the position of the object shown in the training image 361, and the ground truth information about the class to which that object belongs.

[0044] The embodiments will be described in more detail below. The proposed method of this embodiment will be explained using Faster R-CNN (described in Non-Patent Document 1) as an example. However, the proposed method of this embodiment is also applicable to other machine learning-based candidate region extraction and object detection models such as Mask R-CNN (described in Non-Patent Document 2) and YOLO (described in Non-Patent Document 3).

[0045] [1] Identification stage The object detection model of this embodiment consists of three networks: a feature extraction network, a candidate region extraction network (RPN), and an object detection network. The processing in the identification stage of this object detection model is as follows.

[0046] [1.1] Feature extraction network The feature extraction network 14 extracts feature maps from the input image. For example, deep learning models such as VGG16 or ResNet can be used as the feature extraction network.

[0047] VGG16 is described in the following document. Literature: Karen Simonyan, Andrew Zisserman: “Very Deep Convolutional Networks for Large-Scale Image Recognition.” International Conference on Learning Representations (ICLR) 2015.

[0048] ResNet is described in the following document. Literature: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun: “Deep Residual Learning for Image Recognition.” Computer Vision and Pattern Recognition (CVPR) 2016: 770-778.

[0049] In the following explanation, we will use VGG16 as an example. VGG16 consists of multiple convolutional layers, a maximum pooling layer, and a fully connected layer. The input image is input from the first convolutional layer of VGG16 and passed through to the final convolutional layer (the layer immediately preceding the maximum pooling layer). The output is then output as a feature map. VGG16 performs maximum pooling four times. Therefore, the feature map is a 512-dimensional matrix (also called a tensor) with a size of 1 / 16 of the input image in both width and height. Layers after the feature map output are not used in object detection. For example, if the size of the input image is 224 × 224 × 3 (where 3 is the number of channels), the size of both width and height becomes 1 / 16, and the size of the feature map becomes 14 × 14 × 512. Each channel corresponds to, for example, the three primary colors of light: R (red), G (green), and B (blue). For example, if the input image size is 640 x 640 x 3, the width and height will be 1 / 16 of the original size, and the feature map size will be 40 x 40 x 512.

[0050] Figure 2 is a schematic diagram illustrating an example of processing by a feature extraction network. In the illustrated example, the input image size is 640 pixels × 640 pixels × 3 channels. When this image is input, the feature extraction network performs processing such as convolution and maximum pooling, and as a result outputs a feature map. In the illustrated example, the feature extraction network outputs a 40 × 40 × 512 feature map.

[0051] [1.2] Candidate Region Extraction Network (RPN) The Region Proposal Network (RPN) extracts bounding boxes of regions (candidate regions) that are highly likely to contain objects, based on the feature maps output by the feature extraction network described above. In the RPN, the feature maps output from the feature extraction network are input into a 3x3 convolutional layer (512 channels), and then passed through two more 1x1 convolutional layers. This process outputs two maps for each of the anchor boxes (described later) centered on each position in the feature map described above. The first map is an object-likeness map that stores the probability that each anchor box is an object / background. The second map is a deviation map (also referred to as a "deviation map") that represents the amount of deviation from the correct object region.

[0052] Figure 3 is a schematic diagram illustrating the processing overview by the Candidate Region Extraction Network (RPN). As shown in the figure, the Candidate Region Extraction Network (RPN) takes the feature map output by the Feature Extraction Network as input and outputs an object-likeness map and a shift map. In the example shown, the feature map has a size of 40 × 40 × 512 (512 is the number of channels). The Candidate Region Extraction Network (RPN) consists of a preceding CNN and a succeeding CNN. The output from the preceding CNN becomes the input to each of the two succeeding CNNs. These two succeeding CNNs output an object-likeness map and a shift map, respectively. The object-likeness map inherits the vertical and horizontal dimensions of the feature map and has a size of 40 × 40 × 18 (18 is the number of channels). The shift map similarly inherits the vertical and horizontal dimensions of the feature map and has a size of 40 × 40 × 36 (36 is the number of channels). Based on the two maps output from these candidate region extraction networks (RPNs), a non-maximal suppression method can be used to output candidate regions that are likely to be objects.

[0053] The anchor boxes described above are multiple bounding boxes of different sizes, centered on each location in the feature map. Nine anchor boxes are used for each location in the feature map. In other words, in the example shown in Figure 3, since the size of the feature map is 40x40, a total of 40x40x9 = 14,400 anchor boxes are prepared. Each anchor box is centered on each location within the feature map. The center coordinates and size of the anchor boxes are based on the scale of the input image. The size of the nine anchor boxes at each location is predetermined. In this embodiment, three scales with box areas of 128^2 pixels, 256^2 pixels, and 512^2 pixels, and three aspect ratios of 1:1, 1:2, and 2:1 are used. In other words, by combining the box area and aspect ratio, a total of 3x3 = 9 anchor boxes are prepared for each location.

[0054] The object-likeness map shown above has 18 channels. For each position on the feature map, the first 9 channels represent the probability that the 9 anchor boxes are objects, and the last 9 channels represent the probability that the anchor box is part of the background.

[0055] The above displacement map has 36 channels. The displacement map represents the amount of displacement between the anchor box and the correct object region. In other words, for each anchor box, the displacement map has four values: the displacement of the center x coordinate, the displacement of the center y coordinate, the displacement of the horizontal length, and the displacement of the vertical length. Since there are 9 anchor boxes at each position on the feature map, a total of 4 × 9 = 36 values ​​are stored. Each of these 36 values ​​corresponds to one channel. Since the center coordinates and size of the anchor boxes themselves are predetermined, by adding the four displacement amounts mentioned above to them, the center coordinates and size of the object region estimated to be the correct object region, i.e., the candidate region, can be calculated.

[0056] As described above, RPN estimates whether or not an object is reflected in each anchor box, and also estimates the center coordinates and size of the candidate regions of the object. Next, using the Non-Maximum Suppression method, only the candidate regions with a high likelihood of being an object are retained in the final output, and all other superfluous candidate regions are removed. Note that the Non-Maximum Suppression method is an existing technique.

[0057] [1.3] Object detection network The object detection network 18 detects objects within candidate regions. First, based on the feature maps extracted by the feature extraction network described above, it uses the Region of Interest Pooling (ROI) method to extract local feature maps corresponding to the extent of each candidate region. ROI stands for Region of Interest. Even if the candidate regions have different sizes, the local feature maps extracted from them have a unified, constant length. Next, the local feature maps are input to a CNN to identify the object class and simultaneously correct the center coordinates and size of the candidate region. This CNN passes the local feature maps through a total of four convolutional layers and finally outputs the object class classification result and the amount of deviation from the ground truth object region. The amount of deviation from the ground truth object region is similar to the output of the RPN, and by adding these deviations to the center coordinates and size of the candidate region, the bounding box of the candidate region can be corrected.

[0058] [2] Training phase Next, we will explain the training phase of the object detection model. In the training phase, a large amount of training data is loaded into the object detection model, and the parameters of the object detection model are iteratively updated and determined to create an optimally tuned model. One training data set consists of a pair of one training image and the corresponding ground truth data. This ground truth data consists of a set of bounding boxes surrounding each object in the training image and the class labels (class classifications) of those objects. As a training method, we use backpropagation based on the ground truth data. Backpropagation is an existing method.

[0059] [2.1] Data augmentation based on spatial structural destruction The data augmentation unit 34 expands the training data. To improve the performance of an object detection model based on deep learning, larger and more diverse training data that reflects the distribution of target image data is necessary. However, training images containing small objects are difficult to collect and therefore few in number. As a result, conventional object detection models have the problem that their ability to detect small objects is not sufficiently learned compared to normal objects. In this embodiment, this problem is addressed and the training of the object detection model is strengthened by creating new training data from existing training data to increase the amount of training data on small objects.

[0060] One possible approach is to create new training data by pasting images of small objects at arbitrary positions within an arbitrary training image. However, this method has two problems. The first problem is that since training images contain other normal-sized objects, creating new training data increases both normal and small objects, while the proportion of small objects remains low. While it is possible to increase the proportion of small objects by treating normal objects as background, this negatively impacts the learning of the ability to detect normal objects. The second problem is that since images of small objects are square or rectangular, unnatural horizontal and vertical lines appear at their boundaries. If a large amount of such training data is created, these unnatural patterns will be mistakenly learned as cues for detecting small objects. As a result, the object detection model's training will focus on detecting patterns such as horizontal and vertical lines rather than small objects.

[0061] Therefore, in this embodiment, a data augmentation method based on the spatial structure destruction of images is introduced. In the method introduced here, first, a pseudo-background image is created by performing a patch shuffle on an arbitrary training image. Next, new training data is created by combining small object images at arbitrary locations on the background image.

[0062] This embodiment employs patch shuffling to actively destroy information about ordinary objects that may be included in training images, and uses the resulting image as the background image. The aim is to prevent the influence of ordinary objects during training.

[0063] Specifically, this embodiment destroys the spatial structure by dividing the original image into multiple square patches and randomly shuffling the positions of those patches. For example, if the input image is 224 pixels x 224 pixels, the input image can be divided into patches of 7x7 (49 in total) or 14x14 (196 in total). If the input image is 640 pixels x 640 pixels, it can be divided into patches of 8x8 (64 in total) or 16x16 (256 in total). This division of the input image destroys spatial structure information that is normally an important clue to representing an object, such as the shape of an object and the connections between its parts.

[0064] Figure 4 is a schematic diagram showing examples of patch shuffling with different patch sizes. Figure 4(A) shows the original image before shuffling. This example shows an image of a frog with its head above the water. Figures 4(B), (C), and (D) show examples of the result of dividing the image in Figure 4(A) into patches and shuffling them. In the case of Figure 4(B), the number of patches is 7x7 (49 in total). In the case of Figure 4(C), the number of patches is 14x14 (196 in total). In the case of Figure 4(D), the number of patches is 28x28 (784 in total). Each square fragment when the original image is divided is a patch. Shuffling should be performed on a patch-by-patch basis.

[0065] As shown in Figure 4, when the division is coarse (when the number of divided patches is small), the shape and parts of the object are still relatively recognizable. However, when the division is made finer (when the number of divided patches increases), even the parts that make up the object become fragmented, making it difficult to recognize the object.

[0066] Figure 5 is a schematic diagram illustrating an example of how patches are shuffled. In this example, the image before shuffling is divided into 4x4 (16 patches in total) sections. In this example, all patches are randomly shuffled across the entire image. For convenience, this type of shuffling is called a global shuffle. In other words, in the illustrated example, the four patches in the first row before shuffling are placed in the third column of the first row, the fourth column of the first row, the second column of the second row, and the first column of the fourth row after shuffling, showing no regularity in the shuffling pattern. The same applies to the patches in the second row and beyond before shuffling.

[0067] Figure 6 is a schematic diagram illustrating another example of how patches are shuffled. In this example, the image before shuffling is divided into 4x4 (16 patches in total) patches. In this example, the 4x4 patches are first shuffled into 2x2 patch blocks, and then further shuffled within those blocks, using 2x2 patches. In other words, this example performs a hierarchical shuffle. For convenience, this type of shuffling is called a hierarchical shuffle. Specifically, in the illustrated example, the top-left 2x2 block before shuffling is first moved to the bottom-left 2x2 block after shuffling. The same applies to the top-right, bottom-left, and bottom-right 2x2 blocks. After such block-level shuffling, patch-level shuffling is performed within those blocks. In the example shown in Figure 6, a two-level hierarchical shuffle was performed, but generally, a hierarchical shuffle of three or more levels may be used.

[0068] When shuffling patches, a general shuffle as shown in Figure 5 may be performed, a hierarchical shuffle as shown in Figure 6 may be performed, or other methods may be used to shuffle the patches. In any of these patch shuffling methods, the spatial structure of the original image is destroyed. In other words, objects that were present in the original image (relatively large objects) become more difficult to recognize.

[0069] In this embodiment, shuffling the patches as described above not only destroys the information of normal objects in the image, but also generates numerous vertical and horizontal line patterns, such as patch boundaries, within the background of the image. In other words, when a small object is inserted, both the boundaries and the background will contain an equal number of vertical and horizontal line patterns. When training an object detection model using such training data, it is possible to avoid a training method in which the vertical and horizontal line patterns are mistakenly used as clues for detecting small objects. That is, when training an object detection model using such training data, it is possible to concentrate on training to detect the small objects themselves without being affected by unnatural vertical or horizontal line patterns.

[0070] Next, the final background image is generated by combining the original image with the image obtained by patch shuffling (referred to as the "patch shuffle image"). Here, let the original image be A and the patch shuffle image be B, and their respective weights be w. a and w b Let's assume this. The final background image generated based on this weighting is B' in equation (1) below. This final background image is an image that includes a patch shuffle image, as shown in equation (1).

[0071]

number

[0072] weight w a and w b For example, the value of 0 ≤ browser a≦1, 0≦w b ≦1, w a +w b =1. As a specific example of values, w a =0.5, w b =0.5 may be set.

[0073] As described above, after performing the spatial structure destruction processing using patch shuffling, a new training image can be created by pasting a small object image to any position on the background image. Since the positions of bounding boxes for small objects in training data are known, it is easy to cut out small objects from training images. By storing the position where the small object is pasted onto the background image, a new training image, the (correct) position and size of the small object, and the (correct) class label of the small object can be collected together and generated as new training data. By creating a large number of such new training data and adding them to existing training data, the conventional problem of insufficient training data containing small objects can be solved or alleviated.

[0074] Note that at this time, diverse training data can be generated by using small object images belonging to various class labels and pasting the small object images to various positions within the background image. Furthermore, by training an object detection model using such diverse training data, it becomes possible to further improve the performance of the object detection model in detecting small objects.

[0075] FIG. 7 is a schematic diagram showing an example configuration of training data. As illustrated, the training data is configured as pairs of training images and correct answer data. When training an object detection model, a large number of pairs of training images and correct answer data are used. A training image is, for example, a 3-channel image composed of the three primary colors RGB. One or more objects are shown in a training image. The correct answer data includes, for each object included in the training image, a set of data including the correct class classification (correct label), and the center position coordinates and size of the correct object region (bounding box of the correct object).

[0076] Figure 8 is a schematic diagram showing the processing flow of the data augmentation unit 34 performing training data augmentation. As shown in the figure, the data augmentation unit 34 generates additional images based on one training image. Specifically, the data augmentation unit 34 first divides the training image into patches. The shape of the patches is, for example, a rectangle, but other shapes may also be used. For example, the data augmentation unit 34 divides the training image into patches arranged in a grid of M vertically and N horizontally. Next, the data augmentation unit 34 shuffles the patched images in units of patches to generate a patch-shuffled image. Next, the data augmentation unit 34 combines the original training image and the patch-shuffled image using the above equation (1) to create a background image. Next, the data augmentation unit 34 pastes an image of a small object (additional object) (additional image) to a predetermined position on the background image. In this way, the data augmentation unit 34 generates an additional image. In other words, the data augmentation unit 34 generates an additional image by pasting an additional image containing an additional object onto an image containing a patch-shuffled image. The data augmentation unit 34 understands the position and size of the additional images to be pasted. Accordingly, the data augmentation unit 34 generates ground truth data for the additional objects. This ground truth data includes the correct classification (ground truth label) for the additional objects and data on the center position and size of the ground truth object region (bounding box). The data augmentation unit 34 stores these pairs of generated additional images and their corresponding ground truth data for the additional objects in the training data storage unit 36 ​​as additional training data.

[0077] [2.2] Training of the feature extraction network We will pre-train a feature extraction network 14 (using VGG16 as an example) using the ImageNet dataset. ImageNet contains a large number of images and class labels for the objects in each image. VGG16 is a CNN for classification. When an image is input to VGG16, it outputs the probability that an object of each class is present. The cross-entropy error between the probability output by VGG16 and the ground truth data in ImageNet can be calculated as the loss function (error). We will iteratively update the values ​​of the intrinsic parameters of the feature extraction network using backpropagation to minimize the value of this loss function.

[0078] The ImageNet dataset is described in the following document. Literature: Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, Li Fei-Fei: “ImageNet Large Scale Visual Recognition Challenge.” Computing Research Repository (CoRR) (2014)

[0079] [2.3] Training of candidate region extraction network To train the candidate region extraction network 16, first, training images are passed through a pre-trained feature extraction network to obtain feature maps. Next, the feature maps are input to the candidate region extraction network, and object resemblance maps and displacement maps are output. A loss function (error) is prepared based on these outputs and the ground truth values ​​(described later), and the parameters of the candidate region extraction network are iteratively updated using backpropagation to minimize the value of the loss function.

[0080] To prepare the ground-truth values for the objectness map, the IOU (degree of overlap) between each anchor box and the ground-truth bounding box included in the training data is calculated. IOU is an abbreviation for "Intersection over Union". That is, IOU is a value obtained by dividing the area of the intersection of the two target regions by the area of the union of the two regions. When the two target regions completely overlap and match each other, the IOU value is 1. When the two target regions do not overlap at all (have no common area at all), the IOU value is 0. When t1 and t2 are predetermined thresholds respectively, if IOU > t1, the ground-truth value of the probability that the anchor box is an object is set to 1, and the ground-truth value of the probability that the anchor box is the background is set to 0. Further, if IOU < t2, the ground-truth value of the probability that the anchor box is an object is set to 0, and the ground-truth value of the probability that the anchor box is the background is set to 1. The setting of the values of thresholds t1 and t2 will be described later. Portions (anchor boxes) that satisfy t2 ≤ IOU ≤ t1 are portions that can hardly be classified as either an object or the background, and are not used for training (excluded from training targets).

[0081] For the offset map, portions other than those satisfying IOU > t1 (non-object portions) are not used for training. For portions satisfying IOU > t1, the offset between each anchor box and the ground-truth bounding box is calculated as the ground-truth value. Specifically, four ground-truth values are calculated: the offset of the center x-coordinate, the offset of the center y-coordinate, the offset of the horizontal length, and the offset of the vertical length.

[0082] A candidate region extraction network (RPN) is trained using the loss function represented by Equation (2) below.

[0083]

Mathematical Expression

[0084] In Equation (2), i is the index of an anchor box. p´ iThis is the correct value for the probability that the i-th anchor box is an object. That is, if the i-th anchor box is an object, then p' i If = 1 and the i-th anchor box is the background, then p' i = 0. i This is the output value from the candidate region extraction network (RPN). T' i This is the correct vector for the displacement of the i-th anchor box. i This is the output value from the candidate region extraction network (RPN). cls This is the number of anchor boxes when the portion where t2 ≤ IOU ≤ t1 (the portion that is neither clearly an object nor background) is excluded. reg is the number of anchor boxes where IOU > t1. The parameter λ is for adjusting the weighting; for example, λ = 10. cls This is the logarithmic loss, L reg This is the smooth L1 loss.

[0085] [2.4] Adaptive Candidate Region Extraction Learning Not only Faster R-CNN described in Non-Patent Document 1, but most existing (conventional) object detection models train the model using a large number of bounding boxes that resemble anchor boxes. In this process, not all anchor boxes are used for training, but only those anchor boxes whose degree of overlap with the correct bounding box (the IOU value mentioned above) is above a certain threshold (t1). In Faster R-CNN, t1=0.7 and t2=0.3 are set.

[0086] However, this method is not suitable for training the ability to detect small objects. When dealing with small objects, the IOU value between the anchor box and the object is often low, and as a result, many of the anchor boxes containing the small objects are not used in training. Therefore, in the identification stage, there is a problem in that small objects are mistakenly recognized as background.

[0087] In this embodiment, a method is employed to automatically select anchor boxes to be used for training based on the statistical characteristics of the anchor boxes. This method is applied only to the bounding boxes of small objects. Here, objects that occupy less than a predetermined percentage (for example, 12%) of the image area are considered small objects.

[0088] As explained in Section 1.2, "Candidate Region Extraction Network (RPN)," nine anchor boxes are prepared for each position in the feature map, using three scales with box areas of 128^2 pixels, 256^2 pixels, and 512^2 pixels. This method in this embodiment is applied to anchor boxes of the same scale. The following processing is performed for each scale.

[0089] For each small object, select all anchor boxes whose IOU value with respect to G is not 0 within the correct bounding box G. Calculate the mean and standard deviation of the IOU values ​​of these anchor boxes as m1 and s1, respectively. Using these statistics, determine the IOU threshold t1 for this small object using equation (3) below.

[0090]

number

[0091] In equation (3), α is a parameter. In this embodiment, for example, α is set to 2. Another threshold t2 is calculated as follows: For all anchor boxes in the image containing G, the IOU value between G and the anchor box is calculated, and the mean and standard deviation of these IOU values ​​are calculated as m2 and s2, respectively. Using these statistics, the IOU threshold t2 for this small object is calculated using equation (4) below.

[0092]

number

[0093] In equation (4), β is a parameter. In this embodiment, for example, β is set to 1.

[0094] This method allows for the selection of anchor boxes necessary for training small object detection capabilities, adapting to the position and size of the small objects. Compared to conventional techniques that use fixed thresholds t1 and t2, the method of this embodiment allows for the use of more anchor boxes containing small objects in training, thereby improving the balance between normal objects and small objects and enhancing the small object detection performance of the model.

[0095] As a variation, when calculating the mean m1 and standard deviation s1, instead of calculating for each individual object, the mean and standard deviation of the IOU values ​​can be calculated by grouping together anchor boxes with non-zero IOU values ​​for all objects included in the training data set. Modified versions may also be used when calculating the mean m² and standard deviation s². Alternatively, all anchor boxes of the same scale included in the training data set can be grouped together, their IOU values ​​with each small object can be calculated, and then the mean m² and standard deviation s² of the IOU values ​​can be calculated.

[0096] [2.5] Training of object detection networks As explained in Section 1.3, "Object Detection Network," the object detection network 18 outputs two types of information: the result of object classification and the amount of deviation from the ground truth object region.

[0097] If there are a total of K object classes to be detected, the classification result for each candidate region output by the Candidate Region Extraction Network (RPN) is a (K+1) dimensional vector U = (u0, u1, ..., u K It is expressed as ). Here, each element of vector U is a class probability, and its range of value is [0,1]. Note that u0 is the probability of the background class. u1,···,u KEach of these is a probability of one of the K object classes. The label of the correct class for each candidate region is represented by u'. To prepare the label of the correct class, calculate the IOU value between the candidate region and the correct bounding box. If the calculated IOU value is 0.5 or greater, the class of the corresponding object is taken as the correct class. If the IOU value is less than 0.5, the background class is taken as the correct class.

[0098] The deviation from the ground truth object region is similar to the output from the Representation Promoter Network (RPN). Let V be the estimated bounding box deviation for each candidate region. V is a 4-dimensional vector containing four elements: deviation of the center x coordinate, deviation of the center y coordinate, deviation of the horizontal length, and deviation of the vertical length. Let V' be the ground truth value corresponding to V.

[0099] The loss function for training the object detection network is given by equation (5) below.

[0100]

number

[0101] Similar to the loss function of the candidate region extraction network (RPN), in equation (5), L cls This is the logarithmic loss, L reg This is the smooth L1 loss. The value of the parameter λ is set to, for example, λ=1. [u≠0] is Iverson notation, where the value is 1 when u≠0 and 0 when u=0. The parameter values ​​of the object detection network are iteratively updated using backpropagation so that the value of the loss function L(U,V) in equation (5) is minimized.

[0102] [3] Summary The above describes the method for generating training data and the training methods for each network. When actually training the object detection model, the accuracy of each network is improved by training the candidate region extraction network (RPN) and the object detection network alternately. The parameters of the feature extraction network are fixed in both the training of the candidate region extraction network (RPN) and the object detection network.

[0103] Figure 9 is a block diagram illustrating an example of the internal configuration of an object detection system 1, which consists of an object detection device 10 and a learning device 30. The devices comprising the object detection system 1 can be realized using a computer. As shown in the figure, the computer consists of a central processing unit 901, RAM 902, input / output ports 903, input / output devices 904 and 905, etc., and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in programs read from RAM 902, etc. The central processing unit 901 writes data to RAM 902, reads data from RAM 902, and performs arithmetic and logical operations according to each instruction. RAM 902 stores data and programs. Each element contained in RAM 902 has an address and can be accessed using that address. RAM stands for "Random Access Memory". Input / output ports 903 are ports for the central processing unit 901 to exchange data with external input / output devices, etc. Input / output devices 904 and 905 exchange data with the central processing unit 901 via input / output port 903. Bus 906 is a common communication channel used within the computer. For example, the central processing unit 901 reads and writes data to RAM 902 via bus 906. Also, for example, the central processing unit 901 accesses input / output port 903 via bus 906.

[0104] At least some of the functions of the object detection system in the above-described embodiment can be implemented using a computer and a program. In this case, the program for implementing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, DVD-ROMs, USB memory, and storage devices such as hard disks built into a computer system. In other words, "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Moreover, "computer-readable recording medium" may also include those that temporarily and dynamically hold programs, such as communication lines when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in such a case. Furthermore, the above-mentioned program may be for implementing some of the functions described above, and may also be able to implement the above-mentioned functions in combination with a program already recorded in the computer system.

[0105] The method for achieving a predetermined function using machine learning techniques is as follows. Here, the unit of a machine learning-capable function is conveniently referred to as a "function unit." This function unit calculates output data based on the input data. When calculating the output data, the function unit uses the values ​​of internal parameters. These internal parameters can be updated and optimized through the learning process. When training the function unit, training data (also called "training data" as mentioned above) is used. The training data includes the input data to the function unit and the correct output data calculated based on that input data. During the learning process, the function unit reads the input data contained in the training data and, based on that input data and using the values ​​of the internal parameters at that time, refers to the output data. This output data is an estimated value obtained based on the internal parameters at that time and does not necessarily coincide with the correct answer. To update the internal parameters, the difference between the estimated value calculated and output by the function unit based on the input data at that time and the correct answer corresponding to that input data is calculated. This difference is called error, loss, etc. The difference sought here may be, for example, the absolute value of the difference between scalars, the squared error, the cross-entropy error, or a difference calculated by other methods. Based on the obtained difference, the values ​​of the internal parameters can be updated using backpropagation. This operation adjusts the values ​​of the internal parameters in a direction that reduces the error. By performing the above operation multiple times (many times) using a predetermined amount of training data, the values ​​of the internal parameters are optimized. In other words, the above function is adjusted to perform the processing exemplified by the given training data. After a sufficient amount of training has been performed, the training process may be terminated. By storing the set of trained internal parameter values ​​at this time in a memory device, estimation based on the training results becomes possible. The function that includes the above internal parameters and enables data input and output may be called a "model". In other words, a model includes internal parameters.The process of learning the functional components described above can be called "model training." One method for realizing a model is to use a neural network. Depending on the type of processing being performed, an appropriate form of neural network can be used.

[0106] Although embodiments have been described above, the present invention can also be implemented with the following modifications. In the above embodiments, processing based on Faster R-CNN was described. As modifications, the present invention may be applied to candidate region extraction networks or object detection networks using models with other configurations, such as Mask R-CNN or YOLO.

[0107] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Industrial applicability]

[0108] The present invention can be used, for example, as a mechanism for object detection in product manufacturing processes. However, the scope of application of the present invention is not limited to those exemplified herein. [Explanation of Symbols]

[0109] 1. Object detection system 10 Object detection device 12 Image Input Section 14 Feature Extraction Network 16 Candidate Region Extraction Network 18 Object Detection Networks 20 Result Output Section 22 Parameter storage unit 30 Learning device 32 Learning Control Unit 34 Data Expansion Unit 36 Training data storage unit 361 training images 362 correct data points 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. A feature extraction network that extracts features for each position in an image based on the input image and outputs a feature map, A candidate region extraction network configured to be machine learning capable, which outputs, based on the feature map, an object-likeness map that holds the probability that each anchor region including each position in the feature map is an object, and a deviation map that holds the deviation amount representing the degree to which the anchor region deviates from the correct object region, as candidate region information. An object detection network configured to be machine learning capable, which outputs the object classification for each candidate region output by the candidate region extraction network, as well as the location and size of the candidate region, based on the feature map, A training data storage unit stores training data which is a set of pairs of training images and correct data consisting of the classification of objects contained in the training images and the position and size of those objects. A learning control unit (1) supplies the training images included in the training data as input images to the feature extraction network, and causes the feature extraction network, the candidate region extraction network, and the object detection network to process the training images; (2) calculates a loss function value for the object resemblance map and the shift map based on the object resemblance map and the shift map output by the candidate region extraction network, and updates the internal parameters of the candidate region extraction network based on the loss function value for the object resemblance map and the shift map; and (3) calculates a loss function value for the classification and the amount of deviation from the correct object region based on the classification and the amount of deviation from the correct object region output by the object detection network, and updates the internal parameters of the object detection network based on the loss function value for the classification and the amount of deviation from the correct object region. An object detection system comprising, The learning control unit calculates the degree of overlap between the anchor region and the objects included in the correct data with respect to the object-likeness map, and sets the correct value of the probability that the anchor region is an object to 1 if the degree of overlap > t1 using a predetermined threshold t1. The learning control unit calculates the correct value of the displacement amount based on the anchor region and the objects included in the correct data when the degree of overlap > t1 with respect to the displacement map. The learning control unit calculates the degree of overlap between the correct answer for an object included in the image and all anchor regions, and determines the threshold t1 based on the calculated statistical value of the degree of overlap with all anchor regions. Object detection system.

2. A data expansion unit generates a patch-shuffled image by dividing the training image read from the training data storage unit into multiple patches and shuffling these multiple patches; generates an additional image by pasting an additional image containing an additional object onto an image containing at least the patch-shuffled image; and stores pairs of the additional image and the correct data consisting of the classification of the additional object contained in the additional image, the position and size of the additional object, as additional training data in the training data storage unit. The object detection system according to claim 1, further comprising:

3. A feature extraction network that extracts features for each position in an image based on the input image and outputs a feature map, A candidate region extraction network configured to be machine learning capable, which outputs, based on the feature map, an object-likeness map that holds the probability that each anchor region including each position in the feature map is an object, and a deviation map that holds the deviation amount representing the degree to which the anchor region deviates from the correct object region, as candidate region information. An object detection network configured to be machine learning capable, which outputs the object classification for each candidate region output by the candidate region extraction network, as well as the location and size of the candidate region, based on the feature map, A learning method for training the candidate region extraction network and the object detection network of an object detection device comprising the following: The training data storage unit stores training data which is a set of pairs of training images and ground truth data consisting of the classification of objects contained in the training images and the position and size of those objects. The learning control unit, (1) The training images included in the training data are supplied as input images to the feature extraction network, and the feature extraction network, the candidate region extraction network, and the object detection network are made to perform processing based on the training images. (2) Based on the object-likeness map and the misalignment map output by the candidate region extraction network, and the ground truth values ​​of the object-likeness map and the misalignment map, a loss function value for the object-likeness map and the misalignment map is calculated, and the internal parameters of the candidate region extraction network are updated based on the loss function value for the object-likeness map and the misalignment map. (3) Based on the class classification and deviation from the ground truth object region output by the object detection network, and the ground truth for the class classification and the deviation from the ground truth object region, a loss function value related to the class classification and the deviation from the ground truth object region is calculated, and the internal parameters of the object detection network are updated based on the loss function value related to the class classification and the deviation from the ground truth object region. The learning control unit calculates the degree of overlap between the anchor region and the objects included in the correct data with respect to the object-likeness map, and sets the correct value of the probability that the anchor region is an object to 1 if the degree of overlap > t1 using a predetermined threshold t1. The learning control unit calculates the correct value of the displacement amount based on the anchor region and the objects included in the correct data when the degree of overlap > t1 with respect to the displacement map. The learning control unit calculates the degree of overlap between the correct answer for an object included in the image and all anchor regions, and determines the threshold t1 based on the calculated statistical value of the degree of overlap with all anchor regions. Learning methods.

4. The data expansion unit generates a patch-shuffled image by dividing the training image read from the training data storage unit into multiple patches and shuffling these multiple patches, generates an additional image by pasting an additional image containing additional objects onto an image containing at least the patch-shuffled image, and stores pairs of the additional image and the correct data consisting of the classification of the additional objects contained in the additional image and the position and size of the additional objects, as additional training data in the training data storage unit. The learning method according to claim 3.

Images