Information processing device, information processing method, and program

Abstract

To provide an information processing device capable of improving inference accuracy of a learning model to infer an area where an object exists, from an image.SOLUTION: An image processing apparatus according to the present invention learns an object likeness in an image. The image processing apparatus has image acquisition means for acquiring the image, correct-answer data acquisition means for acquiring correct-answer data including at least the object area where the object exists for the object existing in the acquired image, inference means for inferring the candidate area of the object in the image and a first score indicating the object likeness in the candidate area based on a learning model, determination means for determining a second score indicating the object likeness for the object area based on the inferred candidate area and the area included in the acquired correct-answer data, and update means for updating the parameters of the learning model based on a calculated loss value, based on the second score and on the first score indicating the object likeness in the inferred candidate area.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to training a learning model that infers the position of an object from an image. [Background technology]

[0002] In recent years, machine learning technologies, particularly deep learning, have seen rapid advances in image recognition, speech recognition, and machine translation. In particular, the advent of CNNs (Convolutional Neural Networks) has dramatically improved accuracy in the field of object detection, which infers the position and shape of objects in images. Non-Patent Document 1 is an example of object detection using CNNs. Faster-RCNN uses a mechanism for inferring object candidate positions called a Region Proposal Network (RPN) to determine whether each anchor set in a grid pattern within an image is an object. Each anchor is assigned an anchor box, which is a rectangle of various sizes and aspects centered on the anchor. Learning is performed based on the Intersection of Union (IoU), which indicates the degree of overlap between the anchor box and a bounding box, which is the true value (Ground Truth). Anchor boxes with a high IoU with the ground truth are trained as likely objects, while anchor boxes with a low IoU are trained as background. Anchor boxes that are deemed likely objects undergo regression training to approximate their shape to the true bounding box. Furthermore, in Non-Patent Document 1, centerness is learned, with a value that increases toward the center of a bounding box, which is a rectangle surrounding an object. During inference, areas with high centerness are determined to be likely to be objects. Furthermore, Patent Document 1 is an invention that applies Faster-RCNN, and like Faster-RCNN, object-likeness is learned and inferred using an RPN. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2020-119522 [Non-patent literature]

[0004] [Non-Patent Document 1] Ren et al, “Faster-RCNN: Towards Real-Time Object Detection with Region Proposal Network”, NIPS2015. Summary of the Invention [Problem to be solved by the invention]

[0005] In the method disclosed in Patent Document 1, object-likeness is learned using an RPN. As described above, RPN learning is performed according to the IoU between the anchor box set for each anchor and the bounding box that is the true value. Here, the anchor box is set relative to the center of the anchor, and for anchor boxes with the same size and aspect ratio, the anchor box with an anchor closer to the center of the bounding box of the GT has a higher IoU with the bounding box of the GT. For this reason, anchors closer to the object center are more likely to be learned as object-like.

[0006] On the other hand, the degree of overlap between the bounding box regression results of anchor boxes with anchors close to the object center and the bounding box of GT is not necessarily high. This is because the bounding box regression results do not necessarily learn the correct anchors as likely objects, and object-likeness learning and bounding box regression learning are done independently.

[0007] The present invention has been made in view of the above-mentioned problems, and aims to improve the inference accuracy of a learning model that infers an area where an object exists from an image. [Means for solving the problem]

[0008] The image processing device of the present invention is an information processing device that learns the object-likelihood of an object in an image, and includes: an image acquisition means for acquiring an image; a correct data acquisition means for acquiring correct data for an object present in the acquired image, the correct data including at least an object region in which the object exists; an inference means for inferring a candidate region of the object in the image and a first score indicating the object-likelihood of the candidate region based on a learning model; a determination means for determining a second score indicating the object-likelihood of the object region based on the inferred candidate region and a region included in the acquired correct data; and an update means for updating parameters of the learning model based on a loss value calculated based on the second score and the first score indicating the object-likelihood of the inferred candidate region. [Effects of the Invention]

[0009] According to the present invention, it is possible to improve the inference accuracy of a learning model that infers an area where an object exists from an image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 2] A block diagram showing an example of the functional configuration of an information processing device. [Figure 3] 1 is a flowchart showing a processing procedure executed by an information processing device; [Figure 4] An example showing the object likelihood and object shape output by the inference unit [Figure 5] Example showing intersection of union [Figure 6] 1 is a flowchart showing a processing procedure executed by an information processing device; [Figure 7] FIG. 10 is a diagram showing an example of an acquired template image and a search range image. [Figure 8] An example of a Convolutional Neural Network DETAILED DESCRIPTION OF THE INVENTION

[0011] <Embodiment 1> <Detection task learning> An information processing device according to an embodiment of the present invention will be described with reference to the drawings. Components with the same reference numerals in the drawings operate in the same way, and therefore will not be described again. Furthermore, the components shown in the embodiments are merely examples, and are not intended to limit the scope of the present invention to those components alone.

[0012] In this embodiment, a learning method for an object detection task that detects the position and shape of an object from an image will be described.

[0013] 1 is a hardware configuration diagram of an information processing device 1 that performs object detector training in this embodiment. A CPU H101 controls the entire device by executing a control program stored in a ROM H102. A RAM H103 temporarily stores various data from each component. The RAM H103 also expands the program and makes it executable by the CPU H101.

[0014] The storage unit H104 stores data to be processed in this embodiment, and saves data used for learning. The storage unit H104 may be stored in a hard disk drive (HDD), a flash memory, or various optical media.

[0015] 2 is a block diagram showing an example of the functional configuration of the information processing device 1. The information processing device 1 aims to learn a task of detecting an object from an image. The information processing device 1 has an image acquisition unit 201, a GT acquisition unit 202, an inference unit 203, a score calculation unit 204, a determination unit 205, a loss calculation unit 206, and a parameter update unit 207, and each of these components is connected to a storage unit 207. Note that the storage unit 208 may be included in the information processing device 1. Each of the functional components will be briefly described.

[0016] The image acquisition unit 201 acquires images stored in the storage unit 208. The images show objects to be detected, such as people and cars. The image acquisition unit 201 may acquire images captured by an imaging device. The GT acquisition unit 202 acquires ground truth data (Ground Truth) from the storage unit 208, including the positions and sizes of objects present in the images acquired by the image acquisition unit 201. Specifically, the GT (Ground Truth) is an image prepared for learning, to which a bounding box (e.g., a rectangular area) indicating the position and size of the object and a label indicating the object category (class) have been assigned in advance by a person. An object area with a specified bounding box is likely to contain some kind of object regardless of the object category, and can be said to be an area that should be estimated with a high object-likeness score. However, the method of generating a GT is not limited to this.

[0017] The inference unit 203 inputs the image acquired by the image acquisition unit 201 into a neural network, and obtains, as inference results, a candidate region (bounding box) indicating the possibility that an object is included, and a score (first score) indicating the object-likelihood of the candidate region. As shown in FIG. 8, the neural network is configured with convolution, batch-normalization, and rectified linear unit (hereinafter referred to as ReLU). The configuration example in FIG. 8 is merely an example, and instead of ReLU, Leaky ReLU or a sigmoid function, etc., may be used, or MaxPooling or AveragePooling may be combined. This embodiment is not limited to these. As inference results, a bounding box indicating the object-likelihood and the shape of the object is obtained. Note that a bounding box is set for each anchor point set for each coordinate of the target image.

[0018] The score calculation unit 204 calculates a score (second score) of the inferred object shape based on the object shape (candidate area) that is the inference result obtained from the inference unit 203 and the object shape (correct answer data) that is the true value obtained by the GT acquisition unit 202.

[0019] The determination unit 205 determines the GT of object-likeness (the score of a new correct answer region and object-likeness in the target image) based on the score obtained by the score calculation unit 204. As will be described in detail later, a bounding box whose inferred object shape score is larger (greater than or equal to a predetermined value) than the surrounding bounding boxes is determined as a region of interest, and the score of the inferred object shape of the region of interest and the region of interest are treated as the GT of the target image. The loss calculation unit 206 calculates a loss based on the object-likeness that is the inference result obtained by the inference unit 203 and the GT obtained by the object-likeness GT determination unit 205. The parameter update unit 207 updates the parameters of the neural network based on the loss obtained by the loss calculation unit 206, and stores the results in the storage unit 208.

[0020] 3 is a flowchart showing the flow of learning for object detection in this embodiment. In the following description, each process (step) is represented by adding an S to the beginning, and the notation of the process (step) is omitted. However, the information processing device does not necessarily have to perform all of the processes described in this flowchart. The processes executed by the CPU H101 are each shown as a functional block.

[0021] In S301, the image acquisition unit 201 acquires an input image stored in the storage unit 208. The input image is as shown in Fig. 7(a), and an object to be detected is captured in the image.

[0022] In S302, the GT acquisition unit 202 acquires the GT, which is the correct answer data stored in the storage unit 208. As shown in Fig. 4(a), the GT is a bounding box shape 403 and a label 404 for an object 402 to be detected. In this case, the object to be detected is a person, and the person's label and bounding box constitute the GT.

[0023] In S303, the inference unit 203 infers the object likelihood, label, and shape. The shape inferred here is a rectangular bounding box that surrounds the object. Figure 4(b) shows bounding boxes inferred at some anchors. Each grid in 405 represents an anchor. 406, 408, and 410 are anchors, 407 is the bounding box inferred at anchor 406, 409 is the bounding box inferred at anchor 408, and 411 is the bounding box inferred at anchor 410. 406, 408, and 410 are filled in according to the inferred object likelihood value, with a darker color indicating a higher inferred object likelihood.

[0024] FIG. 4(c) shows the parameters of the bounding box to be inferred. Distances 412, 413, 414, and 415 to the top, bottom, left, and right edges of the bounding box are inferred, with anchor 417 as the center. Note that the bounding box parameters to be inferred are not limited to those described above. The vertical and horizontal sizes of the bounding box and the positional deviation from the center of the bounding box may also be inferred. Furthermore, the shape to be inferred does not have to be rectangular like the bounding box. A shape such as a circle may also be inferred. This embodiment is not limited to these.

[0025] In S304, the score calculation unit 204 calculates a BB representing the shape of the object inferred by the inference unit 203. pred and GT BB acquired from GT Acquisition Department 202 GT Based on this, a score that is an index of object-likeness is calculated for each anchor. Specifically, using Equation (1-1), BB pred and BB GT The score is determined based on the degree of overlap (IoU) with the original data.

[0026]

number

[0027] As shown in Figure 5, when there are two bounding boxes 501 and 502, IoU is calculated based on the intersection and union of the two bounding boxes as shown in the following formula (1-2). The range of IoU is [0, 1], and the higher the degree of overlap between the bounding boxes, the higher the value.

[0028]

number

[0029] In S305, the object likelihood GT determination unit 205 determines the GT of object likelihood based on the score obtained by the score calculation unit 204 for the inferred object shape, as shown in equation (1-3).

[0030]

number

[0031] Here, Threshold is a threshold value for the object-likeness score, and if it is below the threshold, the object-likeness GT is set to 0, that is, it is not an object.

[0032] In S306, the loss calculation unit 306 calculates the loss related to the object-likeness and the loss related to the bounding box regression. First, the loss is calculated using formula (1-4) based on the object-likeness obtained by the inference unit 203 and the object-likeness of the GT obtained by the object-likeness GT determination unit 205.

[0033]

number

[0034] Here, N is the total number of anchors. By calculating the sum of squared errors in this way, the output Objectness of the neural network is calculated. pred The value of GT Map Objectness GT If the difference is large, the loss will be large. predand Objectness GT The closer they are, the smaller the loss.

[0035] The loss function is not limited to the sum of squared errors as described above. Loss functions such as Cross-Entropy may also be used. The loss above was calculated based on all anchors, but it may also be calculated by limiting it to only some anchors. For example, Objectness GT We can extract anchors whose eigenvalues ​​are above a given value and below a given value, and calculate the loss only for those anchors. The loss for bounding box regression is calculated as IoU Loss as shown in equation (1-5).

[0036]

number

[0037] BB pred is the inferred bounding box, and BB GT is the bounding box of GT. The smaller the overlap between the two bounding boxes, the larger the loss, and the larger the overlap, the smaller the loss. Note that the loss function is not limited to the IoU Loss described above. Loss functions such as Smooth-L1 can also be used. Finally, the sum of the object likelihood loss and the bounding box regression loss is calculated to obtain the integrated loss.

[0038]

number

[0039] Here, λ is a coefficient for balancing the loss of object-likeness and the loss of bounding box regression, and is determined empirically.

[0040] In S307, the parameter update unit 207 updates the parameters based on the loss calculated in S306. The parameter update is performed based on back propagation using Momentum SGD or the like. Note that while the output of the loss function for one image has been described above, in actual learning, the loss value of equation (1-6) is calculated for multiple different images. The neural network parameters are updated so that the loss values ​​for multiple images are all smaller than a predetermined threshold.

[0041] In S308, the storage unit 208 stores the parameters of the neural network updated by the parameter update unit 207.

[0042] In S309, the parameter update unit 207 determines whether to end the learning. The end of the learning may be determined when the loss value obtained by equation (1-6) becomes smaller than a predetermined threshold, or when learning has been performed a predetermined number of times.

[0043] <Effects of the First Embodiment> By determining the object-likeness of a GT based on the score calculated from the degree of overlap between the inferred bounding box and the bounding box of the GT, as shown in equation (1-1), the system learns that the greater the overlap between the inferred bounding box and the GT, the higher the object-likeness of the anchor, and the lower the overlap, the lower the object-likeness of the anchor. Therefore, when an anchor with a high object-likeness is inferred as an object during inference, a bounding box with a higher degree of overlap with the GT can be obtained, improving the accuracy of bounding box inference.

[0044] In other words, the information processing device of this embodiment determines the candidate area with the highest object-likelihood score from among multiple candidate areas inferred by the learning model, based on the object area (ground truth area) and the object-likelihood score previously assigned to the target image to be learned. By training the learning model with the determined ground truth data, the object detector can be trained more efficiently.

[0045] <Embodiment 2> In this embodiment, a method for learning the likelihood of a specific target being tracked in an object tracking task for detecting a specific target being tracked from an image will be described.

[0046] In this embodiment, the tracking task is learned according to the method of Bertinetto, et al., "Fully-Convolutional Siamese Networks for Object Tracking."

[0047] An example of the hardware configuration and functional configuration of the information processing device 2 during learning is the same as that of the first embodiment and is shown in Figures 1 and 2. A processing flowchart is shown in Figure 6.

[0048] In S601, the image acquisition unit 201 acquires a first image (an image that serves as the basis for the template image) in which the tracking target exists. Furthermore, the GT acquisition unit 202 acquires a GT that indicates the bounding box of the tracking target that exists in the template image. An example of the template image is shown in FIG. 7(a). 701 is the first image acquired by the image acquisition unit, and 703 is the tracking target. 704 is a bounding box that indicates the position and size of the tracking target. Note that in the tracking process, the first image is the image in which the tracking target was detected in the previous tracking process.

[0049] In S602, the image acquisition unit 201 extracts and resizes an image of the periphery of the tracking target in the first image based on the position and size of the tracking target obtained by the GT acquisition unit 202, thereby acquiring a template image. Note that in the tracking process, a partial image including the tracking target can be extracted based on the results of the previous tracking process. This can be done by cropping an image at a constant multiple of the size of the tracking target, with the position of the tracking target at the center. Figure 7(a) 702 is an example of cropping the periphery of the tracking target.

[0050] In S604, the image acquisition unit 201 acquires an image (second image) including a target area for searching for the tracking target. For example, an image taken at a different time in the same sequence as the image selected in S601 is acquired as the second image for searching for the tracking target. In FIG. 7(b), 705 is the second image for searching for the tracking target, and 707 is the tracking target. 708 is the bounding box of the tracking target. Note that in the tracking process, the second image is an image captured at a different time from the first image (for example, an image captured after a predetermined time).

[0051] In S605, the image acquisition unit 201 extracts and resizes an image around the tracking target of the template image based on the bounding box of the tracking target obtained by the GT acquisition unit 202. That is, based on the position and size of the template image in the first image, a corresponding area of ​​the second image is set as a search range. For example, it is sufficient to extract an image centered on the position of the tracking target and a constant multiple of the size of the tracking target. Figure 7(b) 706 is an example of an extracted image to be searched.

[0052] In S603, the inference unit 203 inputs the template image obtained in S601 into the trained neural network to obtain the feature (first feature) of the tracking target.

[0053] In S606, the inference unit 203 inputs the search range image obtained in S604 into the neural network, and obtains features (second features) included in the search range image.

[0054] In S607, the inference unit 203 infers the likelihood of the object being tracked in the second image and its Bounding Box (an object region including a score indicating the likelihood of the object being tracked and an area indicated by the Bounding Box) based on the cross-correlation between the features of the template obtained in S603 (first features) and the features of the search range obtained in S606 (second features).

[0055] In S608, the score calculation unit 204 calculates the score for the inferred object region based on the formula (1-1), as in the first embodiment.

[0056] In S609, the object likelihood GT determination unit 205 calculates the GT of the likelihood of being a tracking target as shown in Equation (2-1). Objectness Compare your score and get a higher score than others. Objectness The anchor having this property is more likely to be a tracking target.

[0057]

number

[0058] In S610, the loss calculation unit 306 calculates the loss for the output result of the neural network in the same manner as in the first embodiment.

[0059] <Effects of the Second Embodiment> As shown in equation (2-1), by calculating the GT of object likelihood using not only the object likelihood score of the anchor of interest but also the object likelihood scores of its surroundings, it becomes easier to obtain an anchor with a higher bounding box accuracy compared to the surrounding anchors during inference.

[0060] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a data communication network or various storage media. Then, a computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. The programs may also be provided by recording them on a computer-readable storage medium.

[0061] Note that, among the above-mentioned processing units, the face detection unit 102 and the like may instead use a trained model that has been trained by machine learning. In this case, for example, multiple combinations of input data and output data for the processing unit are prepared as training data, knowledge is acquired from them by machine learning, and a trained model is generated that outputs output data for the input data as a result based on the acquired knowledge. The trained model can be configured, for example, as a neural network model. Then, the trained model operates in cooperation with a CPU or GPU as a program for performing processing equivalent to that of the processing unit, thereby performing the processing of the processing unit. Note that the trained model may be updated after a certain amount of processing, as necessary. [Explanation of symbols]

[0062] 1. Information processing equipment 201 Image acquisition unit 202 GT Acquisition Department 203 Reasoning section 204 Score Calculation Unit 205 Decision Section 206 Loss calculation section 207 Parameter Update Unit 208 Storage section

Claims

1. An information processing device that learns the object-likeness of an object in an image, image acquisition means for acquiring an image; a correct data acquisition means for acquiring correct data including at least an object region in which an object exists in the acquired image; an inference means for inferring a candidate region of an object in the image and a first score indicating the object-likelihood of the candidate region based on a learning model; a determination means for determining a second score indicating an object-likeliness of an object region based on the degree of overlap between a candidate region corresponding to a position of interest in the image and a region included in the acquired ground truth data, among a plurality of candidate regions inferred by the inference means, and the degree of overlap between another candidate region corresponding to a periphery of the position of interest in the image and an object region in the acquired ground truth data; an update means for updating parameters of the learning model based on a loss value calculated based on the second score and a first score indicating the object-likeness of the inferred candidate region; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the second score is determined based on a similarity between the candidate region and an object region in the acquired ground truth data.

3. The information processing device described in claim 1, characterized in that the determination means determines the second score for the candidate area corresponding to the focus position in the image to be greater than the second score for another candidate area corresponding to the focus position in the image when the degree of overlap between the candidate area corresponding to the focus position in the image and the object area in the acquired correct answer data is greater than the degree of overlap between another candidate area corresponding to the periphery of the focus position in the image and the object area in the acquired correct answer data.

4. an acquisition means for acquiring correct answer data indicating a correct answer region including the position of an object present in a target image; an estimation means for estimating a candidate region in the target image in which an object may exist and a first score indicating the possibility of the object existing in the candidate region by inputting the target image into a learning model that estimates an object region including the position of an object existing in the target image; a score calculation means for calculating a second score indicating a possibility that an object exists in a candidate area based on the degree of overlap between a candidate area corresponding to a position of interest in the target image and the acquired ground truth data, among the plurality of candidate areas estimated by the estimation means, and the degree of overlap between another candidate area corresponding to a periphery of the position of interest in the target image and the acquired ground truth data; a loss calculation means for calculating a loss for the learning model based on the second score and the estimated candidate region; and an update means for updating parameters of the learning model based on the calculated loss, The information processing device is characterized in that the update means learns the learning model by updating the candidate area in the target image where the second score is equal to or greater than a predetermined value as the correct area.

5. An information processing method for learning object-likelihood of an object in an image, comprising: an image acquisition step of acquiring an image; a supervised data acquisition step of acquiring supervised data for an object present in the acquired image, the supervised data including at least an object region in which the object exists; an inference step of inferring a candidate region of an object in the image and a score indicating the object-likeliness of the candidate region based on a learning model; a determination step of determining a score indicating an object-likeliness for an object region based on the degree of overlap between a candidate region corresponding to a position of interest in the image and a region included in the acquired ground truth data, among the plurality of candidate regions inferred in the inference step, and the degree of overlap between another candidate region corresponding to the periphery of the position of interest in the image and an object region in the acquired ground truth data; an updating step of updating parameters of the learning model based on a loss value calculated based on the determined score and the score indicating the object-likeness of the inferred candidate region; An information processing method comprising:

6. A program for causing a computer to function as each of the means included in the information processing device according to any one of claims 1 to 5.

Images