Information processing apparatus, information processing method, and program

Abstract

To enable learning to improve detection accuracy in the center of a detection target area and in divided areas.SOLUTION: An information processing apparatus determines a kernel for acquiring the importance of a correct answer area corresponding to a learning image for each pixel, and determines the importance of the correct answer area for each pixel using the determined kernel. The information processing apparatus learns target areas to be detected from the image based on the importance of the determined correct answer area.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing technique relating to machine learning. [Background technology]

[0002] In recent years, research in the field of image recognition has been actively conducted, and various methods have been proposed to detect object regions in images using neural networks. Region detection methods using deep learning, such as semantic segmentation, are algorithms that assign labels and categories to each pixel in an image. However, the general region detection methods currently in use can sometimes fail to detect the center of the target region, or the detected region can be divided, resulting in unstable detection results.

[0003] To solve such problems, Non-Patent Document 1 discloses a method of performing learning that takes spatial correlation information into consideration by dynamically setting the loss so that adjacent regions are more easily detected. Also, Non-Patent Document 2 discloses a method of prioritizing learning based on positional relationships by determining the center point of the region to be learned and learning so that detection results tend to concentrate at the center point. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Wei Yuan, Wenbo Xu, "NeighborLoss: A Loss Function Considering Spatial Correlation for Semantic Segmentation of Remote Sensing Image", IEEE Access.2021.3082076 [Non-Patent Document 2] Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming-Hsuan Yang, Jan Kautz, "SCOPS: Self-Supervised Co-Part Segmentation", arXiv1905.01298, 2019 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, when detecting a person's head region as an object region, it is desirable to be able to stably detect the center of the head region rather than the ambiguous boundary region between the person's head and the background. Also, it is desirable to be able to accurately detect the person's head region even when an object is blocking the person's head region.

[0006] However, in the method disclosed in Non-Patent Document 1, since the loss is determined based on whether the area around the target pixel is detected or not, it is difficult to perform learning that enables stable detection of the center part of the detection target area such as the head area. Also, in the method disclosed in Non-Patent Document 2, since the loss is calculated based on the distance from the center part of the learning target area, for example, when the head area is divided by an obstruction, it is difficult to accurately detect the divided area.

[0007] Therefore, an object of the present invention is to enable learning that improves the detection accuracy of the center portion and divided regions of a detection target region. [Means for solving the problem]

[0008] The information processing device of the present invention is characterized by having a kernel determination means for determining a kernel for obtaining the importance of a correct answer region corresponding to a training image for each pixel, an importance determination means for determining the importance of the correct answer region for each pixel using the determined kernel, and a learning means for learning a target region to be detected from an image based on the determined importance of the correct answer region. Effect of the Invention

[0009] According to the present invention, learning can be performed to improve the detection accuracy of the center portion of a detection target area and divided areas. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of a learning device. [Diagram 3] FIG. 2 is a diagram illustrating a functional configuration of an importance determining unit. [Figure 4] 4 is a flowchart of information processing according to the first embodiment. [Diagram 5] FIG. 4 is an explanatory diagram of an importance determination process in the first embodiment. [Figure 6] 13 is a flowchart of an importance determination process according to the second embodiment. [Figure 7] FIG. 11 is an explanatory diagram of an importance determination process in the second embodiment. [Figure 8] FIG. 13 is a diagram illustrating a functional configuration according to a third embodiment. [Figure 9] 13 is a flowchart of a reliability acquisition process according to the third embodiment. [Figure 10] FIG. 13 is an explanatory diagram of a reliability acquisition process according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configuration of the embodiments may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (conditions of use, environment of use, etc.). In addition, the present invention may be configured by appropriately combining parts of the embodiments described below. In the following embodiments, redundant descriptions of the same or similar configurations and processing steps will be omitted.

[0012] <First embodiment> FIG. 1 is a diagram showing an example of the hardware configuration of an information processing apparatus according to the present embodiment. The CPU 100 is a central processing unit, and performs calculations and logical judgments for various processes. The ROM (Read-Only-Memory) 110 stores a control program. The RAM (Random Access Memory) 120 is used as a temporary storage area such as the main memory and work area of ​​the CPU 100. The HDD 130 is a storage device for storing the program and various data according to this embodiment. An external storage device may be used to perform the same role as the HDD 130. The external storage device may be realized, for example, by a medium (recording medium) and an external storage drive for realizing access to the medium. Examples of such media include a flexible disk (FD), a CD-ROM, a DVD, a USB memory, an MO, and a flash memory. The external storage device may also be a server device connected via a network.

[0013] The input unit 140 is composed of a keyboard, a touch panel, etc., and accepts input from a user. The display 150 is composed of a liquid crystal display, etc., and can display various data, information processing results, etc. to the user. In addition, this device can communicate with other devices via the communication unit 160. This device may receive user instructions from other devices via the communication unit 160, or may output processing results to other devices. The information processing device of this embodiment can be realized by a general-purpose personal computer, digital camera, smartphone, etc., having the above-mentioned configuration.

[0014] FIG. 2 is a diagram showing an example of the functional configuration of a learning device 200 that uses learning data 210 to learn a target region to be detected from an image, as an application example of the information processing device of this embodiment. For example, if a task is assumed in which the head region of an object is detected as a detection target region from an image and the number of people is counted, the importance of the region that is an ambiguous boundary between the head and the background is thought to decrease, and the importance of stably detecting the center of the head is thought to increase. Also, for example, if it is desired to detect the head region of a person and determine the presence or absence of an obstruction, the importance of the vicinity of the obstruction is thought to increase. In other words, in learning to recognize (detect) a target region from an image, the region that is important during learning often changes depending on the type of task to be learned and the learning target.

[0015] Therefore, the information processing device of this embodiment determines the area that is important during learning based on the information of the correct answer area of ​​the learning target. In the learning device 200 according to this embodiment shown in Fig. 2, in a task such as object area detection, the importance of each pixel is determined according to the correct answer area of ​​the learning target, and learning is performed based on the importance. Each functional unit of the learning device 200 shown in Fig. 2 will be described in detail below.

[0016] The data input unit 220 of the learning device 200 receives learning data 210 including learning images and information on correct answer regions, etc. The data input unit 220 outputs the received learning data 210 to the importance determination unit 230. The importance determination unit 230 calculates the importance of the correct answer region included in the training data 210 for each pixel, and generates an importance map from the importance of each pixel of the correct answer region. Then, the importance determination unit 230 outputs the importance map of the correct answer region and the training data 210 to the detection processing unit 240. FIG. 3 is a diagram showing an example of the functional configuration of the importance determining unit 230. As shown in FIG.

[0017] The information acquisition unit 310 acquires the learning data 210 including information on the learning image and the correct answer region from the data input unit 220 , and sends it to the kernel determination unit 320 . The kernel determination unit 320 determines a kernel (hereinafter, referred to as importance calculation kernel) used to determine the importance of the correct answer region. In this embodiment, the kernel determination unit 320 determines the importance calculation kernel according to the type of task to be learned, such as a task of object region detection, but may determine a kernel input in advance as the importance calculation kernel. Details of the importance calculation kernel determination process in the kernel determination unit 320 will be described later. Then, the kernel determination unit 320 outputs information of the determined importance information kernel to the importance calculation unit 330 together with the learning data 210.

[0018] The importance calculation unit 330 calculates the importance of the correct region for each pixel using the importance calculation kernel determined by the kernel determination unit 320. Furthermore, the importance calculation unit 330 generates an importance map based on the importance of each pixel. In the present embodiment, the importance calculation unit 330 generates, as the importance map, an undetected importance map that determines the loss when undetected and an erroneous detection importance map that determines the loss when erroneous detection occurs, but is not necessarily limited thereto. Details of the importance calculation process in the importance calculation unit 330 will be described later. Then, the importance calculation unit 330 outputs the generated importance map to the detection processing unit 240 together with the learning data 210.

[0019] The detector 241 of the detection processing unit 240 detects a target region (an image region of a head, which is a detection target object, in the example of this embodiment) from a training image included in the training data 210. Then, the detector 241 outputs the detection result of the target region to the loss calculation unit 250 together with the above-mentioned importance map and the training data 210. As a training model (image recognition model) used in the detector 241 to detect the target region, various models such as a neural network such as a CNN (Covolutional Neural Network) are considered. In addition, the detector 241 may detect the target region by a recognition model such as a ViT (Vision Transformer) or an SVM (Support Vector Machine) combined with a feature extractor. The detector 241 is not limited to any of these, but is assumed to be a CNN in this embodiment.

[0020] The loss calculation unit 250 compares the detection result of the object region by the detector 241 with the correct region included in the learning data 210 to calculate the loss. In this embodiment, the loss calculation unit 250 calculates the loss using the undetection importance map and the false detection importance map generated by the importance determination unit 230. In addition, the loss calculation unit 250 at this time weights the undetection importance map and the false detection importance map. For example, when it is desired to suppress false detection with priority over undetection, the loss calculation unit 250 increases the coefficient multiplied by the false detection importance map compared to the undetection importance map. The loss calculation process in the loss calculation unit 250 will be described in detail later.

[0021] The weight update unit 260 updates the weights of the network used in machine learning based on the loss calculated by the loss calculation unit 250. After that, the weight update unit 260 outputs the updated network weight information to the output unit 270 and stores it in the parameter storage unit 242. The weight information stored in the parameter storage unit 242 will be used by the detector 241 during the next learning.

[0022] The output destination of the output unit 270 is not limited to a specific format, and may be a memory of a general-purpose computer or a control circuit inside a camera. As an output destination, for example, an information processing device that performs image recognition processing such as object detection using the weights output from the output unit 270 of the learning device 200 is assumed. As an application example of an information processing device that performs image recognition processing, an object detection circuit for autofocus control in a digital camera is conceivable. As an application example of the image recognition processing according to this embodiment, for example, a program that performs object detection for use in image processing in a smartphone is also conceivable. Of course, these application examples are merely examples, and are not limited to a specific form.

[0023] 2 and 3, the operation of the learning device 200 of this embodiment having the configuration described above will be described in detail below with specific examples. In this embodiment, the purpose is to learn so that a person's head region can be stably detected, and an example will be described in which importance is calculated so that the central part where head regions are concentrated has higher importance.

[0024] Fig. 4 is a flowchart showing the process flow from acquisition of learning data 210 to importance determination and loss calculation in the first embodiment. Fig. 4(a) shows the overall flowchart, and Fig. 4(b) shows the flowchart of the importance determination process. In the following explanation of each flowchart, the symbol S represents a processing step (process). Figs. 5(a) to 5(h) are diagrams used to explain the learning image, correct answer region, importance calculation kernel, importance map, and loss calculation target in the first embodiment.

[0025] First, as the process of S401, the data input unit 220 acquires data to be used for learning. The data acquired by the data input unit 220 at this time includes the learning image and correct answer area information of the learning data 210 described above, as well as learning task information, image state information, and the like. The learning task information is information on what the learning model is intended to detect and for what purpose. The image state information is information that indicates the state of the image, such as brightness information, blur / shake information, and color information of the image. The data acquired by the data input unit 220 is sent to the importance determination unit 230.

[0026] Next, in the process of S402, the information acquisition unit 310 of the importance determination unit 230 acquires an image of one frame from the group of training images included in the training data 210 as a training image to be processed. In this embodiment, it is assumed that the image shown in Fig. 5(a) is acquired as a training image 510 of one frame. Furthermore, the information acquisition unit 310 at this time acquires information on a correct answer region corresponding to the training image 510 from information on the correct answer region included in the training data 210. It is assumed that the correct answer region corresponding to the training image 510 is a correct answer region 520 as shown by the shaded pixels in Fig. 5(b).

[0027] Next, in the process of S403, the importance determination unit 230 performs a process of determining the importance of the correct answer region for each pixel. That is, in the importance determination unit 230, the kernel determination unit 320 determines the importance calculation kernel used to determine the importance of the correct answer region, and the importance calculation unit 330 calculates the importance of the correct answer region for each pixel using the importance calculation kernel.

[0028] The importance determination process in S403 will be described with reference to the flowchart shown in FIG. First, in the process of S 411 , the information acquisition unit 310 acquires information on the correct solution region 520 from the learning data 210 read from the data input unit 220 , and sends the information to the kernel determination unit 320 . Next, in the process of S412 , the kernel determination unit 320 determines an importance calculation kernel for determining the importance of each pixel in the correct answer region 520 .

[0029] As described above, in this embodiment, learning is assumed to enable stable detection of a person's head region, and it is assumed to calculate a higher importance for the central portion where the head regions are concentrated. For this reason, in this embodiment, the importance calculation kernel is a kernel that can calculate a higher importance for an area closer to the center of the detection target area. Also, in this embodiment, the importance of the correct answer area is determined for each pixel. The method of calculating the importance in this embodiment is a method that uses correct answer area information for multiple pixels surrounding a pixel of interest whose importance is to be determined (calculated) in the correct answer area. In this embodiment, the importance calculation kernel defines which pixels around the pixel of interest in the correct answer area are to be included in the calculation range of the importance.

[0030] In this embodiment, the kernel determination unit 320 determines the importance calculation kernel 530 shown in Fig. 5(c) as the importance calculation kernel for calculating the importance of a region closer to the center of the correct answer region. In the importance calculation kernel 530 in Fig. 5(c), the central black pixel among the 9 pixels of 3 x 3 corresponds to the pixel of interest for which the importance is to be determined, and the region consisting of a plurality of pixels surrounding the pixel of interest is used to calculate the importance.

[0031] Next, in the process of S413, the importance calculation unit 330 sequentially sets each pixel included in the correct answer area 520 as a pixel of interest, and calculates the importance of each pixel of interest using the importance calculation kernel 530 illustrated in FIG. 5(c). Here, the importance of the pixel of interest can be calculated by the following formula (1).

[0032]

number

[0033] In formula (1), I represents the calculated importance, and N represents the total number of pixels in the importance calculation kernel. P in formula (1) is a value indicating whether each pixel included in the importance calculation kernel is a pixel in the correct answer region, and is given 1 if it is a pixel in the correct answer region, and 0 if it is not a pixel in the correct answer region. W in formula (1) indicates the weight value of each pixel used when calculating the importance, and in the importance calculation kernel 530, the weights of all pixels surrounding the pixel of interest are given as 1.0. That is, according to formula (1), the importance calculation unit 330 calculates the importance of the pixel of interest based on the number of pixels in the correct answer region in the importance calculation kernel. The weight W can be set to an arbitrary value using learning task information and image state information.

[0034] Then, the importance calculation unit 330 generates an importance map based on the importance calculated for each pixel in the correct answer region 520. In the case of this embodiment, the importance calculation unit 330 generates, as the importance map, an undetected importance map that determines a loss in the event of undetection, and an erroneous detection importance map that determines a loss in the event of erroneous detection.

[0035] 5(d) shows an importance map based on the result of calculating the importance of each pixel included in the correct answer region 520 using the importance calculation kernel 530, and the importance calculation unit 330 sets this importance map as an undetected importance map 540. The importance calculation unit 330 generates an undetected importance map 540 in which the importance of the central part of the head region in the training image 510 is increased, based on the importance of each pixel calculated using formula (1) and the importance calculation kernel 530. When learning is performed using this undetected importance map 540, it is possible to increase the loss when an undetection of the head center occurs, and to train a detector that is likely to detect the center of the head region.

[0036] FIG. 5(e) is an importance map based on the result of calculating the importance of each pixel not included in the correct answer region 520 using the importance calculation kernel 530, and the importance calculation unit 330 sets this importance map as the false detection importance map 550. The importance calculation unit 330 calculates the importance of each pixel using the formula (1) and the importance calculation kernel 530 to generate the false detection importance map 550 in which the importance of the boundary region between the head region and the background region is increased. Note that in this example, the false detection importance map 550 is generated using the formula (1) and the importance calculation kernel 530 used in generating the undetected importance map 540, but the present invention is not limited to this example, and an importance calculation kernel different from the importance calculation kernel 530 may be used. For example, an importance calculation kernel different from the importance calculation kernel 530 may be set based on learning task information or image state information. In addition, in the case of this embodiment, if the importance calculation kernel 530 protrudes from the correct answer region 520, the calculation is not performed as it is not included in the calculation target, but this is not necessarily required. By performing learning using this false detection importance map 550, it is possible to increase the loss when a false detection occurs in an area where it is unclear whether it is a head area or a background area, thereby enabling more stable head area detection.

[0037] Returning to the flowchart of FIG. After the importance determination process in S403 described above, the process proceeds to S404, where the detector 241 of the detection processing unit 240 performs object detection on the learning image 510 and outputs the detection result. Here, it is assumed that the detector 241 obtains a detection result 560 as shown in Fig. 5(f) from the learning image 510 in Fig. 5(a) described above.

[0038] Next, in the process of S405, the loss calculation section 250 compares the correct area 520 with the detection result 560 to calculate the loss. Here, in loss calculation section 250, undetected importance map 540 is used to calculate the undetected loss. Fig. 5(g) is a diagram showing an example of an undetected region 570 obtained by comparing correct answer region 520 with detection result 560, and this undetected region 570 is the target of loss calculation. According to undetected importance map 540, the loss in the center part of the head region is high, so it is possible to learn so that the center part of the head region is more easily detected than the edges.

[0039] Furthermore, in loss calculation section 250, a false positive importance map 550 is used to calculate the false positive loss. Fig. 5(h) is a diagram showing an example of a false positive region 580 obtained by comparing correct answer region 520 with detection result 560, and this false positive region 580 is the target of loss calculation. According to false positive importance map 550, the loss increases the closer the region is to the head region, so it is possible to learn to suppress false positives in regions that are ambiguous as to whether they are head regions or background regions, enabling more stable head region detection.

[0040] Finally, in the process of S406, the learning device 200 checks whether the series of processes has been completed for all frames, and if completed, ends the process, and if not completed, returns to S402 and repeats the processes of S402 to S406 described above.

[0041] In the learning device 200 of this embodiment, in a task such as object region detection, the importance of each pixel is determined according to the correct region of the learning target, and learning is performed based on the importance. In particular, in this embodiment, by increasing the "importance of non-detection of the central region where head regions are concentrated" and the "importance of erroneous detection of regions close to the head region", it is possible to train a learning model for image recognition that can stably detect the center part of a person's head region.

[0042] <Second embodiment> Next, in a second embodiment, an example will be described in which, when a part of a person's head area is covered by an obstruction, etc., the importance of the boundary area between the head area and the obstruction is set high, making it possible to suppress non-detection or erroneous detection of the boundary area. In the second embodiment, the process flow from obtaining input information to determining loss is similar to the process shown in the flowchart of Fig. 4(a) in the first embodiment, and therefore a basic description thereof will be omitted.

[0043] In the second embodiment, the importance determination process in S403 differs from that in the first embodiment, and will be described in detail below with reference to Fig. 6 and Fig. 7. Fig. 6 is a flowchart showing the flow of the importance determination process in S403 according to the second embodiment. Fig. 7(a) to Fig. 7(i) are diagrams used to explain the learning image and correct answer region, the importance calculation kernel, the importance map, and the target of loss calculation in the second embodiment.

[0044] In S601 of Fig. 6, the data input unit 220 reads the learning image and correct answer region of the learning data 210, learning task information, image state information, etc. as data to be used for learning. Fig. 7(a) is a diagram showing an example of a learning image 710, which is an image in which part of the head is covered by an obstruction or the like. Fig. 7(b) is a diagram showing an example of a correct answer region 720 for the learning image 710.

[0045] Next, in the process of S602, the kernel determination unit 320 determines a kernel to be used from among a plurality of predefined importance calculation kernels based on the information input from the information acquisition unit 310. At this time, the kernel determination unit 320 may determine one importance calculation kernel or may determine a plurality of importance calculation kernels.

[0046] The kernel determination unit 320 of the second embodiment determines, as kernels for calculating the importance of undetection, an importance calculation kernel in which the central part of the visible head region is given a higher importance, and an importance calculation kernel in which the boundary region between the head region and an obstructing object is given a higher importance. Fig. 7(c) is a diagram showing an importance calculation kernel 730 in which the central part of the visible head region is given a higher importance, and Fig. 7(d) is a diagram showing an importance calculation kernel 740 in which the boundary region between the head region and an obstructing object is given a higher importance. In addition, the kernel determination unit 320 of the second embodiment determines an importance calculation kernel that has a high importance when a false detection occurs due to an obstruction as an importance calculation kernel for calculating the importance of false detection. Fig. 7(e) is a diagram showing an importance calculation kernel 750 that has a high importance when a false detection occurs due to an obstruction.

[0047] For example, in the importance calculation kernel 740 shown in FIG. 7(d), a weight of −1.0 is set to be added to each pixel in the vertical column to the right of the pixel of interest shown in black when there is no head region. The kernel determination unit 320 calculates the importance using equation (1) as described above. In the case of the second embodiment, P indicates whether each pixel in the importance calculation kernel is a pixel in the correct answer region or not, and is given 1 if it is a pixel in the correct answer region, and −1 if it is not a pixel in the correct answer region. Also, in the importance calculation kernel 750 shown in FIG. 7(e), a weight of 1.0 is set to be added to each pixel in the vertical column to the left of the pixel of interest shown in black when there is a head region.

[0048] In addition to the above-mentioned examples, the kernel determination unit 320 may determine the importance calculation kernel by using at least one of the following first to fourth determination methods. A first determination method in which a kernel input by a user via the input unit 140 is obtained and used as the importance calculation kernel. A second determination method in which the importance of the correct region is calculated using each of a number of predefined importance calculation kernels, and the kernel that has a region with an importance greater than a predefined importance threshold, or a greater number than a predefined value, is selected as the importance calculation kernel. The third decision method calculates the importance of the correct region with each of multiple predefined importance calculation kernels, and selects the kernel with the total importance value for each pixel equal to or greater than a predefined value as the importance calculation kernel. A fourth determination method for determining the importance calculation kernel based on predetermined conditions in response to the image condition information acquired by the data input unit 220.

[0049] Next, in the process of S603, the importance calculation unit 330 selects one from the importance calculation kernel 730 in FIG. 7(c) to the importance calculation kernel 750 in FIG. Furthermore, in the process of S604, the importance calculation section 330 uses one importance calculation kernel selected in S603 to calculate the importance of the correct region for each pixel, and generates an importance map.

[0050] Next, in the process of S605, the importance calculation unit 330 judges whether the importance calculation process is completed for all pixels (referred to as importance calculation areas) in the correct answer area that are the subject of importance calculation. If the calculation process for all the importance calculation areas is not completed, the importance calculation unit 330 returns the process to S603, whereas if the calculation process is completed, the importance calculation unit 330 advances the process to S607. That is, the importance calculation unit 330 performs the loop process of the importance calculation until the process for all the importance calculation areas is completed for each importance calculation kernel determined in S602, and when the process for all the importance calculation areas for each importance calculation kernel is completed, the process proceeds to S606. Note that if the importance is calculated in advance in S602, the importance calculation process may be skipped.

[0051] FIG. 7(f) is a diagram showing an undetected importance map 760 calculated in S604. The undetected importance map 760 is an importance map calculated using the importance calculation kernel 730, and it can be seen that the importance of the central part of the head region visible in the learning image 710 is high. FIG. 7(g) is a diagram showing an undetected importance map 770 calculated in S604. The undetected importance map 770 is an importance map calculated using the importance calculation kernel 740, and it can be seen that the importance of the boundary region between the head region and the obstructing object in the learning image 710 is high. FIG. 7(h) is a diagram showing an erroneous detection importance map 780 calculated in S604. The erroneous detection importance map 780 is an importance map calculated using the importance calculation kernel 750, and it can be seen that the importance of the boundary region between the head region and the obstructing object in the learning image 710 is high.

[0052] Next, in the process of S606, the importance calculation unit 330 determines a combination parameter when combining the importance maps generated as described above. Here, the combination parameter refers to a weight when adding up the importance maps, a round-up threshold when rounding up the importance map values, a round-down threshold when rounding down the importance map values, etc. In this embodiment, each weight of the false detection importance map 780 is set to 1.0, the round-down threshold applied to the undetected importance map 770 is set to 5, and importance of 5 or less is rounded down to 0.

[0053] Next, in the process of S607, the importance calculation unit 330 combines each importance map using the combination parameter determined in S606. In this embodiment, a simple sum is used as an example of the importance map combination process, but the method is not necessarily limited to this example. FIG. 7(i) is a diagram showing an undetected importance map 790 generated by the combination process in S607. In the undetected importance map 790 after the combination process, the importance of both the center part of the head region and the boundary region with the obstruction can be calculated to be high. By using this undetected importance map 790 for learning, it is possible to stably detect the head center and accurately detect the boundary region with the obstruction. Thereafter, in the second embodiment, as in the first embodiment, the loss is calculated from the importance and learning is performed.

[0054] According to the second embodiment, as described above, by using multiple importance calculation kernels, it is possible to set the importance so that the boundary area between the head center and the occluding object has a high importance, and it is possible to train an area detector that is resistant to occluding objects.

[0055] <Third embodiment> Next, in the third embodiment, an example will be described in which the reliability of the detection result by the image recognition process is improved by combining the detection result by the image recognition process using the learning model learned by the learning device of the above-mentioned embodiment with the detection result by a detector for another task. The detector for another task refers to, for example, a detector that detects a face, pupils, nose, mask, etc., or a region detector that detects a foreground region, background region, hair region, etc., and may be in any form as long as it detects a region other than the head region. Hereinafter, the detection result by the detector for another task will be referred to as the detection result for another task.

[0056] FIG. 8 is a diagram showing an example of a functional configuration of an information processing device according to a third embodiment, which improves the detection reliability of image recognition processing by combining a detection result obtained by performing image recognition processing using a learning model learned by the learning device described above with a detection result of another task. The information processing device according to this embodiment is an image recognition device including a target input unit 810, a reliability acquisition unit 820, a different task result input unit 830, and an output unit 840 in addition to a detection processing unit 240, and the reliability acquisition unit 820 includes a reliability calculation unit 821 and an information input unit 822. The detection processing unit 240 is the same as the detection processing unit 240 of the first embodiment, and a description of its basic configuration and processing will be omitted. Also, FIG. 9 is a flowchart showing the flow of processing performed by the reliability calculation unit 821 of FIG. 8.

[0057] An image to be subjected to image recognition processing by a digital camera, a smartphone, or the like (not shown) is input to the object input unit 810. The image input to the object input unit 810 is sent to the detection processing unit 240. The detection processing unit 240 performs image recognition processing using a learning model in which area learning has been performed with the importance of the correct area taken into account as described in the above embodiment. That is, the detector 241 of the detection processing unit 240 detects the target area (head area in this embodiment) by executing a detection process on the image input from the target input unit 810 using the learning parameters stored in the parameter storage unit 242. The detection result of the detector 241 is sent to the reliability calculation unit 821 of the reliability acquisition unit 820.

[0058] FIG. 10(a) is a diagram showing an example of an input image 1010 to be recognized, where the input image 1010 is an image including a person's head region. FIG. 10(b) is a diagram showing an importance calculation kernel 1020 used during learning of the detector 241. The importance calculation kernel 1020 shown in FIG. 10(b) is a kernel used for learning to improve the detection accuracy of the center part of the head region as in the first embodiment. The importance calculation kernel 1020 used during this learning is stored in the parameter storage unit 242. FIG. 10(c) is a diagram showing an example of a detection result 1030 detected by the detector 241.

[0059] The information input unit 822 of the reliability acquisition unit 820 receives the importance calculation kernel 1020 from the parameter storage unit 242, and passes the importance calculation kernel 1020 to the reliability calculation unit 821. The information input unit 822 also receives a separate task detection result acquired by the separate task result input unit 830 from a separate task detector (not shown), and passes it to the reliability calculation unit 821.

[0060] The reliability calculation unit 821 executes a process of acquiring the detection reliability of the detection result 1030 of the detector 241, using the importance calculation kernel 1020 as the kernel of the CNN. The reliability acquisition unit 820 calculates a score value (hereinafter, referred to as a reliability score) indicating the level of detection reliability for each pixel of the detection result 1030 detected by the detector 241. By using the importance calculation kernel 1020 used during learning as the kernel of the CNN, the reliability calculation unit 821 can calculate the reliability of the detection result if the recognition model has progressed in learning. Furthermore, the reliability acquisition unit 820 receives the detection result of another task from the information input unit 822 and combines the detection result of the other task with the detection result of the detector 241, thereby correcting the detection reliability of the detection result of the detector 241.

[0061] Hereinafter, the flow of the reliability calculation process in the reliability calculation unit 821 and the process of combining the results of detection of another task will be described with reference to the flowchart in FIG. First, in the process of S901, the information input unit 822 reads the importance calculation kernel from the parameter storage unit 242, and acquires the other task detection result from the other task result input unit 830. Then, the information acquired by the information input unit 822 is sent to the reliability calculation unit 821. In addition, the reliability calculation unit 821 at this time acquires the detection result 1030 from the detector 241.

[0062] Next, in the process of S902, the reliability calculation unit 821 determines the importance calculation kernel that the information input unit 822 reads from the parameter storage unit 242 as a kernel to be used for calculating the reliability of the detection result of the detector 241. In the case of this embodiment, the reliability calculation unit 821 determines the importance calculation kernel 1020 shown in Fig. 10(b), but when there are multiple importance calculation kernels, the reliability calculation unit 821 may determine the multiple importance calculation kernels.

[0063] Next, in step S903, the reliability calculation unit 821 executes a process of calculating the detection reliability of the detection result of the detector 241. Fig. 10(d) shows a superimposed image 1040 in which the detection result 1030 of the detector 241 is superimposed on the input image 1010 of Fig. 10(a) in which a person's head is captured. As can be seen from the superimposed image 1040 in Fig. 10(d), it is assumed that a false detection area occurs in the detection result 1030 of the detector 241 above the person's head.

[0064] Fig. 10(e) is a diagram showing the reliability calculation result 1050 by the reliability calculation unit 821, where the numerical value in each pixel represents the reliability score. That is, the detection result 1030 of the detector 241 has a false detection area in the upper part of the person's head as shown in the superimposed image 1040 in Fig. 10(d), and therefore the reliability calculation result 1050 also includes the reliability score in the false detection area.

[0065] Here, for example, if the confidence threshold for the confidence score is set to, for example, 8, it can be determined that a portion of the confidence calculation result 1050 having a confidence score exceeding 8 has been correctly detected as a person's head region. Therefore, a portion having a confidence score equal to or less than the confidence threshold of 8 can be excluded from the detection result 1030 as an erroneous detection region.

[0066] Next, in the process of S904, if a different task detection result is input from the information input unit 822, the reliability calculation unit 821 determines a parameter for combining the different task detection result with the detection reliability calculated in S903.

[0067] FIG. 10(f) is a diagram showing an example of a face organ point detection result 1060 obtained as a detection result of another task, including, for example, an pupil detection result in which pupil detection has been performed and a nose detection result in which nose detection has been performed. The face organ point detection result 1060 shows an example in which the pupil and nose have been accurately detected. FIG. 10(g) is a diagram showing a corresponding positional relationship 1070 indicating which pixels correspond to the detection results of the other task. In this embodiment, an example is given in which a reliability score of 3 is set as a weight parameter for combining, and therefore, in the corresponding positional relationship 1070 of FIG. 10(g), the reliability score of each pixel at a position corresponding to the detection result of the other task is 3.

[0068] Next, in the process of S905, the reliability calculation unit 821 performs a process of combining each reliability score of the reliability calculation result 1050 of FIG. 10(e) calculated in S903 with the reliability score of each pixel of the corresponding positional relationship 1070 of FIG. 10(g) determined in S904. In the case of this embodiment, the combining process is, for example, a process of taking the sum of the reliability scores of both detection results, and FIG. 10(h) is a diagram showing the combined result 1080. Note that the combining process is not limited to the process of taking the sum of the reliability scores of both detection results.

[0069] As exemplified in this embodiment, when the reliability threshold is set to 8, it can be seen that even an area (pixel) that would be excluded from the detection result when only the detection result of detector 241 is used exceeds the reliability threshold of 8 in the combined result 1080. Thereafter, the output unit 840 outputs the detection result of the detector 241 and the detection reliability input from the reliability acquisition unit 820.

[0070] According to the third embodiment, in the image recognition process, it is possible to output a highly reliable detection result by using the importance calculation kernel used when learning the detector 241. Furthermore, in the third embodiment, it is possible to calculate the detection reliability more accurately by combining the detection results of other tasks.

[0071] <Other embodiments of the learning device> Each embodiment has been described above, but as long as the importance calculation kernel used during learning and the method of importance calculation are appropriately changed, the methods are not limited to those described in the above-mentioned embodiments, and for example, the following three importance calculation methods may be used. The importance calculation unit may calculate the importance of a pixel of interest, the importance of which is to be determined in the correct answer region, based on the number of adjacent pixels of the correct answer region within the importance calculation kernel. The importance calculation unit may calculate the importance of a pixel of interest, the importance of which is to be determined within the correct answer area, based on the number of consecutive pixels included in the correct answer area in a specific direction within the importance calculation kernel. The importance calculation unit may calculate the importance of a pixel of interest, the importance of which is to be determined within the correct answer region, based on a matching rate between the distribution of pixels included in the correct answer region within the importance calculation kernel and a preset template.

[0072] <Another embodiment of the image recognition device> In the third embodiment, an example is given in which image recognition processing is performed using a learning model trained by the learning device of the first or second embodiment described above, but the present invention is not limited to this example. For example, a user can select a desired model from a plurality of different learning models trained by the learning device of the above-mentioned embodiment through an input to the input unit 140, and image recognition processing, etc. may be performed using the model selected by the user.

[0073] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0074] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A kernel determination means for determining a kernel for obtaining the importance of a correct region corresponding to a learning image for each pixel; an importance determining means for determining the importance of the correct region for each pixel using the determined kernel; a learning means for learning a target region to be detected from an image based on the determined importance of the correct region; 13. An information processing device comprising: (Configuration 2) The information processing device according to configuration 1, wherein the importance determination means determines the importance of a pixel of interest, the importance of which is to be determined in the correct answer region, based on the number of pixels in the correct answer region within the kernel. (Configuration 3) The information processing device according to configuration 1, wherein the importance determination means determines the importance of a pixel of interest, the importance of which is to be determined in the correct answer region, based on the number of adjacent pixels of the correct answer region within the kernel. (Configuration 4) The information processing device according to configuration 1, wherein the importance determination means determines the importance of a pixel of interest, the importance of which is to be determined within the correct answer area, based on the number of consecutive pixels included in the correct answer area in a specific direction within the kernel. (Configuration 5) The information processing device according to configuration 1, wherein the importance determination means determines the importance of a pixel of interest, the importance of which is to be determined in the correct answer region, based on a matching rate between a distribution of pixels included in the correct answer region within the kernel and a preset template. (Configuration 6) 6. The information processing apparatus according to any one of configurations 1 to 5, wherein the importance determining means determines a weight for each position of the kernel. (Configuration 7) The information processing device according to any one of configurations 1 to 6, characterized in that the learning means obtains a loss in a process of detecting a target region from the image based on the importance of the correct region determined by the importance determination means. (Configuration 8) The importance determining means generates an importance map based on the importance of each pixel in the correct solution region; 8. The information processing apparatus according to configuration 7, wherein the learning means acquires the loss based on the importance map, and performs the learning based on the loss. (Configuration 9) the importance determining means generates, as the importance maps, an undetected importance map for determining a loss when an undetected occurs and an erroneous detection importance map for determining a loss when an erroneous detection occurs; 9. The information processing apparatus according to configuration 8, wherein the learning means acquires the loss when the detection is not performed based on the undetected importance map, and acquires the loss when the detection is not performed based on the false detection importance map. (Configuration 10) 10. The information processing device according to any one of configurations 1 to 9, wherein the importance determining means determines the importance of the correct region for each pixel by using a plurality of kernels determined by the kernel determining means. (Configuration 11) the importance determination means generates an importance map based on the importance of each pixel in the correct solution region for each of the plurality of kernels determined by the kernel determination means, and combines the generated plurality of importance maps; 11. The information processing apparatus according to configuration 10, wherein the learning means learns a target region to be detected from the image based on the combined importance map. (Configuration 12) the importance determining means combines the plurality of importance maps using a combination parameter; The information processing device according to configuration 11, characterized in that the combination parameters include at least one of a weight for adding together the importance maps, a rounding-up threshold for rounding up the importance map values, and a rounding-down threshold for rounding down the importance map values. (Configuration 13) 10. The information processing device according to any one of configurations 1 to 9, wherein the kernel determination means selects at least one kernel as the kernel to be determined from a plurality of predefined kernels. (Configuration 14) The information processing device described in configuration 13, characterized in that the kernel determination means calculates the importance of the correct region for each of the predefined multiple kernels, and selects a kernel that has a region where the calculated importance is greater than an importance threshold value or more than a predetermined value. (Configuration 15) The information processing device according to configuration 13, wherein the kernel determination means calculates the importance of the correct region for each of the predefined kernels, and selects a kernel for which the total value of the importance for each pixel is equal to or greater than a predetermined value. (Configuration 16) 16. The information processing apparatus according to any one of configurations 1 to 15, further comprising a detection means for detecting the target region from an image using a learning model learned by the learning means. (Configuration 17) 17. The information processing apparatus according to claim 16, further comprising a reliability acquisition means for acquiring a reliability for each pixel of the target region detected by the detection means based on the kernel and the importance of the correct region. (Configuration 18) The information processing device described in configuration 17, characterized in that the reliability acquisition means acquires the reliability using a result that combines the detection result by the detection means and the detection result detected by another detection process that detects another target area different from the target area. (Configuration 19) An acquisition means for acquiring an image; A detection means for detecting a target region from the image by using a learning model selected by a user from among a plurality of different learning models trained by an information processing device of any one of configurations 1 to 18; 13. An information processing device comprising: (Method 1) A kernel determination step of determining a kernel for acquiring the importance of a correct region corresponding to a learning image for each pixel; an importance determining step of determining an importance of the correct region for each pixel using the determined kernel; a learning step of learning a target region to be detected from the image based on the determined importance of the correct region; 13. An information processing method comprising: (Program 1) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 19. [Explanation of symbols]

[0075] 200: learning device, 220: data input unit, 230: importance determination unit, 240: detection unit, 241: detector, 242: parameter storage unit, 250: loss calculation unit, 260: weight update unit, 270: output unit

Claims

1. A kernel determination means for determining a kernel for obtaining the importance of a correct region corresponding to a learning image for each pixel; an importance determining means for determining the importance of the correct region for each pixel using the determined kernel; a learning means for learning a target region to be detected from an image based on the determined importance of the correct region; 13. An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein said importance determining means determines the importance of a pixel of interest, the importance of which is to be determined in said correct answer region, based on the number of pixels in said correct answer region within said kernel.

3. 2. The information processing apparatus according to claim 1, wherein the importance determining means determines the importance of a pixel of interest, the importance of which is to be determined in the correct answer region, based on the number of adjacent pixels of the correct answer region within the kernel.

4. 2. The information processing device according to claim 1, wherein the importance determination means determines the importance of a pixel of interest, the importance of which is to be determined in the correct answer region, based on the number of consecutive pixels included in the correct answer region in a specific direction within the kernel.

5. 2. The information processing apparatus according to claim 1, wherein the importance determining means determines the importance of a pixel of interest, the importance of which is to be determined in the correct answer region, based on a matching rate between a distribution of pixels contained in the correct answer region within the kernel and a preset template.

6. 2. The information processing apparatus according to claim 1, wherein said importance determining means determines a weight for each position of said kernel.

7. 2 . The information processing apparatus according to claim 1 , wherein the learning means acquires a loss in a process of detecting a target region from the image based on the importance of the correct region determined by the importance determination means.

8. The importance determining means generates an importance map based on the importance of each pixel in the correct solution region; 8. The information processing apparatus according to claim 7, wherein the learning means acquires the loss based on the importance map, and performs the learning based on the loss.

9. the importance determining means generates, as the importance maps, an undetected importance map for determining a loss when an undetected occurs and an erroneous detection importance map for determining a loss when an erroneous detection occurs; 9. The information processing apparatus according to claim 8, wherein the learning means acquires the loss when the detection is not performed based on the undetected importance map, and acquires the loss when the detection is not performed based on the false detection importance map.

10. 2. The information processing apparatus according to claim 1, wherein the importance determining means determines the importance of the correct region for each pixel by using a plurality of kernels determined by the kernel determining means.

11. the importance determination means generates an importance map based on the importance of each pixel in the correct solution region for each of the plurality of kernels determined by the kernel determination means, and combines the generated plurality of importance maps; 11. The information processing apparatus according to claim 10, wherein said learning means learns a target region to be detected from said image based on said combined importance map.

12. the importance determining means combines the plurality of importance maps using a combination parameter; 12. The information processing apparatus according to claim 11, wherein the combination parameters include at least one of a weight for adding together the importance maps, a round-up threshold for rounding up the importance map values, and a round-down threshold for rounding down the importance map values.

13. 2. The information processing apparatus according to claim 1, wherein the kernel determining means selects at least one kernel as the kernel to be determined from a plurality of predefined kernels.

14. The information processing device according to claim 13, characterized in that the kernel determination means calculates the importance of a correct region for each of the plurality of predefined kernels, and selects a kernel having a region in which the calculated importance is greater than an importance threshold value or more than a predetermined value.

15. 14. The information processing apparatus according to claim 13, wherein the kernel determination means calculates the importance of a correct region for each of the plurality of predefined kernels, and selects a kernel for which a total value of the importance for each pixel is equal to or greater than a predetermined value.

16. 2. The information processing apparatus according to claim 1, further comprising a detection unit that detects the target region from an image using a learning model learned by the learning unit.

17. 17. The information processing apparatus according to claim 16, further comprising a reliability acquisition means for acquiring a reliability for each pixel of the target region detected by the detection means, based on the kernel and the importance of the correct region.

18. The information processing device according to claim 17, characterized in that the reliability acquisition means acquires the reliability using a result of combining the detection result by the detection means and the detection result detected by another detection process that detects another target area different from the target area.

19. An acquisition means for acquiring an image; A detection means for detecting a target region from the image by using a learning model selected by a user from a plurality of different learning models trained by the information processing device according to any one of claims 1 to 18; 13. An information processing device comprising:

20. A kernel determination step of determining a kernel for acquiring the importance of a correct region corresponding to a learning image for each pixel; an importance determining step of determining an importance of the correct region for each pixel using the determined kernel; a learning step of learning a target region to be detected from an image based on the determined importance of the correct region; 13. An information processing method comprising:

21. Computer, A kernel determination means for determining a kernel for obtaining the importance of a correct region corresponding to a learning image for each pixel; an importance determining means for determining the importance of the correct region for each pixel using the determined kernel; a learning means for learning a target region to be detected from an image based on the determined importance of the correct region; A program that causes the device to function as an information processing device having the above-mentioned configuration.

Images