Information processing device, learning method, program, storage medium, and detection device

Abstract

To provide an information processing device which effectively reduces a position error when detecting a key point from an image, a detection device, a learning method, a program, and a storage medium.SOLUTION: An information processing device 1 that performs learning of a learning model used for detecting a key point of a subject included in an image, comprises: a score map estimation unit that obtains an estimation score map used for estimating the key point by using the learning model from the image; a partial region loss calculation unit that calculates a loss as a difference between a score map value of the estimation score map and one of a correct answer score map for each partial region; a contributing region estimation unit that calculates a contributing region on the basis of at least one of the correct answer score map and the estimation score map; a weight setting unit that sets a weight for each partial region on the basis of the contributing region; a score map loss calculation unit that obtains the loss of the estimation score map on the basis of the weight for each partial region and the loss for each partial region; and an update unit that updates the learning model on the basis of the loss of the estimation score map.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for detecting the position of an object from an image. [Background technology]

[0002] Conventionally, organ detection technology has been used as a method for detecting facial organ points such as eyes (Patent Document 1). In Patent Document 1, based on an input face image, a score map with high values ​​at positions where organ points exist is estimated, and the coordinates of the organ points are determined by detecting the peaks of the score map.

[0003] A method for estimating a score map from an input face image can be obtained by preparing an input image and a correct score map and using machine learning techniques such as a neural network. For example, the input image is subjected to forward processing of a neural network to obtain a score map, which is then compared with the correct score map, and the neural network is updated by error backpropagation processing so that the error between the two is reduced. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-223459 Summary of the Invention [Problem to be solved by the invention]

[0005] However, even if the organ detection score map estimation method is trained to reduce the error between the score map estimated from the input image and the correct score map, the position error of the organ coordinates may not improve. This is because, in organ detection, the organ coordinates are obtained by detecting peaks in the score map estimated from the input image. On the other hand, in score map training, training is performed to reduce the overall error, so training is also performed to reduce errors in areas other than the peaks. In other words, even if only the errors in areas other than the peaks are reduced, training of the entire score map progresses. However, there are cases where the position error of the organ coordinates obtained by peak detection does not change. As such, there was a problem that the position error of the organ coordinates does not improve even if the score map error is reduced in areas that do not affect the peak detection results.

[0006] An exemplary object of the present invention is to efficiently improve position errors when detecting keypoints from an image. [Means for solving the problem]

[0007] In order to solve the above problem, the present invention provides an information processing device that trains a learning model used to detect key points of a subject included in an image, the information processing device including: a score map estimation means that uses the learning model to obtain an estimated score map from the image that is used to estimate the key points; a first loss calculation means that calculates a loss, which is a difference in score map value between the estimated score map and a ground truth score map, for each partial region; before The estimated score P The learning model is characterized by comprising a contributing area estimation means for estimating a contributing area based on the contribution area, a weight setting means for setting a weight for each of the partial areas based on the contribution area, a second loss calculation means for calculating a loss in the estimated score map based on the weight for each of the partial areas and the loss for each of the partial areas, and an update means for updating the learning model based on the loss in the estimated score map. [Effects of the Invention]

[0008] According to the present invention, it is possible to efficiently improve position errors when detecting key points from an image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an apparatus according to a first embodiment. [Figure 2] 4 is a flowchart showing a process according to the first embodiment. [Figure 3] FIG. 10 is a diagram showing the relationship between the score map value and the weight when γ is changed. [Figure 4] 5A to 5C are diagrams illustrating the contributing region estimation process according to the first embodiment. [Figure 5] 10A to 10C are diagrams illustrating a contributing region estimation process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings, parts having the same functions are designated by the same numbers, and repeated explanations thereof will be omitted. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings.

[0011] (First embodiment) Fig. 1 is a diagram illustrating an apparatus according to a first embodiment. First, the configuration of a computer apparatus (information processing apparatus) according to the first embodiment will be described with reference to Fig. 1(A). Fig. 1(A) is a block diagram illustrating an example of the hardware configuration of the information processing apparatus according to the first embodiment.

[0012] The CPU 101a is a central processing unit that controls the entire computer system 100a. The ROM 102a is a read-only memory that stores programs and parameters that do not require modification. The RAM 103a is a random-access memory that temporarily stores computer programs and data supplied from external devices. The external storage device 104a is a storage device such as a hard disk or memory card that is fixedly installed in the computer system 100a. The external storage device 104a may also include optical disks such as flexible disks (FDs) and compact disks (CDs), magnetic or optical cards, IC cards, and memory cards that are detachable from the computer system 100a. The input device interface 105a is an interface with an input device 109a, such as a pointing device or keyboard, that receives user operations and inputs data. The output device interface 106a is an interface with a monitor 110a that displays data held by the computer system 100a and supplied data. The communication interface 107a is a communication interface for connecting to a network line 111a, such as the Internet. The system bus 108a is a transmission path that connects the units 101a to 107a so that they can communicate with each other. Each process described below is performed by the CPU 101a executing a computer program stored in a computer-readable storage medium such as the ROM 102a.

[0013] [Configuration of organ detection device] The configuration of the organ detection device 1 according to the first embodiment will be described with reference to FIG. 1(B). FIG. 1(B) is a block diagram showing an example of the functional configuration of the organ detection device 1 according to the first embodiment. The organ detection device 1 can be configured by incorporating a computer program related to organ detection into a general-purpose or dedicated information processing device (for example, the computer device 100a described above). The organ detection device 1 is a device that uses a learning model to detect a specific part of a subject (a human body in this case) included in an image. As an example, the organ detection device 1 according to this embodiment performs learning so that it can detect the left eye. The organ detection device 1 includes a learning unit 100b and an inference unit 120b. Note that the learning unit 100b and the inference unit 120b are described here as a single organ detection device 1, but the learning unit 100b and the inference unit 120b may be incorporated into separate information processing devices. In other words, the learning unit 100b and the inference unit 120b may each be independent devices.

[0014] The learning unit 100b learns a method for estimating a score map from an image based on the image and the correct score map. The inference unit 120b detects organ points from the image using the score map estimation method acquired by the learning unit 100b. Note that here, organ points are points where the organ to be detected is located, and in this specification, key points are points where the detection target is located.

[0015] First, the components of the learning unit 100b will be described. The image storage unit 101b stores images used for organ detection learning. Specifically, the images are stored in an external storage device 104a. All images contain at least one face or no faces at all. The vertical and horizontal sizes of all images have been resized to predetermined sizes in advance.

[0016] The correct score map storage unit 102b stores a correct score map corresponding to the image stored in the image storage unit 101b. Here, the correct score map is a score map including correct position information of the organ to be detected, and serves as training data.

[0017] Each pixel (partial region) of the score map takes a value between 0 and 1. This value is called the score map value. A value of 0 indicates that no organ point is present. The score map can have the same vertical and horizontal dimensions as the image stored in the image storage unit 101b. In this embodiment, to detect one left eye, the score map value takes a higher value at the position corresponding to the left eye in the image stored in the image storage unit 101b. The score map may take values ​​greater than 0 not only at the position of the left eye but also in its surrounding region. For example, it may take values ​​that follow a Gaussian distribution centered on the left eye. If the image does not include a face, each partial region of the score map takes a value of zero.

[0018] The score map may be a different size from the image stored in the image storage unit 101b. In that case, the coordinates of organ points on the image are converted to coordinates of organ points on the score map, and the score at that position becomes the score map with the largest score.

[0019] The score map estimation unit 103b processes an image using a learning model to estimate a score map. In this specification, the score map generated by the score map estimation unit 103b is referred to as an estimated score map. Specifically, the score map estimation unit 103b performs processing using Convolutional Neural Networks (CNN), a type of neural network. CNN extracts abstracted information from an input image by repeatedly performing processing on the input image, which includes convolution processing, nonlinear processing, and pooling processing. In this case, a processing unit including convolution processing, nonlinear processing, and pooling processing is often referred to as a layer. There are several well-known nonlinear processing methods used in this case, including a method called Rectified Linear Unit (ReLU). There are also several well-known pooling methods, including a method called maximum pooling. In this embodiment, to generate a score map having score map values ​​ranging from 0 to 1 from the input image using CNN, nonlinear processing is performed using a Sigmoid function in the final layer.

[0020] The network storage unit 104b stores the network structure and weight parameters of the CNN used in the score map estimation unit 103b. Specifically, these are stored in the external storage device 104a.

[0021] The detection unit 105b detects peak points of the score map values ​​from the score map. The estimated score map obtained by the score map estimation unit 103b is processed to generate a peak detection result in which only peak values ​​are left and the rest are set to zero. Note that the detection unit 105b may be applied to the correct score map of the correct score map holding unit 102b.

[0022] The peak detection process involves, for example, sliding a 3x3 window and storing the maximum score at the center point of the window, and otherwise storing zero, thereby obtaining a map in which all values ​​except the peak value are changed to zero from the score map. Note that the window size may be a different size, such as 5x5. The peak detection method is not limited to these.

[0023] The contributing area estimation unit 106b estimates an area in which reducing the error between the score map values ​​of the estimated score map and the correct score map will contribute to improving the position error of the organ coordinates. Specifically, for the correct score map, the area where the organ exists is considered to be the contributing area. This is because if the area where the organ exists can be sufficiently reproduced in the estimated score map, the position error will be reduced. In this embodiment, since the score map value of the correct score map is greater than 0 in the area where the organ exists, the area with a value greater than 0 is considered to be the contributing area. On the other hand, for the estimated score map, the area that affects peak detection is considered to be the contributing area. For example, the peak coordinate (peak point) obtained by peak detection by the detection unit 105b and its surrounding area (hereinafter, the peak point and its surrounding area will be collectively referred to as the "peak area") are considered to be the contributing area. This is because when calculating organ coordinates, peaks are detected in the estimated score map, and the peak area is an area that contributes to determining the final organ coordinates. Therefore, improving the error between this area and the correct score map will improve the position error of the organ coordinates.

[0024] The contributing regions obtained from the correct score map and the estimated score map are integrated to obtain one contributing region. Details of the contributing region estimation process will be described later in the explanation of the flowchart in Figure 2(B). Note that in the correct score map, the surrounding area of ​​the peak coordinates may also be used as the contributing region.

[0025] The subregion loss calculation unit 107b calculates the error (loss) between the score map values ​​of the estimated score map and the correct score map for each pixel (subregion) of the score map. Specifically, the loss for each pixel of the score map is calculated using the L2 loss function shown in the following formula (1) or the cross-entropy loss function shown in formula (2).

number

[0026] The weight setting unit 108b sets a weight for each pixel (partial region) of the score map. Specifically, the weight is set according to the following formulas (3) and (4) using the contributing region determined by the contributing region estimation unit 106b.

number

[0027] The score map loss calculation unit 109b calculates the loss of the entire score map from the loss and weight for each pixel calculated by the partial region loss calculation unit 107b and the weight setting unit 108b. Specifically, based on the loss in the following formula (5), the sum of the product of the loss and weight for each pixel is calculated and divided by the area S of the score map to calculate the overall loss.

number

[0028] The update unit 110b updates the weights of the network to reduce the loss based on the loss calculated by the score map loss calculation unit 109b. The network can be updated using a known technique such as backpropagation.

[0029] The weight setting unit 108b determines the weights for areas outside the contributing region according to the score map value in accordance with Equation (3), but a fixed value smaller than 1 may be set. However, determining the weights according to the score map value, as shown in Equation (3), is preferable because it allows for larger weights to be assigned to larger errors. Equation (3) assigns larger weights to larger errors, and this can be adjusted by adjusting γ. Figure 3 illustrates the relationship between the score map value and the weight when γ is changed. The horizontal axis represents the score map value, and the vertical axis represents the weight. γ is indicated in the legend. It can be seen from the figure that increasing γ reduces the weight when the score map value is small. Outside the contributing region, the score map value of the correct score map is zero, so the closer the score map value of the estimated score map is to zero, the more accurate the score map value. Therefore, increasing γ reduces the impact of small errors on the overall loss. As a result, learning can be focused on areas with large errors, enabling learning that is less likely to result in large errors.

[0030] In addition, when the score map value of the correct score map outside the contributing region is other than zero, the weight may be set to be larger as the deviation between the correct score map and the estimated score map increases, as shown in the following equation (6).

number

[0031] Furthermore, the weight setting unit 108b sets the weight of the contributing region to a fixed value of 1 according to equation (3). However, a larger weight may be set closer to the center of the contributing region, i.e., the peak point. For example, the weight may be set according to a Gaussian distribution. In order to set a larger weight than for regions other than the contributing region, the weight calculated by the Gaussian distribution may be calculated by adding 1. This places more importance on the center that contributes to determining the organ coordinates, and allows learning to more easily obtain a score map value closer to the correct answer closer to the center. As such, the method of setting weights based on the contributing region is not limited to these.

[0032] Furthermore, the weight setting unit 108b does not need to perform normalization using equation (4). However, if normalization is not performed, as the contributing region becomes smaller, the overall loss calculated by the score map loss calculation unit 109b (described later) may become smaller. As the loss becomes smaller, the amount of network updates performed by the update unit 110b may decrease, slowing down learning. For example, if the contributing region determined from the correct score map and the contributing region determined from the estimated score map begin to overlap, the contributing region may become smaller, gradually slowing down the learning speed, which may be a problem. To solve this problem, it is desirable to perform normalization.

[0033] Next, the components 121b to 124b included in the inference unit 120b will be described. The image input unit 121b accepts input of an image to which organ detection is to be applied. For example, an image stored in the external storage device 104a is displayed on the monitor 110a, and an image selected by the input device 109a is received. Alternatively, the inference unit 120b may receive an image received from another device via the communication interface 107a. Furthermore, the inference unit 120b itself may include an imaging unit such as a camera that captures an image of a subject, and the image input unit 121b may receive an image captured by the imaging unit.

[0034] The score map estimation unit 122b is almost the same as the score map estimation unit 103b of the learning unit 100b, and the detection unit 123b is almost the same as the detection unit 105b of the learning unit 100b. The network holding unit 124b is also almost the same as the network holding unit 104b of the learning unit 100b, and holds a network that is a learning model acquired by learning. The organ detection process will be described later using the flowchart in Figure 2(C).

[0035] [Network Update Process] 2A is a flowchart showing the network update process of the first embodiment. Each operation (step) shown in this flow diagram can be executed by the control of each unit of the CPU 101a. This process shows the process of updating the network, which is a learning model, from one image, i.e., the learning process of the learning model. Therefore, the learning process is realized by repeatedly applying this process (S201a to S207a in the figure) to various images until the overall loss of the estimated score map generated using the network converges to a small value.

[0036] In S201a, the image holding unit 101b and the correct score map holding unit 102b obtain (or store) images and correct score maps to be used for learning.

[0037] In S202a, an estimated score map is generated from the image obtained in S201a. Specifically, the image is input to score map estimation unit 103b, and an estimated score map is obtained by forward processing of the neural network based on the network held in network holding unit 104b.

[0038] In S203a, the estimated score map obtained in S202a is compared with the correct score map obtained in S201a to calculate the loss of pixels (subregions) of the score map. Specifically, the subregion loss calculation unit 107b calculates the loss of each pixel of the score map based on the above-mentioned formula (1) or formula (2).

[0039] In S204a, the contributing region is estimated using the contributing region estimation unit 106b. The process of estimating the contributing region will be described later with reference to FIG.

[0040] In S205a, the contributing region and the estimated score map are used to set pixel-by-pixel weights for the score map. Specifically, the weight setting unit 108b calculates pixel-by-pixel weights for the score map based on the above-mentioned formulas (3) and (4).

[0041] In S206a, the loss of the entire score map is calculated from the loss and weight of each pixel of the score map. Specifically, the score map loss calculation unit 109b performs the calculation based on the above-mentioned formula (5).

[0042] In S207a, the update unit 110b updates the weights of the network so as to reduce the loss calculated in S206a.

[0043] [Contributing area estimation process] 2B is a flowchart showing the contributing region estimation process of the first embodiment. This process is performed in S204a of the flowchart in FIG. 2A. Unless otherwise specified, the process of this flowchart is performed by the contributing region estimation unit 106b.

[0044] In S201b, organ regions are extracted from the correct score map. Since the correct score map has values ​​greater than zero in regions where organ points exist, organ regions are obtained by extracting regions with values ​​greater than zero.

[0045] In S202b, the detection unit 105b detects a peak point from the estimated score map.

[0046] In S203b, a predetermined number of peak points are selected from the peak points obtained in S202b. In this embodiment, the number of peak points selected is one. This is because there is only one organ point to be detected. When there are multiple peak points, they are selected in descending order of the peak point score map value.

[0047] In S204b, an expansion process is performed on the peak point of the estimated score map. The expansion process is a process of expanding the points to a 3x3 area around the peak point. The surrounding area may be 5x5 or the like. Alternatively, the expansion size may be determined according to the size of the organ area of ​​the correct score map. The expansion process method is not limited to these.

[0048] In S205b, the contributing regions of the estimated score map and the correct score map are integrated. Specifically, the contributing region of the correct score map obtained in S201b and the contributing region of the estimated score map obtained in S204b are used, and when a contributing region is included in one of the contributing regions, it is determined as the final contributing region, thereby performing the integration.

[0049] [Example of contributing area estimation process] A specific example of the contributing area estimation process will be described with reference to Fig. 4. Fig. 4 is a diagram illustrating the contributing area estimation process of the first embodiment. The score map takes values ​​from 0 to 1, but for ease of explanation, values ​​obtained by multiplying the score map values ​​by 10 are shown here. In addition, for ease of understanding, the face, which is the subject of the input image, is drawn on the score map with a dashed line.

[0050] In the correct score map 401, a high score map value is set around the left eye of the face. In S201b, this is processed to identify an area with a value greater than 0, and a contributing area 402 of the correct score map is generated.

[0051] The estimated score map 403 is a score map estimated from the input image, and has values ​​greater than 0 in some areas. In S202b, this is processed to detect peaks in the estimated score map and extract peak point 404 of the estimated score map. Here, a 3x3 peak detection filter is applied, and as a result, three peak points (peak points 404a to 404c) are extracted. Next, in S203b, one peak point is selected from the three peak points. Here, peak point 404a, which indicates the largest score map value, score 9, is selected. Then, in S204b, peak point 404a is expanded to obtain a contributing area 405 of the estimated score map.

[0052] Finally, in S205b, the contributing region 402 of the correct score map and the contributing region 405 of the estimated score map are integrated to obtain an integrated contributing region 406.

[0053] [Organ detection processing] 2(C) is a flowchart showing the organ detection process according to the first embodiment. This process is a process for obtaining organ detection points from an input image using a network obtained by repeatedly applying the network update process.

[0054] In S201c, the score map estimation unit 122b acquires the image received by the image input unit 121b.

[0055] In S202c, the score map estimation unit 122b estimates a score map of the image obtained in S201c. Specifically, the score map estimation unit 122b performs a score map estimation process using the network of the network holding unit 124b.

[0056] In S203c, the detection unit 123b applies peak detection processing to the estimated score map obtained in S202c.

[0057] In S204c, a predetermined number of peak points are selected from the peak points obtained in S203c. In this embodiment, since the number of organ points to be detected is one, one peak point is selected. When there are multiple peaks, the peak points are selected in descending order of their score map values.

[0058] In S205c, the peak point is converted into coordinates on the input image. Coordinate values ​​are obtained when the upper left corner of the image is the origin. The obtained coordinates indicate the coordinates of the left eye on the input image, and are stored in the external storage device 104a or the like in association with the input image. Alternatively, an image in which organ points are superimposed on the input image may be displayed on the monitor 110a. The method of outputting organ points is not limited to these.

[0059] [Variations] In this embodiment, the loss is calculated using only one image, but in neural network training, multiple images are often used as a mini-batch. Similarly, the loss for multiple images may be calculated before updating the neural network.

[0060] In this embodiment, peak detection is also performed when selecting one peak point from the estimated score map. However, when there is only one peak point, the maximum value of the score map values ​​in the estimated score map may be set as the peak point.

[0061] Furthermore, in this embodiment, the contributing region is obtained by threshold processing for the correct score map, but the contributing region may be obtained by applying processing such as peak detection as in the case of the estimated score map.

[0062] Furthermore, although the present embodiment has been described with reference to an example in which only the left eye is detected, multiple organ points may be detected. For example, a case may be considered in which five organ points, namely, the left and right eyes, the nose, and the left and right corners of the mouth, are detected. In this case, in S201a of FIG. 2(A), a correct score map exists for each of the five organ points. Then, in S202a, an estimated score map is obtained for each of the five organ points. Then, the processes of S203a to S206a are performed for each organ point, and an overall loss is calculated for each organ point score map. Then, in S207a, the network is updated based on the sum or average of the overall losses for the five organ points. In this way, multiple organ points may be detected.

[0063] [Effects of this embodiment] The loss calculation method of this embodiment promotes reducing errors in peak regions of the estimated score map and organ regions of the correct score map, rather than errors in the entire score map. Therefore, position errors in organ detection can be efficiently improved. This is because only a portion of the score map affects the peak detection results that determine organ coordinates, and efficiently improving the score map in these partial regions allows for efficient improvement of position errors in organ detection.

[0064] Furthermore, according to this embodiment, the learning model is updated to reduce the loss of the contributing region. Reducing the loss of the contributing region leads to a reduction in the position error of organ detection. This is because the contributing region is the region where the organ points of the correct score map exist, so bringing it closer to this region leads to a reduction in the position error. In addition, since the contributing region is the peak region of the estimated score map, it is a region that affects peak detection during inference of organ detection, so bringing it closer to the correct region leads to a reduction in the position error.

[0065] This allows learning to proceed so as to reduce positional errors, thereby efficiently improving the accuracy of organ detection.

[0066] In addition, when there are multiple peak regions in the estimated score map, only one is used as the organ detection result, so only one peak region is used for processing. This allows for efficient learning of only the parts that affect the organ detection results.

[0067] (Second embodiment) In the above embodiment, an example of detecting only the left eye has been described. Also, a method of estimating a score map for each organ point detected when detecting multiple organ points has been described. In this embodiment, an example of detecting multiple organ points from one score map will be described. Specifically, in this embodiment, as an example, both the left and right eyes are detected from one score map.

[0068] The configuration of the organ detection device of this embodiment will be described with reference to Fig. 1(B). The main difference from the first embodiment is the correct score map storage unit 102b.

[0069] The correct score map storage unit 102b stores the correct score map in the same manner as in the first embodiment, except that the stored correct score map has high score values ​​at the positions of both the left and right eyes.

[0070] The network update process of this embodiment (FIG. 2A) is almost the same as that of the first embodiment, but the contributing region estimation process (FIG. 2B) and S203b and S204c of the organ detection process (FIG. 2C) are different.

[0071] In both S203b and S204c, a predetermined number of peak points are selected from the results of peak detection in the estimated score map. In this case, only one peak point is selected in the first embodiment, but two peak points are selected in this embodiment. When there are multiple peak points, a predetermined number (here, two) of peaks are selected in descending order of score.

[0072] Next, a specific example of the contributing area estimation process will be described with reference to Fig. 5. Fig. 5 is a diagram illustrating the contributing area estimation process of the second embodiment. The score map takes values ​​from 0 to 1, but for ease of explanation, values ​​obtained by multiplying the score map values ​​by 10 are shown here. In addition, for ease of understanding, the face, which is the subject of the input image, is drawn on the score map with a dashed line.

[0073] In the correct score map 501, high score map values ​​are set around the left and right eyes of the face. In S201b, this is processed to identify areas with values ​​greater than 0, and the contributing areas 502a and 502b of the correct score map are generated.

[0074] Estimated score map 503 is a score map estimated from the input image, and some regions have values ​​greater than 0. In S202b, this is processed to detect peak points in the estimated score map, generating peak point 504 of the estimated score map. Here, a 3x3 peak detection filter is applied, resulting in the extraction of three peak points (peak points 504a to 504c). Next, in S203b, two peak points are selected from the three peak points in descending order of score. In this case, peak points 504a and 504c, which indicate scores of 9 and 4, are selected. Then, in S204b, peak points 504a and 504c are expanded to obtain contributing regions 505a and 505c of the estimated score map. Finally, in S205b, contributing regions 502a and 502c of the correct score map and contributing regions 505a and 505c of the estimated score map are integrated to obtain integrated contributing region 506.

[0075] [Variations] Furthermore, in the present embodiment and the first embodiment, examples of detecting facial organ points have been described. However, other examples of detecting key points from an image can be applied. For example, a configuration may be used to estimate body parts such as a person's joint points. Specifically, when a whole-body image of a person is input, the positions of 12 points are detected: the center of the head, left and right shoulders, left and right elbows, left and right hands, waist centers, left and right knees, and left and right toes. Alternatively, key points of an artificial object may be detected. For example, in the assembly process of a printer, digital camera, etc., the housing has predetermined screw holes. These screw holes may be detected as key points. The key points to be detected are not limited to these. In any of the key point examples, one key point or multiple key points may be assigned to one score map.

[0076] Furthermore, in the present embodiment and the first embodiment, the number of key points detected from the score map is fixed. However, this embodiment can also be applied to cases where this number is not fixed. In this case, all detected peak points are selected in the peak point selection in S203b or S204c. Note that a score map value threshold may be set for the selected peak points, and peak points that are equal to or greater than the threshold may be selected. In this case, the peak region remaining after applying peak detection to the estimated score map is considered to be the contributing region. If the peak detection filter size is large, small peaks near large peaks are suppressed. Therefore, results different from those obtained by simply thresholding the estimated score map are obtained. Because these small peaks do not affect peak detection, suppressing them is unlikely to reduce position errors. Therefore, even when an arbitrary number of key points are detected from the score map, position errors can be effectively reduced by using the peak region as the contributing region.

[0077] [Effects of this embodiment] This method can be applied to cases where multiple keypoints are detected from the score map. The peak score in S203b is determined according to the number of keypoints assigned to the score map, and the contributing area is determined based on that. These contributing areas affect peak detection during organ detection inference, so bringing them closer to the correct answer will lead to a reduction in positional errors.

[0078] In addition, when there are multiple peak regions, by narrowing down the number of keypoints used during inference, it is possible to efficiently learn only the parts that affect the keypoint detection results.

[0079] (Other embodiments) Although the embodiments have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0080] The present invention can be embodied as, for example, a system, an apparatus, a method, a program, or a storage medium, etc. Specifically, the present invention may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device.

[0081] The present invention can also be realized by executing the following process: software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. [Explanation of symbols]

[0082] 1. Organ detection device 100a Computer equipment (information processing equipment) 100b Learning Department 103b,122b Score map estimation part 104b, 124b Network holding section 106b Contributing area estimation section 107b Partial area loss calculation unit 108b Weight setting section 109b Score map loss calculation part 110b Update section 120b Reasoning part

Claims

1. An information processing device that performs learning of a learning model used to detect key points of a subject included in an image, a score map estimation means for obtaining an estimated score map from the image using the learning model, the estimated score map being used to estimate the key points; a first loss calculation means for calculating a loss, which is a difference between a score map value of the estimated score map and a correct score map, for each partial region; a contributing area estimation means for determining a contributing area based on the estimated score map; a weight setting means for setting a weight for each of the partial regions based on the contributing region; a second loss calculation means for calculating a loss of the estimated score map based on the weight for each partial region and the loss for each partial region; and updating means for updating the learning model based on the loss of the estimated score map.

2. The information processing device described in Claim 1, characterized in that the contributing area estimation means determines the contributing area based on the correct score map and the estimated score map.

3. 2. The information processing apparatus according to claim 1, wherein the contributing region estimation means determines a region including a peak point of the score map value as the contributing region.

4. 4. The information processing apparatus according to claim 3, wherein the contributing region estimation means selects, as the contributing regions, a predetermined number of regions in descending order of the score map value from the region including the peak point.

5. An information processing device that trains a learning model used to detect key points of a subject included in an image, a score map estimation means for obtaining an estimated score map from the image using the learning model, the estimated score map being used to estimate the key points; a first loss calculation means for calculating a loss, which is a difference between a score map value of the estimated score map and a correct score map, for each partial region; a contributing region estimation means for estimating a predetermined number of regions in descending order of score map value from a region including a peak point of the score map value of at least one of the correct score map and the estimated score map as contributing regions; a weight setting means for setting a weight for each of the partial regions based on the contributing region; a second loss calculation means for calculating a loss of the estimated score map based on the weight for each partial region and the loss for each partial region; and updating means for updating the learning model based on the loss of the estimated score map.

6. 6. The information processing apparatus according to claim 1, wherein the contributing region estimation means determines a region having the score map value equal to or greater than a predetermined threshold value as a contributing region.

7. 7. The information processing apparatus according to claim 1, wherein the weight setting means sets a weight for a partial region other than the contributing region to be smaller than that for the contributing region.

8. The information processing apparatus according to claim 1 , wherein the weight setting means sets weights for partial regions other than the contributing region based on the score map values ​​of the estimated score map.

9. 9. The information processing apparatus according to claim 1, wherein the weight setting means performs normalization so that the sum of the weights of the partial regions becomes 1.

10. The information processing device according to claim 1 , wherein the learning model includes a neural network.

11. The information processing apparatus according to claim 1 , wherein the update means updates the learning model so as to reduce a loss in the estimated score map.

12. 12. The information processing apparatus according to claim 1, wherein the key point is a predetermined part of a human body.

13. A method for training a learning model used to detect key points of a subject included in an image, comprising: determining an estimated score map from the image using the learning model for use in estimating the keypoints; Calculating a loss, which is a difference between the estimated score map and a correct score map, for each subregion; determining a contributing region based on the estimated score map; setting a weight for each of the partial regions based on the contributing region; calculating a loss of the estimated score map based on the weight for each subregion and the loss for each subregion; A learning method, characterized in that the learning model is updated based on the loss of the estimated score map.

14. A program that causes a computer to execute a learning method for a learning model used to detect key points of a subject included in an image, The learning method includes: determining an estimated score map from the image using the learning model for use in estimating the keypoints; Calculating a loss, which is a difference between the estimated score map and a correct score map, for each subregion; determining a contributing region based on the estimated score map; setting a weight for each of the partial regions based on the contributing region; calculating a loss of the estimated score map based on the weight for each subregion and the loss for each subregion; A program that updates the learning model based on the loss of the estimated score map.

15. A computer-readable storage medium storing the program according to claim 14.

16. A detection device for detecting key points of a subject included in an image, a score map estimation means for obtaining an estimated score map from the image using a learning model to be used for estimating the key points; a detection means for detecting a peak point from the estimated score map to detect the key point; The learning model is determining an estimated score map from the image using the learning model for use in estimating the keypoints; Calculating a loss, which is a difference between the estimated score map and a correct score map, for each subregion; determining a contributing region based on the estimated score map; setting a weight for each of the partial regions based on the contributing region; calculating a loss of the estimated score map based on the weight for each subregion and the loss for each subregion; A detection device characterized in that the learning model is updated based on the loss of the estimated score map.

Images