Image processing device, image processing method, and program
The image processing device addresses false detections by adjusting error weights based on object boundaries, improving the accuracy of facial organ detection and subsequent tasks.
Patent Information
- Application Number
- JP2020207376
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-12-15
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-12-15
AI Technical Summary
Existing image processing systems face challenges in accurately suppressing false detections when negative examples are similar to positive examples, leading to inaccurate object detection.
An image processing device that includes an estimation means for object positioning and a learning mechanism to update parameters based on error within a predetermined range, increasing the weight of errors outside the object range to suppress false detections.
Effectively reduces false detections by adjusting error weights based on object boundaries, enhancing the accuracy of subsequent tasks utilizing facial organ detection.
Smart Images

Figure 0007577526000001 
Figure 0007577526000002 
Figure 0007577526000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for processing images. [Background technology]
[0002] In recent years, in the task of detecting a specific object in an image, highly accurate object detection has become possible by learning (optimizing) parameters of a multi-layered CNN (Convolutional Neural Network) using a large amount of correct answer data. In addition, Non-Patent Document 1 discloses a method of suppressing false detection by collecting and learning negative examples that are erroneously detected in object detection. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] Abhinav Shrivastava, Abhinav Gupta, Ross Girshick, "Training Region-Based Object Detectors with Online Hard Example Mining", in CVPR, 2016. Summary of the Invention [Problem to be solved by the invention]
[0004] However, when negative examples are very similar to positive examples, it is difficult to accurately remove negative examples using learning alone, and false positives may occur.
[0005] Therefore, an object of the present invention is to make it possible to suppress the occurrence of erroneous detection. [Means for solving the problem]
[0006] The image processing device of the present invention comprises: The present invention has an estimation means for executing an estimation process for estimating a position of an object from an image, and a learning means for updating parameters used in the estimation process so that an error determined based on the position of the object estimated by the estimation means and correct answer information regarding the position of the object is reduced, and the learning means updates the parameters by weighting the error so that the error is larger when the position of the object estimated by the estimation means is outside a predetermined range in the image than when the position is within the predetermined range. The present invention is characterized by the above. Effect of the Invention
[0007] According to the present invention, it is possible to suppress the occurrence of erroneous detection. [Brief description of the drawings]
[0008] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing apparatus. [Diagram 2] FIG. 1 illustrates an example of a functional configuration of an image processing apparatus according to a first embodiment. [Diagram 3] 4 is a flowchart of a process according to the present embodiment. [Figure 4] 13 is a flowchart of a process for calculating a loss of face parts based on face frame correct information. [Diagram 5] 11A to 11C are diagrams illustrating examples of an estimated map, a correct answer map, and a weight map of face organ positions. [Figure 6] FIG. 11 is a diagram illustrating an example of a functional configuration of an image processing apparatus according to a second embodiment. [Figure 7] FIG. 13 is a diagram illustrating an example of a functional configuration of an image processing apparatus according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. <First embodiment> 1 is a diagram showing a schematic hardware configuration of a computer device 100 which is an application example of an image processing device according to a first embodiment. The image processing device of this embodiment may be realized by a single computer device, or may be realized by distributing each function among a plurality of computer devices as necessary. When the image processing device is configured by a plurality of computer devices, the plurality of computer devices are connected by a LAN (Local Area Network) or the like so as to be able to communicate with each other.
[0010] In FIG. 1, a central processing unit (CPU) 101 controls the entire computer device 100. The CPU 101 executes a program to perform image processing according to the present embodiment, which will be described later. A read only memory (ROM) 102 stores programs and parameters that do not require modification. A random access memory (RAM) 103 temporarily stores programs and data supplied from an external device. An external storage device 104 reads and writes data from a storage medium fixed to the computer device 100 or a removable storage medium. Examples of the storage medium fixed to the computer device 100 include a hard disk and a memory card, and examples of the removable storage medium include optical disks such as flexible disks (FDs) and compact disks (CDs), magnetic cards, optical cards, IC cards, and memory cards. In this embodiment, the external storage device 104 stores large amounts of learning data to be stored in a data storage unit 208, which will be described later. An input IF 105 is an interface with an input device 108, such as a pointing device or a keyboard, which receives a user's operation to input data such as numerical values. The output IF 106 is an interface with an output device 109 such as a monitor for outputting data held by the computer device 100, execution results of programs, and the like. The system bus 107 connects the above-mentioned units, CPU 101 to output IF 106, so that they can communicate with each other. Note that one or more dedicated hardware devices different from the CPU may be provided, and at least a part of the processing by the CPU may be executed by the dedicated hardware devices. Examples of the dedicated hardware devices include a GPU (graphics processing unit), an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).
[0011] FIG. 2 is a diagram illustrating an example of a functional configuration of the image processing apparatus according to the first embodiment. The data storage unit 208 stores learning data 209 prepared in advance. The learning data 209 is data in which data of learning images including at least a first target region and a second target region that are estimation targets are linked to correct answer information such as the positions and sizes of the target regions and managed. In the case of this embodiment, for example, the first target region is a face region, and the second target region is a region of facial organs such as the eyes, nose, and mouth (corners of the mouth) in the face. The correct answer information is information such as the position coordinates and size (e.g., the number of pixels in the vertical and horizontal directions) of the face and facial organs. For example, the position coordinates of the face are stored as the center coordinates of the face in the image and the position coordinates of the frame (face frame) surrounding the face, and the size of the face frame is stored as the size of the face.
[0012] The image acquisition unit 201 acquires images to be used for learning according to this embodiment from learning data 209 stored in a data storage unit 208 .
[0013] The feature extraction unit 202 receives an image acquired by the image acquisition unit 201, and extracts features including information effective for face frame estimation and organ estimation processing, which will be described later, from the input image. In this embodiment, the feature extraction unit 202 is configured as a multi-layered CNN (Convolutional Neural Network), and is connected to a face frame estimation unit 203 and an organ estimation unit 205, which will be described later. In this embodiment, the face frame estimation unit 203 and the organ estimation unit 205 are each a multi-layered neural network (NN) including a CNN, and as a whole, they constitute a multi-layered NN. By configuring the image processing device of this embodiment in this way, learning results from two tasks, face frame estimation and organ estimation, which will be described later, are fed back to the feature extraction unit 202, and the feature extraction unit 202 extracts features suitable for both tasks.
[0014] The face frame estimation unit 203 performs a range estimation process to estimate the range of the face, which is the first target region. In this embodiment, the face frame estimation unit 203 receives the feature amount extracted by the feature extraction unit 202 as an input, and estimates at least information related to the position and size of the face in the image. As described above, in this embodiment, the face frame estimation unit 203 is configured with a NN including a multi-layer CNN. Any method may be used for the estimation process in the face frame estimation unit 203 as long as the position and size of the face can be estimated. In this embodiment, the face frame estimation unit 203 outputs a likelihood map in which the likelihood of the face appearing is high at a position corresponding to the face center in the image, while being low at a position other than the face center. In addition, the face frame estimation unit 203 outputs three maps, including the likelihood map, a map in which the vicinity of the face center is set to the numerical value of the width of the face, and a map in which the vicinity is set to the numerical value of the height of the face.
[0015] Here, when multiple faces appear in an image, multiple points (pixels showing sufficiently high values compared to the surroundings) also appear in the face-centered likelihood map, but by referring to the pixel at the same position as the pixel at the face center position, the width and height values of the face corresponding to the face center can be obtained. For this reason, in this embodiment, the above-mentioned three map configuration is adopted. However, this is not limited to this, and for example, when there is a constraint that there is only one face, the width and height of the face may be estimated as numerical values rather than maps.
[0016] The face frame loss calculation unit 204 compares the estimation result output by the face frame estimation unit 203 with the correct data of the face frame for the same image generated by the correct data generation unit 207 described later, and calculates the loss required for learning from the error. In the case of this embodiment, three maps of the face position and the face frame size (length, width) are output as the estimation result, so the correct data generation unit 207 generates correct maps corresponding to the three maps from the learning data 209 as correct data. Therefore, the face frame loss calculation unit 204 calculates the error of the value for each pixel between each map of these estimation results and each map of the correct data corresponding to each of them, and calculates the overall loss. The error for each pixel value may be the commonly used square error, and the overall loss may be calculated by calculating the average of the square errors of all the pixels, a value generally known as MSE (Mean Square Error).
[0017] The organ estimation unit 205 estimates the position of a second target area that appears in a first target area in an image. In this embodiment, the organ estimation unit 205 estimates the positions of face organ points as so-called landmark positions, using the feature amount extracted by the feature extraction unit 202 as an input. Here, the face organ points to be detected are those within the face frame that is the estimation target by the face frame estimation unit 203. In this embodiment, the organ estimation unit 205 is configured with a NN including a multi-layer CNN, as described above. Although the estimation method is not limited, the organ estimation unit 205 outputs a likelihood map of each organ point position by the same method as the face frame estimation unit 203. For example, when estimating five organ points, namely the centers of the left and right eyes, the nose, and both ends of the mouth (corners of the mouth), the organ estimation unit 205 outputs five likelihood maps that estimate the positions of the five organ points.
[0018] The organ loss calculation unit 206, in the same manner as the face frame loss calculation unit 204, calculates the organ loss from the error obtained by comparing the estimation result of the organ estimation unit 205 with the correct answer data on the organ position generated by the correct answer data generation unit 207. At this time, the organ loss calculation unit 206 also acquires correct answer information on the face frame position and size of the learning data 209 via the correct answer data generation unit 207. The organ loss calculation unit 206 refers to the correct answer information on the face frame position and size thus obtained, and corrects the estimation result of the organ point appearing outside the face frame so that the error becomes large. In the case of the map output as in this embodiment, the organ loss calculation unit 206 increases the weight (gives weight) to the error when it is outside the correct face frame, so that the error value outside the face frame becomes large. Note that the error calculation method may be MSE or the like, as described in the face frame loss calculation unit 204. In addition, the flow of processing when the map is used will be described later with reference to FIG. 4 and FIG. 5.
[0019] The correct answer data generating unit 207 acquires correct answer information such as position coordinates and size related to the face frame and face organs associated with the image acquired by the image acquiring unit 201 from the learning data 209 in the data storage unit 208, and generates correct answer data that can be compared with the estimation results of each estimation unit to calculate an error. As described above, the face frame estimation unit 203 and the organ estimation unit 205 output maps as the corresponding estimation results, so the correct answer data generating unit 207 generates a correct answer likelihood map as correct answer data corresponding to each output from the acquired correct answer information. In this embodiment, as described above, the face frame estimation unit 203 estimates three maps, including a likelihood map of the face center position and a map in which the vicinity of the face center position is a numerical value of each of the vertical and horizontal sizes of the face frame. The map indicating the size is, for example, a map in which a numerical value indicating the vertical or horizontal size of the face is given to coordinates within a predetermined range (for example, 2 pixels) from the center position of the face, and a pixel value of 0 is given to other coordinates where there is no face frame. The predetermined range also depends on the degree of overlap of the targets. The map may be a map showing a numerical value indicating the size relative to the center position, but is set in the vicinity so that the center position of the target can be estimated even if it is slightly shifted. At this time, the correct answer data generating unit 207 refers to the position coordinates of the face center from the acquired learning data 209 as the correct answer data of the likelihood map of the face center position. Then, the correct answer data generating unit 207 generates a likelihood map of the same size as the estimated map, in which the closer to the position coordinates, the closer to 1, and the other coordinates, the 0, as the correct answer likelihood map. In addition, the correct answer data generating unit 207 also generates a map as the correct answer map in which the vertical and horizontal sizes of the face frame are set to the value of the size within a predetermined range centered on the position coordinates of the face center. The estimation of each organ position by the organ estimation unit 205 is similar to the estimation of the face center position described above, and the correct answer data generating unit 207 refers to the coordinate values of each organ from the learning data 209 to generate the correct likelihood map of each organ.
[0020] The learning unit 210 updates the parameters of each NN of the feature extraction unit 202, the face frame estimation unit 203, and the organ estimation unit 205 so that the losses calculated by the face frame loss calculation unit 204 and the organ loss calculation unit 206 are reduced (closer to 0 when MSE Loss is used as in the example of this embodiment). Here, the method of optimizing the parameters is not particularly limited as long as the parameters can be updated to reduce the overall loss. For example, the parameters may be updated by a general-purpose optimization process such as SGD (Stochastic Gradient Descent).
[0021] As described above, in the image processing device of this embodiment, the organ loss calculation unit 206 and the learning unit 210 perform a learning process using the learning image and the correct answer information to reduce the error between the estimation result of the position of the face organ point by the estimation process and the correct answer information. Also, in this embodiment, the organ loss calculation unit 206 obtains the range of the face frame from the learning image, and sets a weight for the error from the correct answer information based on the estimation result by the estimation process of the position of the face organ point and the range of the face frame.
[0022] Next, the processing procedure in the image processing device of this embodiment will be described with reference to the flowchart in Fig. 3. In the following flowcharts, the reference character "S" indicates a processing step. First, in S301, the image acquisition unit 201 acquires image data to be used for learning from the learning data 209 in the data storage unit 208. In this embodiment, similar to general NN learning, learning is performed while randomly sampling a plurality of unselected image data from the large-scale learning data 209. After all image data of the learning data 209 is acquired in this manner, the process of randomly sampling image data from all the learning data is repeated again.
[0023] Next, in S302, the feature extraction unit 202 extracts features from the image data. That is, the image data acquired in S301 is used as input to obtain the output of the previously configured multi-layer CNN. When the overall learning has progressed sufficiently, the output includes feature information suitable for estimating the face frame and organs in the image.
[0024] Next, at 303, the face frame estimation unit 203 receives the features extracted by the feature extraction unit 202 and outputs an estimation result regarding the position and size of the face. As described above, in this embodiment, the face frame estimation unit 203 outputs three maps: a face-centered likelihood map, and maps of the width and height of the face.
[0025] Furthermore, in S304, the organ estimation unit 205 outputs an estimation result regarding the face organ positions using the features extracted by the feature extraction unit 202 as an input. In this embodiment, as described above, the organ estimation unit 205 outputs a likelihood map for each of the five face organ positions.
[0026] Next, in S305, the face frame loss calculation unit 204 acquires the estimation result of the face frame estimation unit 203 and the correct answer data generated by the correct answer data generation unit 207 based on the correct answer information of the learning data 209, and calculates the error between them. As described above, the correct answer information here is information on the face frame, and is information indicating the center coordinates of the face and the horizontal and vertical sizes of the face. In addition, since the error is calculated by the face frame loss calculation unit 204, the correct answer data is three correct answer maps corresponding to the estimation results in S303 in this embodiment.
[0027] Next, in S306, the organ loss calculation unit 206 obtains the estimation result of the organ estimation unit 205 and the correct answer data generated by the correct answer data generation unit 207 based on the correct answer information of the learning data 209, and calculates the error between them. Note that the correct answer information here is information on the position coordinates of the face organs, as described above. In addition, since the organ loss calculation unit 206 calculates the error, the correct answer data is five correct answer maps corresponding to the estimation results in S304 in this embodiment.
[0028] Next, in S307, the organ loss calculation unit 206 also acquires correct answer information on the face frame position and size of the learning data 209 via the correct answer data generation unit 207, and performs a process of increasing the weight of the error value in S306. Details of this process will be described later with reference to the flowchart shown in FIG.
[0029] Next, in S308, the learning unit 210 updates the parameters of each of the NNs in the feature extraction unit 202, face frame estimation unit 203, and organ estimation unit 205 so as to reduce the loss obtained from the error obtained in the processes up to S307. Next, in S309, the learning unit 210 determines whether or not there is unprocessed data in the learning data 209. If there is unprocessed data, the learning unit 210 returns the process to S301, acquires the unprocessed data, and repeats the above process, whereas if there is no unprocessed data, the learning unit 210 transitions the process to S310.
[0030] When the process proceeds to S310, the learning unit 210 determines whether the number of times all data in the learning data 209 has been learned does not exceed a preset number of times. If the number of times has not exceeded the preset number, the learning unit 210 proceeds to S311. On the other hand, if the number of times learning has been performed exceeds the preset number of times, the process of this flow ends. Note that the end of learning does not have to be determined by the number of times. For example, a validation data set that does not include learning data may be prepared, and learning may end when the accuracy is no longer updated. Alternatively, a period of the video to be processed may be determined, and learning may be continued, and a model with a peak estimation accuracy may be adopted at the stage where learning for the predetermined period is completed. Thereafter, in S311, the image processing device resets the learning data 209 to a state in which all data has not been acquired, and returns the process to S301, from which the above-mentioned processes are repeated.
[0031] The process in S307 will be described below with reference to the flowchart in Fig. 4. It is assumed that the processes up to S306 in Fig. 3 have already been performed when the process in the flowchart in Fig. 4 starts. First, in S401 , the organ loss calculation unit 206 acquires correct answer information on the face frame position and size of the learning data 209 via the correct answer data generation unit 207 . Furthermore, in S402, the organ loss calculation unit 206 performs weighting for a predetermined range based on the estimated face position and size, that is, sets the weight of the error outside the face frame.
[0032] Next, in S403, the organ loss calculation unit 206 determines whether the organ point position estimated by the organ estimation unit 205 is outside the range of the face frame acquired in S401. If the face frame is outside the range, the organ loss calculation unit 206 transitions the process to S404, whereas if it is within the range of the face frame, the organ loss calculation unit 206 uses the error already calculated in S306 as is and transitions to S405.
[0033] When the process proceeds to S404, the organ loss calculation unit 206 performs processing on the estimation results of organ points outside the face frame so that the error value from the correct answer becomes even larger. In this embodiment, the organ loss calculation unit 206 multiplies (multiplies) a preset weight. This makes it possible to utilize the feature that distinguishes whether inside or outside the face frame, which is included in the output of the feature extraction unit 202, and even if there is an image similar to a face organ outside the face frame, learning proceeds so that the detection likelihood is switched based on the information on whether it is inside or outside the frame, and the overall loss is reduced. Note that in this embodiment, a likelihood map is output as an estimation result of the face organ position, but the processing in such a case will be described later with reference to FIG. 5.
[0034] When the process proceeds to S405, the organ loss calculation unit 206 calculates the loss based on the error calculated up to this point and the set weight of the error. In this embodiment, the organ loss calculation unit 206 calculates the average of the squared error for all pixels as the loss, as described above. This process corresponds to the calculation of the loss related to the face organs in S308.
[0035] A series of processes from S401 to S404 will be described below with reference to FIG. 5(a) to FIG. 5(c). FIG. 5(a) is a diagram showing an example of a map of the estimation results by the organ estimation unit 205. In reality, a map for each organ point is output, but in this example, it is assumed to be a map of the estimation results centered on the left eye. Black parts in the figure have a value of 0, and the higher the likelihood estimation value, the whiter the color becomes. Here, we consider the case where there is one face to be detected. In the example of FIG. 5(a), there are two parts with high likelihood estimation values, but of these, the upper left one is assumed to be a false detection part that appears outside the size of the face frame.
[0036] Fig. 5(b) is a diagram showing an example of a correct answer map for Fig. 5(a), which is correct answer data generated by the correct answer data generating unit 207 from the correct answer information of the learning data 209. In general learning, the aim is to be able to estimate such that the estimated map matches the correct answer map (i.e., the loss due to error becomes zero).
[0037] Here, the organ loss calculation unit 206 refers to the correct answer information of the face frame position and size acquired via the correct answer data generation unit 207, and further generates a map of the same size (number of pixels) that gives a weight to the error of the organ estimation. FIG. 5(c) is a diagram showing an example, and a rectangular part 501 is the range of the face position and size in the image in this example. In the case of this embodiment, the organ loss calculation unit 206 calculates the loss by multiplying the square error for each pixel in FIG. 5(a) and FIG. 5(b) by the weight of the same pixel position in FIG. 5(c). At this time, a predetermined weight value is set outside the rectangular part 501 so that the error becomes large when the organ is estimated outside the face frame. The weight value is a value to be multiplied by the error of the same pixel position, and a value larger than 1 is set outside the face frame. On the other hand, since the inside of the rectangular part 501 is within the face frame, the weight value is set to 1 so that correction does not work even when the square error in FIG. 5(a) and FIG. 5(b) is applied.
[0038] The weighting value to be set may be determined so as to obtain a better accuracy while observing the progress of the evaluation of the detection accuracy by learning. The weighting value does not need to be a fixed value, and may be changed during the learning process. For example, once the estimation of the face frame can be stably performed, the features related to the face frame will also stably appear in the feature extraction unit 202, so the weighting of the error outside the face frame may be gradually increased according to the progress of the learning of the face frame estimation unit 203, that is, after the learning has progressed sufficiently. Also, the weighting of the outside of the frame does not need to be the same, and for example, the weighting may be increased according to the distance from the frame.
[0039] For example, if learning is performed by setting a uniform loss inside and outside the face frame, it is difficult to suppress erroneous detection outside the face frame, but in this embodiment, learning is performed by heavily weighting the loss outside the face frame, thereby suppressing erroneous detection outside the face frame. That is, according to this embodiment, for example, it becomes possible to suppress erroneous detection in which a texture appearing outside the face frame is similar to the detection target.
[0040] In the above explanation, the output of the organ estimation unit 205 is a likelihood map corresponding to the input image, and an example was given of using a map like that in Figure 5(c) as a method for calculating the weighted squared error, but the method is not important as long as it is possible to make the error of the estimated value outside the range of the frame larger.
[0041] As described above, according to the present embodiment, it is possible to suppress erroneous detection of face organs, and therefore it is expected that the accuracy of the subsequent tasks using face organ detection can be improved. For example, a unique effect of improving the accuracy of face recognition using an image normalized by the position of the face organs can be expected. In addition, according to the present embodiment, since erroneous detection of face organs can be removed during the learning process, post-processing such as removing organ points outside the face frame is not required. This makes it possible to configure an end-to-end NN from organ detection to subsequent face recognition, and it is expected that it is possible to achieve both the suppression of erroneous detection of organ points and the learning of a detector suitable for recognition.
[0042] As described above, in this embodiment, the range of the object as the first object region is acquired, and the error weight (i.e., the learning weight) is switched depending on whether the detected second object region is within the object range. As a result, according to this embodiment, it is possible to strongly suppress erroneous detection of the position of the second object region (landmark) that appears outside the range of the first object region. Therefore, according to this embodiment, it is possible to accurately perform subsequent processing using the detection result. Note that the subsequent processing is, for example, classification, matching processing, tracking processing, etc., using the detection result, and may be any processing using the detection result.
[0043] <Second embodiment> In the first embodiment, an example of performing learning by multitasking face organ estimation and face frame estimation has been described. By configuring the multitasking in this way, it is possible to simultaneously learn to acquire with high accuracy feature information required for face frame estimation as the output of the feature extraction unit 202, and therefore it is possible to proceed with learning so that the organ estimation unit 205 uses this information to stably suppress erroneous detection outside the frame. On the other hand, even if the face frame estimation task is not performed, by switching the weighting of the error depending on whether or not it is within the range of the face frame size according to the correct information, it is expected that if learning progresses sufficiently, learning will progress so as to reduce the overall loss by extracting information about the face frame size using the feature extraction unit 202.
[0044] Fig. 6 is a diagram showing an example of a configuration of an image processing device according to the second embodiment that does not estimate a face frame or perform learning related to the face frame, that is, a configuration that does not include the face frame estimation unit 203 and the face frame loss calculation unit 204 described above. In Fig. 6, the image acquisition unit 201 to the learning data 209 are the same as those described above, so their description will be omitted. The image processing device according to the second embodiment can be realized with a simple configuration as shown in Fig. 6, and the processing cost during learning and inference can be suppressed.
[0045] <Third embodiment> In the first embodiment, correct answer information on the face frame size is acquired and the weight of the error is determined based on this information, but the weight may be determined from the estimation result of the face position and size output by the face frame estimation unit 203. Fig. 7 is a diagram showing an example of the configuration of an image processing device according to a third embodiment in which the weight is determined from the estimation result of the face position and size output by the face frame estimation unit 203. Note that the processing in each component in Fig. 7 is basically the same as the processing in each component in Fig. 2 described above, and therefore description thereof will be omitted.
[0046] In the third embodiment, as shown in FIG. 7, the output of the face frame estimation unit 203 is used by the organ loss calculation unit 206, and the organ loss calculation unit 206 does not receive correct answer information of the face frame. In this way, correct answer information of the face frame is not necessary for the learning data of the organ estimation, so that separate data sets can be used. In machine learning, it is very important to prepare a large amount of learning data to which correct answer information is added, and it is significant to expand the selection of data sets. However, since the estimation result is used, there is a possibility that the learning of the organ estimation will not converge until the face frame has been learned to a certain extent. Therefore, in the third embodiment, the face frame estimation unit 203 and the organ estimation unit 205 may be trained separately at first, and then, after the accuracy of the face frame estimation unit 203 exceeds a certain level, learning using the face frame estimation result may be performed.
[0047] <Fourth embodiment> In the first embodiment, a weight map (FIG. 5(c)) was generated during learning, but since correct answer information regarding the face frame has already been added to the learning data 209, in the fourth embodiment, a weight map such as that shown in FIG. 5(c) is created in advance. As a result, according to the fourth embodiment, the processing cost during learning can be suppressed, and learning can be performed efficiently.
[0048] <Fifth embodiment> In the first embodiment, the face range is acquired as a rectangular face frame determined by the center position and the vertical and horizontal sizes, but the present invention is not limited to this. If an area including face organs can be acquired, it is not necessary to use a frame. In the fifth embodiment, for example, the face outline is acquired as correct answer information. In addition, if a map of error weights is created at this time, weights are applied to the inside of the outline rectangle. Note that, when the outline is used, it is sufficient to perform simultaneous learning with tasks such as outline estimation and face area segmentation that utilize outline information, rather than simultaneous learning with the face frame estimation task.
[0049] Sixth embodiment In the first embodiment, an example was described in which the first target area in an image is a face area of a person, and the landmark estimated as the second target area is an organ included in the face, but this is not limited thereto. There is no particular limitation thereto as long as it always appears within the range of the first target area in an image at least when photographed from a certain direction. In the sixth embodiment, a case is given in which the first target area is, for example, a car or a license plate. For example, when a fixed camera installed at the entrance gate of a parking lot photographs a car entering the parking lot from the front, the photographed image shows the area of the car body seen from the front with the license plate inside. In this case, the first target area is the car body, and the landmark estimated as the second target area is the license plate. Alternatively, even in the same image, if the focus is on the license plate, the first target area is the license plate, and the numbers and letters written on the license plate can be considered as the second target area (landmark). In the case of the sixth embodiment, if these images and correct answer information can be acquired as learning data, the configuration of the first embodiment described above can be applied. In this way, the present invention can be widely applied to cases where a landmark, which is a second target region, is expected to appear within a first target region in an image.
[0050] <Other embodiments> A program that realizes one or more functions in the control processing of the present invention can be supplied to a system or device via a network or a storage medium, and can be realized by being read and executed by one or more processors of the computer of that system or device. The above-described embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main characteristics. [Explanation of symbols]
[0051] 101: CPU, 102: ROM, 103: RAM, 104: External storage device, 201: Image acquisition unit, 202: Feature extraction unit, 203: Face frame estimation unit, 204: Face frame loss calculation unit, 205: Organ estimation unit, 206: Organ loss calculation unit, 207: Correct answer data generation unit, 208: Data storage unit, 209: Learning data, 210: Learning unit
Claims
1. An estimation means for executing an estimation process for estimating a position of an object from an image; a learning means for updating a parameter used in the estimation process so that an error determined based on the position of the object estimated by the estimation means and correct answer information regarding the position of the object is reduced; The image processing device is characterized in that the learning means updates the parameters by weighting the error so that when the position of the object estimated by the estimation means is outside a predetermined range in the image, the error is larger than when the position is within the predetermined range.
2. The image processing device according to claim 1, characterized in that the specified range is defined in the correct answer information.
3. Having a range estimation means for estimating the predetermined range, 2. The image processing apparatus according to claim 1, wherein the learning means performs the weighting based on the predetermined range estimated by the range estimation means.
4. The predetermined range corresponds to a face area of a person, 4. The image processing apparatus according to claim 1, wherein the object is a facial part of a person.
5. The predetermined range corresponds to an area of a vehicle license plate, 4. The image processing apparatus according to claim 1, wherein the object is a character string on a license plate.
6. An image processing device as described in any one of claims 1 to 5, characterized in that the learning means updates both the parameters used in the estimation process to estimate the specified range and the parameters used in the estimation process to estimate the position of the target object using the weighted error.
7. The estimation means generates a likelihood map representing a position of an object in the image by the estimation process, The image processing device according to any one of claims 1 to 6, characterized in that the learning means generates a correct likelihood map based on the correct answer information, and performs the weighting by multiplying the error between the correct likelihood map and the likelihood map by a weight map in which a larger weight is assigned to areas outside the specified range.
8. An image processing device as described in any one of claims 1 to 7, characterized in that the learning means performs weighting so that when the distance between the position of the object estimated by the estimation means and the specified range is a first distance, a greater weight is set than when the distance is a second distance shorter than the first distance.
9. An image processing device as described in any one of claims 1 to 7, characterized in that the learning means sets the weight to be larger according to the progress of the learning.
10. An image processing method executed by an image processing device, comprising: an estimation step of executing an estimation process for estimating a position of an object from an image; a learning process of updating parameters used in the estimation process so that an error determined based on the position of the object estimated by the estimation process and correct answer information regarding the position of the object is reduced; The image processing method is characterized in that the learning process updates the parameters by weighting the error so that when the position of the object estimated by the estimation process is outside a specified range in the image, the error is larger than when the position is within the specified range.
11. A program for causing a computer to operate as the image processing device according to any one of claims 1 to 9.
Citation Information
Patent Citations
Learning device, processing device, neural network, learning method, and program
JP2020091543A
Image analysis system and image analysis method
JP2020140236A
Learning of detection model using loss function
WO2020194111A1
