Image processing apparatus, imaging apparatus, learning device, control method, and program

Abstract

To provide a technique to detect a desired portion at high speed.SOLUTION: An image processing apparatus comprises: feature quantity extraction means that extracts a feature quantity from an image; portion estimation means that estimates a plurality of predetermined portions to be detected from the image on the basis of the feature quantity; priority area estimation means that estimates a priority area from the image on the basis of the feature quantity; and selection means that selects one predetermined portion from the plurality of predetermined portions on the basis of the priority area.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, a learning device, a control method, and a program. [Background technology]

[0002] Current imaging devices are equipped with an automatic focus adjustment (AF) function that automatically adjusts the lens to focus on the subject. When photographing people or animals in particular, there is a need to focus on the eyes or the pupils of the eyes (hereinafter sometimes simply referred to as the eyes) in order to take impressive facial photos.

[0003] In Patent Document 1, when the pupil detection AF mode is set, the eyes are detected from the captured image and the focus is adjusted to match the eyes. In this case, the left and right eyes are detected from the captured image during AF, and the direction of the face is detected, and the focus is adjusted to the eye that is closer to the photographer (imaging device) based on the direction of the face. As a result, even when taking a picture with the face tilted relative to the imaging device, for example, it is possible to focus on the eye closest to the imaging device, and a good facial photo can be taken.

[0004] In recent years, many methods have been proposed for detecting objects by highly processing images of objects captured in a captured image. Among these, methods for learning the features of objects in an image and recognizing the position and type of the object using a multi-layered neural network called a deep net (also called a deep neural network or deep learning) have been actively researched. Non-Patent Document 1 discloses a method for detecting face parts from an image using a deep net.

[0005] Non-Patent Document 2 describes that there are two methods for estimating facial organ points: a direct regression method and a heat map method, and it is known that the heat map method has higher accuracy. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 5669549 [Non-patent literature]

[0007] [Non-Patent Document 1] RetinaFace: Single-shot Multi-level Face Localization in the Wild, Jiankang Deng et al., 2020 [Non-Patent Document 2] Fast Facial Landmark Detection and Applications: A Survey, Kostiantyn Khabarlak et al., 2021 [Non-Patent Document 3] Fully convolutional networks for semantic segmentation, Jonathan Long et al., 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Summary of the Invention [Problem to be solved by the invention]

[0008] However, the method described in Patent Document 1 detects multiple facial organ points (eyes, mouth, nose, chin, etc.) and estimates the face direction from the positional relationship of these organ points. In order to estimate the face direction, it is necessary to detect multiple organ points, and it is costly to prepare correct answer data for the facial organ points. In addition, if a heat map method is used to detect multiple facial organ points, one heat map for estimating x and y coordinates is required to estimate one organ point. Therefore, when multiple organ points are detected, the time required for detection increases in proportion to the number of organ points.

[0009] The present invention has been made in consideration of the above problems, and has an object to provide a technique for quickly detecting a desired portion. [Means for solving the problem]

[0010] To achieve the above object, an image processing device according to one aspect of the present invention comprises: A feature extraction means for extracting features from an image; a part estimation means for estimating a plurality of predetermined parts of a detection object from the image based on the feature amount; a priority area estimation means for estimating a priority area from the image based on the feature amount; a selection means for selecting one predetermined portion from the plurality of predetermined portions based on the priority region; The present invention is characterized by comprising: Effect of the Invention

[0011] According to the present invention, it is possible to quickly detect a desired portion. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a hardware configuration diagram of the imaging apparatus according to the first embodiment. [Diagram 2] FIG. 1A is a functional configuration diagram of an image processing unit according to a first embodiment, and FIG. 1B is a functional configuration diagram of an image processing unit according to a second embodiment. [Diagram 3] 1 is a flowchart showing a processing flow of the first embodiment. [Figure 4] FIG. 2 is an explanatory diagram of the network structure of the neural network of the first embodiment. [Diagram 5] FIG. 2 is a functional configuration diagram of a learning device for performing learning according to the first embodiment. [Figure 6] 5A and 5B are schematic diagrams illustrating the inference results of the face direction and the eye-proximal area in the first embodiment. [Figure 7] 5 is a schematic diagram of learning images and correct answer information of the eye center position correct answer map according to the first embodiment. FIG. [Figure 8] 4 is a flowchart showing the flow of learning processing according to the first and second embodiments. [Figure 9] FIG. 11 is an explanatory diagram of the network structure of the neural network of the second embodiment. [Figure 10] 10 is a flowchart showing the flow of processing according to the second embodiment. [Figure 11] Schematic diagram of the areas around the two eyes and the skin area in embodiment 2. [Figure 12] FIG. 4 is a schematic diagram of a method for creating an eye-proximal region correct map according to the first embodiment. [Figure 13] FIG. 2 is a diagram showing the functional configuration of an imaging device according to the first and second embodiments, an image processing unit included in the imaging device, and a learning device for performing learning. [Figure 14] 10 is a flowchart showing the flow of processing of a method for creating an eye-proximal region correct map according to the first embodiment. [Figure 15] FIG. 13 is a schematic diagram of an example of creating an eye proximity area correct map. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] (Embodiment 1) In this embodiment, multiple predetermined parts of the detection target (e.g., parts of the same type (left and right eyes, left and right ears, etc.)) are estimated from the image based on the feature amounts extracted from the image, and further, a priority region (e.g., an eye proximity region that is a region that includes eyes that are relatively close to the camera) is estimated from the image based on the feature amounts. Then, one predetermined part (one eye) is selected from the multiple predetermined parts (left and right eyes) based on the priority region.

[0015] In this embodiment, an imaging device will be described as an example. However, this embodiment can be implemented in any electronic device that detects an object area in a moving image. Such electronic devices include, but are not limited to, imaging devices such as digital cameras and digital video cameras, as well as personal computers with camera functions, mobile phones, drive recorders, robots, drones, etc.

[0016] <Hardware configuration> 1 shows an example of the hardware configuration of an imaging device according to this embodiment. The imaging device 100 according to this embodiment includes an imaging unit 101, a RAM 102, a ROM 103, an image processing unit 104, an input / output unit 105, and a control unit 106. The individual units are configured to be able to communicate with each other and are connected via a bus or the like.

[0017] The imaging unit 101 includes an imaging lens, an imaging element, an A / D converter, an aperture control unit, a focus control unit, and the like. The imaging lens may include a fixed lens, a zoom lens, a focus lens, an aperture, and an aperture motor. The imaging element includes a CCD, a CMOS, or the like that converts an optical image of a subject into an electric signal. The A / D converter converts an analog signal into a digital signal. The imaging unit 101 converts an image of a subject formed on the imaging surface of the imaging element by the imaging lens into an electric signal, applies signal processing of A / D conversion processing to the electric signal by the A / D converter, and supplies the image data to the RAM 102. The aperture control unit controls the operation of the aperture motor and changes the aperture opening diameter to control the aperture of the imaging lens. The focus control unit controls the operation of the focus motor based on the phase difference between a pair of focus detection signals obtained from the imaging element, and controls the focus state of the imaging lens by driving the focus lens.

[0018] The RAM 102 stores image data obtained by the imaging unit 101 and image data to be displayed on the input / output unit 105. The RAM 102 has a storage capacity sufficient to store a predetermined number of still images and a predetermined length of video. The RAM 102 also serves as a memory for image display (video memory), and supplies image data for display to the input / output unit 105.

[0019] The ROM 103 is a storage device such as a magnetic storage device or a semiconductor memory, and stores programs that are loaded based on the operations of the image processing unit 104 and the control unit 106, data that must be stored for a long time, and the like.

[0020] The image processing unit 104 detects and selects object candidate regions from an image and outputs the results. The object candidates here refer to various categories of objects such as people, animals, and vehicles, as well as local parts such as heads, faces, and eyes. In this embodiment, the image processing unit 104 performs object detection by outputting the position and size of a specific object candidate region and a likelihood that it is an object as a detection result. The configuration and operation of the image processing unit 104 will be described in detail later.

[0021] The input / output unit 105 is composed of a group of input devices such as switches, buttons, keys, and touch panels for the user to input instructions to the imaging device 100, and a display such as an LCD or an organic EL display. The control unit 106 detects the input through the group of input devices via a bus, and the control unit 106 controls each unit to realize an operation according to the input. The input / output unit 105 may be a touch panel, in which case the touch detection surface becomes the display surface of the display. The touch panel may be any of various touch panels such as a resistive film type, a capacitive type, or an optical sensor type. The input / output unit 105 displays a live view image by sequentially transferring and displaying image data.

[0022] The control unit 106 is composed of a CPU (Central Processing Unit). The control unit 106 executes programs stored in the ROM 103 and realizes the functions of the imaging device 100. The control unit 106 also controls the imaging unit 101 and performs aperture control, focus control, and exposure control. For example, the control unit 106 executes AE (automatic exposure) processing to automatically determine exposure conditions (shutter speed or accumulation time, aperture value, sensitivity) based on subject luminance information of image data obtained by the imaging unit 101. The control unit 106 can also automatically set a focus detection area by using the detection result of the subject area by the image processing unit 104, and can realize a tracking AF processing function for an arbitrary subject area. Furthermore, the control unit 106 can also perform AE processing based on luminance information of the focus detection area, and perform image processing (for example, gamma correction processing and AWB (auto white balance) adjustment processing, etc.) based on pixel values ​​of the focus detection area.

[0023] The control unit 106 also performs display control by controlling the input / output unit 105. For example, the control unit 106 causes an indicator indicating the current position of the subject area (for example, a rectangular frame surrounding the area) to be superimposed on the display image based on the result detected by the image processing unit 104.

[0024] <Functional configuration> 2(a) is a diagram for explaining the functional configuration of the image processing unit according to the first embodiment. The image processing unit 104 includes an image input unit 210, a feature extraction unit 220, a first part estimation unit 230, a priority area estimation unit 240, and a first part selection unit 250. The AF unit 260 may be included in the control unit 106 (not shown). In this embodiment, the first part is a person's eyes, the priority area is an area including eyes relatively close to the image capture device 100 that captured the image (hereinafter, eye-proximate area), and the priority is close to the image capture device 100.

[0025] The image input unit 210 inputs images from a time-series video captured by the imaging unit 101. For example, the image input unit 210 inputs image data of 1600×1200 pixels in real time (60 frames per second). The feature extraction unit 220 processes the images input by the image input unit 210 and extracts image features. The first part estimation unit 230 estimates eye frames that are highly likely to represent the center of the eye frame based on the features extracted by the feature extraction unit 220, and selects one or two of them.

[0026] Based on the features extracted by the feature extraction unit 220, the priority area estimation unit 240 estimates the likelihood that the area contains a specific part that is relatively close to the imaging device 100 that captured the image (the likelihood that the area is an eye-proximity area), and outputs the estimate as an area map.

[0027] First part selection section 250 selects one eye from the multiple eyes obtained by first part estimation section based on the priority order. In this embodiment, one eye close to imaging device 100 is selected based on the likelihood (score) indicating the likelihood of being an eye-proximate region obtained by priority region estimation section 240. First part selection section 250 acquires the likelihood on the region map for each position of the multiple predetermined parts, and selects the predetermined part (one eye) with the greater likelihood from the multiple predetermined parts (left and right eyes).

[0028] If no eyes are obtained by first part estimation section 250, a part is selected based on the area information obtained from priority area estimation section 240. AF section 260 performs AF processing on the part selected by first part selection section 250.

[0029] <Processing> An example of selecting a notable eye frame from a captured video and performing AF processing in the imaging device 100 according to an embodiment of the present invention will be described below. FIG. 3 shows a flowchart illustrating the flow of processing. In the following description, each process (step) is represented by adding an S to the beginning, and the description of the process (step) will be omitted. However, the imaging device 100 does not necessarily have to perform all of the processes described in this flowchart. (S300) In S300, the image input unit 210 acquires an image from a time-series video captured by the imaging unit 101, and inputs the image to the feature extraction unit 220. The image acquired in S300 is, for example, bitmap data expressed in RGB 8 bits.

[0030] (S301) In S301, the feature extraction unit 220 processes the image input by the image input unit 210 and extracts the feature of the image. As a feature extraction method, various known methods are possible, such as a feature vector that is an aggregate of pixel colors, textures, etc. Any method can be applied as long as it is a map-like feature, and the method is not limited to a specific method.

[0031] The feature amount is information for estimating the eye region and the eye-proximate region. For the eye region, the likelihood of the region being the center of the frame (center position map), the width of the frame, and the height of the frame (size map) are estimated and output as a map. For the eye-proximate region, the likelihood (score) of the region being an eye-proximate region where an eye close to the imaging device 100 exists is estimated and output as an eye-proximate region map.

[0032] 6(a) and 6(b) are schematic diagrams showing the face direction and the inference results of the eye-proximate region. In Fig. 6(a), 600 is an input image showing the face of a person facing to the right, 601 is the eye (right eye) closer to the image capture device 100, 602 is the eye (left eye) farther from the image capture device 100, and 603 is the nose. In addition, 604 is a region with a high score for the eye-proximate region (a region drawn with dots in the figure), and 605 is a region with a low score for the eye-proximate region (a region drawn with vertical lines in the figure).

[0033] In addition, in Fig. 6(b), 606 is an input image showing the face of a person facing forward, 607 is the right eye, 608 is the left eye (the right eye 607 and the left eye 608 are almost equidistant from the image capture device 100), and 609 is the nose. Also, 610 is an area with a high eye-proximate score (an area drawn with dots in the figure). As shown in Fig. 6(b), when facing forward, the distance between the right eye and the image capture device and the distance between the left eye and the image capture device are approximately the same. Therefore, the entire range of the face is an area with a high likelihood of being an eye-proximate area.

[0034] 6(a), when facing right, the distance between the face surface including the right eye and the imaging device 100 becomes relatively small, and the distance between the face surface including the left eye and the imaging device 100 becomes relatively large. Therefore, the left half of the face becomes an area with a high likelihood of being an eye-proximate area, and the right half of the face becomes an area with a low likelihood of being an eye-proximate area. In this embodiment, this relationship is utilized to select the eye closest to the imaging device 100.

[0035] (S302) In S302, first part estimation unit 230 selects one or two eye frames in order of the likelihood of the eye frame being the center, calculated based on the features extracted by feature extraction unit 220, exceeding a preset threshold value and in descending order of the likelihood of the eye frame being the center.

[0036] In this embodiment, the functions of the feature extraction unit 220, the first part estimation unit 230, and the priority area estimation unit 240 are realized using a neural network. Here, FIG. 4 shows a schematic diagram of an example of a network structure using a neural network and the output results of a center position map, a size map, a distance map, and an eye frame. 400 represents an input image, 401 and 402 represent a person's eyes, and 403 represents a person's nose. 404 is a center position map representing the magnitude of the likelihood that the eyes are the center. 405 and 406 represent the magnitude of the likelihood of the person's eyes 401 and 402 in the input image 400. Also, 407 is a size map representing the width of the eyes.

[0037] Reference numerals 408 and 409 represent the width between the person's eyes 401 and 402 in the input image 400. Reference numeral 410 represents a size map representing the height of the eyes. Reference numerals 411 and 412 represent the height between the person's eyes 401 and 402 in the input image 400. Reference numeral 413 represents an eye proximity region map representing a region including eyes relatively close to the image capture device 100. Reference numeral 414 represents the magnitude of the likelihood that the region includes eyes relatively close to the image capture device 100. Reference numeral 416 represents a result of superimposing an eye frame 417 obtained by first part estimation section 230 on the input image 400.

[0038] The neural network in FIG. 4 has a network structure used in the face part detection process described in Non-Patent Document 1. Such a network outputs intermediate features by inputting an image into a network called a backbone. This backbone part corresponds to the feature extraction unit 220. The features obtained through the backbone are input into networks separated by tasks for estimating the eye position, eye size, and eye proximity area. The network for estimating the eye position and eye size corresponds to the first part estimation unit 230. The network for estimating the eye proximity area corresponds to the priority area estimation unit 240. From these networks, an eye center position map 404 indicating the center position of the eye, two eye size maps 407 and 410 indicating the width and height of the frame surrounding the eye, and an eye proximity area map 413 indicating the likelihood of an area including an eye relatively close to the imaging device 10 are obtained. Each map is a two-dimensional array and is expressed as a grid.

[0039] The eye center position map 404 is used to infer the likelihood of the sequence being the center of the eye. The eye center position map 404 indicates that the closer to the center of the circle the higher the likelihood, and thus the higher the likelihood of the location of the eye.

[0040] The eye size maps 407 and 410 are two maps of width and height, and the width and height of the eye are inferred when the position on the map is the center of the eye. The size map indicates the magnitude of the value by the length of the arrow. Values ​​indicating the width and height are inferred at the center position of the eye.

[0041] In the eye-proximal region map 413, a likelihood that a position on the map is an eye-proximal region is inferred. In this way, the number of maps required to estimate the eye close to the image capture device 100 is four in total.

[0042] The eye frame is defined by the center coordinates of a rectangle surrounding the eye in the image, and the width and height of the rectangle. The center position map estimates the likelihood that the eye is the center of the eye. A threshold is set in advance for the likelihood, and elements having a value exceeding the threshold are obtained as candidates for the center position of the eye. When center position candidates are obtained between adjacent elements, the element with the higher likelihood is determined to be the center position of the eye. Since the resolution of the center position map is lower than the resolution of the original image, the center position of the eye on the image is obtained by scaling the center position obtained from the center position map to the image size. In addition, the width and height of the frame surrounding the eye are obtained from the element of the size map corresponding to the detected center position of the eye, and a candidate eye frame is obtained.

[0043] (S303) In S303, first part estimation section 230 determines whether or not one or more eyes have been detected in accordance with a control signal from control section 106. If it is determined that one or more eyes have been detected, the process proceeds to S304. On the other hand, if one or more eyes have not been detected, the process proceeds to S311.

[0044] (S304) In S304, first part estimation section 230 determines, in accordance with a control signal from control section 106, whether or not two eyes have been detected.

[0045] (S305) In S305, the first part selection unit 250 obtains a score of the eye proximity region map obtained by the priority region estimation unit 240 based on the central position of the two eye frame candidates obtained in S302. In this embodiment, this is realized by obtaining a score from the element of the eye proximity region map that corresponds to the x coordinate and y coordinate of the central position of the two eye candidate frame obtained in S302.

[0046] (S306) In S306, the first part selection unit 250 calculates the absolute value of the difference between the scores of the two eyes obtained in S305, and determines whether the absolute value of the score difference is equal to or greater than a preset threshold. If the absolute value of the score difference is equal to or greater than the threshold, the process proceeds to S307. On the other hand, if the absolute value of the score difference is less than the threshold, the process proceeds to S308.

[0047] (S307) In S307, the first part selection section 250 selects one eye having a large score as an AF region candidate.

[0048] (S308) In S308, the first part selection unit 250 selects an eye close to the image center as an AF area candidate using, for example, the method described in Patent Document 1. If the first part selection unit 250 does not determine in S306 that the absolute value of the difference in the scores of the two eyes is equal to or greater than the threshold, this means that the distances between the two eyes and the imaging device are almost the same. Therefore, since the distances from the imaging device 100 to the two eyes are approximately equal, there is no significant difference even if either eye is focused on. Therefore, as an example, an eye close to the image center is selected as an AF area candidate. Note that the process does not need to be limited to selecting an eye close to the image center, and in S308, the first part selection unit 250 may select an eye with a high score in the eye proximity area map as an AF area candidate.

[0049] (S309) In S309, the first part selection section 250 selects one eye detected in S302 as an AF area candidate.

[0050] (S310) In S310, AF section 260 performs AF processing using one eye selected by first part selection section 250 as an AF candidate area, or using the center of gravity position of the eye proximity area map as an AF candidate area. As an AF processing method, for example, AF using a phase difference detection method described in Patent Document 1 can be used.

[0051] (S311) In S311, the first part selection unit 250 selects an AF candidate area based on the eye proximity area map. For example, the center of gravity of an area equal to or greater than a preset threshold (0.5) is calculated, and the center of gravity is set as the AF candidate area. Alternatively, a position that has the maximum likelihood of the eye proximity area map may be selected as the AF candidate area.

[0052] <Learning method of embodiment 1> Next, a learning method according to this embodiment will be described. Fig. 13 is a diagram showing the configuration of the imaging device 100 according to this embodiment, the image processing unit 104 included in the imaging device 100, and a learning device 500 for performing learning. The image processing unit 104 acquires image data to be processed and outputs the processing result. The learning device 500 learns parameters for the image processing unit 104 to perform processing. Hereinafter, a method for learning parameters for enabling the image processing unit 104 to execute processing will be described.

[0053] <Functional configuration of the learning device> 5 is a diagram showing the functional configuration of a learning device for performing learning according to this embodiment. Learning device 500 includes learning data storage unit 510, learning data acquisition unit 520, image acquisition unit 530, detection target estimation unit 540, teacher data creation unit 550, error calculation unit 560, and parameter learning unit 570. Here, detection target estimation unit 540 performs the same processes as feature amount extraction unit 220, first part estimation unit 230, and priority area estimation unit 240 in image processing unit 104 in FIG.

[0054] The learning data storage unit 510 stores and holds learning data for the learning device 500 to learn the image processing unit 104. The learning data includes a set of a learning image and correct answer information of the eyes of a person in the learning image to be analyzed (coordinates of the eye center position, eye size (eye width and height), coordinates of the face center position, face size (width and height of the face), and coordinates of the nose center). It is assumed that the learning data storage unit 510 stores a sufficient number and variety of learning data for the learning device 500 to learn the image processing unit 104.

[0055] The learning data acquisition unit 520 acquires the learning data stored in the learning data storage unit 510 .

[0056] The image acquisition unit 530 acquires a training image from the training data acquisition unit 520. The acquired image is input to the detection target estimation unit 540. Note that the image acquisition unit 530 may perform data expansion on the image before inputting it to the detection target estimation unit 540. For example, robustness can be improved by enlarging or reducing the image, adding noise, or changing the brightness or color of the image. Note that when data expansion involving geometric transformation such as image rotation or enlarging or reducing is performed, the correct answer information of the training data must also be converted to correspond to the geometric transformation.

[0057] The detection target estimation unit 540 is configured to include the functions of the feature amount extraction unit 220, the first part estimation unit 230, and the priority area estimation unit 240 described with reference to Fig. 2(a). The detection target estimation unit 540 outputs an eye center position map, an eye size map, and an eye proximity area map indicating an area including eyes relatively close to the imaging device 100 of the person in the training image. In the first embodiment, the detection target estimation unit 540 is configured by the above-mentioned neural network.

[0058] The teacher data generating unit 550 generates, from the correct answer information obtained from the learning data acquiring unit 520, an eye center position correct answer map, an eye size correct answer map, and an eye proximity area correct answer map as teacher data that serve as target values.

[0059] The error calculation unit 560 calculates a center position error, which is the error between the eye center position map output by the detection object estimation unit 540 and the eye center position correct map created by the teacher data creation unit 550. The error calculation unit 560 also calculates a size error, which is the error between the size map output by the detection object estimation unit 540 and the eye size correct map created by the teacher data creation unit 550. The error calculation unit 560 also calculates a region error, which is the error between the eye proximity region map output by the detection object estimation unit 540 and the eye proximity region correct map created by the teacher data creation unit 550.

[0060] The parameter learning section 570 updates the parameters of the detection object estimation section 540 so as to reduce the center position error, size error, and area error.

[0061] FIG. 7 shows a schematic diagram of the training image and correct answer information of the eye center position correct map. Reference numeral 700 represents a training image showing a person's face. Reference numeral 710 represents correct answer information on input image coordinates. Reference numeral 720 represents correct answer information on correct answer map coordinates. Reference numeral 730 represents a correct answer map of the eye center position when the vicinity of the person's eyes is enlarged. Reference numeral 740 represents a correct answer map of the eye size when the vicinity of the person's eyes is enlarged.

[0062] As learning data, a learning image 700 in which a person is shown, and the person's right eye center coordinates (X, Y) = (900, 430), eye size (25), the person's left eye center coordinates (X, Y) = (960, 340), and eye size (20) are given as correct answer information. Also, the person's face center coordinates (X, Y) = (910, 440), face size (240), and the person's nose center coordinates (X, Y) = (985, 440) are given as correct answer information. The eye center coordinates, face center coordinates, and nose center coordinates are coordinates on the input image. Also, the eye size and face size are sizes on the input image. In the first embodiment, the size of the input image is 1600 × 1200 pixels. The eye center position correct answer map, which is teacher data of the eye center position, is matrix data of the same size as the eye center position map output by the detection target estimation unit 540. In the first embodiment, the size is 320 x 240, which is 1 / 5 of the input image in both length and width. Therefore, the eye center coordinates and eye size on the map must also be 1 / 5 of the input image. Specifically, the eye center coordinates are (X, Y) = (180, 60), and the eye size is 4.

[0063] The eye center position correct map 730 is a map in which correct cases are labeled at the eye center positions. As shown in the figure, a heat map of a circular area with a diameter of eye size 4, centered at (X, Y) = (180, 60), is assigned as a label. Since the correct map takes a value range of 0 to 1, the center is set to the maximum value of 1, and the value gradually decreases as it approaches the circumferential direction of the heat map.

[0064] In the eye size correct answer map 740, a label of a correct case is assigned within a frame whose center is the eye center coordinate (X, Y) = (180, 60) and whose length of one side is the same as the eye size. As shown in the figure, the value of the label is the value (0.1) obtained by dividing the eye size on the map (20) by the maximum size on the map (200). In addition, elements outside the frame are set to blank values ​​so as not to contribute to learning.

[0065] Next, a method for creating an eye-proximate area correct map will be described. Fig. 12(a) to Fig. 12(c) are schematic diagrams of a method for creating an eye-proximate area correct map. In Fig. 12(a), 1200 is an input image showing a person's face facing right. 1201 is an eye (right eye) closer to the imaging device 100, 1202 is an eye (left eye) farther from the imaging device 100, and 1203 is a nose. In Fig. 12(b), 1204 (dashed line) is a line segment connecting the right eye and the left eye, and 1205 (dash-dotted line) is a line segment perpendicular to 1204 (dashed line) and passing through the center position of the nose.

[0066] In Fig. 12(c), 1206 is a region (a region drawn with dots in the figure) that includes eyes relatively close to the image capture device 100 in a circle whose center is the face center position and whose radius is the face size. 1207 is a region (a region drawn with vertical lines in the figure) that includes eyes relatively far from the image capture device 100 in a circle whose center is the face center position and whose radius is the face size. 1208 is a neighborhood region (a region drawn with diagonal lines in the figure) of 1205 (dashed line) between the right eye and the left eye in a circle whose center is the face center position and whose radius is the face size. 1209 is an eye proximity region correct map.

[0067] <Processing> 14 is a flowchart showing the process flow of the method for creating an eye-proximity area correct map. In S1401, the teacher data creation unit 550 draws a line segment 1204 connecting the right eye 1201 and the left eye 1202. In S1402, the teacher data creation unit 550 draws a perpendicular line 1205 that intersects the line segment 1204 connecting the right eye 1201 and the left eye 1202 and passes through the center of the nose. In S1403, the teacher data creation unit 550 calculates the inter-eye distance between the right eye 1201 and the nose 1203 and the inter-eye distance between the left eye 1202 and the nose 1203, and stores the calculation results.

[0068] In S1404, the teacher data creation unit 550 determines whether the difference between the eye-to-nose distance calculated in S1403 is equal to or greater than a threshold. If the difference between the eye-to-nose distance is equal to or greater than the threshold, the process proceeds to S1405. On the other hand, if the difference between the eye-to-nose distance is less than the threshold, the process proceeds to S1408. In S1405, the teacher data creation unit 550 calculates the eye-to-eye distance, which is the distance between the right eye 1201 and the left eye 1202, and holds the calculation result.

[0069] In S1406, the teacher data creation unit 550 sets the vicinity of the perpendicular line 1205 as a null region so as not to contribute to learning. This is because learning is difficult to converge if the boundary region between the positive region and the negative region is used for learning, and setting the null region makes learning easier. As a method for setting a null region near the perpendicular line 1205, the distance between the eyes calculated in S1405 is used. For example, in a circle whose center is the face center position and whose radius is the face size, a region that is included in the circular region and whose distance from the perpendicular line 1205 drawn in S1402 is a predetermined multiple (for example, 0.2 times) of the distance between the eyes (for example, 108) is set as a null region. Also, a region whose distance from the perpendicular line 1205 is within a predetermined value without using the distance between the eyes may be set as a null region. 1208 in FIG. 12(c) corresponds to that region. Note that in this embodiment, an example of using a null region has been shown, but learning may be performed without setting a null region.

[0070] In S1407, the teacher data creation unit 550 determines the region where the eye (right eye 1201) with the larger distance between the eye and the nose is located based on the distance calculated in S1403 as a positive region, and the region where the eye (left eye 1202) with the smaller distance between the eye and the nose is located as a negative region. For example, the positive region is determined to be 1206, which is a region including the right eye 1201 close to the imaging device 100 and does not overlap with the null region. The positive region is labeled with a value of (1) as a positive example. Then, the negative region is labeled with a value of (0) as a negative example. Note that in this embodiment, an example in which the negative region is labeled with (0) has been described, but the negative region may be labeled with a label of (-1) and the region that is neither a positive region nor a negative region may be labeled with a label of (0).

[0071] In S1408, the teacher data creation unit 550 determines the inside of a circle whose center is the face center position and whose radius is the face size as a positive area. The positive area is labeled with the value (1) as a positive example. This completes the series of processes in FIG. 14.

[0072] <Processing> 8 shows a flowchart illustrating the flow of the learning process. In S801, the learning data acquisition unit 520 acquires learning data stored in the learning data storage unit 510. In S802, the image acquisition unit 530 acquires a learning image from the learning data acquisition unit 520. In S803, the detection target estimation unit 540 estimates a detection target from the learning image acquired by the image acquisition unit 530, and outputs an eye center position map, an eye size map, and an eye proximity area map. In S804, the teacher data creation unit 550 creates an eye center position correct map, an eye size correct map, and an eye proximity area correct map from the correct answer information of the learning data acquired in S801 according to the teacher data creation method described above.

[0073] In S805, the error calculation unit 560 calculates a center position error based on the eye center position correct map created in S804 and the eye center position map 404 obtained in S803. The error calculation unit 560 also calculates a size error based on the eye size correct map created in S804 and the eye size map 407 obtained in S803. Furthermore, the error calculation unit 560 calculates a region error based on the eye proximity region correct map created in S804 and the eye proximity region map 413 obtained in S803.

[0074] In S806, parameter learning unit 570 learns the parameters of detection object estimation unit 540 so as to reduce the center position error, size error, and eye proximity region error obtained in S805. In S807, parameter learning unit 570 determines whether to continue learning, and if so, executes each step again from acquiring learning data in S801. If learning is not to be continued, processing ends. The determination of whether to continue learning may be based on a pre-specified number of learning times or learning time, or on other criteria. This completes the series of processes in FIG. 8.

[0075] In this embodiment, the center positions of multiple eyes and the output of the eye proximity region are used in steps S302 to S308 to select the eye closest to the image capture device 100. However, the part is not limited to the eyes, and can be applied to any part that exists on the face. For example, the left and right ears (not shown) and the ear proximity regions may be inferred, and the ear with the larger score of the ear proximity region may be selected as the ear closest to the image capture device 100.

[0076] In addition, in this embodiment, when creating the eye proximity area correct map, the eye closest to the imaging device 100 is selected using the eye-to-nose distance, but this is not limited to the eye-to-nose distance. Any part that is approximately equidistant from both eyes when facing forward can be applied. For example, the mouth, chin, top of the head, and space between the eyebrows (not shown) can be included, and the distance between the above parts and the eyes can be used to infer and select the area with the eyes that are far away as the area with the eyes closest to the imaging device 100. In addition, in this embodiment, a person has been described as the target, but if the target is a bird, the distance between the tip of the beak and the eyes can be used.

[0077] Furthermore, in this embodiment, if the first part selection unit 250 does not determine in S306 that the difference between the two eyes is equal to or greater than the threshold, the eye closest to the center of the image is selected as the AF area candidate in S308. However, a tracking unit (not shown) that tracks a subject in a video may be used to select the eye closest to the coordinates of the eye tracked in the previous frame (previous frame). Alternatively, the larger eye may be selected.

[0078] In this embodiment, if one or more eyes are detected in S303, the eyes are selected as the AF candidate area. If one or more eyes are not detected in S303, the center of gravity of the eye proximity area equal to or larger than the threshold is selected as the AF candidate area. If no eyes are detected and there is no eye proximity area equal to or larger than the threshold, frame information may be acquired from a face frame estimation unit, a head frame estimation unit, a person area estimation unit, or the like (not shown) and selected as the AF candidate area. Specifically, the AF candidate area may be selected by assigning priority to the smallest frame size based on various frame information of the face frame, head frame, and person area surrounding the entire body of the person acquired from each estimation unit.

[0079] In this embodiment, whether or not two eyes are detected is determined in S304, but this is based on the assumption that one person is photographed. If multiple people are photographed, there is a possibility that more than two eyes are detected, but in that case, the eye closest to the imaging device 100 can be selected using a similar method. For example, there is a method of acquiring a face frame, a head frame, or a person area by a face frame estimation unit, a head frame estimation unit, or a person area estimation unit (not shown). Specifically, as a process prior to S304, there is a method of limiting the eyes to those included within the face frame, head frame, or person area acquired from each estimation unit. With this method, even if multiple people's eyes are detected, it is possible to limit the eyes to two eyes of one person.

[0080] In the present embodiment, the priority order for selecting one eye from a plurality of eyes is an example in which the eye closer to the imaging device 100 is given priority. However, the present invention is not limited to this example, and the priority order may be an eye farther from the imaging device 100. Also, the priority may be an eye included in the face frame.

[0081] As described above, in this embodiment, the eye proximity region map is estimated, and the eye with the higher likelihood (score) of the eye proximity region map is selected as the eye closer to the imaging device.

[0082] By performing such processing, the eye-proximate region can be learned and estimated from information about the area around the eyes even when the nose is hidden, so that the eye closest to the imaging device can be selected with high accuracy.

[0083] Even if eye detection fails, the center of gravity of the eye proximity area map can be used as the AF candidate area to achieve a better AF position than the face or head. Furthermore, compared to face direction estimation using multiple facial organ points, the cost of preparing correct answer data can be reduced, while the number of maps required for estimation can be reduced, allowing for faster selection of eye areas.

[0084] <Modification of the first embodiment> In the first embodiment, the teacher data creation unit 550 creates an eye-proximity region correct map using two-dimensional coordinates, which are coordinates on the input image, in S1401 to S1407 in Fig. 14. In this modification, an example will be described in which teacher data is created and learned based on correct information using three-dimensional coordinates.

[0085] FIG. 15 is a schematic diagram of an example of creating an eye-proximate area correct map. Reference numeral 600 denotes an input image showing a face of a person facing rightward. Reference numeral 601 denotes an eye (right eye) closer to the imaging device 100, 602 denotes an eye (left eye) farther from the imaging device 100, and 603 denotes a nose. Reference numeral 1501 denotes an area (area drawn with dots in the figure) including an eye relatively closer to the imaging device 100 in a circular area having a center at the face center position and a radius of the face size. Reference numeral 1502 denotes an area (area drawn with vertical lines in the figure) including an eye relatively farther from the imaging device 100 in a circular area having a center at the face center position and a radius of the face size. Reference numeral 1503 denotes an area (area drawn with horizontal lines in the figure) near the midpoint between the right eye and the left eye in a circular area having a center at the face center position and a radius of the face size.

[0086] 15, when facing right, the distance between the face area including the right eye and the image capture device 100 becomes relatively small, and the distance between the face area including the left eye and the image capture device 100 becomes relatively large. In this modification, this relationship is utilized to select the eye closest to the image capture device 100.

[0087] <How to create an area> A method for creating the near region 1501, far region 1502, and nearby region 1503 will be described below. The teacher data creation unit 550 creates the near region 1501, far region 1502, and nearby region 1503 by using the three-dimensional coordinates of the midpoints of the eyes. For example, it is assumed that a learning image 700 (FIG. 7) containing a person, the center coordinates of the person's right eye (X, Y, Z)=(900, 430, 150), and the center coordinates of the person's left eye (X, Y, Z)=(960, 340, 200) are given as learning data as correct answer information. It is also assumed that the center coordinates of the person's face (X, Y, Z)=(910, 440, 300) and the face size (240) are given as correct answer information. The eye center coordinates, face center coordinates, and nose center coordinates are coordinates on the input image. The eye size and face size are sizes on the input image.

[0088] The three-dimensional coordinates of the object are assumed to be acquired in advance using various distance measurement devices. Various distance measurement devices can be used. Specifically, a distance measurement device using a binocular stereo method, an active distance measurement device using a pattern projection method, etc. can be used.

[0089] The teacher data creation unit 550 calculates the coordinate of the midpoint between the right eye and the left eye. Here, the Z-axis coordinate of the midpoint is (175) by calculating (150+200) / 2. The teacher data creation unit 550 uses a value (160) obtained by subtracting a predetermined value (15) from the Z-axis coordinate (175) of the midpoint between the right eye and the left eye as a reference, and determines an area whose Z-axis coordinate is smaller than 160 and is included in the circular area of ​​the face as a near area 1501. Also, using a value (190) obtained by adding a predetermined value (15) to the Z-axis coordinate (175) of the midpoint between the right eye and the left eye as a reference, a area whose Z-axis coordinate is larger than (190) and is included in the circular area of ​​the face as a far area 1502. Also, a area whose Z-axis coordinate is equal to or larger than (160) and equal to or smaller than (190) and is included in the circular area of ​​the face as a near area 1503.

[0090] As described above, in this modified example, a correct eye-proximal region map is generated using three-dimensional coordinates. By performing such processing, a highly accurate correct eye-proximal region map that takes into account three-dimensional directional information can be learned, compared to a method that uses two-dimensional coordinates.

[0091] Furthermore, the unit for generating correct information for the eye proximity area map is not limited to using this method, and may also use a method in which, for example, a three-dimensional model of a face created using CG is fitted to a two-dimensional image of a face. The three-dimensional model information can be used to determine which face is closer to the imaging device, so a correct map of the eye proximity area can be created.

[0092] (Embodiment 2) In the second embodiment, a method of selecting a first part to be prioritized based on the result of estimating a region of a specific category as the priority region, rather than on a proximity region map of the first part, will be described. In this embodiment, the first part is a person's eyes, the priority region is the person's skin region, and the priority is that there is a lot of skin region around the person's eyes.

[0093] In this embodiment, multiple predetermined parts of the detection target (e.g., parts of the same type (left and right eyes, left and right ears, etc.)) are estimated from the image based on the feature amounts extracted from the image, and further, a priority area (e.g., skin area) is estimated from the image based on the feature amounts. Then, one predetermined part (one eye) is selected from the multiple predetermined parts (left and right eyes) based on the priority area.

[0094] 2(b) is a diagram showing the functional configuration of an image processing unit 270 according to this embodiment. The image processing unit 270 corresponds to the image processing unit 104 of the first embodiment. The image processing unit 270 includes an image input unit 210, a feature extraction unit 271, a first part estimation unit 230, a priority area estimation unit 240, an area calculation unit 273, and a first part selection unit 250. The AF unit 260 is included in, for example, the control unit 106 (not shown). The feature extraction unit 271, the first part estimation unit 230, and the priority area estimation unit 240 are configured by neural networks as in the first embodiment.

[0095] Fig. 9 shows an example of a network structure using a neural network. In Fig. 9, reference numerals are given corresponding to Fig. 4. 900 to 912, 916, and 917 in Fig. 9 are the same as 400 to 412, 416, and 417 described in Fig. 4.

[0096] Reference numeral 918 denotes a skin region map indicating the skin region of a person, and 919 denotes the magnitude of the likelihood of the skin region of a person. The network as a whole outputs a skin region map indicating the likeliness of skin, in addition to an eye center position map indicating the center position of the eye, and two eye size maps indicating the width and height of the frame surrounding the eye.

[0097] The priority region estimation unit 240 outputs a person's skin region map 918 based on the feature amount extracted by the feature amount extraction unit 271. The region calculation unit 273 calculates the number of pixels in the skin region around the eye frames (the number of pixels in the priority region) based on the two eye frames with high likelihood obtained by the first part estimation unit 230 and the skin region estimated by the priority region estimation unit 240.

[0098] <Processing> Next, Fig. 10 shows a flowchart showing the flow of processing according to the second embodiment. In Fig. 10, step numbers are assigned in correspondence with the flowchart in Fig. 3 described in the first embodiment. Steps S1000 to S1004 and S1008 to S1010 in Fig. 10 are the same processing as steps S300 to S304 and S308 to S310 in Fig. 3. In this embodiment, if it is determined in S1004 that two eyes have been detected, the process proceeds to S1011. On the other hand, if it is determined that two eyes have not been detected, the process proceeds to S1009.

[0099] In S1009, the first part selection unit 250 selects one of the detected eyes. In S1011, the priority area estimation unit 240 acquires a person's skin area map based on the feature amount extracted by the feature amount extraction unit 271. The skin area is acquired as a skin area map when the likelihood exceeds a preset threshold. The priority area estimation unit 240 is an estimation unit that has been trained to output the person's skin area as a map, and is trained using, for example, the learning method described in Non-Patent Document 3. The technology described in Non-Patent Document 3 is a technology called semantic segmentation, which is a technology that determines what category of object is captured in each pixel in an image. In this embodiment, regression learning is performed to output a likelihood corresponding to the likelihood that the skin area is a single category of a person. During inference, the likelihood of the person's skin area is output as a map.

[0100] In S1012, the area calculation unit 273 counts the number of pixels of the skin area around the two eyes (the number of pixels of the priority area) from the two eyes obtained in S1002 and the skin area obtained in S1011. In this embodiment, the number of pixels of the skin area included in a rectangular area surrounding the center position of the eye is counted.

[0101] FIG. 11 shows a schematic diagram of the peripheral areas of the two eyes and the skin area. Reference numeral 1100 denotes an input image showing the face of a person facing to the right. Reference numeral 1101 denotes the right eye, 1102 denotes the left eye, 1103 denotes a rectangular area for counting the right eye, and 1104 denotes a rectangular area for counting the left eye. Reference numeral 1105 denotes a skin area (area drawn with dots in the figure). As the size of the rectangular area, the Euclidean distance between the right eye 1101 and the left eye 1102 is calculated, and counting is performed as a rectangle whose width and height are, for example, the size obtained by multiplying the distance by (0.5). In this embodiment, the rectangular area of ​​the right eye is entirely the skin area, and the rectangular area of ​​the left eye includes a background area in addition to the skin area. Therefore, the number of skin areas counted is greater for the right eye than for the left eye.

[0102] In S1013, the region calculation unit 273 determines whether the difference in the counts between the skin regions of one eye and the other eye is equal to or greater than a threshold. If the difference in the counts is equal to or greater than the threshold, the process proceeds to S1007. On the other hand, if the difference in the counts is less than the threshold, the process proceeds to S1008. For example, if the threshold is set to (40), and the counts of the skin region of one eye are (81) and the counts of the region adjacent to the other eye are (35), the difference is (46), and it is determined that the difference is equal to or greater than the threshold. In this embodiment, the method of comparing the counted value with the threshold has been exemplified, but the counts may be divided by the area of ​​a rectangle to obtain a ratio, and then compared with the threshold. In other words, the ratio of the priority region to the respective predetermined peripheral regions of a plurality of predetermined parts may be calculated.

[0103] In S1007, first part selection section 250 selects the eye with the largest skin region count. In the above example, the eye with the count (81) is selected. If there is a lot of background around the eye as an AF target region, there is a risk that AF will fail due to the background being mistakenly detected. Therefore, by selecting the eye with the largest skin region count, the number of AF failures can be reduced. Alternatively, first part selection section 250 may select a specified part that has a high proportion of the priority region calculated by region calculation section 273 from among multiple specified parts.

[0104] In S1008, first part selection unit 250 selects the eye closest to the center of the image. Alternatively, a tracking unit (not shown) that tracks a subject in a video may be used to select the eye closest to the coordinates of the eye tracked in the previous frame (previous frame). Alternatively, the larger eye may be selected.

[0105] In S1010, AF section 260 performs AF processing on one eye, which is the AF candidate area selected by first part selection section 250.

[0106] In this embodiment, an example in which a skin region is used as a priority region has been described, but the application of this embodiment is not limited to skin regions. For example, a person region (whole body region), a head region, a face region, etc. can also be used as a priority region. By counting the number of person regions near the eyes, the same effect as that of a skin region can be obtained.

[0107] As described above, in this embodiment, the number of pixels in the priority area in the predetermined peripheral area of ​​each of the plurality of predetermined parts is calculated. Then, when the difference between the number of pixels in the priority area in the peripheral area of ​​a first predetermined part (e.g., the left eye) among the plurality of predetermined parts and the number of pixels in the priority area in the peripheral area of ​​a second predetermined part (e.g., the right eye) is equal to or greater than a threshold, the predetermined part with the larger number of pixels is selected.

[0108] More specifically, in this embodiment, the skin area (priority area) around the eyes is counted, and the eye with the most skin area around it is selected. By performing such processing, erroneous AF processing on the background can be prevented, and AF processing can be performed on the eye intended by the user.

[0109] <Other embodiments> Although the embodiment has been described above in detail, the present invention can be embodied as, for example, a system, an apparatus, a method, a program, or a recording medium (storage medium), etc. Specifically, the present invention may be applied to a system composed of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to an apparatus composed of a single device.

[0110] The disclosure of this specification includes the following image processing device, imaging device, learning device, control method, and program.

[0111] (Item 1) A feature extraction means for extracting features from an image; a part estimation means for estimating a plurality of predetermined parts of a detection object from the image based on the feature amount; a priority area estimation means for estimating a priority area from the image based on the feature amount; a selection means for selecting one predetermined portion from the plurality of predetermined portions based on the priority region; An image processing device comprising:

[0112] (Item 2) 2. The image processing device according to item 1, wherein the priority region estimation means estimates, as the priority region, a region including a predetermined part that is relatively close to an imaging device that has acquired the image.

[0113] (Item 3) 3. The image processing device according to item 1 or 2, wherein the plurality of predetermined regions are a plurality of predetermined regions of the same type.

[0114] (Item 4) 4. The image processing device according to item 3, wherein the plurality of predetermined parts of the same type are the left and right eyes or the left and right ears.

[0115] (Item 5) the priority region estimation means estimates a likelihood representing a likelihood that a region includes a predetermined portion relatively close to an imaging device that acquired the image, and outputs the likelihood as a region map; 5. The image processing device according to any one of items 1 to 4, characterized in that the selection means acquires likelihoods on the region map for the respective positions of the plurality of specified parts, and selects the specified part with the greater likelihood from among the plurality of specified parts.

[0116] (Item 6) 6. The image processing device according to item 5, wherein the selection means selects, from the plurality of predetermined regions, a predetermined region with a greater likelihood when a difference in likelihood on the region map at each position of the plurality of predetermined regions is equal to or greater than a threshold value.

[0117] (Item 7) 7. The image processing device according to item 6, characterized in that, when the difference is less than a threshold value, the selection means selects a predetermined part that is closest to the center of the image, or a predetermined part that is closest to the predetermined part selected in the image of the previous frame.

[0118] (Item 8) further comprising area calculation means for calculating the number of pixels of the priority area in a predetermined peripheral area of ​​each of the plurality of predetermined sites, or a ratio of the priority area to the predetermined peripheral area of ​​each of the plurality of predetermined sites, 5. The image processing device according to any one of items 1 to 4, characterized in that the selection means selects, from among the plurality of specified regions, a specified region in which the number of pixels of the priority region calculated by the region calculation means is large or the proportion of the priority region is high.

[0119] (Item 9) further comprising area calculation means for calculating the number of pixels of the priority area in a predetermined peripheral area of ​​each of the plurality of predetermined parts; The image processing device described in any one of items 1 to 4, characterized in that the selection means selects the specified part with the larger number of pixels when a difference between the number of pixels of the priority region in the peripheral region of a first specified part among the plurality of specified parts and the number of pixels of the priority region in the peripheral region of a second specified part among the plurality of specified parts is equal to or greater than a threshold value.

[0120] (Item 10) 10. The image processing device according to item 9, wherein the selection means selects a predetermined part that is closest to the center of the image or a predetermined part that is closest to the predetermined part selected in the image of the previous frame when the difference is less than a threshold value.

[0121] (Item 11) 11. The image processing device according to item 9 or 10, wherein the priority region is a skin region, a head region, a face region, or a whole body region of the detection target.

[0122] (Item 12) An image processing device according to any one of items 1 to 11, an imaging apparatus comprising: an AF unit that performs AF processing on the predetermined portion selected by the selection unit;

[0123] (Item 13) A learning device, comprising: An acquisition means for acquiring an image and correct answer information from a storage unit that stores an image including a predetermined portion and correct answer information; an estimation means for estimating a center position map of the predetermined part in the image, a size map of the predetermined part, and a proximity area map of the predetermined part; a generating means for generating correct answer maps of the center position, size, and adjacent area of ​​the predetermined portion as teacher data from the correct answer information; A calculation means for calculating each error between each map extracted by the estimation means and corresponding teacher data; learning means for learning parameters of the estimation means based on each of the errors; A learning device comprising:

[0124] (Item 14) 14. The learning device according to item 13, wherein the learning means learns the parameters so as to reduce the errors.

[0125] (Item 15) 15. The learning device according to item 13 or 14, wherein the predetermined part is an eye.

[0126] (Item 16) The learning device according to any one of items 13 to 15, characterized in that when a difference between a first inter-eye distance between the left eye and the nose and a second inter-eye distance between the right eye and the nose is equal to or greater than a threshold, the creation means creates teacher data for the adjacent area by labeling the area in which the eye with the larger inter-eye distance is located as a positive area and labeling the area in which the eye with the smaller inter-eye distance is located as a negative area and labeling it as a negative example.

[0127] (Item 17) Item 17. The learning device according to item 16, wherein the creation means, when the difference is less than a threshold, creates teacher data for the adjacent area by labeling the face area as a positive example as a positive area.

[0128] (Item 18) 18. The learning device according to any one of items 13 to 17, wherein the creation means creates teacher data for the proximity area based on three-dimensional coordinates of the positions of the eyes and nose.

[0129] (Item 19) A control method for an image processing device, comprising: A feature extraction step of extracting features from the image; a part estimation step of estimating a plurality of predetermined parts of the detection object from the image based on the feature amount; a priority region estimation step of estimating a priority region from the image based on the feature amount; a selection step of selecting one predetermined site from the plurality of predetermined sites based on the priority region; 13. A method for controlling an image processing apparatus comprising the steps of:

[0130] (Item 20) A method for controlling a learning device, comprising: An acquisition step of acquiring an image and correct answer information from a storage unit that stores an image including a predetermined portion and correct answer information; an estimation step of estimating a center position map of the predetermined part in the image, a size map of the predetermined part, and a proximity area map of the predetermined part; a creating step of creating correct maps of the center position, size, and adjacent area of ​​the predetermined portion as teacher data from the correct answer information; a calculation step of calculating each error between each map extracted by the estimation step and corresponding teacher data; a learning step of learning parameters for the estimation step based on the errors; A method for controlling a learning device comprising:

[0131] (Item 21) 21. A program for causing a computer to execute the control method according to item 19 or 20.

[0132] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0133] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0134] 100: imaging device, 104: image processing unit, 210: image input unit, 220: feature amount extraction unit, 230: first estimation unit, 240: priority area estimation unit, 250: first part selection unit, AF unit, 260

Claims

1. A feature extraction means for extracting features from an image; a part estimation means for estimating a plurality of predetermined parts of a detection object from the image based on the feature amount; a priority area estimation means for estimating a priority area from the image based on the feature amount; a selection means for selecting one predetermined portion from the plurality of predetermined portions based on the priority region; An image processing device comprising:

2. The image processing apparatus according to claim 1 , wherein the priority area estimation means estimates, as the priority area, an area including a predetermined part that is relatively close to an imaging device that has acquired the image.

3. 2. The image processing apparatus according to claim 1, wherein the plurality of predetermined portions are a plurality of predetermined portions of the same type.

4. 4. The image processing apparatus according to claim 3, wherein the plurality of predetermined parts of the same type are the left and right eyes or the left and right ears.

5. the priority region estimation means estimates a likelihood representing a likelihood that a region includes a predetermined portion relatively close to an imaging device that acquired the image, and outputs the likelihood as a region map; 2 . The image processing device according to claim 1 , wherein the selection means acquires a likelihood on the region map for each of the positions of the plurality of predetermined regions, and selects the predetermined region having the greatest likelihood from among the plurality of predetermined regions.

6. The image processing device according to claim 5, characterized in that the selection means selects, from the plurality of predetermined regions, the predetermined region with the greater likelihood when a difference in likelihood on the region map at each position of the plurality of predetermined regions is equal to or greater than a threshold value.

7. 7. The image processing device according to claim 6, wherein the selection means selects, when the difference is less than a threshold value, a predetermined portion that is closest to a center of the image or a predetermined portion that is closest to a predetermined portion selected in the image of a previous frame.

8. further comprising area calculation means for calculating the number of pixels of the priority area in a predetermined peripheral area of ​​each of the plurality of predetermined sites, or a ratio of the priority area to the predetermined peripheral area of ​​each of the plurality of predetermined sites, 2. The image processing device according to claim 1, wherein the selection means selects, from the plurality of predetermined regions, a predetermined region in which the number of pixels of the priority region calculated by the region calculation means is large or in which the priority region occupies a large proportion.

9. further comprising area calculation means for calculating the number of pixels of the priority area in a predetermined peripheral area of ​​each of the plurality of predetermined parts; The image processing device according to claim 1, characterized in that the selection means selects the specified part with the larger number of pixels when a difference between the number of pixels of the priority area in a peripheral area of ​​a first specified part among the plurality of specified parts and the number of pixels of the priority area in a peripheral area of ​​a second specified part among the plurality of specified parts is equal to or greater than a threshold value.

10. 10. The image processing device according to claim 9, wherein the selection means selects, when the difference is less than a threshold value, a predetermined portion that is closest to a center of the image or a predetermined portion that is closest to a predetermined portion selected in the image of a previous frame.

11. The image processing apparatus according to claim 9 , wherein the priority region is a skin region, a head region, a face region, or a whole body region of the detection target.

12. The image processing device according to claim 1 ; an imaging device comprising: an AF unit for performing AF processing on the predetermined portion selected by the selection unit;

13. A learning device, comprising: An acquisition means for acquiring an image and correct answer information from a storage unit that stores an image including a predetermined portion and correct answer information; an estimation means for estimating a center position map of the predetermined part in the image, a size map of the predetermined part, and a proximity area map of the predetermined part; a generating means for generating correct answer maps of the center position, size, and adjacent area of ​​the predetermined portion as teacher data from the correct answer information; A calculation means for calculating each error between each map extracted by the estimation means and corresponding teacher data; learning means for learning parameters of the estimation means based on each of the errors; A learning device comprising:

14. 14. The learning device according to claim 13, wherein said learning means learns said parameters so as to reduce said respective errors.

15. 14. The learning device according to claim 13, wherein the predetermined part is an eye.

16. The learning device according to claim 13, characterized in that when a difference between a first inter-eye distance between a left eye and a nose and a second inter-eye distance between a right eye and the nose is equal to or greater than a threshold, the creation means creates teacher data for the adjacent area by labeling the area in which the eye with the larger inter-eye distance is located as a positive area and labeling the area in which the eye with the smaller inter-eye distance is located as a negative area and labeling it as a negative example.

17. 17. The learning device according to claim 16, wherein the creating means creates teacher data for the adjacent region by labeling the face region as a positive example when the difference is less than a threshold value.

18. 14. The learning device according to claim 13, wherein said creating means creates the teacher data of the proximity area based on three-dimensional coordinates of the positions of the eyes and nose.

19. A control method for an image processing device, comprising: A feature extraction step of extracting features from the image; a part estimation step of estimating a plurality of predetermined parts of the detection object from the image based on the feature amount; a priority region estimation step of estimating a priority region from the image based on the feature amount; a selection step of selecting one predetermined site from the plurality of predetermined sites based on the priority region; 13. A method for controlling an image processing apparatus comprising the steps of:

20. A method for controlling a learning device, comprising: An acquisition step of acquiring an image and correct answer information from a storage unit that stores an image including a predetermined portion and correct answer information; an estimation step of estimating a center position map of the predetermined part in the image, a size map of the predetermined part, and a proximity area map of the predetermined part; a creating step of creating correct maps of the center position, size, and adjacent area of ​​the predetermined portion as teacher data from the correct answer information; a calculation step of calculating each error between each map extracted by the estimation step and corresponding teacher data; a learning step of learning parameters for the estimation step based on the errors; A method for controlling a learning device comprising:

21. A program for causing a computer to execute the control method according to claim 19 or 20.

Images