Image processing apparatus, control method therefor, and program
The image processing device enhances human body detection accuracy for underrepresented postures by generating rotated images, performing detection, and removing low-confidence candidates, addressing the limitations of existing DNN-based methods.
Patent Information
- Application Number
- JP2023072672
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-26
- Publication Date
- 2024-11-08
AI Technical Summary
Existing human body detection technologies using Deep Neural Networks (DNNs) struggle with low accuracy for human postures not sufficiently represented in training data, such as falling postures, and result in increased false detections when using rotated images.
An image processing device that generates multiple rotated images of an input image at preset angles, performs human body detection on each, and removes candidate regions based on likelihood scores to enhance detection accuracy while reducing false positives.
Improves human body detection accuracy for challenging postures by reducing false detections through a method that includes generating rotated images, performing detection, and removing low-confidence candidates.
Smart Images

Figure 2024157967000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing apparatus, a control method thereof, and a program. [Background technology]
[0002] Human body detection technology in an image is a technology that detects a human body rectangle (typically a circumscribing rectangle) that contains the human body area of an object in the image, and is used for human behavior recognition and number counting. In recent years, many human body detection technologies using machine learning have been proposed, and among them, a method using a human body detection model using a Deep Neural Network (hereinafter referred to as DNN) has shown high recognition accuracy (for example, Non-Patent Document 1).
[0003] The human body detection model is obtained by learning learning data consisting of a pair of an image in which a person is captured (hereinafter, a human image) and a correct label as a human body rectangle of the person. The human posture that the human body detection model can detect is limited to a human posture that is sufficiently included in the learning data. For example, while a person's standing posture is generally photographed relatively frequently, a person's fallen posture is photographed relatively infrequently and is not sufficiently included in the learning data. Therefore, while it is easy to detect a human body of a person in a standing posture, it is difficult to detect a human body of a person in a fallen posture that is not sufficiently included in the learning data. As a solution to this problem, it is possible to create learning data of a person in a fallen posture. However, for a large number of images, it is necessary to manually create the human body rectangle of the person in the image as a correct label, and the human cost of this work is high, and it is not easy to build sufficient learning data.
[0004] As a method for enabling human body detection in a human posture that is difficult to detect, such as a fallen posture, a method of rotating an image to create a pseudo image of a person, such as an upright posture that is easy to detect, and performing human body detection on the pseudo image can be considered. Patent Document 1 can be cited as a technology for estimating a person's position by rotating an image. Patent Document 1 proposes a posture estimation technology that rotates a human image to create multiple rotated images, performs posture estimation for estimating the positions of the joint points of the person in the images as postures for the multiple rotated images, and outputs the position of the joint points with the highest reliability. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] Patent No. 7066122 [Non-patent literature]
[0006] [Non-Patent Document 1] Wang, Chien-Yao, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors.” arXiv preprint arXiv:2207.02696 (2022). Summary of the Invention [Problem to be solved by the invention]
[0007] However, when applying the posture estimation technology of Patent Document 1 to human body detection, creating multiple rotated images of a person image, performing human body detection on the multiple rotated images, and outputting the most reliable human body rectangle, false positives occur in each rotated image, which increases the rate of false positives compared to when multiple rotated images are not used.
[0008] The present invention has been made in consideration of such problems, and aims to provide a technique for improving the detection accuracy of human body detection for human poses that make human body detection difficult, while suppressing an increase in false detections. [Means for solving the problem]
[0009] In order to solve this problem, for example, an image processing device according to the present invention has the following arrangement. 1. An image processing device for detecting a human body region representing a human body in an image, comprising: An acquisition means for acquiring an image; a rotation means for rotating the acquired image by a plurality of preset angles to generate a plurality of rotated images; a detection means for performing a human body detection process on each of the plurality of rotated images and acquiring detection information including information representing a candidate region of the detected human body and information representing a likelihood of the candidate region being a human body; and a removal means for determining a candidate region to be removed from the human body region in accordance with each likelihood in the detection information obtained by the detection means and removing the candidate region. Effect of the Invention
[0010] According to the present invention, it is possible to improve the detection accuracy of human body detection for human poses in which human body detection is difficult, while suppressing an increase in erroneous detections. [Brief description of the drawings]
[0011] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram showing an example of the functional arrangement of an image processing apparatus according to a first embodiment; [Diagram 3] 1 is a flowchart showing the operation of the image processing device according to the first embodiment. [Figure 4] Schematic diagram showing an image of embodiment 1 [Diagram 5] FIG. 1 is a schematic diagram showing a rotated image according to the first embodiment; [Figure 6] FIG. 1 is a schematic diagram showing a human body detection result according to the first embodiment; [Figure 7] Schematic diagram showing a human body rectangle before rotation in embodiment 1. [Figure 8] FIG. 1 is a schematic diagram showing grouping of human body detection results according to the first embodiment; [Figure 9] FIG. 11 is a block diagram showing an example of the functional arrangement of an image processing apparatus according to a second embodiment of the present invention; [Figure 10] 11 is a flowchart showing the operation of the image processing apparatus according to the second embodiment. [Figure 11] FIG. 13 is a schematic diagram showing a human body detection result according to the second embodiment; [Figure 12] Schematic diagram showing a human body rectangle before rotation in embodiment 2 [Figure 13] FIG. 13 is a schematic diagram showing a human body likelihood distribution according to the second embodiment; [Figure 14] FIG. 11 is a block diagram showing an example of the functional arrangement of an image processing apparatus according to a third embodiment. [Figure 15] 11 is a flowchart showing the operation of the image processing device according to the third embodiment. [Figure 16] Schematic diagram showing an image of embodiment 3 [Figure 17] Schematic diagram showing a rotated image of embodiment 3. [Figure 18] FIG. 13 is a schematic diagram showing a human body detection result according to the third embodiment. [Figure 19] Schematic diagram showing a human body rectangle before rotation in embodiment 3. [Figure 20] FIG. 13 is a schematic diagram showing an allocation map according to the third embodiment; [Figure 21] FIG. 13 is a schematic diagram showing a rotation angle table of a removal target according to the third embodiment. [Figure 22] FIG. 13 is a schematic diagram showing grouping of human body detection results according to the third embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.
[0013] [Embodiment 1] In this embodiment, human body detection is performed on multiple rotated images obtained by rotation at multiple rotation angles, and a process is described in which human body rectangles that are likely to be erroneously detected are removed from the human body rectangles obtained from rotated images at a specified rotation angle.
[0014] FIG. 1 is a block diagram showing the hardware configuration of an image processing apparatus 100 according to the first embodiment.
[0015] The image processing device 100 includes a CPU 101, a ROM 102, a RAM 103, a secondary storage device 104, an imaging device 105, an input device 107, a network I / F 108, and a bus 109 that connects these devices together so that they can communicate with each other.
[0016] The CPU 101 executes instructions according to programs stored in the ROM 102 and the RAM 103. The ROM 102 is a non-volatile memory (e.g., an EEPROM) and stores the program of this embodiment and other programs and data required for control. The RAM 103 is a volatile memory and stores temporary data such as image data and human body detection results.
[0017] The secondary storage device 104 is a rewritable secondary storage device such as a hard disk drive or flash memory, and stores image information, programs, various settings, etc. This information is transferred to the RAM 103, and the CPU 101 executes the programs and uses the data.
[0018] The imaging device 105 is composed of an imaging lens, an imaging sensor such as a CCD or CMOS, a video signal processing unit, and the like, and captures video (30 frames per second).
[0019] The input device 106 is a keyboard, a mouse, or the like, and transmits instructions input by the user to the CPU 101 .
[0020] The display device 107 is a cathode ray tube (CRT) or a liquid crystal display, and displays the processing results, etc., to the user. For example, the display device 107 displays a list of four images shown in Fig. 7(a) to (d), which will be described in detail later, with a frame, a likelihood, and a rotation angle indicating that a human body has been detected superimposed on each of the images.
[0021] The network I / F 108 is a modem or a LAN interface that connects to a network such as the Internet or an intranet. The type of communication may be either wired or wireless.
[0022] The CPU 101 loads software that implements processes corresponding to each step of a flowchart described later from the secondary storage device 104 onto the RAM 103 and executes the software.
[0023] 2 is a functional configuration diagram when the CPU 101 is executing image processing software in the embodiment. In this state, the image processing device 100 includes an image acquisition unit 201, a rotation unit 202, a human body detection unit 203, a reverse rotation unit 204, a removal unit 205, and an integration unit 206. Note that some of the illustrated configuration may be realized by dedicated circuits instead of the CPU 101.
[0024] Fig. 3 is a flowchart showing the processing flow of the image processing device 100 of this embodiment. Fig. 4 is a schematic diagram showing an image in this embodiment. Fig. 5 is a schematic diagram showing a rotated image in this embodiment. Fig. 6 is a schematic diagram showing a human body detection result in this embodiment. Fig. 7 is a schematic diagram showing a pre-rotation human body rectangle in this embodiment. Fig. 8 is a schematic diagram showing grouping of the human body detection result in this embodiment.
[0025] Hereinafter, a procedure for detecting a human body rectangle in an image in this embodiment will be described with reference to FIGS.
[0026] First, the overall processing operation will be described with reference to the flowchart in FIG.
[0027] In S101, the image acquisition unit 201 acquires an image obtained by imaging using the imaging device 105. Fig. 4 is an example of this acquired image 400. This image 400 shows that two people, person 401 and person 402, are captured, and also shows a non-person 403. The number of people included in the image is arbitrary, and may be zero or three or more. The example in Fig. 4 should be recognized as an example of two people by chance.
[0028] In S102, the rotation unit 202 rotates the image 400 acquired by the image acquisition unit 201 to generate rotated images 501 to 504 shown in Figs. 5(a) to 5(d). The rotated image 501 is an image obtained by rotating the image 400 by 0 degrees, the rotated image 502 is an image obtained by rotating the image 400 by 90 degrees, the rotated image 503 is an image obtained by rotating the image 400 by 180 degrees, and the image 504 is an image obtained by rotating the image 400 by 270 degrees. In the following description, the rotation direction is positive in the counterclockwise direction. The rotation angle is expressed within the range of 0 to 360 degrees. For example, a rotation angle of 270+180 degrees is equivalent to a rotation angle of 90 degrees. The number of rotated images and the rotation angle are arbitrary, and the present embodiment does not limit them. The rotation unit 202 also assigns information 505 to 508 indicating the rotation angle to each of the rotated images 501 to 504.
[0029] In S103, the human body detection unit 203 performs human body detection processing on the rotated images 501 to 504 obtained by the rotation unit 202 according to a human body detection method using DNN described in Non-Patent Document 1, thereby generating human body detection results 601 to 607 (detection information indicating a candidate region of a human body) shown in Figs. 6(a) to 6(d). The examples given here are merely examples, and any method capable of performing human body detection may be used, and the present embodiment is not limited to these. The human body detection results 601 to 607 each include a circumscribing rectangle of a human body (hereinafter, a human body rectangle), a human body likelihood, and a rotation angle of a rotated image to be detected. The human body likelihood is a real value between 0 and 1, and the closer to 1, the more likely a human body is to exist within the human body rectangle. In the embodiment, a human body likelihood equal to or greater than a preset threshold (for example, 0.2) is determined as a human body candidate. This threshold can be changed and set by the user operating the input device 106.
[0030] In order to speed up the human body detection process, the rotated image may be converted to a predetermined size (for example, the size of image 400) by scaling and padding before performing human body detection using DNN. This is because the process of changing the shape of the entire DNN model, which is required every time the input shape of the DNN model changes, can be reduced by converting to a predetermined size. Furthermore, multiple rotated images converted to a predetermined size may be converted to a mini-batch of size B x H x W x C by connecting them on a new axis, and then human body detection may be performed. Here, B is the number of rotated images that are simultaneously used for human body detection, H is a predetermined size (height), W is a predetermined size (width), and C is a predetermined size (number of channels). In addition, multiple rotated images that are simultaneously used for human body detection may be mapped onto one mapping image before human body detection. Here, the mapping image has a size that allows multiple rotated images to be mapped without overlapping. This is because the more elements of the image input to the DNN, the more effectively the parallel processing performance of the processor can be utilized.
[0031] Fig. 6(a) shows that person 401 was detected as a human body candidate, and person 402 was not detected as a human body candidate. Fig. 6(b) shows that both people 401 and 402 were detected as human body candidates. Fig. 6(c) shows that person 402 and subject body 403 were detected as human body candidates, and human body 402 was not detected as a human body candidate. Fig. 6(d) shows that person 401 and non-person 403 were detected as human body candidates, and person 402 was not detected as a self candidate.
[0032] In S104, the reverse rotation unit 204 converts (reverse rotates) the human body rectangles (coordinates representing) of the human body detection results 601 to 607 obtained by the human body detection unit 203 into coordinates in the image 400 based on the respective rotation angles, thereby generating pre-rotation human body rectangles 701 to 707 shown in Figure 7.
[0033] In S105, the removal unit 205 removes pre-rotation human body rectangles 703, 705, 706, and 707 other than the top k=2 human body likelihoods from pre-rotation human body rectangles 702-707 having a rotation angle other than 0°. The example given here is merely an example, and k may be any natural number equal to or greater than 1, such as 1 or 3, and may be set by the user. Furthermore, k may be determined based on the number n of human body detection results, as shown in the following formula (1). k = Ceiling(n × r) …(1) Here, Celing(x) is a ceiling function that returns the smallest integer that is not less than the real number x. Also, r is a parameter that adjusts the percentage of human body detection results to be removed relative to the number of human body detection results, and may be any value such as 0.1 or 0.2, or may be selected from within a range preset by the user.
[0034] Also, instead of removing pre-rotation human body rectangles other than the top k from all pre-rotation human body detection results 702 to 707 having a rotation angle other than 0°, pre-rotation human body rectangles other than the top k may be removed for each rotation angle. Pre-rotation human body rectangles less than a predetermined threshold may be removed instead of removing the ones other than the top k. Here, the rotation angle is set to be other than 0°, but pre-rotation human body rectangles having other rotation angles may be removed. Also, human body detection may be applied to the evaluation data, the number of correct detections for each rotation angle may be calculated, and the pre-rotation human body rectangles of which rotation angle are to be removed may be determined from the number of correct detections. Any method may be used as long as it can remove pre-rotation human body rectangles that are highly likely to be erroneously detected from pre-rotation human body rectangles obtained from a rotated image having a rotation angle with a small number of detectable human body rectangles, and this embodiment is not limited thereto.
[0035] In S106, the integration unit 206 calculates an intersection over union (hereinafter, IoU) representing an evaluation index between the pre-rotation human body rectangles after the removal process in S105. IoU is a value indicating the degree of agreement between rectangles. Human body detection results corresponding to pre-rotation human body rectangles whose IoU is equal to or greater than a predetermined threshold are grouped into a group of human body detection results for the same person, as shown in Figs. 8(a) and 8(b). The example given here is an example, and human body detection results corresponding to pre-rotation human body rectangles whose distance between center coordinates is equal to or less than a predetermined threshold may be grouped as human body detection results for the same person, and any method that can group human body detection results for the same person may be used. This embodiment does not limit these. Finally, among the grouped human body detection results, the one with the highest human body likelihood is selected. The example given here is an example, and the one with the largest size of the human body rectangle may be selected, or when images of multiple frames are targeted, the one with the smallest positional variation of the human body rectangle between each frame may be selected. Any method that can generate accurate human body detection results may be used, and this embodiment does not limit these.
[0036] As described above, in the first embodiment, human body detection is performed on images rotated at a plurality of rotation angles, and human body rectangles with low confidence are removed from among those obtained from images rotated at a predetermined rotation angle. By removing human body rectangles with low confidence from among those obtained from images rotated at a rotation angle at which the number of detectable human body rectangles is small, the first embodiment can improve the detection accuracy of human body detection for human poses in which human body detection is difficult while suppressing an increase in false positives.
[0037] [Embodiment 2] In the second embodiment, a process will be described in which human body detection is performed on rotated images at a plurality of rotation angles, and human body rectangles that are highly likely to be erroneously detected are removed using the distribution of human body likelihood.
[0038] The hardware configuration diagram of the image processing device is assumed to be the same as that of FIG. 1 of the first embodiment, and a description thereof will be omitted. FIG. 9 is a block diagram showing the basic functional configuration of the image processing device in the second embodiment. FIG. 9 is also a functional configuration diagram when the CPU 101 is executing software for image processing in the second embodiment. In this state, the image processing device 100 has an image acquisition unit 201, a rotation unit 202, a human body detection unit 203, a reverse rotation unit 204, a distribution calculation unit 901, a removal unit 205, and an integration unit 206. Note that part of the illustrated configuration may be realized by a circuit other than the CPU 101.
[0039] Fig. 10 is a flowchart showing the flow of processing in the present embodiment 2. Fig. 11 is a schematic diagram showing a human body detection result in the present embodiment 2. Fig. 12 is a schematic diagram showing a human body rectangle before rotation in the present embodiment 2. Fig. 13 is a schematic diagram of a human body likelihood distribution in the present embodiment 2. Figs. 4 and 5 are also referred to in the present embodiment 2.
[0040] Hereinafter, a procedure for detecting a human body rectangle of a person in an image in the second embodiment will be described with reference to FIGS.
[0041] First, in S201, the image acquisition unit 201 acquires a captured image (reference numeral 400 in FIG. 4). Since details of the process are the same as those in the first embodiment, a description thereof will be omitted.
[0042] In S202, the rotation unit 202 performs a rotation process on the image 400 acquired by the image acquisition unit 201 to generate rotated images 501 to 504 shown in Figures 5(a) to 5(d). The details of the process are the same as those in the first embodiment, and therefore will not be described.
[0043] In S203, the human body detection unit 203 executes human body detection processing on the rotated images 501-504 obtained by the rotation unit 202 by the human body detection method using DNN described in Non-Patent Document 2, thereby generating human body detection results 1101-1112 shown in Fig. 11(a)-(d). The details of the processing are the same as those in the first embodiment, and therefore will not be described.
[0044] In S204, the reverse rotation unit 204 converts the human body rectangles of the human body detection results 1101 to 1112 obtained by the human body detection unit 203 into coordinates in the image 400 based on their respective rotation angles, thereby generating pre-rotation human body rectangles 1201 to 1212 shown in Figures 12(a) to 12(d).
[0045] In S205, the distribution calculation unit 901 calculates IoU, which represents an evaluation index, between the pre-rotation human body rectangles, and groups the human body detection results corresponding to the pre-rotation human body rectangles whose IoU is equal to or greater than a predetermined threshold into a group of human body detection results for the same person. The example given here is an example, and the human body detection results corresponding to the pre-rotation human body rectangles whose distance between the center coordinates is equal to or less than a predetermined threshold may be grouped as human body detection results for the same person, and any method that can group the human body detection results for the same person may be used. This embodiment is not limited to these. The distribution calculation unit 901 calculates the distribution of human body likelihood for each rotation angle from the last grouped human body detection results, and generates human body likelihood distributions 1301 to 1303 shown in the target human body candidates by classifying the results into regions at the same position in the original image as shown in Figs. 13(a) to 13(c).
[0046] In S206, the removal unit 205 removes the corresponding human body detection result for each of the human body likelihood distributions 1301 to 1303 shown in Figures 13(a) to (c), on the condition that the absolute value of the difference between the maximum human body likelihood and the human body likelihood of the corresponding region obtained from the rotated image with a rotation angle obtained by adding 180° to the rotation angle of the maximum human body likelihood is less than a predetermined threshold.
[0047] Here, for example, the threshold value is set to 0.3. In this case, only for the human body likelihood distribution 1303, the absolute value of the difference between the maximum human body likelihood and the human body likelihood obtained from a rotated image with a rotation angle obtained by adding 180° to the rotation angle of the maximum human body likelihood is 0.1, which is less than the threshold value, and the corresponding human body detection results 1102, 1105, 1108, and 1111 are removed. This is because, in the case of a person, since it is divided into the head and the legs, there is a difference in appearance in the up and down directions, that is, since the direction dependency is high, the change in the human body likelihood due to the rotation angle tends to be large. However, in the case of other erroneously detected objects with low direction dependency, the change in the human body likelihood due to the rotation angle tends to be small, and by utilizing this, it is possible to remove the erroneously detected objects. The example given here is just an example, and the threshold value can be any value such as 0.5 or 0.2, as long as it is a value according to the value range of the human body likelihood. Alternatively, instead of the absolute value of the difference between the maximum human body likelihood and the human body likelihood at a rotation angle obtained by adding 180° to the rotation angle of the maximum human body likelihood, a corresponding human body detection result may be removed when the variance of the human body likelihood is less than a predetermined threshold, or when the detection likelihood at a plurality of rotation angles does not satisfy a predetermined threshold or more. Any method may be used as long as it can remove pre-rotation human body rectangles that are highly likely to be erroneously detected using the human body likelihood distribution, and the present embodiment is not limited to these.
[0048] In S207, the integration unit 206 calculates an evaluation index IoU between the pre-rotation human body rectangles. Human body detection results corresponding to pre-rotation human body rectangles with IoU equal to or greater than a predetermined threshold are grouped into a group of human body detection results for the same person. Details of the grouping process are the same as in the first embodiment, and therefore will not be described. Finally, from the grouped human body detection results, the one with the highest human body likelihood is selected. Details of the selection process of the human body detection results are the same as in the first embodiment, and therefore will not be described.
[0049] In the above-mentioned embodiment 2, human body detection is performed on rotated images with a plurality of rotation angles, and human body rectangles with a high probability of erroneous detection are removed using the distribution of human body likelihood. With embodiment 2, even if the human body likelihood of a human body rectangle with a high probability of erroneous detection is higher than the human body likelihood of a human body rectangle for a person, it can be removed.
[0050] [Third embodiment] In embodiment 3, we will explain a process in which human body detection is performed on rotated images at multiple rotation angles for an image captured by a fisheye camera, and human body rectangles that are likely to be erroneously detected are removed from the human body rectangles obtained from rotated images at different rotation angles depending on the position of the human body rectangle on the image.
[0051] The hardware configuration diagram of the image processing device is assumed to be the same as that of FIG. 1 of the first embodiment, and a description thereof will be omitted. FIG. 14 is a block diagram showing the basic functional configuration of the image processing device in the third embodiment. FIG. 14 is also a functional configuration diagram when the CPU 101 is executing the image processing software in the second embodiment. In this state, the image processing device 100 has an image acquisition unit 201, a rotation unit 202, a human body detection unit 203, a reverse rotation unit 204, an allocation unit 1401, a removal unit 205, and an integration unit 206. Note that part of the illustrated configuration may be realized by a circuit other than the CPU 101.
[0052] FIG. 15 is a flowchart showing the flow of processing in this embodiment. FIG. 16 is a diagram showing an example of a captured image in this embodiment. FIGS. 17(a) to 17(d) are schematic diagrams showing rotated images in this embodiment 3. FIGS. 18(a) to 18(d) are schematic diagrams showing human body detection results in this embodiment. FIGS. 19(a) to 19(d) are schematic diagrams showing pre-rotation human body rectangles in this embodiment. FIG. 20 is a schematic diagram showing an allocation map when the angle of view range of the imaging device 105 in this embodiment is divided into a plurality of angle of view areas (four in the figure). FIG. 21 is a schematic diagram showing a table in this embodiment that associates each angle of view area with the rotation angle of the removal target. This table is assumed to be stored and held in, for example, the secondary storage device 104. FIGS. 22(a) and 22(b) are schematic diagrams showing grouping of human body detection results in this embodiment.
[0053] A procedure for detecting a human body rectangle in an image in the third embodiment will be described below with reference to FIGS.
[0054] First, in S301, the image acquisition unit 201 acquires an image captured by the imaging device 105 using a fisheye lens. Fig. 16 shows an example of the acquired image 1600. The illustrated image 1600 shows two people, person 1601 and person 1602. Note that the number of people in the image is arbitrary, and may be zero or three or more, with no particular limit on the number of people.
[0055] In S302, the rotation unit 202 executes a rotation process on the image 1600 acquired by the image acquisition unit 201 to generate rotated images 1701 to 1704 shown in FIGS. 17(a) to 17(d).
[0056] In S303, the human body detection unit 203 performs human body detection on the rotated images 1701 to 1704 obtained by the rotation unit 202 by the human body detection method using DNN described in Non-Patent Document 1, thereby generating human body detection results 1801 to 1806 shown in Figures 18(a) to 18(d). Since the details of the process are the same as those in the first embodiment, the description will be omitted.
[0057] In S304, the reverse rotation unit 204 converts (reverse rotates) the human body rectangles of the human body detection results 1701 to 1706 obtained by the human body detection unit 203 into coordinates in the image 1600 based on their respective rotation angles, thereby generating pre-rotated human body rectangles 1901, 1905, and 1906 shown in Figures 19(a) to (d).
[0058] In S305, the assignment unit 1401 assigns labels 1 to 4 to the pre-rotation human body rectangles 1901 to 1906 obtained by the reverse rotation process by the reverse rotation unit 204, depending on which label area in the assignment map 2000 (FIG. 20) the center coordinates of the human body rectangles correspond to. In the case of this embodiment, a label of 1 is assigned to the pre-rotation human body rectangles 1901 and 1903, and a label of 3 is assigned to the pre-rotation human body rectangles 1902 and 1904 to 1906. The number of labels and the positions of the label areas are arbitrary, and this embodiment does not limit them.
[0059] In S306, the removal unit 205 refers to the rotation angle table 2100 (assumed to be stored in a non-volatile memory) of the removal target shown in FIG. 21, and removes pre-rotation human body rectangles 1902-1904 whose rotation angles correspond to the removal target rotation angles corresponding to the labels and whose human body likelihoods are less than a predetermined threshold (0.5) from the pre-rotation human body rectangles 1901-1906. The example given here is an example, and the threshold may be any value such as 0.6 or 0.3, as long as it is a value according to the value range of the human body likelihood. Instead of removing pre-rotation human body rectangles less than the predetermined threshold, pre-rotation human body rectangles other than the top k may be removed from the pre-rotation human body rectangles having the removal target rotation angles. The removal target rotation angle may be any, and human body detection may be applied to the evaluation data, the number of positive detections for each rotation angle may be calculated, and the removal target rotation angle may be determined from the number of positive detections. Any method can be used as long as it can remove pre-rotated human body rectangles that are likely to be misdetected from pre-rotated human body rectangles obtained from a rotated image with a rotation angle that has a small number of detectable human body rectangles, and this embodiment is not limited to these.
[0060] In S306, the integrating unit 206 calculates an evaluation index IoU between the pre-rotation human body rectangles. Then, the integrating unit 206 groups the human body detection results corresponding to the pre-rotation human body rectangles whose evaluation index IoU is equal to or greater than a predetermined threshold into a group of human body detection results for the same person, as shown in Figs. 22(a) and (b). The details of the grouping process are the same as in the first embodiment, and therefore will not be described. Finally, from the grouped human body detection results, the one with the highest human body likelihood is selected. The details of the selection process of the human body detection results are the same as in the first embodiment, and therefore will not be described.
[0061] As described above, according to the third embodiment, human body detection is performed on images rotated at a plurality of rotation angles for images captured by a fisheye camera, and human body rectangles that are likely to be erroneously detected are removed from the human body rectangles obtained from the rotated images at different rotation angles according to the position of the human body rectangle on the image. According to the third embodiment, even when the appearance of a person with the same posture varies greatly depending on the position on the image, such as in images captured by a fisheye camera, human body rectangles that are likely to be erroneously detected can be removed.
[0062] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
[0063] The disclosure of this specification includes the following image processing device, its control method, and program. (Item 1) 1. An image processing device for detecting a human body region representing a human body in an image, comprising: An acquisition means for acquiring an image; a rotation means for rotating the acquired image by a plurality of preset angles to generate a plurality of rotated images; a detection means for performing a human body detection process on each of the plurality of rotated images and acquiring detection information including information representing a candidate region of the detected human body and information representing a likelihood of the candidate region being a human body; a removal means for determining a candidate region to be removed from the human body region in accordance with each likelihood in the detection information obtained by the detection means, and removing the candidate region; 13. An image processing device comprising: (Item 2) The image processing device according to item 1, characterized in that the removal means removes detection information that is less than a predetermined top k (k is a natural number) of detection information detected by the detection means. (Item 3) The removing means is a classification means for classifying each of the rotated images detected by the detection means into corresponding candidate regions; a calculation means for calculating a difference between a maximum likelihood of a candidate region of interest classified by said classification means and a likelihood of said candidate region of interest obtained from an image rotated by 180 degrees with respect to the rotated image in which said maximum likelihood was obtained; a comparison means for comparing the difference calculated by the calculation means with a preset threshold value, When a result of the comparison by the comparing means indicates that the difference is less than the threshold value, the target candidate region is removed from the candidates as a human body region. 3. The image processing device according to item 1 or 2. (Item 4) 4. The image processing device according to any one of items 1 to 3, further comprising an imaging unit that uses a fisheye lens to capture an image of the target to be acquired by the acquisition unit. (Item 5) a storage means for storing a table that stores a correspondence between each of the angle of view areas and a rotation angle of the object to be removed, the table storing the correspondence between each of the angle of view areas and a rotation angle of the object to be removed, The removing means is It is determined in which field of view area before rotation the candidate area of interest detected by the detection means is located, and the candidate area of interest is removed from candidates for the human body area on the condition that the rotation angle of the removal target obtained by referring to the table matches the rotation angle of the rotated image in which the candidate area of interest exists, and the likelihood of the candidate area of interest is less than a preset threshold value. 5. The image processing device according to item 4, (Item 6) A method for controlling an image processing device for detecting a human body region representing a human body in an image, comprising: An acquisition step of acquiring an image; a rotation step of rotating the acquired image by a plurality of preset angles to generate a plurality of rotated images; a detection step of performing a human body detection process on each of the plurality of rotated images to obtain detection information including information representing a candidate region of the detected human body and information representing a likelihood of the candidate region being a human body; a removal step of determining and removing candidate regions to be removed from the human body region in accordance with the respective likelihoods in the detection information obtained by the detection step; 13. A method for controlling an image processing apparatus comprising the steps of: (Item 7) A program that, when read and executed by a computer, causes the computer to function as each of the means possessed by the device described in any one of items 1 to 5.
[0064] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]
[0065] 101: CPU, 102: ROM, 103: RAM, 104: secondary storage device, 105: imaging device, 106: input device, 107: display device, 108: network I / F, 201: image acquisition unit, 202: rotation unit, 203: human body detection unit, 204: reverse rotation unit, 205: removal unit, 206: integration unit
Claims
1. 1. An image processing device for detecting a human body region representing a human body in an image, comprising: An acquisition means for acquiring an image; a rotation means for rotating the acquired image by a plurality of preset angles to generate a plurality of rotated images; a detection means for performing a human body detection process on each of the plurality of rotated images and acquiring detection information including information representing a candidate region of the detected human body and information representing a likelihood of the candidate region being a human body; a removal means for determining a candidate region to be removed from the human body region in accordance with each likelihood in the detection information obtained by the detection means, and removing the candidate region; 13. An image processing device comprising:
2. 2 . The image processing apparatus according to claim 1 , wherein the removal unit removes detection information that is less than a predetermined top k (k is a natural number) pieces of detection information detected by the detection unit.
3. The removing means is A classification means for classifying each of the rotated images detected by the detection means into corresponding candidate regions; a calculation means for calculating a difference between a maximum likelihood of a candidate region of interest classified by said classification means and a likelihood of said candidate region of interest obtained from an image rotated by 180 degrees with respect to the rotated image from which said maximum likelihood was obtained; a comparison means for comparing the difference calculated by the calculation means with a preset threshold value, When a result of the comparison by the comparing means indicates that the difference is less than the threshold, the target candidate region is removed from the candidates as a human body region.
2. The image processing device according to claim 1,
4. 2. The image processing apparatus according to claim 1, further comprising an image capturing unit that captures an image of the target to be acquired by said acquisition unit using a fisheye lens.
5. a storage means for storing a table that stores a correspondence between each of the angle of view areas and a rotation angle of the object to be removed, the table storing the correspondence between each of the angle of view areas and a rotation angle of the object to be removed, The removing means is It is determined in which field of view area before rotation the candidate area of interest detected by the detection means is located, and the candidate area of interest is removed from candidates for the human body area on the condition that the rotation angle of the removal target obtained by referring to the table matches the rotation angle of the rotated image in which the candidate area of interest exists, and the likelihood of the candidate area of interest is less than a preset threshold value.
5. The image processing device according to claim 4.
6. A method for controlling an image processing device for detecting a human body region representing a human body in an image, comprising: An acquisition step of acquiring an image; a rotation step of rotating the acquired image by a plurality of preset angles to generate a plurality of rotated images; a detection step of performing a human body detection process on each of the plurality of rotated images to obtain detection information including information representing a candidate region of the detected human body and information representing a likelihood of the candidate region being a human body; a removal step of determining and removing candidate regions to be removed from the human body region in accordance with the respective likelihoods in the detection information obtained by the detection step; 13. A method for controlling an image processing apparatus comprising the steps of:
7. A program for causing a computer to execute each step of the method according to claim 6 when the program is read and executed by the computer.
Citation Information
Patent Citations
Attitude estimation device, attitude estimation method, and program
JP7066122B2
