Information processing device, information processing method, and program

Abstract

The information processing device includes: a storage unit that stores color information of a human body as reference color information; a detection unit that detects a human body candidate from an image captured by the fisheye camera; and a human body determination unit that acquires, from the storage unit, the reference color information corresponding to a detection region in which the human body candidate is detected, and determines whether or not the human body candidate is a human body on the basis of the degree of similarity between the acquired reference color information of the detection region and the color information of the human body candidate.

Description

Technical Field

[0001] The present invention relates to an information processing device and an information processing method. Background Art

[0002] In recent years, in the factory automation (FA) market, applications for analyzing human motions using fisheye cameras and seeking process improvements through line production and cell production methods have been used. In order to analyze human motions, it is required to improve the accuracy of human body detection. Patent document 1 discloses a technology that determines a face candidate region that may be a person's face as a face region by determining whether the hue of the face candidate region extracted from a captured image is skin color.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2009-123081 Summary of the invention

[0006] Problems to be solved by the invention

[0007] In bird's-eye images captured by a fisheye camera installed on a ceiling, etc., the appearance of a person changes depending on the position in the image, and the color information of the person to be detected varies depending on the position being detected. Therefore, even if color information is used in images captured by a fisheye camera, it is sometimes difficult to reduce false detections.

[0008] An object of one aspect of the present invention is to provide a technology for reducing false detection of a human body in a bird's-eye view image captured by a fisheye camera.

[0009] Means for solving problems

[0010] In order to achieve the above-mentioned object, the present invention adopts the following structure.

[0011] The first aspect of the present disclosure is an information processing device, which includes: a storage unit, which stores color information of a human body as reference color information; a detection unit, which detects human body candidates from images taken with a fisheye camera; and a human body determination unit, which obtains reference color information corresponding to a detection area in which the human body candidate is detected from the storage unit, and determines whether the human body candidate is a human body based on the similarity between the reference color information of the obtained detection area and the color information of the human body candidate.

[0012] The information processing device compares the color information of the human candidate with the reference color information corresponding to the position (detection area) where the human candidate is detected, thereby being able to accurately determine whether the detected human candidate is a human body, thereby reducing the false detection of human bodies in the bird's-eye image captured by the fisheye camera.

[0013] The storage unit may store color information of a human body detected in each of a plurality of regions obtained by dividing the image captured by the fisheye camera as reference color information. By preparing reference color information for each region, the information processing device can accurately determine whether a human body candidate detected in the region is a human body.

[0014] The information processing device may further include a generating unit that generates reference color information of the region based on the color information of the human body detected in the region and stores the information in the storage unit. By including the generating unit, the information processing device can generate or update the reference color information while performing human body detection processing. The information processing device can continuously update the reference color information, and can suppress the reduction in the accuracy of human body detection due to changes in the background and the characteristics of the person over time.

[0015] The generation unit may generate reference color information corresponding to the regions based on color information of human bodies detected in the regions of the plurality of captured images. The information processing device can create a more average color information map by generating reference color information corresponding to each region based on the plurality of captured images for learning.

[0016] The generation unit may also update the reference color information of the region based on the color information of the human body newly detected in the region when the correlation coefficient between the color information of the human body newly detected in the region and the reference color information corresponding to the region is above a first threshold, and store the color information of the human body newly detected in the region in the storage unit as new reference color information of the region when the correlation coefficient between the color information of the human body newly detected in the region and the reference color information of the region is below a second threshold, wherein the second threshold is below the first threshold. The information processing device stores new reference color information in the storage unit for color information with different characteristics based on the captured images for learning. By associating each region with a plurality of reference color information, the information processing device can detect a human body with high accuracy even in the case of detecting a person with different color characteristics.

[0017] The reference color information may be generated based on the pixel values ​​within the frame surrounding the human body detected in the region, and the color information of the human body candidate may be generated based on the pixel values ​​within the frame surrounding the human body candidate. The information processing device can suppress false detection based on the difference in pixel values ​​between the human body and the human body candidate.

[0018] The reference color information may be generated based on the pixel values ​​of the portion of the captured image removed from the frame surrounding the human body detected in the region, and the color information of the human body candidate may be generated based on the pixel values ​​of the portion of the captured image removed from the frame surrounding the human body candidate. The information processing device generates color information by removing the background portion, and can generate basic color information and color information based only on the actual human body and the portion of the human body candidate, thereby enabling detection of the human body with high accuracy.

[0019] The reference color information may also be a histogram of pixel values ​​within a frame surrounding a human body detected in the region, and the color information of the human body candidate may also be a histogram of pixel values ​​within a frame surrounding the human body candidate. The information processing device can suppress false detection based on the difference in distribution of pixel values ​​between the human body and the human body candidate.

[0020] When the correlation coefficient between the histogram of the color information of the human candidate and the histogram of the reference color information corresponding to the detection area is above a predetermined threshold, the human body determination unit may also determine that the human body candidate is a human body. The information processing device can determine whether the human body candidate is a human body based on the correlation coefficient (similarity) between the histograms. The information processing device can adjust the detection accuracy of the difference in color features by changing the predetermined threshold.

[0021] The histogram of pixel values ​​may also be a histogram of each RGB. In addition, the human body determination unit may determine whether the human body candidate is a human body based on the average value, maximum value, or minimum value of each correlation coefficient of the histogram of each RGB of the color information of the human body candidate and the histogram of each RGB of the reference color information corresponding to the detection area. The information processing device can suppress erroneous detection when the color characteristics of the human body and the human body candidate are different.

[0022] The reference color information may also be the mode (most frequent value) or average value of RGB of pixels in a frame surrounding a human body detected in the region, and the color information of a human body candidate may also be the mode or average value of RGB of pixels in a frame surrounding the human body candidate. The information processing device can detect a human body with high accuracy by comparing the color information of a human body candidate with the reference color information through simple calculations.

[0023] The reference color information may be information obtained by averaging the color information of human bodies detected in a region in a plurality of captured images. In addition, the reference color information may be a learned model that uses the color information of human bodies detected in a region in a plurality of captured images as teacher data for learning and outputs whether the color information of the input human body candidate is the color information of the human body detected in the detection region. The information processing device can create a more average color information map by generating reference color information corresponding to each region based on a plurality of captured images for learning.

[0024] The region may correspond to a plurality of reference color information. Even in the case of detecting a person having different color characteristics, the information processing device can detect a human body with high accuracy.

[0025] The information processing device may further include an output unit that presents the human body candidate determined as a human body by the human body determination unit to the user as the human body detection result. The information processing device can present the human body detection result taking the color information into consideration to the user.

[0026] The storage unit may also store reference color information corresponding to the distance from the center position of the captured image by the fisheye camera. By preparing reference color information corresponding to the distance from the center position of the captured image, the information processing device can accurately determine whether a human candidate detected at the distance is a human body.

[0027] The second aspect of the present invention is an information processing method, which includes the following steps executed by a computer: a storage step, in which, for each of a plurality of areas obtained by dividing an image captured by a fisheye camera, color information of a human body detected in an area is stored in a storage unit as reference color information of the area; a detection step, in which a human body candidate is detected from the captured image; and a human body determination step, in which reference color information corresponding to a detection area in which a human body candidate is detected is obtained from the storage unit, and whether the human body candidate is a human body is determined based on the obtained reference color information of the detection area and the color information of the human body candidate.

[0028] The present invention can also be understood as a program for implementing the method by a computer, or a recording medium that non-temporarily records the program. In addition, each of the above-mentioned processes can be combined with each other as much as possible to constitute the present invention.

[0029] Effects of the Invention

[0030] According to the present invention, it is possible to reduce erroneous detection of a human body in a bird's-eye view image captured by a fisheye camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a diagram for explaining an application example of the information processing device according to the embodiment.

[0032] Figure 2 This is a diagram illustrating the hardware configuration of an information processing device.

[0033] Figure 3 This is a diagram illustrating the functional configuration of an information processing device.

[0034] Figure 4 is a flowchart illustrating a human body detection process.

[0035] Figure 5 This is a diagram for explaining color information according to Modification Example 1.

[0036] Figure 6 This is a diagram for explaining color information according to Modification Example 2.

[0037] Figure 7 This is a diagram for explaining a color information map according to Modification 3.

[0038] Figure 8 This is a diagram for explaining a color information map according to Modification 4. DETAILED DESCRIPTION

[0039] Hereinafter, embodiments according to one aspect of the present invention will be described with reference to the drawings.

[0040] <Application Examples>

[0041] Figure 1 The information processing device detects a subject recognized as a human body (hereinafter referred to as a human body candidate) from a camera image (captured image) captured by a camera, and uses color information to determine whether the human body candidate is a human body.

[0042] The information processing device divides the captured image into a plurality of regions and obtains (generates) color information of a human body detected in each region. Figure 1 In the example, the captured image 1 for learning is divided into 4×4 rectangular areas. The information processing device generates a histogram of each RGB as the color information of the human body for the pixels in the frame surrounding the human body detected in area A1. The generated color information is stored in the storage unit as color information that becomes a reference for determining whether the human body candidate is a human body (hereinafter referred to as reference color information). The reference color information is the color information of the human body detected in each area, and is generated for each of the multiple areas obtained by dividing the captured image.

[0043] The information processing device detects a human candidate in a captured image of a detection target and generates color information of the human candidate. Figure 1 The lower part of FIG. 2 shows an example in which the information processing device detects a personal computer (PC) as a human candidate in area A1 of captured image 2. The information processing device generates a histogram for each RGB as color information of the human candidate for pixels within a frame surrounding the detected PC.

[0044] The information processing device compares the color information of the PC detected in the area A1 of the captured image 2 with the reference color information of the area A1, and determines whether the PC detected as a human candidate is a human body based on their similarity. The information processing device can, for example, calculate the correlation coefficient between the histogram of the color information of the human candidate (PC) and the histogram of the reference color information as the similarity, and when the correlation coefficient is above a predetermined threshold, determine that the human candidate is a human body.

[0045] The correlation coefficient is calculated as a value greater than or equal to 0 and less than or equal to 1 by various well-known methods. Figure 1 In the example of , the information processing device calculates the correlation coefficient between each RGB histogram, and can use the average of three correlation coefficients as the correlation coefficient for determining whether it is a human body. The predetermined threshold value can be set to a value of 0.6 or more and 0.9 or less, for example. By increasing the predetermined threshold value, the information processing device can determine whether it is a human body with higher accuracy.

[0046] The camera used for shooting for human detection is an ultra-wide-angle camera equipped with a fisheye lens that can obtain image information over a wide range. A camera equipped with a fisheye lens is also called a fisheye camera, an omnidirectional camera, a panoramic camera, etc., but the term "fisheye camera" is used in this manual.

[0047] The image captured by a fisheye camera distorts the appearance of the subject depending on the position within the captured image. For example, when the fisheye camera is configured to look down from the ceiling, the image of a person is captured with the feet facing the center and the top of the head facing the outside. The human body is viewed from the front, back, or side in the periphery of the captured image, and from the top in the center of the captured image.

[0048] Therefore, even for the same person or a person wearing the same uniform, the color information generated is different depending on the detected area. The information processing device can accurately determine whether the human candidate is a human by comparing the color information of the human candidate with the reference color information corresponding to the area (detection area) where the human candidate is detected. Therefore, the information processing device can reduce the false detection of human bodies in the bird's-eye image captured by the fisheye camera.

[0049] In addition, when detecting a human candidate from an image captured by a camera set at a specific location, the information processing device detects the human candidate mainly based on the background difference obtained by removing the background image from the captured image. In this case, the possibility of an object included in the background difference being detected as a human candidate becomes higher even if it is not a human body. In particular, when detecting a human body using a model that has been learned to detect a human body in a specific environment, an object that is not included in the background but has a different color may sometimes be detected as a human body. The information processing device can reduce the false detection of objects from the background difference by using color information to determine whether it is a human body.

[0050] <Implementation Method>

[0051] (Hardware Structure)

[0052] Reference Figure 2 , an example of the hardware structure of the information processing device 10 is described. Figure 2 1 is a diagram illustrating a hardware configuration of the information processing device 10. The information processing device 10 includes a processor 101, a main storage unit 102, an auxiliary storage unit 103, a communication interface (I / F) 104, and a display unit 105.

[0053] The processor 101 reads the program stored in the auxiliary storage unit 103 to the main storage unit 102 and executes it, thereby realizing Figure 3 The main storage unit 102 is a semiconductor memory such as RAM (Random Access Memory) or ROM (Read Only Memory). The auxiliary storage unit 103 is a non-volatile memory such as a hard disk drive or a solid state drive.

[0054] The communication interface 104 is an interface for communicating by wire (USB cable, LAN cable, etc.) or wirelessly (WiFi, etc.). The display unit 105 is a display, etc., for displaying the detection result of the human body.

[0055] The information processing device 10 may be a general-purpose computer such as a personal computer, a server computer, a tablet terminal, or a smart phone, or may be an embedded computer such as an on-board computer. The information processing device 10 may also implement a portion of the processing of each functional unit through a cloud server. In addition, a portion of the processing of each functional unit of the information processing device 10 may also be implemented by a dedicated hardware device such as an FPGA or an ASIC.

[0056] The information processing device 10 is connected to the camera 20 by wire or wirelessly, and receives image data (captured image) captured by the camera 20. The camera 20 is an imaging device having an optical system including a lens and an imaging element (image sensor such as CCD or CMOS).

[0057] In addition, part of the processing of the information processing device 10 may be performed by the camera 20. In addition, the result of human body detection based on the information processing device 10 may be sent to an external device and presented to the user. Furthermore, the information processing device 10 and the camera 20 may be configured as one body.

[0058] (Functional Structure)

[0059] Reference Figure 3 , an example of the functional structure of the information processing device 10 is described. Figure 3 2 is a diagram illustrating a functional configuration of the information processing device 10. The information processing device 10 includes a color information generating unit 11, a detecting unit 12, a human body determining unit 13, an output unit 14, and a color information database 15 (color information DB 15).

[0060] The color information generating unit 11 obtains the captured image captured by the camera 20, and generates reference color information of each of the plurality of regions obtained by dividing the captured image based on the color information of the human body detected in the region. The data of the reference color information corresponding to each region obtained by dividing the captured image is also called a color information map.

[0061] The color information is generated based on the pixel values ​​in the frame surrounding the detected human body. For example, the color information generating unit 11 generates information representing the pixel values ​​(brightness values) of the pixels in the frame surrounding the human body as a histogram as the reference color information of the area where the human body is detected. The color information generating unit 11 may also generate a histogram of each RGB as the reference color information. The color information generating unit 11 associates the generated reference color information with the area where the human body is detected and stores it in the color information database 15.

[0062] The color information generating unit 11 can generate color information of a human candidate detected from a captured image, similarly to the reference color information. The human candidate color information is used by the human determination unit 13 to determine whether the human candidate is a human. The color information generating unit 11 is an example of a “generating unit”.

[0063] The detection unit 12 obtains a captured image captured by the camera 20 and detects a human candidate from the captured image. The detection unit 12 can use a general object recognition algorithm to detect a human candidate. For example, the detection unit 12 can use a recognizer that combines image features such as HoG or Haar features and a boosting algorithm to detect a human candidate. In addition, the detection unit 12 can also use a human recognition algorithm based on deep learning (for example, R-CNN, Faster R-CNN, YOLO, SSD, etc.) to detect a human candidate.

[0064] The human body determination unit 13 determines whether the human body candidate detected by the detection unit 12 is a human body. The human body determination unit 13 determines whether the human body candidate is a human body based on the similarity between the reference color information corresponding to the detection area where the human body candidate is detected among the multiple areas of the captured image and the color information of the human body candidate. The detection area can be set to, for example, the area where the center position of the rectangular frame surrounding the human body candidate is detected.

[0065] The output unit 14 outputs the human body candidate determined to be a human body by the human body determination unit 13 as a human body detection result. The output unit 14 can present the human body detection result to the user by, for example, superimposing and displaying a rectangle surrounding the detected human body on the captured image.

[0066] The color information database 15 stores a color information map created in advance by the color information generating unit 11. The color information map includes data of regions obtained by dividing the captured image and reference color information corresponding to the regions. The reference color information of the color information map may be updated while performing the human body detection process. The color information database 15 is an example of a "storage unit".

[0067] (Human body detection processing)

[0068] Reference Figure 4 , the overall process of human body detection processing is explained. Figure 4 2 is a flowchart illustrating a human body detection process. The human body detection process is started, for example, when the power of the camera 20 is turned on and the information processing device 10 receives a captured image from the camera 20. Figure 4 The human body detection process shown is a process executed for each frame (captured image) of image data received from the camera 20. It is assumed that a color information map is created in advance by the color information generating unit 11 and stored in the color information database 15.

[0069] In step S101, the detection unit 12 obtains a captured image. The detection unit 12 can obtain the captured image from the camera 20 via the communication interface 104. When the information processing device 10 and the camera (imaging unit) are integrally configured, the color information generation unit 11 obtains the captured image captured by the imaging unit.

[0070] In step S102, the detection unit 12 detects a human candidate from the captured image acquired in step S101. The detection unit 12 can detect a human candidate using a well-known technique such as deep learning. When multiple human candidates are detected in step S102, the human determination process L1 from step S103 to step S106 is repeated for each human candidate.

[0071] In step S103, the human body determination unit 13 determines whether the human body candidate detected in step S102 is a human body. The human body determination unit 13 first obtains reference color information corresponding to the detection area where the human body candidate is detected among the multiple areas obtained by dividing the captured image from the color information database 15. Next, the human body determination unit 13 determines whether the human body candidate is a human body based on the similarity between the obtained reference color information of the detection area and the color information of the detected human body candidate. The color information of the human body candidate is obtained by the color information generation unit 11.

[0072] like Figure 1 As shown in FIG. 1 , it is assumed that the color information is a histogram for each RGB. In this case, the human body determination unit 13 calculates the correlation coefficient between the histogram for each RGB of the color information of the human body candidate and the histogram for each RGB of the reference color information.

[0073] In step S104, the human body determination unit 13 determines whether the correlation coefficient calculated in step S103 is greater than a predetermined threshold value. For example, when the average value, maximum value, or minimum value of each correlation coefficient of the RGB histogram is greater than a predetermined threshold value, the human body determination unit 13 determines that the human body candidate is a human body. In addition, when each correlation coefficient of the RGB histogram is greater than a predetermined threshold value, the human body determination unit 13 may also determine that the human body candidate is a human body.

[0074] If the correlation coefficient is greater than the predetermined threshold value and the human candidate is determined to be a human body (step S104: YES), the process proceeds to step S105. If the correlation coefficient is less than the predetermined threshold value and the human candidate is determined to be not a human body (step S104: NO), the process proceeds to step S106.

[0075] In step S105, the human body determination unit 13 adopts the human body candidate for determination as the human body detection result and stores it in the auxiliary storage unit 103, etc. In step S106, the human body determination unit 13 removes the human body candidate for determination as a false detection.

[0076] If the human body determination process L1 is performed on each human body candidate detected in step S102, the process proceeds to step S107. In step S107, the output unit 14 outputs the information of the human body adopted as the detection result in step S105. The output unit 14 may also display the detection result on the display unit 105. The output unit 14 can present the detection result of the human body to the user by, for example, overlapping and displaying a rectangle surrounding the detected human body on the captured image. The output unit 14 may also output the detection result to an external device and display it on a display of the external device.

[0077] When the human body detection process for the current frame (captured image) is completed, the human body detection process for the next frame is started. The information processing device 10 repeatedly executes Figure 4 The human body detection process is continued until the shooting process based on the camera 20 stops.

[0078] In the above-mentioned embodiment, the information processing device 10 detects a human candidate from a captured image, and determines whether the human candidate is a human based on the similarity between the reference color information corresponding to the detection area where the human candidate is detected and the color information of the human candidate. In an image captured by a fisheye camera, the appearance varies depending on the captured position, but the information processing device 10 can determine whether the detected human candidate is a human with high accuracy by comparing it with the reference color information corresponding to the position (detection area) where the human candidate is detected. Therefore, the information processing device 10 can reduce the false detection of a human in a bird's-eye image captured by a fisheye camera.

[0079] <Variation 1>

[0080] The above-mentioned embodiments show an example in which the color information is a histogram of pixel values. Modification 1 is an example in which the color information is set to the mode of the RGB of the pixels in the frame surrounding the human body or the human body candidate. In addition, the color information is not limited to the mode, and the color information may also be the average value of the RGB of the pixels in the frame surrounding the human body or the human body candidate.

[0081] Figure 5 This is a diagram for explaining color information according to Modification 1. A captured image 1 is a captured image for learning to generate reference color information, and a captured image 2 is a captured image of a human body to be detected.

[0082] The color information generating unit 11 obtains the mode (r1, g1, b1) of RGB for the pixels in the frame surrounding the human body detected in the area A1 of the captured image 1, and stores it in the color information database 15 as the reference color information of the area A1. In addition, the color information generating unit 11 obtains the mode (r2, g2, b2) of RGB for the pixels in the frame surrounding the PC (human body candidate) detected in the area A1 of the captured image 2 as the color information of the human body candidate. The mode is the pixel value that appears most among the pixels in the frame surrounding the human body or the human body candidate. r1, g1, b1, r2, g2, and b2 are represented by values ​​from 0 to 255.

[0083] The human body determination unit 13 calculates the distance d between the RGB mode (r1, g1, b1) of the reference color information and the RGB mode (r2, g2, b2) of the color information of the human body candidate by the following (Equation 1).

[0084] [Mathematical formula 1]

[0085]

[0086] The human body determination unit 13 can determine that the human body candidate is a human body when the distance d is below a predetermined threshold value, and can determine that the human body candidate is not a human body when the distance d is greater than the predetermined threshold value. In addition, the correlation coefficient can also be calculated by defining a conversion formula in a manner that the correlation coefficient increases as the distance d decreases. In this case, the human body determination unit 13 can determine whether the human body candidate is a human body by comparing the correlation coefficient with the predetermined threshold value, similar to the case of the histogram.

[0087] According to Modification 1, information processing device 10 can detect a human body with high accuracy by comparing the color information of a human body candidate with the reference color information through simple calculation.

[0088] <Variation 2>

[0089] Modification 2 is an example of generating color information based on pixel values ​​in a portion excluding the background from a frame surrounding a human body or a human body candidate. By excluding pixels in the background portion, the color information generating unit 11 can generate color information of a human body or a human body candidate more accurately.

[0090] Figure 6 This is a diagram for explaining the color information involved in Modification Example 2. Figure 6 In the example of generating color information of a detected human candidate based on the captured image 60 of the detection target, the color information generating unit 11 can also generate reference color information of a human body based on the captured image for learning in the same manner.

[0091] The detection unit 12 detects a human candidate from the captured image 60. Since the rectangular frame 61 surrounding the human candidate includes the background image, when color information is generated based on the pixel values ​​within the frame 61, the generated color information includes information on the color of the background portion other than the human candidate. Therefore, the color information generation unit 11 obtains the background difference 63 that is the difference between the captured image 60 and the background image 62, and generates color information based on the human candidate 64 in the background difference 63.

[0092] According to Modification 2, the information processing device 10 generates color information by removing the background portion, and can generate basic color information and color information based only on the actual human body and human body candidate portions, thereby enabling the detection of the human body with high accuracy.

[0093] <Variation 3>

[0094] The above-mentioned embodiment shows an example of generating reference color information corresponding to each region from one captured image for learning. Modification 3 is an example of generating reference color information corresponding to each region from a plurality of captured images for learning and creating a color information map.

[0095] Figure 7 This is a diagram for explaining a color information diagram according to Modification 3. The color information generating unit 11 generates reference color information 74 corresponding to area A1 by averaging color information 71 and color information 73 of a human body detected in area A1 of captured images 70 and 72, for example.

[0096] The color information generating unit 11 is not limited to the two captured images 70 and 72, and may generate reference color information based on three or more captured images. In addition, the color information generating unit 11 may generate a learned model as the reference color information, which is a learned model that has been learned using the color information of the human body detected in the area A1 in a plurality of captured images for learning as teacher data. The learned model outputs whether the color information of the input human body candidate is the color information of the human body detected in the detection area.

[0097] According to Modification 3, the information processing device 10 can create a more average color information map by generating reference color information corresponding to each area based on a plurality of captured images for learning.

[0098] <Variation 4>

[0099] Modification 4 is similar to Modification 3 in that it is an example of generating reference color information corresponding to each region based on a plurality of captured images for learning and creating a color information map. Modification 3 is an example of generating one reference color information based on a plurality of captured images for learning, whereas Modification 4 is an example of generating a plurality of reference color information based on a plurality of captured images for learning. In Modification 4, when color information having a correlation coefficient with the current reference color information less than a predetermined threshold is generated based on any of the captured images for learning, the color information is stored as new reference color information.

[0100] Figure 8 This is a diagram for explaining a color information diagram according to Modification 4. The color information generating unit 11 first adopts color information 81 in which a human body is detected in the captured image 80 as reference color information 81 in the area A1.

[0101] Next, the color information generating unit 11 generates color information 83 of the human body newly detected in the captured image 82, and calculates the correlation coefficient with the reference color information 81. When the correlation coefficient is greater than a predetermined threshold value (first threshold value), the color information generating unit 11 updates the reference color information 81 based on the color information 83. The color information generating unit 11 updates the reference color information 84 by averaging the reference color information 81 and the color information 83, for example. The first threshold value can be set to a value of, for example, 0.6 or more and 0.9 or less.

[0102] Furthermore, the color information generating unit 11 generates color information 86 of a human body newly detected in the captured image 85, and calculates a correlation coefficient with the reference color information 84. When the correlation coefficient is less than a predetermined threshold value (second threshold value), the color information generating unit 11 adopts the color information 86 as new reference color information 86 in the area A1, and stores it in the color information database 15. The second threshold value is a value lower than the first threshold value, and can be set to a value of 0.3 or more and 0.6 or less, for example.

[0103] Furthermore, in the case where there are other captured images for learning, the color information generating unit 11 also calculates the correlation coefficients between the color information of the newly detected human body and the reference color information 84 and the reference color information 86. If the correlation coefficient is greater than the first threshold, the color information generating unit 11 updates the reference color information based on the color information of the newly detected human body. In addition, when the correlation coefficient is less than the second threshold for any of the reference color information, the color information generating unit 11 adopts the color information of the newly detected human body as the new reference color information in the area A1 and stores it in the color information database 15.

[0104] According to Modification 4, when the information processing device 10 generates color information with different characteristics based on the captured image for learning, it stores it as new reference color information in the color information database 15. By associating each area with a plurality of reference color information, the information processing device 10 can detect a human body with high accuracy even when detecting a person whose uniform is different in color.

[0105] Furthermore, by using a plurality of reference color information in each area, information such as the frequency of detecting a person having different characteristics can be obtained, and this information can be used as information useful for process improvement in factory automation.

[0106] <Variant 5>

[0107] In the above-mentioned embodiment, an example of a color information map prepared in advance including reference color information for each area is described, but the color information map may be generated and updated while performing human body detection processing. Figure 4 When the human candidate is determined to be a human body in step S105 of the human body detection process shown, the human body determination unit 13 updates the reference color information by averaging the color information of the human candidate determined to be a human body and the current reference color information.

[0108] According to Modification 5, the information processing device 10 can continuously update the reference color information, and thus can suppress a decrease in the accuracy of human body detection due to changes over time in the background and features of a person.

[0109] <Other>

[0110] The above-mentioned embodiments and various modifications are merely illustrative of the configuration examples of the present invention. The present invention is not limited to the above-mentioned specific forms, and various combinations and modifications are possible within the scope of the technical concept.

[0111] In addition, in the above-mentioned embodiment and each modified example, an example of performing human body detection processing by dividing the captured image into 4×4 rectangular areas is described, but the method of dividing the captured image is not limited to 4×4 rectangular areas. The number of divided areas can be more or less than 4×4. In addition, the shape of the divided area is not limited to a rectangle, and it can also be set as a concentric circle area. In addition, the information processing device 10 is not limited to the case where the reference color information is prepared for each divided area, and the reference color information corresponding to the distance from the center position of the captured image can also be prepared.

[0112] <Note>

[0113] (1) The information processing device (10) comprises:

[0114] A storage unit (15) storing color information of a human body as reference color information;

[0115] A detection unit (12) detects a human candidate from an image captured by a fisheye camera; and

[0116] A human body determination unit (13) obtains the reference color information corresponding to the detection area where the human body candidate is detected from the storage unit, and determines whether the human body candidate is a human body based on the similarity between the reference color information of the detection area and the color information of the human body candidate.

[0117] (2) In the information processing method, the following steps are included, which are performed by a computer:

[0118] A storing step of storing the color information of the human body in a storage unit as reference color information;

[0119] A detection step (step S102), detecting a human candidate from an image captured by a fisheye camera; and

[0120] The human body determination step (steps S103 to S106) obtains the reference color information corresponding to the detection area where the human body candidate is detected from the storage unit, and determines whether the human body candidate is a human body based on the similarity between the reference color information of the detection area and the color information of the human body candidate.

[0121] Description of Reference Numerals

[0122] 10 information processing device; 11 color information generating unit; 12 detection unit; 13 human body determination unit; 14 output unit; 15 color information database; 20 camera; 101 processor; 102 main storage unit; 103 auxiliary storage unit; 104 communication interface; 105 display unit.

Claims

1. An information processing device comprising: A storage unit, storing the color information of the human body as reference color information; A detection unit that detects a human candidate from an image captured by a fisheye camera; and The human body determination unit obtains the reference color information corresponding to the detection area where the human body candidate is detected from the storage unit, and determines whether the human body candidate is a human body based on the similarity between the reference color information of the detection area and the color information of the human body candidate.

2. The information processing device according to claim 1, wherein: The storage unit stores, for each of a plurality of regions obtained by dividing the image captured by the fisheye camera, color information of a human body detected in the region as the reference color information.

3. The information processing device according to claim 2, further comprising: The generating unit generates the reference color information of the area based on the color information of the human body detected in the area and stores the reference color information in the storage unit.

4. The information processing device according to claim 3, wherein: The generating unit generates the reference color information corresponding to the region based on color information of a human body detected in each of the regions of the plurality of captured images.

5. The information processing device according to claim 3 or 4, wherein: When a correlation coefficient between the color information of a human body newly detected in the area and the reference color information corresponding to the area is greater than a first threshold, the generating unit updates the reference color information of the area based on the color information of the human body newly detected in the area, When the correlation coefficient between the color information of the human body newly detected in the area and the baseline color information of the area is less than a second threshold, the generation unit stores the color information of the human body newly detected in the area in the storage unit as the new baseline color information of the area, wherein the second threshold is below the first threshold.

6. The information processing device according to any one of claims 2 to 5, wherein: The reference color information is generated based on pixel values ​​within a frame surrounding a human body detected in the area, The color information of the human candidate is generated based on pixel values ​​within a frame surrounding the human candidate.

7. The information processing device according to claim 6, wherein: The reference color information is generated based on pixel values ​​in a portion of the captured image excluding a background from a frame surrounding the human body detected in the region, The color information of the human candidate is generated based on pixel values ​​in a portion excluding a background portion of the captured image from within a frame surrounding the human candidate.

8. The information processing device according to claim 6 or 7, wherein: The reference color information is a histogram of pixel values ​​within a frame surrounding a human body detected in the region, The color information of the human candidate is a histogram of pixel values ​​within a frame surrounding the human candidate.

9. The information processing device according to claim 8, wherein: The human body determination unit determines that the human body candidate is a human body when a correlation coefficient between a histogram of the color information of the human body candidate and a histogram of the reference color information corresponding to the detection area is equal to or larger than a predetermined threshold value.

10. The information processing device according to claim 8 or 9, wherein: The histogram of pixel values ​​is a histogram of each RGB.

11. The information processing device according to claim 10, wherein: The human body determination unit determines whether the human body candidate is a human body based on the average value, maximum value or minimum value of each correlation coefficient of the histogram of each RGB of the color information of the human body candidate and the histogram of each RGB of the reference color information corresponding to the detection area.

12. The information processing device according to claim 6 or 7, wherein: The reference color information is the mode or average value of RGB of pixels within a frame surrounding a human body detected in the region, The color information of the human candidate is the mode or average value of RGB of pixels within a frame surrounding the human candidate.

13. The information processing device according to any one of claims 2 to 7, wherein: The reference color information is information obtained by averaging color information of a human body detected in the region in the plurality of captured images.

14. The information processing device according to any one of claims 2 to 7, wherein: The reference color information is a learned model, and the learned model uses the color information of the human body detected in the area in the plurality of captured images as teacher data for learning, and outputs whether the color information of the input human body candidate is the color information of the human body detected in the detection area.

15. The information processing device according to any one of claims 2 to 7, wherein: The region is associated with a plurality of pieces of the reference color information.

16. The information processing device according to any one of claims 1 to 15, further comprising: The output unit presents the human body candidate determined as a human body by the human body determination unit to a user as a human body detection result.

17. The information processing device according to claim 1, wherein: The storage unit stores the reference color information corresponding to a distance from a center position of an image captured by a fisheye camera.

18. An information processing method, the information processing method comprising the following steps executed by a computer: A storing step of storing the color information of the human body as reference color information in a storage unit; A detection step, detecting a human candidate from an image captured by a fisheye camera; as well as The human body determination step obtains the reference color information corresponding to the detection area where the human body candidate is detected from the storage unit, and determines whether the human body candidate is a human body based on the similarity between the reference color information of the detection area and the color information of the human body candidate.

19. A program for causing a computer to execute the following steps: A storing step of storing the color information of the human body as reference color information in a storage unit; A detection step, detecting a human candidate from an image captured by a fisheye camera; as well as The human body determination step obtains the reference color information corresponding to the detection area where the human body candidate is detected from the storage unit, and determines whether the human body candidate is a human body based on the similarity between the reference color information of the detection area and the color information of the human body candidate.

Images