Object recognition device and object recognition method

Abstract

The present invention provides an object recognition device and an object recognition method, which can accurately detect distant objects while suppressing cost increases. The object recognition device (100) includes: a camera unit (10) that acquires multiple thermal images of an object; a stereo matching unit (11) that uses the multiple thermal images to perform stereo matching and calculates the distance to the object; a specific temperature region extraction unit (12) that extracts a specific temperature region from the thermal image; and an area ratio estimation unit (14) that estimates the ratio of the area of ​​the object included in the pixel based on the distance to the object calculated by the stereo matching unit for the specific temperature region.

Description

Technical Field

[0001] The invention relates to an object recognition device and an object recognition method. Background Art

[0002] When an object radiates heat, it radiates far infrared rays. The far infrared imaging sensor can detect the far infrared rays radiated from the detection object. By using units such as photoelectric conversion and signal processing, the detected far infrared rays can be converted into a thermal image that shows the temperature distribution of the detection object.

[0003] Since humans are warm-blooded animals, by using a far-infrared imaging sensor, it is possible to detect a person even in a place with a poor light environment such as at night. Therefore, for example, Patent Documents 1 and 2 disclose a technology for detecting a person using a far-infrared imaging sensor.

[0004] For example, Patent Document 1 discloses a pedestrian detection device that uses a linear relationship, which is used to calculate the temperature that a pedestrian may be in based on the road surface temperature, to extract pedestrian candidates with high accuracy. Specifically, Patent Document 1 states that "when the temperature difference between the temperature of a person candidate detected from a thermal image and the temperature of a person calculated by a person temperature calculation unit is greater than a preset value, it is determined that a person candidate adjacent to the person candidate is not a person."

[0005] In addition, Patent Document 2 discloses a pedestrian recognition device that can determine the necessity of issuing an alarm with a high degree of reliability when recognizing pedestrians near the vehicle to issue an alarm to the driver. Specifically, Patent Document 2 states that "when the temperature distribution detected by the temperature distribution detection unit is inconsistent with the temperature distribution of a human being, it is determined that the pedestrian is not a person."

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Publication No. 2012-53724

[0009] Patent Document 2: Japanese Patent Application Publication No. 2007-133644 Summary of the invention

[0010] Technical problem to be solved by the invention

[0011] However, when using a far-infrared camera sensor to identify an object, if the resolution of a distant object is insufficient, the surrounding environment and other objects are integrated into one pixel, so the shape of the object cannot be recognized, making it difficult to detect a distant object (such as a person). Although it is possible to consider using multiple camera sensors, the cost will increase in this case.

[0012] Therefore, an object of the present invention is to provide an object recognition device capable of accurately detecting a distant object while suppressing a cost increase.

[0013] Technical means of solving problems

[0014] The following is a brief description of the representative technical solutions disclosed in this application.

[0015] A representative embodiment of the present invention provides an object recognition device, which includes: a camera unit, which acquires multiple thermal images of an object; a stereo matching unit, which uses the multiple thermal images to perform stereo matching and calculates the distance to the object; a specific temperature region extraction unit, which extracts a specific temperature region from the thermal image; and an area ratio estimation unit, which estimates the ratio of the area of ​​the object contained in the pixel based on the distance to the object calculated by the stereo matching unit for the specific temperature region.

[0016] Effects of the Invention

[0017] The technical effects that can be obtained by representative ones of the technical solutions disclosed in this application are briefly described as follows.

[0018] That is, according to the representative embodiment of the present invention, it is possible to accurately detect a distant object while suppressing a cost increase. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a block diagram showing the structure of embodiment 1 of the present invention.

[0020] Figure 2 It is a flowchart showing the processing flow of Example 1 of the present invention.

[0021] Figure 3 It is an explanatory diagram showing an example of a real image and an image in an image of thermal radiation of a detection target located at a distant place.

[0022] Figure 4 It is an explanatory diagram showing an example of a real image and an image in an image of thermal radiation of a detection target located nearby.

[0023] Figure 5 It is an explanatory diagram showing a process of searching for a specific temperature region in the specific temperature region extraction unit of the present invention.

[0024] Figure 6 This is an explanatory diagram of a process of estimating the center point position of a real image of a person's face using the center point estimating unit of the present invention.

[0025] Figure 7 It is an explanatory diagram showing a first calculation example of center point position estimation according to the present invention.

[0026] Figure 8 It is an explanatory diagram showing a second calculation example of the center point position estimation according to the present invention.

[0027] Fig. 9 It is an explanatory diagram showing a process of estimating the center point position of a real image of a person's face using the center point estimating unit of the present invention.

[0028] Fig.10 This is a diagram for explaining an example of a detection temperature range used in the detection temperature range adjustment unit according to the first embodiment of the present invention, by comparison with the prior art.

[0029] Fig.11 It is a block diagram showing the structure of embodiment 2 of the present invention.

[0030] Fig.12 This is a flowchart showing the processing flow of Example 2 of the present invention. DETAILED DESCRIPTION

[0031] (Implementation Method 1)

[0032] The embodiments of the present invention are described below with reference to the accompanying drawings. Each embodiment described below is an example for implementing the present invention and does not limit the technical scope of the present invention. In addition, the same symbols are marked on the components with the same functions in the embodiments, and their repeated description is omitted except in particularly necessary cases.

[0033] <Outline>

[0034] In this embodiment, when the resolution of a distant object is insufficient, the distance of the distant object is calculated based on stereo matching of multiple thermal images, and the detection temperature range of the object is adjusted accordingly according to the calculated distance. Since the detection temperature range is adjusted according to the distance, it is possible to accurately detect distant objects while suppressing cost increases.

[0035] <Structure of object recognition device>

[0036] Figure 1 FIG. 1 is a structural diagram showing an example of the structure of an object recognition device according to Embodiment 1 of the present invention. Figure 1 As shown, the object recognition device 100 includes an imaging unit 10 , a stereo matching unit 11 , a specific temperature region extraction unit 12 , a center point estimation unit 13 , an area ratio estimation unit 14 , a detection temperature range adjustment unit 15 and a detection object determination unit 16 .

[0037] The imaging unit 10 is a functional module that detects far infrared rays radiated from the detection object and generates a thermal image. Figure 1As shown, the first far infrared imaging sensor 10a, the second far infrared imaging sensor 10b, the first lens 10c corresponding to the first far infrared imaging sensor 10a, and the second lens 10d corresponding to the second far infrared imaging sensor 10b are made of materials capable of transmitting far infrared rays.

[0038] The first far-infrared imaging sensor 10a and the second far-infrared imaging sensor 10b detect the far-infrared rays radiated when the detection object radiates heat, and perform photoelectric conversion and signal processing on the detected far-infrared rays. As a result, the first far-infrared imaging sensor 10a and the second far-infrared imaging sensor 10b respectively generate thermal images representing the temperature distribution of the detection object based on the detected far-infrared rays. The thermal image generated by the first far-infrared imaging sensor 10a is called the first thermal image, and the thermal image generated by the second far-infrared imaging sensor 10b is called the second thermal image. In these thermal images, each pixel represents the detected temperature.

[0039] These thermal images are generated based on the far infrared rays detected at the same time. Hereinafter, the first thermal image and the second thermal image are sometimes referred to as thermal images or a pair of thermal images. The thermal image represents the temperature of the object detected in each pixel, and the brightness value of each pixel is correlated with the temperature.

[0040] As the first far-infrared imaging sensor 10a and the second far-infrared imaging sensor 10b, for example, quantum type or uncooled type sensors are preferably used, but the present invention is not limited thereto, and various far-infrared sensors can be used.

[0041] The imaging unit 10 is connected to the stereo matching unit 11 and the specific temperature region extraction unit 12. The imaging unit 10 outputs a pair of generated thermal images (a first thermal image and a second thermal image) to the stereo matching unit 11 and the specific temperature region extraction unit 12.

[0042] The stereo matching unit 11 is connected to the camera unit 10 and the area ratio estimation unit 14. The stereo matching unit 11 performs stereo matching on a pair of thermal images consisting of the first thermal image and the second thermal image, and calculates the distance to the object. Then, the stereo matching unit 11 outputs the calculated distance to the object to the area ratio estimation unit 14. In addition, the stereo matching unit 11 can also be set in the camera unit 10.

[0043] The specific temperature region extraction unit 12 is connected to an external input device (such as a register and a GUI, etc.) 17, the imaging unit 10, and the area ratio estimation unit 14. The specific temperature region extraction unit 12 extracts a specific temperature region from the thermal image, the specific temperature region being composed of pixels that detect a temperature within a predetermined specific temperature range. The specific temperature range is a temperature range for detecting the detection object, and is set based on specific temperature range setting information input from the external input device 17, for example. The specific temperature range is a temperature range assumed as the temperature of the detection object, and different specific temperature ranges can be set for each detection object.

[0044] The specific temperature region extraction unit 12 acquires the temperature of the extracted specific temperature region and the coordinates of the specific temperature region, and outputs the acquired temperature of the specific temperature region and the coordinates of the specific temperature region to the center point estimation unit 13 .

[0045] The center point estimation unit 13 is connected to the specific temperature region extraction unit 12 and the area ratio estimation unit 14. The center point estimation unit 13 estimates the center position of the object located in the specific temperature region based on the temperature of the specific temperature region and the coordinates of the specific temperature region output from the specific temperature region extraction unit 12. The center point estimation unit 13 outputs the estimated center position of the object to the area ratio estimation unit 14.

[0046] The area ratio estimation unit 14 is connected to the external input device 17, the stereo matching unit 11, the center point estimation unit 13, and the detection temperature range adjustment unit 15. The area ratio estimation unit 14 estimates the ratio of the area of ​​the object in the thermal image to the area of ​​the environment area based on the distance to the object output from the stereo matching unit 11 and the center point position of the object output from the center point estimation unit 13. For example, the area ratio estimation unit 14 estimates (calculates) the ratio of the area of ​​the object to the area of ​​the environment area for all pixels in which the object is captured.

[0047] Then, the area ratio estimation unit 14 outputs the ratio of the estimated area of ​​the object and the area of ​​the environmental area to the detection temperature range adjustment unit 15. At this time, for example, the estimated area of ​​the object and the area of ​​the environmental area themselves may be output, or the ratio of the area of ​​the object and the area of ​​the environmental area and the area of ​​one of the object and the environmental area may be output as a pair of information.

[0048] In addition, when the detection object is a person, for example, it is expected that the area of ​​the face of the object, i.e., the person, will be smaller than the actual situation due to wearing a hat or a mask, etc. Therefore, the estimated area of ​​the object and the area of ​​the environment area can be adjusted as needed by operating the external input device 17.

[0049] The detection temperature range adjustment unit 15 is connected to the external input device 17, the area ratio estimation unit 14 and the detection object determination unit 16. The detection temperature range adjustment unit 15 is a functional module for adjusting the detection temperature range of the detection object set in advance. For example, in the case where there are no pixels occupied only by the object in the thermal image, that is, all pixels of the object are integrated into the surrounding environment, etc., the temperature of the object cannot be accurately detected based on the thermal image.

[0050] In this case, the detection temperature range adjustment unit 15 adjusts the detection temperature range of the detection object based on the preset detection temperature range of the detection object and the ratio of the area of ​​the object in the thermal image output from the area ratio estimation unit 14 to the area of ​​the environmental area, and calculates the adjusted detection temperature range of the detection object. Then, the detection temperature range adjustment unit 15 outputs the calculated adjusted detection temperature range to the detection object determination unit 16. In addition, the detection temperature range of the detection object can also be adjusted by input from the external input device 17.

[0051] On the other hand, when there are pixels occupied only by an object in the thermal image, the detection temperature range adjustment unit 15 may not adjust the preset detection temperature range.

[0052] The detection object determination unit 16 is connected to the detection temperature range adjustment unit 15 and the external output device 18. The detection object determination unit 16 determines whether the thermal image contains an object of the detection object using the adjusted detection temperature range output from the detection temperature range adjustment unit 15. The detection object determination unit 16 outputs the determination result of the object captured in the thermal image to the external output device 18. At this time, the detection object determination unit 16 may also output the determination result only when it is determined that the object captured in the thermal image is the detection object.

[0053] Examples of the external output device 18 include a display, an on-vehicle ECU (Electronic Control Unit), etc. In the ECU, based on the determination result output from the detection target object determination unit 16, control such as braking is performed.

[0054] <Object recognition method>

[0055] Next, the object recognition method is described, taking the case where the detection target is a person as an example. Figure 2 is a flowchart showing an example of an object recognition method according to the first embodiment of the present invention. In object recognition using a thermal image, for example, Figure 2 Steps S201 to S208 are shown.

[0056] Step S201 is a step in which the imaging unit 10 acquires a thermal image. Figure 3 This is an explanatory diagram showing an example of a real image of thermal radiation of a distant object and an image in a thermal image corresponding to the real image. Figure 3 501 in (a) is a real image of the thermal radiation of the person and his / her surroundings when there is an object, i.e., a person, in the distance. The real image 501 represents the thermal radiation of the person and his / her surroundings in an environment with a constant temperature (e.g., the ambient temperature T B =20° C.) and a person is present in the distance, and the observation result of the thermal radiation of the real image 501 observed from the imaging unit 10. Figure 3 In (a), 505 is a person in the distance, and 503 is the face of the person. Figure 3 502 in (b) is a thermal image obtained by capturing the real image 501 by the imaging unit 10 . Figure 3 In (b), 506 is a person in the thermal image 502 , and 504 is the face of the person in the thermal image 502 .

[0057] Figure 3 507 in (c) is a diagram in which a face 503 of a person 505 representing a real image 501 is superimposed on a face 504 of a thermal image 502. Figure 3 As shown in (c), the face 503 of the real image 501 is included in four pixels of the face 504 of the thermal image 502. In addition, in each pixel, the face 503 is photographed in a state of being mixed with the background of the person 505.

[0058] Thus, in a state where the face and the background exist in one pixel and there is no pixel occupied only by the face, the thermal image 502 observes the average temperature of the face and the background, and cannot detect the actual temperature of the face 503. In addition, in the following description, the state where there is no pixel occupied only by the face of the detection object, that is, the person, is referred to as a state of insufficient resolution.

[0059] then, Figure 4 This is an explanatory diagram showing an example of a real image of thermal radiation of an object located nearby and an image in a thermal image corresponding to the real image. Figure 4 601 in (a) is a real image of the heat radiation of the person and his / her surroundings when the object, i.e., the person, is located nearby. The real image 601 represents the heat radiation of the person and his / her surroundings in a constant temperature environment (e.g., Figure 3 Likewise, the ambient temperature T B =20°C), and the observation result of thermal radiation of the real image 601 observed from the imaging unit 10 when a person exists nearby. Figure 4 In (a), 605 is a person located nearby, and 603 is the face of the person located nearby. Figure 4 602 in (b) is a thermal image obtained by capturing the real image 601 by the imaging unit 10 . Figure 4In (b), 606 is a person in the thermal image 602 , and 604 is the face of the person in the thermal image 602 .

[0060] Figure 4 In (c), 607 is a diagram in which a face 603 of a person 605 representing a real image 601 is superimposed on a face 604 of a thermal image 602. Figure 4 As shown in (c), the face 604 in the thermal image includes pixels occupied only by the face. Figure 3 Unlike the example of , the actual temperature of the face 603 of the real image 601 can be detected by the thermal image 602. In addition, in the following description, the case where only pixels occupied by the detection target object are included will be referred to as a state of sufficient resolution.

[0061] The imaging unit 10 of this embodiment includes two far-infrared imaging sensors and lenses corresponding to the far-infrared imaging sensors, so that two thermal images captured at the same time can be obtained. The imaging unit 10 outputs the obtained pair of thermal images to the stereo matching unit 11 and the specific temperature region extraction unit 12.

[0062] In step S202, the stereo matching unit 11 calculates the distance to the object. The stereo matching unit 11 performs stereo matching processing on a pair of thermal images output from the camera unit 10. The stereo matching unit 11 calculates the distance to the object based on the parallax information between the two thermal images, the first thermal image and the second thermal image. The distance calculated here is specifically the distance from the object to the camera unit 10 (for example, a far-infrared camera sensor). When a plurality of objects are included in the thermal image, the stereo matching unit 11 calculates the distances of all the objects separately. Alternatively, the distance may be calculated only for objects included in a specific area of ​​the thermal image. In addition, the parallax information is pre-set information and can be appropriately changed according to the structure of the camera unit 10.

[0063] As stereo matching methods, for example, methods such as BM (Block Matching), SGBM (Semi-Global BM), and GC (Graph Cut) can be used. When BM is used, algorithms such as SAD (Sum of Absolute Differences) and SSD (Sum of Squared Differences) can be used. However, the stereo matching method is not limited to this, and various other methods can also be applied.

[0064] In step S203 , the specific temperature region extraction unit 12 extracts a specific temperature region composed of pixels detecting a temperature within a preset specific temperature range from one or more thermal images outputted from the imaging unit 10 .

[0065] For example, when extracting a specific temperature area, the specific temperature area extraction unit 12 sets a search area consisting of a total of 4 pixels, 2 pixels in the vertical and horizontal directions, searches for pixels that detect temperatures within the specific temperature range in each search area, and extracts the search area including pixels that detect temperatures within the specific temperature range as the specific temperature area.

[0066] When there are pixels detecting temperatures within a specific temperature range in multiple adjacent search areas, the specific temperature area extraction unit 12 selects a search area with the highest average detected temperature of the four pixels constituting the search area if the ambient temperature is lower than the specific temperature range, and extracts the selected search area as the specific temperature area.

[0067] On the other hand, when there are pixels detecting temperatures within a specific temperature range in multiple adjacent search areas, the specific temperature area extraction unit 12 selects a search area with the lowest average detected temperature of the four pixels constituting the search area if the ambient temperature is higher than the specific temperature range, and extracts the selected search area as the specific temperature area.

[0068] Here, the method of setting a specific temperature range is described by taking the case where the detection object is a person as an example. In this case, the specific temperature area extraction unit 12 uses, for example, the temperature of the face of the person exposed to the outside for detection. The temperature of the face varies from person to person. In addition, due to various factors such as before and after exercise, whether to wear a mask, and ambient temperature, the temperature of the face fluctuates (deviations). The specific temperature range is set based on consideration of such fluctuations in the temperature of the face. In addition, the specific temperature range can also be adjusted by input from the external input device 17. In addition, the temperature of each pixel in each thermal image is statistically processed, and the temperature with the highest frequency is used as the ambient temperature.

[0069] Figure 5 It is a diagram for explaining the extraction process of a specific temperature region. Figure 5 In FIG. 7 , image 701 is a magnified representation of an area including a specific temperature area in thermal image 502 and several pixels around it.

[0070] The brightness value of each pixel in the thermal image does not represent the temperature itself. However, for ease of explanation, it is assumed that the brightness value of each pixel represents the temperature itself, and the temperature of each pixel is shown in the figure.

[0071] like Figure 5 As shown, it is assumed that the image 701 includes a black ambient temperature area (eg, ambient temperature T B =20°C), a specific temperature region 702 (e.g., specific temperature TD = 32°C) and other regions in the shaded area (e.g., temperature 25°C) shown in white. In image 701, the most frequent temperature is 20°C, so the detected ambient temperature T B= 20°C is identified as the ambient temperature area.

[0072] In addition, the region where the specific temperature TD=32° C. is detected is considered to be within the specific temperature range, and thus the region with a temperature of 32° C. is extracted as the specific temperature region 702 .

[0073] Figure 5 701 is used as an example to illustrate the extraction of a specific temperature region, but in fact, it is necessary to extract the specific temperature region in the entire region of the thermal image 502. The specific temperature region extraction unit 12 generates specific temperature region information including temperature information of the extracted specific temperature region and coordinate information of the extracted specific temperature region, and outputs the generated specific temperature region information to the center point estimation unit 13. The specific temperature region information includes, for example, the coordinates of all pixels constituting the specific temperature region and the detected temperature values.

[0074] In step S204 , the center point estimation unit 13 estimates the center point position of the object located in the specific temperature region based on the specific temperature region information output from the specific temperature region extraction unit 12 .

[0075] Figure 6 This is a diagram showing an example of a face and its center of a real image in a specific temperature range. Figure 6 , it is shown that the center point position 801 of the real image face 503 is located in the specific temperature region 702. The center point estimation unit 13 estimates the center point position 801 of the real image face 503, for example, using the temperature value and coordinates of each pixel of the specific temperature region 702 included in the specific temperature region information. At this stage, since the area of ​​the real image face 503 has not yet been calculated, the boundary of the real image face 503 is indicated by a dotted line. The method for calculating the area of ​​the real image face 503 will be described later.

[0076] Figure 7 This is a diagram for explaining an example of a method for estimating the center point position of an object. Figure 7 Expressed Figure 5 The center point estimation unit 13 first pre-sets the maximum temperature value of the specific temperature area 702. For example, if the detection target is a person, the maximum temperature value Tmax is set to 35°C, which is the average temperature of a human face.

[0077] Figure 7 The positions (i, j) shown in FIG. 1 represent the abscissa and ordinate of the position where the maximum temperature value is detected, respectively. Figure 7 and Figure 6 By comparison, it can be seen that the position (i, j) of the maximum temperature value is the center point position 801 of the face 503 of the real image.

[0078] Then, if Figure 7 As shown, in the specific temperature region 702, for example, four pixels having the same temperature value (e.g., 32° C.) are selected so as to surround the position (i, j) of the maximum temperature value. For the sake of convenience, the positions of these pixels are assumed to be (0, 0), (1, 1), (0, 1), (0, 1). That is, the temperatures P(0, 0), P(1, 1), P(0, 1), P(0, 1) at the positions (0, 0), (1, 1), (0, 1), (0, 1) are 32° C.

[0079] Position (i, 0) is the position of the maximum temperature P(i, 0) in the horizontal direction, and position (0, j) is the position of the maximum temperature P(0, j) in the vertical direction. The maximum temperature value P(i, 0) and the maximum temperature value P(0, j) in the vertical direction are equal to the maximum temperature value Tmax = 35°C.

[0080] The position of the maximum temperature value is calculated using, for example, the following equations (1) and (2).

[0081]

[0082]

[0083] Formula 1 is a formula for calculating the horizontal coordinate of the position of the maximum temperature value based on the ratio of the difference between the temperature at each position and the maximum temperature value. Formula 2 is a formula for calculating the vertical coordinate of the position of the maximum temperature value based on the ratio of the difference between the temperature at each position and the maximum temperature value.

[0084] According to the temperature values ​​at the above positions and equations 1 and 2, the position of the maximum temperature value is calculated to be (0.5, 0.5). Figure 7 As shown in , when using 4 pixels with the same temperature value, the position (i, j) of the maximum temperature value is the center of the 4 pixels. In other words, if using Figure 7 According to the method, the center point position 801 of the face 503 of the real image is the center of the coordinates of the four pixels.

[0085] Next, another method of calculating the center point position will be described. Figure 7 In the example of , pixels with the same temperature value in a specific temperature region are selected. Here, a method for calculating the center point position when pixels with different temperature values ​​are selected is described.

[0086] Figure 8 This is a diagram for explaining another example of a method for estimating the center point position of an object. Figure 8In the equation, it is assumed that four pixels at positions (0, 0), (1, 0), (0, 1), and (1, 1) surrounding the position of the maximum temperature value are selected. In addition, it is assumed that the temperature values ​​detected by the selected four pixels are P(0, 0) = 28°C, P(1, 0) = 34°C, P(0, 1) = 28°C, and P(1, 1) = 34°C. Substituting these values ​​into equations 1 and 2, the maximum value position (i, j) is calculated to be (0.5, 0.8). Therefore, in this case, the coordinates of the center point position 1001 of the face 503 of the real image are (0.5, 0.8).

[0087] The center point estimation unit 13 outputs information on the calculated (estimated) center point position of the object (eg, the center point position of the face 503 of the real image) to the area ratio estimation unit 14 .

[0088] In addition, two methods of estimating the center position of an object by selecting four pixels surrounding the maximum temperature value position are described here, but the method of estimating the center position of an object is not limited to this. For example, the center position of an object can also be estimated using various methods such as interpolation method and quadratic polynomial fitting method.

[0089] In step S205 , the area ratio estimation unit 14 estimates the ratio between the area of ​​the object (eg, the real image face 503 ) and the area of ​​the environment based on the distance to the object calculated by the stereo matching unit 21 and the center point position of the object estimated by the center point estimation unit 23 .

[0090] Fig. 9 1 is a diagram for explaining a method of calculating the ratio of the area of ​​an object to the area of ​​an environment. The area ratio estimating unit 14 obtains the area information of the detection object (e.g., the area of ​​a face, etc.). The area information of the detection object is a preset value. The area information of the detection object may be a value stored in the object recognition device or a value input from the external input device 17.

[0091] The area ratio estimation unit 14 calculates the area S of the face 503 of the real image, which is the object, based on the distance to the object calculated in step S202. F (Unit is number of pixels). In addition, when the face area of ​​a person is reduced due to wearing a hat or mask, the actual area can be adjusted relative to the area information of the detection object. The adjustment of the area information of the detection object is performed by the external input device 17, for example.

[0092] For example, the face area of ​​a person at a distance of 100m is 350cm 2 In the case of F That is, in this case, the face 503 of the real image is S FBy calculating the area S of the face 503 F , the area ratio estimation unit 14 can calculate the area of ​​the ambient temperature region 1101 within the specific temperature region 702. If the area of ​​the specific temperature region 702 is S (pixels) and the area of ​​the ambient temperature region 1101 is S B (pixels), then S = S F +S B The area ratio estimation unit 14 calculates the area S of the ambient temperature region 1101 using this relational expression: B .

[0093] Then, the area ratio estimation unit 14 calculates (estimates) the area S of the face 503. F and the area S of the ambient temperature region 1101 B The calculated area S of the face 503 is F The ratio of the area of ​​the ambient temperature region 1101 is output to the detection temperature range adjustment unit 15 .

[0094] Alternatively, the area ratio estimating unit 14 may calculate the area S of the face 503. F and the area S of the ambient temperature region 1101 B , and the area S of the face 503 F The area S of the ambient temperature region 1101 B The ratio of the face area S 503 may be output to the detection temperature range adjustment unit 15. F and the area S of the ambient temperature region 1101 B The ratio of only the area S of the face 503 F and the area S of the ambient temperature region 1101 B The detected temperature range is output to the detection temperature range adjustment unit 15 .

[0095] In step S206, the detection temperature range adjustment unit 15 adjusts the detection temperature range of the detection object (e.g., a person). The detection temperature range adjustment unit 15 obtains temperature information of the detection object (e.g., the temperature of a face, etc.). The temperature information of the detection object is a preset value. The temperature information of the detection object may be a value stored in the object recognition device or a value input from the external input device 17.

[0096] The detection temperature range adjustment unit 15 uses the detected ambient temperature (eg, T B =20°C) and the face area (S F )(the area of ​​the object) to calculate the temperature threshold T of a specific temperature area assuming the presence of a person S . Temperature threshold T SFor example, the following formula 3 is used for calculation. In formula 3, T F is the temperature of an object (for example, the temperature of a person's face).

[0097]

[0098] Fig.10 This is a diagram for explaining a method of setting a detection temperature range. Fig.10 2 shows the detection temperature range in this embodiment and the conventional detection temperature range in comparison. Fig.10 This shows a case where the detection target is a person.

[0099] The relationship diagram 1201 shows the relationship between the detection target distance and the detection temperature range of a person in this embodiment.

[0100] The relationship diagram 1204 shows the relationship between the detection object distance and the detection temperature range of the person in the prior art.

[0101] The horizontal axes of the relationship graphs 1201 and 1204 are the detection target distance, which indicates the distance from the object to the imaging unit 10 , and the vertical axes of the relationship graphs 1201 and 1204 are the temperature of the object detected by the far-infrared imaging sensor.

[0102] 1206, 1207, 1208, and 1209 are thermal images of the detected object, i.e., a person. In these thermal images, the white portion indicates the real image of the person. Thermal image 1206 is the image taken at the closest point among them. Therefore, the face of the person in thermal image 1206 is captured as large as possible. Thermal image 1207 is an image taken at a location farther than thermal image 1206. Thermal image 1208 is an image taken at a location farther than image 1207, and is an image at the limit distance where the resolution of the person's face is sufficient. Thermal image 1209 is an image taken at a distance where the resolution of the person himself is insufficient.

[0103] Conditions such as the body temperature and facial area of ​​a person fluctuate. If the temperature range for detecting a person is not set based on such fluctuations, even though a person is actually captured in the thermal image, only the person that meets the predetermined conditions can be detected. Specifically, in step S205, the facial area of ​​a person at a distance of 100m is set to 350cm 2 However, if the detection temperature range is not set, it will not be possible to detect, for example, an area of ​​400cm 2 character.

[0104] Therefore, the detection temperature range adjustment unit 15 sets the temperature range and the area range of the face of the person, respectively, calculates the upper limit detection temperature and the lower limit detection temperature, and sets the detection temperature range of the person.

[0105] For example, set the face area to 300 to 400 cm 2 , the temperature range of the person's face is set to 34-36° C. The area range and the temperature range may be values ​​stored in the object recognition device or values ​​input from the external input device 17 .

[0106] The detection temperature range adjustment unit 15 minimizes the facial area (300cm 2 ) and the face temperature (surface temperature) is the highest (36°C) as the upper limit detection temperature. On the other hand, the detection temperature range adjustment unit 15 sets the detection temperature of the person with the largest face area (400cm 2 ), the detection temperature of a person under the condition of the lowest face temperature (34°C) is set as the lower limit detection temperature.

[0107] As shown in the relationship diagram 1201, the detection temperature range of a person is a range from the lower limit detection temperature to the upper limit detection temperature. In this way, by setting the detection temperature range of a person, a person whose temperature and face area fluctuate within a certain range can be detected as a detection object.

[0108] However, as described above, in a state of insufficient resolution where a face and a background exist in one pixel and there is no pixel occupied only by a face, the average temperature of the face and the background is observed, and the detected temperature is out of the detection temperature range. Therefore, using the existing detection temperature range, it is impossible to detect the detection object in a state of insufficient resolution.

[0109] Therefore, in the present embodiment, the detection temperature range is adjusted in a state where the detection object is far away and the resolution is insufficient. In the case where the detection object is a person, if the ambient temperature is lower than the face temperature of the person, the detection temperature of the pixel of the face 503 captured in the real image will be lower than the face temperature. Therefore, the detection temperature range adjustment unit 15 sets the lower limit detection temperature in the state of insufficient resolution to be lower than the state where the detection object distance is short and the resolution is sufficient. The adjustment amount of the lower limit detection temperature here can use a preset value, or it can be calculated by calculation, such as multiplying the lower limit detection temperature before adjustment in the state where the detection object distance is short and the resolution is sufficient by a predetermined ratio.

[0110] An example of a method for adjusting the detection temperature range will be specifically described. Pixels occupied by a person's face exist in thermal image 1206 and thermal image 1207. At a distance where these thermal images can be obtained, the detection temperature range is not adjusted.

[0111] In contrast, at the distance where the thermal image 1208 is obtained, there are no pixels occupied only by the face, so the face temperature will be detected to be lower than the actual temperature. Fig.10As shown, when the distance of the object is far away and exceeds the distance where the resolution is insufficient, the lower limit detection temperature is set to be lower.

[0112] In addition, if Fig.10 As shown, compared with the distance corresponding to the thermal image 1208 at which the resolution reaches its limit, the lower limit detection temperature may be gradually lowered starting from a shorter distance, that is, the detection temperature range may be gradually expanded.

[0113] On the other hand, if the ambient temperature is higher than the detected temperature of the object, the detected temperature range adjustment unit 15 adjusts the detected temperature range by increasing the upper limit detected temperature. Fig.10 As shown, in a state where the resolution is insufficient, the detection temperature range is set larger than in a state where the resolution is sufficient.

[0114] In step S207, the detection object determination unit 207 determines whether the object in the specific temperature range is a detection object. Figure 5 In the example of FIG. 1 , the specific temperature of the specific temperature region 702 is TD=32° C., and the detection target object determination unit 16 determines whether the specific temperature TD is within the detection temperature range 1202 of a person.

[0115] When the specific temperature TD is within the detection temperature range 1202 of a person ("Yes"), the detection object determination unit 16 determines that the object within the specific temperature region is the detection object (e.g., a person). In this case, the process proceeds to step S208, and the detection object determination unit 16 outputs information such as the thermal image and the specific temperature region information to the external output device 18, thereby ending the object detection process.

[0116] On the other hand, when the specific temperature TD is not within the detection temperature range 1202 of a person in step S207 ("No"), the detection object determination unit 16 determines that the object within the specific temperature region is not a detection object (e.g., a person). In this case, the thermal image, specific temperature region information, and other information are not output, and the object detection process based on the thermal image is directly terminated.

[0117] <Main Effects of the Present Embodiment>

[0118] According to the present embodiment, the detection temperature range of the object within the specific temperature range is adjusted based on the distance to the object calculated using stereo matching of thermal images to determine whether the object within the specific temperature range is a detection target object.

[0119] This structure eliminates the need to use multiple sensors, making it possible to accurately detect distant objects while suppressing cost increases.

[0120] In addition, according to the present embodiment, the area ratio estimation unit 14 estimates the ratio of the area of ​​the object to the area of ​​the environment based on the distance to the object and the center point position of the specific temperature zone. In addition, the detection temperature range adjustment unit 16 calculates the temperature threshold of the object and adjusts the detection temperature range for the specific temperature zone based on the distance to the object and the area ratio. With this structure, the detection temperature range can be appropriately set.

[0121] In addition, according to this embodiment, the detection temperature range adjustment unit 15 changes the detection temperature range according to the environmental temperature. With this configuration, it is possible to set an appropriate detection temperature range according to the environment.

[0122] Furthermore, according to this embodiment, the range of the detection determination temperature can be changed by the external input device 17. According to this configuration, an appropriate detection determination temperature can be set according to the state of the object or the detection target object.

[0123] Furthermore, according to this embodiment, the set value of the area of ​​the object can be changed by the external input device 17. According to this configuration, an appropriate detection determination temperature can be set according to the state of the object or the detection target object.

[0124] <Other>

[0125] The case of human detection is used as an example for explanation here, but the detection object is not limited to human beings. By adjusting information such as a specific temperature range, the actual area of ​​the detection object, and the temperature of the detection object, various warm-blooded animals and warm-blooded objects can be set as detection objects. For example, in the case of detecting warm-blooded animals with a lower temperature than human beings, these detections can be achieved by lowering the detection temperature range according to Formula 3. In addition, for example, in the case of detecting warm-blooded animals and warm-blooded objects with a higher temperature than human beings, these detections can be achieved by increasing the detection temperature range according to Formula 3.

[0126] In addition, when detecting warm-blooded animals and warm-blooded objects larger than a person, by replacing the area (S) of the face 503 of the real image with the area (S) in equation 3, F ) is set to a larger value, these detections can also be achieved. In the case of detecting warm-blooded animals smaller than humans, by replacing the area of ​​the real image face 503 (S F ) is set to a smaller value, these detections can also be achieved.

[0127] [Comparison with prior art]

[0128] As already explained above, if Fig.10As shown, the prior art does not adjust the detection temperature range corresponding to the distance to the object. Therefore, when the resolution limit of the object is exceeded, the temperature detected by the pixel is affected by the environment, etc., and the accurate temperature of the object cannot be detected, so it is difficult to reliably detect the detection object.

[0129] For example, suppose that the ambient temperature is lower than the temperature of the object. When the object is located near the camera unit 10 and the resolution is sufficient, there are pixels occupied only by the object, so the actual temperature of the object can be detected. On the other hand, when the resolution is insufficient, because the environment and other objects are integrated into one pixel, the detected temperature is averaged, and the detected temperature is lower than the temperature of the object. In the prior art, because the detection temperature range is not adjusted according to the distance, when the resolution exceeds the limit, the detection object cannot be accurately detected.

[0130] (Implementation Method 2)

[0131] Next, the second embodiment will be described. Hereinafter, descriptions of parts overlapping with the above-mentioned embodiment will be appropriately omitted.

[0132] In this embodiment, the detection temperature range is adjusted according to the distance, and the object within the detection temperature range is judged. However, when setting the detection temperature range, it is necessary to detect the temperature of the object in advance. In this regard, this embodiment estimates the temperature of the object based on the distance of the object calculated by stereo matching of the far-infrared camera sensor. With this embodiment, since the actual temperature of the object can be estimated, it can be applied to the temperature detection of objects with insufficient resolution. This embodiment is also not limited to human detection.

[0133] Fig.11 This is a structural diagram showing an example of the structure of an object recognition device according to embodiment 2 of the present invention. Fig.11 The object recognition device 300 and Figure 1 The object recognition device 100 is similar to the one described above, and the detection temperature range adjustment unit 15 and the detection target object determination unit 16 are replaced with the object temperature estimation unit 37 .

[0134] The object temperature estimation unit 37 is as follows Fig.11 As shown, the object temperature estimation unit 37 is connected to the area ratio estimation unit 14 and the external output device 18. The object temperature estimation unit 37 calculates (estimates) the temperature of the object based on the ratio of the area of ​​the object in the specific temperature area and the area of ​​the environment area output from the area ratio estimation unit 14. Then, the object temperature estimation unit 37 outputs the calculated temperature of the object (estimated temperature) to the external output device 18.

[0135] Fig.12 This is a flowchart showing an example of an object recognition method according to the second embodiment of the present invention. Fig.12 and Figure 2 similar, Figure 2 Steps S206 to S208 are replaced by steps S409 to S410.

[0136] Fig.12 The processing of steps S201 to S204 is the same as that of embodiment 1. Step S205 is substantially the same as that of embodiment 1. In embodiment 1, the face of a person is used as an example for explanation, but in this embodiment, an object of a known size is used as an example for explanation. For example, suppose there is an object with an area of ​​350 cm in the distance. 2 known objects.

[0137] The area ratio estimation unit 14 calculates the distance to the object based on the distance to the object calculated in step S202 and the known area of ​​the object, 350 cm 2 , estimate the area of ​​the object in the thermal image of the object (e.g., the area of ​​the face S F The area ratio estimating unit 14 outputs the estimated area ratio to the object temperature estimating unit 37.

[0138] In step S409, the object temperature estimation unit 37 estimates the actual temperature T of the object. F In addition, the actual temperature of an object is referred to as the actual temperature below. F For example, it is estimated using the following formula 4.

[0139]

[0140] The object temperature estimation unit 37 uses the detected temperature T in the specific temperature region. K (For example, T K =32℃), the detected ambient temperature T B (For example, T B =20℃), the area S of the object is calculated F and the area S of the ambient temperature region B ratio to estimate the actual temperature T of the object F .

[0141] In step S410, the object temperature estimation unit 37 calculates the actual temperature T of the object. F Output to the external output device 18. When step S410 is completed, the object recognition process is completed.

[0142] According to this embodiment, the actual temperature of a distant object with insufficient resolution can be estimated. Thus, this embodiment can also be applied to the temperature detection of an object with insufficient resolution.

[0143] In addition, the present invention is not limited to the above-mentioned embodiments, and includes various modified examples. For example, the above-mentioned embodiments are detailed descriptions for easy understanding of the present invention, and are not limited to all the structures described. In addition, a part of the structure of a certain embodiment can be replaced with the structure of other embodiments, and the structure of other embodiments can be added to the structure of a certain embodiment.

[0144] In addition, for a part of the structure of each embodiment, other structures can be added, deleted, or replaced. Moreover, the above-mentioned structures, functions, processing parts, processing units, etc. can be partially or completely implemented in hardware, for example, through integrated circuit design, etc. The above-mentioned structures, functions, etc. can also be implemented in software by a processor interpreting and executing programs that implement each function. Information such as programs, tables, files, etc. that implement each function can be stored in a recording device such as a memory, a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

[0145] Description of Reference Numerals

[0146] 10...imaging unit, 11...stereo matching unit, 12...specific temperature extraction unit, 13...center point estimation unit, 14...area ratio estimation unit, 15...detection temperature range adjustment unit, 16...detection object determination unit, 100, 300...object recognition device.

Claims

1. An object recognition device, characterized in that: include: An imaging unit for acquiring a plurality of thermal images of the captured object; A stereo matching unit, which uses the plurality of thermal images to perform stereo matching and calculate the distance to the object; a specific temperature region extracting unit, which extracts a specific temperature region from the thermal image; an area ratio estimating unit for estimating, for the specific temperature region, a ratio of an area of ​​the object included in a pixel based on the distance to the object calculated by the stereo matching unit; a detection temperature range adjustment unit, which adjusts the detection temperature range of the object; and The detection target object determination unit determines whether the object existing in the specific temperature region is a detection target object based on the adjusted detection temperature range.

2. The object recognition device according to claim 1, wherein: comprising a center point estimating unit for estimating the center point position of the specific temperature area, The area ratio estimating unit estimates the ratio of the area of ​​the object to the area of ​​the environment based on the distance to the object calculated by the stereo matching unit and the center point position estimated by the center point estimating unit. The detection temperature range adjustment unit calculates the temperature threshold of the object for the specific temperature region based on the distance calculated by the stereo matching unit and the ratio, and adjusts the detection temperature range. The detection target object determination unit determines whether the object existing in the specific temperature region is the detection target object by determining whether the temperature of the specific temperature region is within the adjusted detection temperature range.

3. The object recognition device according to claim 2, wherein: The detection temperature range adjustment unit changes the detection temperature range according to an ambient temperature.

4. The object recognition device according to claim 2, wherein: The range of the detection determination temperature can be changed by an external input device provided outside the object recognition device.

5. The object recognition device according to claim 2, wherein: The setting of the area of ​​the object can be changed by an external input device provided outside the object recognition device.

6. The object recognition device according to claim 1, wherein: An object temperature estimating unit is included, which estimates the temperature of the object based on the distance calculated by the stereo matching unit for the specific temperature area.

7. The object recognition device according to claim 6, characterized in that: comprising a center point estimating unit for estimating the center point position of the specific temperature area, The area ratio estimating unit estimates the ratio of the area of ​​the object to the area of ​​the environment based on the distance to the object calculated by the stereo matching unit and the center point position estimated by the center point estimating unit. The object temperature estimation unit calculates an actual temperature of the detection target object of the object based on the distance to the object calculated by the stereo matching unit and the ratio for the specific temperature area.

8. An object recognition method, characterized in that: The camera unit acquires multiple thermal images of the object. A specific temperature region extracting unit extracts a specific temperature region from the thermal image. The area ratio estimating unit estimates the ratio of the area of ​​the object included in the pixel for the specific temperature region based on the distance to the object calculated by stereo matching using the plurality of thermal images. The detection temperature range adjustment unit adjusts the detection temperature range of the object. The detection target object determination unit determines whether the object existing in the specific temperature region is a detection target object based on the adjusted detection temperature range.

9. The object recognition method according to claim 8, characterized in that: The center point of the specific temperature area is estimated by a center point estimation unit. The area ratio estimation unit estimates the ratio of the face area of ​​the person to the environment area based on the distance calculated by the stereo matching and the estimated position of the center point. The detected temperature range adjustment unit calculates a temperature threshold of the object for the specific temperature region based on the distance calculated by the stereo matching and the ratio, and adjusts the detected temperature range. The detection target object determination unit determines whether the temperature of the specific temperature region is within the adjusted detection temperature range, thereby determining whether the object existing in the specific temperature region is the detection target object.

10. The object recognition method according to claim 8, characterized in that: The object temperature estimating unit estimates the temperature of the object in the specific temperature region based on the distance to the object calculated by the stereo matching.

11. The object recognition method according to claim 10, characterized in that: The center point of the specific temperature area is estimated by a center point estimation unit. The area ratio estimating unit estimates the ratio of the area of ​​the object to the area of ​​the environment based on the distance to the object and the position of the center point. The object temperature estimating unit calculates an actual temperature of the object based on the calculated distance to the object and the ratio for the specific temperature area.

Images