Object detection apparatus and object detection method

Abstract

To provide an object detection apparatus and an object detection method capable of properly recognizing a pedestrian and a person who is on a bicycle, a motorcycle, and the like.SOLUTION: An object detection apparatus 100 includes: a video acquisition unit 110 which acquires a captured video; an object detection unit 120 which detects a person in a first mode or a person in a second mode using a first mode detection model trained with videos of the first mode of persons and a cyclist detection model trained with videos of the second mode of persons, for the video; and a determination unit 130 which analyzes, based on a result of the detection in the video, change in the video in a lower part of a detection range, for the detection range in which a detection score of the first mode detection model is equal to or higher than a first threshold and a detection score of the second detection model is equal to or higher than a second threshold which is higher than the first threshold, and determines, when the change is significant, that a person in the first mode has been detected for the detection range.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an object detection device and an object detection method.

Background Art

[0002] In recent years, the implementation of information processing systems for detecting pedestrians and persons riding bicycles, motorcycles, etc., i.e., cyclists, riders, etc., from images captured using infrared cameras, has been progressing in vehicles, smart poles, etc. In a smart pole, it is required to distinguish and detect pedestrians, bicycles ridden by persons, motorcycles ridden by persons, etc. When detecting a person from an image, it may be desirable to be able to distinguish a pedestrian, a cyclist, and a rider. For example, Patent Document 1 discloses a technique for determining whether a detected person is a pedestrian or a cyclist based on the detection presence or absence of wheels with respect to the lower part of the detected person.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a person is moving horizontally as viewed from the camera, it is easy to detect the wheels, so it is easy to recognize whether the person is a pedestrian, a cyclist, or a rider. On the other hand, when a person is moving in a direction toward the camera or away from the camera, it may be difficult to detect the wheels, and there is room for improvement in appropriately recognizing whether the person is a pedestrian, a cyclist, or a rider. Also, it has been required to appropriately recognize a cyclist and a rider.

[0005] The present disclosure has been made in view of such problems, and provides an object detection device and an object detection method that can appropriately recognize pedestrians, persons riding bicycles or motorcycles, etc.

Means for Solving the Problems

[0006] The object detection device according to the present disclosure includes: a video acquisition unit that acquires a video captured by an infrared camera; an object detection unit that detects a person in the first form or a person in the second form by using a first form detection model that has learned a video of a person in the first form and a second form detection model that has learned a video of a person in the second form for the acquired video; a determination unit that analyzes fluctuations in the video in a predetermined portion of the detection target range based on the detection results of the first form and the second form in the acquired video, and when the fluctuations are large, determines that a person in the first form is detected for the detection target range, where the detection score by the first form detection model is equal to or higher than a first threshold value and the detection score by the second form detection model is equal to or higher than a second threshold value that is higher than the first threshold value; and is provided with.

[0007] The object detection method according to the present disclosure includes: a step of acquiring a video captured by an infrared camera; a step of detecting a person in the first form or a person in the second form by using a first form detection model that has learned a video of a person in the first form and a second form detection model that has learned a video of a person in the second form for the acquired video; a step of analyzing fluctuations in the video in a predetermined portion of the detection target range based on the detection result of the person in the first form or the person in the second form, and when the fluctuations are large, determining that a person in the first form is detected for the detection target range, where the detection score by the first form detection model is equal to or higher than a first threshold value and the detection score by the second form detection model is equal to or higher than a second threshold value that is higher than the first threshold value; This is executed by the object detection device.

Advantages of the Invention

[0008] According to the present disclosure, it is possible to provide an object detection device and an object detection method that can appropriately recognize pedestrians, persons riding bicycles or motorcycles, etc.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0010] <Embodiment 1> First, with reference to FIG. 1, the configuration of the object detection device 100 and the object detection system 400 including the object detection device 100 according to the present disclosure will be described. The object detection system 400 is an information processing system that detects a person or the like from an image captured using an infrared camera. As shown in FIG. 1, the object detection system 400 includes an object detection device 100, a camera 200, and an output device 300.

[0011] The object detection system 400 is provided in a device that needs to detect people such as pedestrians, cyclists, and riders. Devices that need to detect these people include, for example, smart poles. A smart pole is a multifunctional pole equipped with various sensors including a camera and a communication device, installed near roads and the like, and used for traffic monitoring, safety support, crime prevention, etc. The object detection device according to the embodiment is mounted on such a smart pole, detects detection target objects such as people and vehicles from a thermal image captured by a far-infrared camera, and outputs information such as the type, coordinates, and moving speed of the detected detection target objects. The information output from the smart pole is used as information to give warnings to vehicles running around the smart pole. The object detection system 400 is applicable not only to smart poles but also to other applications such as vehicles and monitoring devices. The object detection system 400 detects a bicycle on which a person is riding by detecting a cyclist. The object detection system 400 detects a motorcycle on which a person is riding by detecting a rider. The object detection system 400 also detects automobiles and the like.

[0012] The camera 200 is an infrared camera, a (FIR: Far Infrared Rays) camera for capturing an image based on far-infrared rays emitted by an object, i.e., a so-called thermal image. The camera 200 captures the thermal image in the shooting range as a moving image, for example, at 15 to 30 frames per second. The installation location of the camera 200 is not particularly limited. The camera 200 is installed at a location where it captures the range through which a person riding a bicycle, i.e., a cyclist, a person riding a motorcycle, i.e., a rider, and a pedestrian pass. The camera 200 is installed, for example, near a road or at a bicycle parking lot. Also, the camera 200 is installed in a direction that captures the traveling direction of a vehicle in order to notify the vehicle and the like that need to detect a person riding a bicycle, i.e., a cyclist, a person riding a motorcycle, i.e., a rider, and a pedestrian of the detected information. The camera 200 is communicable with the object detection device 100. The communication method between the camera 200 and the object detection device 100 is not particularly limited and may be wired or wireless.

[0013] The structure of the camera 200 is not shown because it is the structure of a general far-infrared camera, and it includes sensors such as a lens, a shutter, and a microbolometer.

[0014] The output device 300 is a device that outputs the result detected by the object detection device 100. The output device 300 may be, for example, a device possessed by an administrator who manages the location where the camera 200 is installed. The communication method between the object detection device 100 and the output device 300 is not particularly limited and may be wired or wireless.

[0015] When the object detection system 400 is provided in a smart pole or the like, the output device 300 is a device that outputs the result detected by the object detection device 100 to a server or the like that centrally manages the smart pole using a known communication technology. Also, it may be a device that outputs the result detected by the object detection device 100 to a vehicle traveling in the vicinity of the smart pole using a known wireless communication technology. When the object detection system 400 is provided in a moving body such as a vehicle, the output device 300 may be a device such as a monitor that notifies a driver or an operator who operates the moving body such as a vehicle of the result detected by the object detection device 100 using video output or the like.

[0016] The object detection device 100 is an information processing device that detects a person or the like based on the shape of the heat distribution or the like from the video captured by the camera 200 and discriminates the movement type of the person. Specifically, the object detection device 100 discriminates whether the person detected from the video captured by the camera 200 is a pedestrian or a cyclist. A pedestrian includes not only a person who is walking but also a person who is standing still. As shown in FIG. 1, the object detection device 100 includes a video acquisition unit 110, an object detection unit 120, a determination unit 130, and an output control unit 140 as functional blocks realized by processing such as a program.

[0017] The video acquisition unit 110 performs a process of acquiring the video captured by the camera 200. The video acquired by the video acquisition unit 110 shows the thermal distribution in the shooting range by luminance values, and the presence or absence of a person in the shooting range is shown as the presence or absence of a heat source. As a known technique for the video acquisition unit 110 to acquire the video captured by the camera 200, which is a far-infrared camera, it includes a defective pixel correction unit (not shown), a NUC (Non-Uniformity Correction) unit, and the like.

[0018] The object detection unit 120 calculates a person detection score for the video acquired by the video acquisition unit 110 using a detection model that has learned various forms of a person. As an example showing various forms of a person, a pedestrian, a cyclist, or a rider may be defined as a first form or a second form. Further, as an example of a detection model that has learned various forms of a person, a pedestrian detection model, a cyclist detection model, or a rider detection model may be defined as a first form detection model or a second form detection model.

[0019] The object detection unit 120 detects the person in the first form or the person in the second form by using the first form detection model obtained by learning the video of the first form of a person and the second form detection model obtained by learning the video of the second form of a person. Specifically, the object detection unit 120 performs a process of detecting a person by calculating a pedestrian detection score, a cyclist detection score, and a rider detection score. Specifically, the object detection unit 120 calculates a pedestrian detection score by using a pedestrian detection model, calculates a cyclist detection score by using a cyclist detection model, and calculates a rider detection score by using a rider detection model. The pedestrian detection model is a model obtained by learning the video of a pedestrian. The pedestrian detection score numerically indicates the similarity between the object detected in the score calculation area and the pedestrian detection model. The cyclist detection model is a model obtained by learning the video of a cyclist who is a person riding a bicycle. The cyclist detection model is a model obtained by learning the video in a range including the bicycle on which the cyclist is riding in addition to the cyclist. The cyclist detection score numerically indicates the similarity between the object detected in the score calculation area and the cyclist detection model. The rider detection model is a model obtained by learning the video of a rider who is a person riding a motorcycle. The rider detection model is a model obtained by learning the video in a range including the motorcycle on which the rider is riding in addition to the rider. The rider detection score numerically indicates the similarity between the object detected in the score calculation area and the rider detection model.

[0020] For each frame of the video acquired by the video acquisition unit 110, the object detection unit 120 scans a detection window and calculates a pedestrian detection score, a cyclist detection score, and a rider detection score within the detection target range defined by the detection window. That is, detection processing using a pedestrian detection model, a cyclist detection model, and a rider detection model is performed on the detection target range defined by the detection window. The pedestrian detection score, cyclist detection score, and rider detection model calculated by the object detection unit 120 are calculated as numerical values in the range of, for example, 0.0 to 1.0. The higher the likelihood that a pedestrian, cyclist, or rider is included in the detection target range, the larger the numerical value of the detection score. When the detection score using the pedestrian detection model is 0.8 or higher for the detection target range, the object detection unit 120 determines that a pedestrian is included in the detection target range. When the score using the cyclist detection model is 0.8 or higher, it is determined that a cyclist is included in the detection target range. When the score using the rider detection model is 0.8 or higher, it is determined that a rider is included in the detection target range. The fact that a pedestrian, cyclist, or rider is included in the detection target range means that a pedestrian, a bicycle with a person on it, or a motorcycle with a person on it is captured within the detection target range defined by the detection window in the video acquired by the video acquisition unit 110.

[0021] Based on the detection results of the first-type person and the second-type person by the object detection unit 120, the determination unit 130 detects the first-type person or the second-type person. Specifically, the determination unit 130 analyzes the variation of the video in a predetermined part of the detection target range for a detection target range where the detection score by the first-type detection model is equal to or higher than the first threshold value and the detection score by the second-type detection model is equal to or higher than the second threshold value, which is a value higher than the first threshold value. Then, based on the analysis result, when the variation is large, the determination unit 130 determines that a first-type person is detected for the detection target range. In the present embodiment, the determination unit 130 analyzes the variation of the video in the lower part of the detection target range.

[0022] The second threshold value for the determination unit 130 to determine the form of a person is a numerical value indicating that it is highly likely that the detection target within the detection target range is a person in the first form or a person in the second form, or clearly indicates that it is a person in the first form or a person in the second form. When the detection score is detected in the range of 0.0 to 1.0, the second threshold value is set to, for example, 0.8. On the other hand, the first threshold value is a numerical value indicating that it is not highly likely that the detection target within the detection target range is a person in the first form or a person in the second form, and although it is not obvious, there is a possibility. When the detection score is detected in the range of 0.0 to 1.0, the first threshold value is set to, for example, 0.6.

[0023] In the present embodiment, the determination unit 130 performs a process of determining whether the detected person is a pedestrian or a cyclist based on the detection result of the object detection unit 120. Specifically, for a detection target range where the detection score by the cyclist detection model defined as the first form detection model is equal to or higher than the second threshold value t2, the determination unit 130 determines that a cyclist is included. In addition, for a detection target range where the detection score by the pedestrian detection model defined as the second form detection model is equal to or higher than the second threshold value t2, the determination unit 130 determines that a pedestrian is included. Further, when the detection score by the cyclist detection model is not equal to or higher than the second threshold value t2, is equal to or higher than the first threshold value t1 which is a value lower than the second threshold value t2, and the detection score by the pedestrian detection model is equal to or higher than the second threshold value t2, the determination unit 130 analyzes the variation of the video in the lower part of the detection target range. When the variation of the video in the lower part of the detection target range is large, the determination unit 130 determines that a cyclist is included in the detection target range. When the video acquired by the video acquisition unit 110 shows the heat distribution as a luminance value, the determination unit 130 may analyze the variation of the luminance value in the lower part of the detection target range. The output control unit 140 outputs the determination result of the determination unit 130 to the output device 300.

[0024] The determination unit 130 analyzes the fluctuations in the video at the lower part of the detection target range, and determines that a cyclist is detected in the detection target range when the fluctuations at the lower part of the detection target range occur alternately from left to right. Further, the determination unit 130 analyzes the fluctuations in the vertical direction of the lower end position of the video at the lower part of the detection target range, and determines that a cyclist is detected in the detection target range when the fluctuations in the lower end position occur alternately from left to right at the lower part of the detection target range.

[0025] The fluctuations in the video analyzed by the determination unit 130 are, for example, a state in which in the range constituting a person in a thermal image, a state where the temperature is high and a state where the temperature is low occur alternately to the extent that a person is detected. The determination unit 130 particularly detects the fluctuations in the part corresponding to the feet of the person and the fluctuations in the part corresponding to the arms of the person. Hereinafter, the lower part of the detection target range is the range corresponding to the feet of the person. The upper part of the detection target range is the range corresponding to the arms of the person, and is not the uppermost part of the detection target range, but the part below the range corresponding to the head of the person. The lower part of the detection target range may be above the central part in the vertical direction of the detection target range, and the lower part of the detection target range may be below the central part in the vertical direction of the detection target range. Also, the determination that the fluctuations occur alternately from left to right may be the case where the fluctuations occur alternately at positions symmetric with respect to the central part in the left-right direction of the detection target range.

[0026] Next, with reference to FIG. 2, the flow of the object detection method executed by the object detection device 100 according to the present disclosure will be described. FIG. 2 is a flowchart showing the flow of the object detection method according to the present disclosure. The object detection method shown in FIG. 2 is constantly executed when the object detection system 400 is provided in a smart pole or the like. In other words, the object detection device 100 constantly executes shooting by the camera 200, and detects pedestrians and cyclists with respect to the video shot by the camera 200. Since the camera 200 of the object detection device 100 according to the present disclosure is a far-infrared camera, the object detection method may be executed at night or when the ambient illuminance is low.

[0027] In addition, when the object detection method shown in FIG. 2 is provided in a moving body such as a vehicle in the object detection system 400, it is executed during the period when the moving body such as the vehicle is operating. Since the camera 200 of the object detection device 100 according to the present disclosure is a far-infrared camera, the object detection method may be executed at night or when the ambient illuminance is low while the moving body such as the vehicle is operating.

[0028] In the object detection method shown in FIG. 2, first, the video acquisition unit 110 acquires the video captured by the camera 200 (step S101). Next, the object detection unit 120 detects a person in the video acquired in step S101 (step S102). Specifically, the object detection unit 120 uses a pedestrian detection model trained with the video of pedestrians and a cyclist detection model trained with the video of a person riding a bicycle to detect a pedestrian or a cyclist in the acquired video.

[0029] Next, the determination unit 130 determines whether the cyclist detection score is equal to or greater than a second threshold value t2 for the detection target range in which the detection was executed in step S102 (step S103). The second threshold value t2 in this case is a numerical value indicating that the detection target within the detection target range is likely to be a cyclist or is clearly a cyclist. For example, 0.8 is set when the cyclist detection score is detected in the range of 0.0 to 1.0.

[0030] When the cyclist detection score is equal to or greater than the second threshold value t2 (step S103: Yes), the determination unit 130 determines that the person shown in the detection target range, that is, the detection target, is a cyclist (step S109). When the cyclist detection score is equal to or greater than the second threshold value t2, since the characteristics of a cyclist appear in the video within the detection target range, the possibility of being misrecognized as a pedestrian is low, and it is determined that the person is a cyclist. When the cyclist detection score is less than the second threshold value t2 (step S103: No), it is determined whether the cyclist detection score determined to be less than the second threshold value t2 is equal to or greater than a first threshold value t1 (step S104).

[0031] When the cyclist detection score is less than the first threshold value t1 (step S104: No), the determination unit 130 determines whether the pedestrian detection score is equal to or greater than the second threshold value t2 for the detection target range that is the same as the detection target range determined in step S103 (step S105). When the pedestrian detection score is equal to or greater than the second threshold value t2 (step S105: Yes), the determination unit 130 determines that the person appearing in the detection target range, that is, the detection target, is a pedestrian (step S106). The second threshold value t2 in this case is also a numerical value indicating that the detection target within the detection target range is likely to be a pedestrian or is clearly a pedestrian. For example, 0.8 is set when the pedestrian detection score is detected in the range of 0.0 to 1.0. When the pedestrian detection score is less than the second threshold value t2 (step S105: No), the determination unit 130 determines that neither a pedestrian nor a cyclist is included in the detection target range, and ends the process.

[0032] When the cyclist detection score is equal to or greater than the first threshold value t1 (step S104: Yes), the determination unit 130 determines whether the pedestrian detection score is equal to or greater than the second threshold value t2 for the detection target range that is the same as the detection target range determined in step S103 (step S107). When the pedestrian detection score is less than the second threshold value t2 (step S107: No), the determination unit 130 determines that the detection target is a cyclist (step S109). When the result in step S107 is No, although the cyclist detection score is lower than the second threshold value t2, the characteristics of a cyclist appear in the video within the detection target range to such an extent that it is equal to or greater than the first threshold value t1. Further, since the pedestrian detection score is less than the second threshold value, it is determined that the target is a cyclist rather than a pedestrian.

[0033] When the pedestrian detection score is greater than or equal to the second threshold value t2 (step S107: Yes), the determination unit 130 analyzes the variation in the video at the lower part of the detection target range and determines whether there is any variation (step S108). The analysis of the variation in the video at the lower part of the detection target range will be described later. When the video varies at the lower part of the detection target range (step S108: Yes), the determination unit 130 determines that the detection target is a cyclist (step S109). When the video does not vary at the lower part of the detection target range (step S108: No), the determination unit 130 determines that the detection target is a pedestrian (step S106).

[0034] If the result in step S107 is Yes, the cyclist detection score is greater than or equal to the first threshold value t1, and for the same detection target range, the pedestrian detection score is greater than or equal to the second threshold value t2. That is, although the characteristics as a cyclist appear, the pedestrian detection score is also a value indicating a high possibility of being a pedestrian. Therefore, even if it is actually a cyclist, there is a possibility of being misdetected as a pedestrian. For this reason, based on the determination in step S108, it is determined whether the detection target is a cyclist or a pedestrian.

[0035] Next, with reference to FIGS. 3 to 5, the analysis of the variation in the video at the lower part of the detection target range performed in step S108 will be described in detail.

[0036] FIG. 3 is an example of a video to be subjected to person detection. The left side of FIG. 3 is an example of a video captured by camera 200. The right side of FIG. 3 is an enlarged view of the dashed-line portion in the left side of FIG. 3. In the video shown in FIG. 3, a cyclist facing away from the camera is captured. Object detection unit 120 calculates a pedestrian detection score for the video captured by camera 200 using a pedestrian detection model, and calculates a cyclist detection score using a cyclist detection model. For example, in detection target range A defined by the detection window shown in FIG. 3, when the cyclist detection score is less than second threshold value t2 and equal to or greater than first threshold value t1, and further, when the pedestrian detection score is equal to or greater than second threshold value t2, determination unit 130 analyzes the variation of the video in lower portion B of detection target range A. When rowing a bicycle, the legs are often moved up and down more greatly than when walking. Therefore, in the detection target range in which a cyclist is shown, the variation of the video in the lower portion of the detection target range is greater than that in the detection target range in which a pedestrian is shown. Accordingly, determination unit 130 can determine whether the person included in the detection target range is a pedestrian or a cyclist by analyzing the variation of the video in the lower portion of the detection target range.

[0037] When cycling, the cyclist moves their legs up and down alternately from side to side. Therefore, in the detection target range where the cyclist is shown, fluctuations in the video occur alternately in the lower left and right areas. Thus, the determination unit 130 may determine that the person shown in the detection target range is a cyclist when fluctuations occur alternately from side to side in the lower part of the detection target range. Further, the determination unit 130 may determine that the person shown in the detection target range is a cyclist when fluctuations occur alternately from side to side at a predetermined interval in the lower part of the detection target range. For example, the determination unit 130 may determine that the person shown in the detection target range is a cyclist when the fluctuations are repeated at a frequency of about 0.5 to 1.5 times per second. Also, in a situation where the cyclist can move forward a certain distance even without pedaling, such as when going downhill, the cyclist may sometimes get on the bicycle without moving for a while with their feet on the pedals. In such a case, in the detection target range where the cyclist is shown, there may be a difference in the height of the video in the lower left and right areas. Thus, the determination unit 130 may analyze the difference in height between the left and right in the lower part of the detection target range and determine that the person shown in the detection target range is a cyclist when there is a difference in height between the left and right in the lower part of the detection target range.

[0038] When the video acquired by the video acquisition unit 110 shows the heat distribution as luminance values, the determination unit 130 may determine whether fluctuations occur alternately from side to side in the lower part of the detection target range by analyzing the fluctuations in the luminance values in the lower part of the detection target range. FIG. 4 is an example of the case of analyzing the fluctuations in the luminance values. The two videos shown in FIG. 4 are of the detection target range A including the same person taken at different times. In the left video of FIG. 4, the left leg of the cyclist is shown in the lower left part C1 of the detection target range A, and the luminance value in the area D1 is high. On the other hand, in the right video of FIG. 4, the right leg of the cyclist is shown in the lower right part C2 of the detection target range A, and the luminance value in the area D2 is high. The determination unit 130 may determine that a cyclist is shown in the detection target range A when changes in the luminance value of a predetermined level or more occur alternately in the lower left and right parts C1, C2.

[0039] The determination unit 130 may determine whether fluctuations occur alternately to the left and right in the lower part of the detection target range by analyzing the vertical fluctuations of the lower end position, that is, the lowermost position, in the lower part of the detection target range. FIG. 5 is an example of analyzing the fluctuations of the lowermost position. The two videos shown in FIG. 5 are of the detection target range A including the same person taken at different times. In the video on the left in FIG. 5, since the cyclist has lifted the right foot, the lower end position E2 in the lower right part C2 is higher than the lower end position E1 in the lower left part C1. On the other hand, in the video on the right in FIG. 5, since the cyclist has lifted the left foot, the lower end position E2 in the lower right part C2 is lower than the lower end position E1 in the lower left part C1. When changes in the lower end position of a predetermined amount or more occur alternately in the left and right lower parts C1 and C2, the determination unit 130 may determine that a cyclist is captured in the detection target range A.

[0040] In this way, the object detection device 100 can appropriately recognize a pedestrian or a person riding a bicycle.

[0041] Note that the object detection device 100 includes a processor, a memory, and a storage device as a configuration not shown. Further, a computer program in which the processing of the object detection method according to the present disclosure is implemented is stored in the storage device. Then, the processor causes the memory to read the computer program from the storage device and executes the computer program. Thereby, the processor realizes the functions of the video acquisition unit 110, the object detection unit 120, the determination unit 130, and the output control unit 140.

[0042] Alternatively, each component of the object detection device 100 may be implemented by dedicated hardware. Also, some or all of the components of each device may be implemented by general-purpose or dedicated circuitry, processors, etc., or a combination thereof. These may be constituted by a single chip, or may be constituted by a plurality of chips connected via a bus. Some or all of the components of each device may be implemented by a combination of the above-described circuitry, etc. and a program. Also, as the processor, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), quantum processor (quantum computer control chip), etc. can be used.

[0043] Also, when some or all of the components of the object detection device 100 are implemented by a plurality of information processing devices, circuitry, etc., the plurality of information processing devices, circuitry, etc. may be centrally arranged or may be distributed. For example, the information processing devices, circuitry, etc. may be implemented in a form in which each is connected via a communication network, such as a client-server system, a cloud computing system, etc. Also, the functions of the object detection device 100 may be provided in the form of SaaS (Software as a Service).

[0044] <Embodiment 2> In Embodiment 1, the case where a cyclist is defined as the first form and a pedestrian is defined as the second form was described. However, the definitions of the first form and the second form are not limited to this. In Embodiment 2, the case where a cyclist is defined as the first form and a rider is defined as the second form will be described. Note that descriptions overlapping with those in Embodiment 1 will be omitted as appropriate.

[0045] In this embodiment, the determination unit 130 performs a process of determining whether the detected person is a rider or a cyclist based on the detection result of the object detection unit 120. Specifically, the determination unit 130 determines that a cyclist is included in a detection target range where the detection score by the cyclist detection model defined as the first form detection model is equal to or greater than a second threshold value t2. Further, the determination unit 130 determines that a rider is included in a detection target range where the detection score by the rider detection model defined as the second form detection model is equal to or greater than the second threshold value t2. Furthermore, when the detection score by the cyclist detection model is not equal to or greater than the second threshold value t2, is equal to or greater than a first threshold value t1 which is a value lower than the second threshold value t2, and the detection score by the rider detection model is equal to or greater than the second threshold value t2, the determination unit 130 analyzes the variation of the video in the lower part of the detection target range. When the variation of the video in the lower part of the detection target range is large, the determination unit 130 determines that a cyclist is included in the detection target range.

[0046] The determination unit 130 analyzes the variation of the video in the lower part of the detection target range, and when the variation occurs alternately left and right in the lower part of the detection target range, determines that a cyclist is detected in the detection target range. Further, the determination unit 130 analyzes the variation in the vertical direction of the lower end position of the video in the lower part of the detection target range, and when the variation of the lower end position occurs alternately left and right in the lower part of the detection target range, determines that a cyclist is detected in the detection target range.

[0047] Next, with reference to FIG. 6, the flow of the object detection method executed by the object detection device 100 according to the present disclosure will be described. In the object detection method shown in FIG. 6, first, the video acquisition unit 110 acquires the video captured by the camera 200 (step S201). Next, the object detection unit 120 detects a person in the video acquired in step S201 (step S202). Specifically, the object detection unit 120 uses a rider detection model trained with the video of a person riding on a rider and a cyclist detection model trained with the video of a person riding on a bicycle to detect a rider or a cyclist in the acquired video.

[0048] Next, the determination unit 130 determines whether the cyclist detection score is equal to or greater than the second threshold value t2 for the detection target range in which the detection was executed in step S202 (step S203). The second threshold value t2 in this case is a numerical value indicating that the detection target within the detection target range is likely to be a cyclist or is clearly a cyclist. For example, 0.8 is set when the cyclist detection score is detected in the range of 0.0 to 1.0.

[0049] When the cyclist detection score is equal to or greater than the second threshold value t2 (step S203: Yes), the determination unit 130 determines that the person shown in the detection target range, that is, the detection target, is a cyclist (step S209). When the cyclist detection score is equal to or greater than the second threshold value t2, since the characteristics as a cyclist appear in the video within the detection target range, the possibility of being misrecognized as a rider is low, and it is determined to be a cyclist. When the cyclist detection score is less than the second threshold value t2 (step S203: No), it is determined whether the cyclist detection score determined to be less than the second threshold value t2 is equal to or greater than the first threshold value t1 (step S204).

[0050] When the cyclist detection score is less than the first threshold value t1 (step S204: No), the determination unit 130 determines whether the lidar detection score is equal to or greater than the second threshold value t2 for the detection target range that is the same as the detection target range determined in step S203 (step S205). When the lidar detection score is equal to or greater than the second threshold value t2 (step S205: Yes), the determination unit 130 determines that the person shown in the detection target range, that is, the detection target, is a rider (step S206). The second threshold value t2 in this case is also a numerical value indicating that the detection target within the detection target range is likely to be a rider or is clearly a rider. For example, 0.8 is set when the lidar detection score is detected in the range of 0.0 to 1.0. When the lidar detection score is less than the second threshold value t2 (step S205: No), the determination unit 130 determines that neither a rider nor a cyclist is included in the detection target range and ends the process.

[0051] When the cyclist detection score is equal to or greater than the first threshold value t1 (step S204: Yes), the determination unit 130 determines whether the lidar detection score is equal to or greater than the second threshold value t2 for the detection target range that is the same as the detection target range determined in step S203 (step S207). When the lidar detection score is less than the second threshold value t2 (step S207: No), the determination unit 130 determines that the detection target is a cyclist (step S209). When the result in step S207 is No, although the cyclist detection score is lower than the second threshold value t2, the characteristics of a cyclist appear in the video within the detection target range to such an extent that it is equal to or greater than the first threshold value t1. Furthermore, since the lidar detection score is less than the second threshold value, it is determined that the object is a cyclist rather than a rider.

[0052] When the rider detection score is equal to or higher than the second threshold value t2 (step S207: Yes), the determination unit 130 analyzes the variation in the video at the lower part of the detection target range and determines whether there is any variation (step S208). The analysis of the variation in the video at the lower part of the detection target range is performed in the same manner as in the first embodiment. When the video varies at the lower part of the detection target range (step S208: Yes), the determination unit 130 determines that the detection target is a cyclist (step S209). When the video does not vary at the lower part of the detection target range (step S208: No), the determination unit 130 determines that the detection target is a rider (step S206).

[0053] In this way, the object detection device 100 can appropriately recognize a person riding a motorcycle or a person riding a bicycle.

[0054] <Embodiment 3> In Embodiment 3, a case will be described where a pedestrian is defined as the first form, and a cyclist and a rider are defined as the second form. In this embodiment, the determination unit 130 performs a process of determining whether the detected person is a pedestrian, a rider, or a cyclist based on the detection result of the object detection unit 120. Specifically, the determination unit 130 determines that a pedestrian is included in a detection target range where the detection score by the pedestrian detection model defined as the first form detection model is equal to or higher than the second threshold value t2. Further, the determination unit 130 determines that a cyclist or a rider is included in a detection target range where the detection score by the cyclist detection model or the rider detection model defined as the second form detection model is equal to or higher than the second threshold value t2. Furthermore, when the detection score by the pedestrian detection model is not equal to or higher than the second threshold value t2, is equal to or higher than the first threshold value t1 which is a value lower than the second threshold value t2, and the detection score by the cyclist detection model or the rider detection model is equal to or higher than the second threshold value t2, the determination unit 130 analyzes the variation of the video in the upper part of the detection target range. When the variation of the video in the upper part of the detection target range is large, the determination unit 130 determines that a pedestrian is included in the detection target range. When the variation of the video in the upper part of the detection target range is small, the determination unit 130 determines that a cyclist or a rider is included in the detection target range.

[0055] When the determination unit 130 determines that a cyclist or a rider is included in the detection target range, the determination unit 130 analyzes the variation of the video in the lower part of the detection target range. The determination unit 130 analyzes the variation of the video in the lower part of the detection target range, and when the variation occurs alternately left and right in the lower part of the detection target range, the determination unit 130 determines that a cyclist is detected in the detection target range. Further, the determination unit 130 analyzes the variation in the vertical direction of the lower end position of the video in the lower part of the detection target range, and when the variation of the lower end position occurs alternately left and right in the lower part of the detection target range, the determination unit 130 determines that a cyclist is detected in the detection target range.

[0056] Next, with reference to FIG. 7, the flow of the object detection method executed by the object detection device 100 according to the present disclosure will be described. In the object detection method shown in FIG. 7, first, the video acquisition unit 110 acquires the video captured by the camera 200 (step S301). Next, the object detection unit 120 detects a person from the video acquired in step S301 (step S302). Specifically, the object detection unit 120 uses a pedestrian detection model that has learned the video of pedestrians, a rider detection model that has learned the video of a person riding a rider, and a cyclist detection model that has learned the video of a person riding a bicycle to detect a pedestrian, a rider, or a cyclist from the acquired video.

[0057] Next, the determination unit 130 determines whether the pedestrian detection score is equal to or greater than a second threshold value t2 for the detection target range in which the detection was executed in step S302 (step S303). The second threshold value t2 in this case is a numerical value indicating that the detection target within the detection target range is likely to be a pedestrian or is clearly a pedestrian. For example, 0.8 is set when the pedestrian detection score is detected in the range of 0.0 to 1.0.

[0058] When the pedestrian detection score is equal to or greater than the second threshold value t2 (step S303: Yes), the determination unit 130 determines that the person appearing in the detection target range, that is, the detection target, is a pedestrian (step S311). When the pedestrian detection score is equal to or greater than the second threshold value t2, since the characteristics of a pedestrian appear in the video within the detection target range, the possibility of being misrecognized as a cyclist or a rider is low, and it is determined that the person is a pedestrian. When the pedestrian detection score is less than the second threshold value t2 (step S303: No), it is determined whether the pedestrian detection score determined to be less than the second threshold value t2 is equal to or greater than a first threshold value t1 (step S304).

[0059] When the pedestrian detection score is less than the first threshold value t1 (step S304: No), the determination unit 130 determines whether the cyclist detection score or the rider detection score is greater than or equal to the second threshold value t2 for the same detection target range as the detection target range determined in step S303 (step S305). When the cyclist detection score or the rider detection score is greater than or equal to the second threshold value t2 (step S305: Yes), the determination unit 130 analyzes the variation of the video in the lower part of the detection target range and determines whether there is a variation (step S306). The analysis of the variation of the video in the lower part of the detection target range is performed in the same manner as in the first embodiment. When the video in the lower part of the detection target range is varying (step S306: Yes), the determination unit 130 determines that the detection target is a cyclist (step S307). When the video in the lower part of the detection target range is not varying (step S306: No), the determination unit 130 determines that the detection target is a rider (step S308). When the cyclist detection score and the rider detection score are less than the second threshold value t2 (step S305: No), the determination unit 130 determines that none of a pedestrian, a rider, or a cyclist is included in the detection target range, and ends the process.

[0060] When the pedestrian detection score is greater than or equal to the first threshold value t1 (step S304: Yes), the determination unit 130 determines whether the cyclist detection score or the rider detection score is greater than or equal to the second threshold value t2 for the same detection target range as the detection target range determined in step S303 (step S309). When the cyclist detection score and the rider detection score are less than the second threshold value t2 (step S309: No), the determination unit 130 determines that the detection target is a pedestrian (step S311). In the case of No in step S309, although the pedestrian detection score is lower than the second threshold value t2, features as a pedestrian appear in the video within the detection target range to such an extent that it is greater than or equal to the first threshold value t1, and further, since the cyclist detection score and the rider detection score are less than the second threshold value, it is determined that the object is a pedestrian rather than a cyclist or a rider.

[0061] When the cyclist detection score or rider detection score is equal to or greater than a second threshold value t2 (step S309: Yes), the determination unit 130 analyzes the variation in the video at the upper part of the detection target range and determines whether there is a variation (step S310). If the result in step S309 is Yes, the pedestrian detection score is equal to or greater than a first threshold value t1, and for the same detection target range, the cyclist detection score or rider detection score is equal to or greater than the second threshold value t2. That is, although the characteristics as a cyclist or rider appear, the pedestrian detection score also indicates a high possibility of being a pedestrian. Therefore, even if it is actually a cyclist or rider, there is a possibility of being misdetected as a pedestrian. Therefore, based on the determination in step S310, it is determined whether the detection target is a cyclist or rider or a pedestrian.

[0062] The analysis of the variation in the video at the upper part of the detection target range performed in step S310 is performed in the same manner as the analysis of the variation in the video at the lower part of the detection target range described in Embodiment 1. When walking, the hands are moved more significantly compared to when riding a bicycle or a motorcycle. Therefore, in the detection target range where a pedestrian appears, the variation in the video at the upper part of the detection target range is larger than that in the detection target range where a cyclist or rider appears. Therefore, the determination unit 130 can determine whether the person included in the detection target range is a pedestrian or a cyclist or rider by analyzing the variation in the video at the upper part of the detection target range.

[0063] When the video is varying at the upper part of the detection target range (step S310: Yes), the determination unit 130 determines that the detection target is a pedestrian (step S311). When the video is not varying at the upper part of the detection target range (step S310: No), the determination unit 130 determines that the detection target is a cyclist or rider and proceeds to step S306.

[0064] In this way, the object detection device 100 can appropriately recognize a pedestrian, a person riding a bicycle, or a person riding a motorcycle.

[0065] <Embodiment 4> In Embodiment 4, a modification example is described when a pedestrian is defined as the first form and a cyclist is defined as the second form. In this embodiment, the determination unit 130 performs a process of determining whether the detected person is a pedestrian or a cyclist based on the detection result of the object detection unit 120. Specifically, the determination unit 130 determines that a pedestrian is included in a detection target range in which the detection score by a pedestrian detection model defined as a first form detection model is equal to or greater than a second threshold value t2. In addition, the determination unit 130 determines that a cyclist is included in a detection target range in which the detection score by a cyclist detection model defined as a second form detection model is equal to or greater than a second threshold value t2. Further, when the detection score by the pedestrian detection model is not equal to or greater than the second threshold value t2, is equal to or greater than a first threshold value t1 which is a value lower than the second threshold value t2, and the detection score by the cyclist detection model is equal to or greater than the second threshold value t2, the determination unit 130 analyzes the variation of the video in the detection target range. When the variation location is in the lower part of the detection target range, the determination unit 130 determines that a cyclist is included in the detection target range. When the variation location is in the upper part of the detection target range, the determination unit 130 determines that a pedestrian is included in the detection target range.

[0066] Next, with reference to FIG. 8, the flow of the object detection method executed by the object detection device 100 according to the present disclosure will be described. In the object detection method shown in FIG. 8, first, the video acquisition unit 110 acquires the video captured by the camera 200 (step S401). Next, the object detection unit 120 detects a person from the video acquired in step S401 (step S402). Specifically, the object detection unit 120 detects a pedestrian or a cyclist from the acquired video using a pedestrian detection model that has learned the video of a pedestrian and a cyclist detection model that has learned the video of a person riding a bicycle.

[0067] Next, the determination unit 130 determines whether the pedestrian detection score is greater than or equal to a second threshold value t2 for the detection target range in which detection was performed in step S402 (step S403). The second threshold value t2 in this case is a numerical value indicating that the detection target within the detection target range is likely to be a pedestrian or is clearly a pedestrian. For example, when the pedestrian detection score is detected in the range of 0.0 to 1.0, 0.8 is set.

[0068] When the pedestrian detection score is greater than or equal to the second threshold value t2 (step S403: Yes), the determination unit 130 determines that the person appearing in the detection target range, that is, the detection target, is a pedestrian (step S410). When the pedestrian detection score is greater than or equal to the second threshold value t2, since the characteristics of a pedestrian appear in the video within the detection target range, the possibility of being misrecognized as a cyclist is low, and it is determined to be a pedestrian. When the pedestrian detection score is less than the second threshold value t2 (step S403: No), it is determined whether the pedestrian detection score determined to be less than the second threshold value t2 is greater than or equal to a first threshold value t1 (step S404).

[0069] When the pedestrian detection score is less than the first threshold value t1 (step S404: No), the determination unit 130 determines whether the cyclist detection score is greater than or equal to the second threshold value t2 for the same detection target range as the detection target range determined in step S403 (step S405). When the cyclist detection score is greater than or equal to the second threshold value t2 (step S405: Yes), the determination unit 130 determines that the person appearing in the detection target range, that is, the detection target, is a cyclist (step S411). When the cyclist detection score is less than the second threshold value t2 (step S405: No), the determination unit 130 determines that neither a pedestrian nor a cyclist is included in the detection target range, and ends the process.

[0070] When the pedestrian detection score is greater than or equal to the first threshold value t1 (step S404: Yes), the determination unit 130 determines whether the cyclist detection score is greater than or equal to the second threshold value t2 for the same detection target range as the detection target range determined in step S403 (step S406). When the cyclist detection score is less than the second threshold value t2 (step S406: No), the determination unit 130 determines that neither a pedestrian nor a cyclist is included in the detection target range, and ends the process.

[0071] When the cyclist detection score is greater than or equal to the second threshold value t2 (step S406: Yes), the determination unit 130 analyzes the variation of the video in the detection target range and determines whether there is a variation (step S407). The analysis of the video variation in the detection target range performed in step S407 is performed in the same manner as the analysis of the video variation in the detection target range described in Embodiment 1. If it is Yes in step S407, the pedestrian detection score is greater than or equal to the first threshold value t1, and for the same detection target range, the cyclist detection score is greater than or equal to the second threshold value t2. That is, although the characteristics as a cyclist appear, the pedestrian detection score is also a value indicating a high possibility of a pedestrian. Therefore, even if it is actually a cyclist, there is a possibility of being misdetected as a pedestrian. Therefore, based on the determinations in steps S408 and S409, it is determined whether the detection target is a cyclist or a pedestrian.

[0072] When the determination unit 130 determines that there is a variation in the video in the detection target range, in other words, when the variation in the video in the detection target range is detected, it is determined in which part of the detection target range the variation is. Specifically, the determination unit 130 determines whether the variation in the video detected in step S407 is in the lower part or the upper part of the detection target range.

[0073] If the answer is Yes in step S407, the determination unit 130 determines whether the changed portion is at the lower part of the detection target range (step S408). If the changed portion is at the lower part of the detection target range (step S408: Yes), the determination unit 130 determines that the detection target is a cyclist (step S411). If the changed portion is not at the lower part of the detection target range (step S408: No), the determination unit 130 determines whether the changed portion is at the upper part of the detection target range (step S409). If the changed portion is at the upper part of the detection target range (step S409: Yes), the determination unit 130 determines that the detection target is a pedestrian (step S410). If the changed portion is not at the upper part of the detection target range (step S409: No), the determination unit 130 determines that neither a pedestrian nor a cyclist is included in the detection target range, and ends the process. If the answer is No in step S407, the determination unit 130 determines that the detection target is a cyclist (step S411).

[0074] In this way, the object detection device 100 can appropriately recognize a pedestrian or a person riding a bicycle.

[0075] Note that the present disclosure is not limited to the above-described embodiment, and can be appropriately changed without departing from the gist. For example, in the fourth embodiment, a cyclist may be defined as the first form and a pedestrian may be defined as the second form, and in this case, in the process of FIG. 8, a process in which a pedestrian and a cyclist are interchanged is performed. Further, the present disclosure may be implemented by appropriately combining the embodiments and examples thereof.

Description of Reference Numerals

[0076] 100 Object detection device 110 Video acquisition unit 120 Object detection unit 130 Determination unit 140 Output control unit 200 Camera 300 Output device 400 Object detection system

Claims

1. an image acquisition unit that acquires an image captured by an infrared camera; an object detection unit that detects a person in the first form or a person in the second form from the acquired video using a first form detection model that has been trained with an image of a person in the first form and a second form detection model that has been trained with an image of a person in the second form; a determination unit that analyzes fluctuations in an image in a predetermined portion of a detection target range in which a detection score by the first form detection model is equal to or greater than a first threshold and a detection score by the second form detection model is equal to or greater than a second threshold that is higher than the first threshold, based on detection results of the first form and the second form in the acquired image, and determines that a person of the first form has been detected in the detection target range when the fluctuations are large; An object detection device comprising:

2. the first form is a cyclist, and the first form detection model is a cyclist detection model; the second form is a pedestrian, and the second form detection model is a pedestrian detection model; The determination unit analyzes fluctuations in the image in the lower part of the detection target range, and when the fluctuations are large, determines that a cyclist has been detected in the detection target range. The object detection device according to claim 1 .

3. the first form is a cyclist, and the first form detection model is a cyclist detection model; the second form is a LIDAR, and the second form detection model is a LIDAR detection model; The determination unit analyzes fluctuations in the image in the lower part of the detection target range, and when the fluctuations are large, determines that a cyclist has been detected in the detection target range. The object detection device according to claim 1 .

4. The determination unit analyzes fluctuations in the image in a lower portion of the detection target range, and determines that a cyclist has been detected in the detection target range when fluctuations in the lower portion of the detection target range occur alternately from left to right. The object detection device according to claim 2 or 3.

5. The determination unit analyzes fluctuations in the up-down direction of the bottom end position of the image at the bottom of the detection target range, and determines that a cyclist has been detected in the detection target range when the fluctuations in the bottom end position occur alternately left and right at the bottom of the detection target range. The object detection device according to claim 4.

6. the first form is a pedestrian, and the first form detection model is a pedestrian detection model; the second form is a cyclist or a rider, and the second form detection model is a cyclist detection model or a rider detection model; the determination unit analyzes a variation in the image in an upper portion of the detection target range, and when the variation is large, determines that a pedestrian has been detected in the detection target range. The object detection device according to claim 1 .

7. The determination unit analyzes a fluctuation in an image in a lower part of the detection target range where it has been determined that a cyclist or rider has been detected in the detection target range, and when the fluctuation in the image in the lower part of the detection target range is large, it determines that a cyclist has been detected in the detection target range. The object detection device according to claim 6.

8. the determination unit analyzes a difference in height between the left and right at a lower portion of the detection target range in which it has been determined that a cyclist or rider has been detected in the detection target range, and when there is a difference in height between the left and right at the lower portion of the detection target range, it determines that a cyclist has been detected in the detection target range. The object detection device according to claim 6.

9. The determination unit analyzes the presence or absence of a heat source in a lower part of the detection target range where it has been determined that a cyclist or rider has been detected in the detection target range, and when a heat source has been detected in the lower part of the detection target range, determines that a rider has been detected in the detection target range. The object detection device according to claim 6.

10. The image acquisition unit acquires an image showing a heat distribution by a brightness value, The determination unit analyzes a variation in luminance value in a predetermined portion of the detection target range. The object detection device according to claim 1 .

11. acquiring an image captured by an infrared camera; detecting a person of the first form or a person of the second form from the acquired video using a first form detection model trained with an image of a person of a first form and a second form detection model trained with an image of a person of a second form; a step of analyzing image fluctuations in a predetermined portion of a detection target range in which a detection score by the first form detection model is equal to or greater than a first threshold and a detection score by the second form detection model is equal to or greater than a second threshold that is higher than the first threshold, based on a detection result of the person in the first form or the person in the second form, and determining that a person in the first form has been detected in the detection target range when the fluctuations are large; The object detection method is executed by the object detection device.

Images