Object detection apparatus and object detection method

Abstract

To provide an object detection device and an object detection method capable of appropriately recognizing a pedestrian and a person riding on a bicycle or a motorcycle.SOLUTION: An object detection apparatus 100 according to the present disclosure includes a video acquisition unit 110 that acquires a captured video, an object detection unit 120 that detects a person in a first form or a person in a second form by using a first form detection model that has learned a video of the first form of a person with respect to the video and a cyclist detection model that has learned a video of the second form of a person with respect to the video, and a determination unit 130 that analyzes, based on a detection result in the video, a change in the video for a detection target range in which a detection score by the first form detection model is equal to or greater than a first threshold value and a detection score by the second form detection model is equal to or greater than a second threshold value that is higher than the first threshold value, and determines whether the person in the detection target range is in the first form or the second form based on a portion where the change is large.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an object detection device and an object detection method. [Background technology]

[0002] In recent years, information processing systems that detect pedestrians and people riding bicycles or motorcycles, i.e., cyclists and riders, from images captured using infrared cameras have been increasingly implemented in vehicles and smart poles. Smart poles are required to be able to distinguish between and detect pedestrians, bicycles with people on them, motorcycles with people on them, and so on. When detecting people from video, it is sometimes desirable to be able to distinguish between pedestrians, cyclists, and riders. For example, Patent Document 1 discloses a technology that determines whether a detected person is a pedestrian or a cyclist based on whether or not a wheel is detected below the person. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 158983 Summary of the Invention [Problem to be solved by the invention]

[0004] When a person is moving sideways from the camera's perspective, the wheels can be easily detected, making it easy to determine whether the person is a pedestrian, cyclist, or rider. However, when the person is moving toward or away from the camera, it can be difficult to detect the wheels, leaving room for improvement in properly determining whether the person is a pedestrian, cyclist, or rider. There was also a need for proper recognition of cyclists and riders.

[0005] The present disclosure has been made in consideration of such problems, and provides an object detection device and an object detection method that are capable of appropriately recognizing pedestrians and people riding bicycles or motorcycles. [Means for solving the problem]

[0006] The object detection device according to the present disclosure comprises: an image acquisition unit that acquires an image captured by an infrared camera; an object detection unit that detects a person in the first form or a person in the second form from the acquired video using a first form detection model that has been trained with a video of a person in the first form and a second form detection model that has been trained with a video of a person in the second form; a determination unit that analyzes fluctuations in the video for a detection target range in which the detection score by the first form detection model is equal to or greater than a first threshold and the detection score by the second form detection model is equal to or greater than a second threshold that is higher than the first threshold, based on detection results of the first form and the second form in the acquired video, and determines whether the person in the detection target range is in the first form or the second form based on the location of large fluctuations; Equipped with.

[0007] The object detection method according to the present disclosure includes: acquiring an image captured by an infrared camera; detecting a person of the first form or a person of the second form from the acquired video using a first form detection model trained with a video of a person of the first form and a second form detection model trained with a video of a person of the second form; a step of analyzing fluctuations in the image for a detection target range in which the detection score by the first form detection model is equal to or greater than a first threshold and the detection score by the second form detection model is equal to or greater than a second threshold that is higher than the first threshold, based on the detection results of the first form and the second form in the acquired image, and determining whether the person in the detection target range is in the first form or the second form based on the location of large fluctuations; The object detection device executes the above. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to provide an object detection device and an object detection method that are capable of appropriately recognizing pedestrians and people riding bicycles or motorcycles. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating a configuration of an object detection system according to the present disclosure. [Figure 2] 1 is a flowchart illustrating a flow of an object detection method according to the present disclosure. [Figure 3] 10 is an example of a video in which a person is detected. [Figure 4] This is an example of analyzing fluctuations in brightness values. [Figure 5] This is an example of analyzing the fluctuation of the bottom end position. [Figure 6] 1 is a flowchart illustrating a flow of an object detection method according to the present disclosure. [Figure 7] 1 is a flowchart illustrating a flow of an object detection method according to the present disclosure. [Figure 8] 1 is a flowchart illustrating a flow of an object detection method according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] <Embodiment 1> First, the configuration of an object detection device 100 according to the present disclosure and an object detection system 400 including the object detection device 100 will be described with reference to Fig. 1. The object detection system 400 is an information processing system that detects people and the like from video captured using an infrared camera. As shown in Fig. 1, the object detection system 400 includes the object detection device 100, a camera 200, and an output device 300.

[0011] The object detection system 400 is provided in a device that needs to detect people, such as pedestrians, cyclists, and riders. Such a device is, for example, a smart pole. A smart pole is a multi-function pole equipped with various sensors, including a camera, and a communication device, installed near roads, and is used for traffic monitoring, safety support, crime prevention, and the like. An object detection device according to an embodiment is implemented in such a smart pole, detects target objects, such as people and vehicles, from thermal images captured by a far-infrared camera, and outputs information about the detected target objects, such as their type, coordinates, and movement speed. The information output from the smart pole is used to provide warnings to vehicles traveling near the smart pole. The object detection system 400 is not limited to smart poles, but can also be applied to other applications, such as vehicles and surveillance devices. The object detection system 400 detects bicycles ridden by detecting cyclists. The object detection system 400 detects motorcycles ridden by detecting riders. The object detection system 400 also detects automobiles and other vehicles.

[0012] The camera 200 is an infrared camera, specifically a Far Infrared Rays (FIR) camera, for capturing images based on far-infrared rays emitted by objects, known as thermal images. The camera 200 captures thermal images of the capture range as moving images at, for example, 15 to 30 frames per second. The installation location of the camera 200 is not particularly limited. The camera 200 is installed in a location where it can capture images of people riding bicycles (i.e., cyclists), people riding motorcycles (i.e., riders), and pedestrians passing by. The camera 200 is installed, for example, near a road or in a bicycle parking lot. The camera 200 is also installed in a direction that captures images in the direction of travel of vehicles that need to detect people riding bicycles (i.e., cyclists), people riding motorcycles (i.e., riders), and pedestrians. The camera 200 is capable of communicating with the object detection device 100. The communication method between the camera 200 and the object detection device 100 is not particularly limited, and may be wired or wireless.

[0013] The structure of the camera 200 is not shown because it is a typical far-infrared camera structure, but it includes a lens, a shutter, a sensor such as a microbolometer, and the like.

[0014] The output device 300 is a device that outputs the results of detection by the object detection device 100. The output device 300 may be, for example, a device carried by an administrator who manages the location where the camera 200 is installed. The communication method between the object detection device 100 and the output device 300 is not particularly limited, and may be wired or wireless.

[0015] When the object detection system 400 is provided on a smart pole or the like, the output device 300 is a device that outputs the results of detection by the object detection device 100 to a server or the like that centrally manages the smart pole using a known communication technology. The output device 300 may also be a device that outputs the results of detection by the object detection device 100 to a vehicle traveling near the smart pole using a known wireless communication technology. When the object detection system 400 is provided on a mobile body such as a vehicle, the output device 300 may be a device such as a monitor that notifies the driver or operator of the mobile body such as a vehicle of the results of detection by the object detection device 100 by using a video output or the like.

[0016] The object detection device 100 is an information processing device that detects a person or the like from an image captured by a camera 200 based on the shape of heat distribution, etc., and determines the type of movement of the person. Specifically, the object detection device 100 determines whether the person detected from the image captured by the camera 200 is a pedestrian or a cyclist. A pedestrian is not limited to a person who is walking, but also includes a person who is standing still. As shown in FIG. 1, the object detection device 100 includes an image acquisition unit 110, an object detection unit 120, a determination unit 130, and an output control unit 140 as functional blocks realized by processing such as a program.

[0017] The image acquisition unit 110 performs processing to acquire images captured by the camera 200. The images acquired by the image acquisition unit 110 indicate the heat distribution in the image capture range using brightness values, and the presence or absence of a person in the image capture range is indicated as the presence or absence of a heat source. The image acquisition unit 110 includes a defective pixel correction unit and a NUC (Non-Uniformity Correction) unit (not shown) as well-known techniques for acquiring images captured by the camera 200, which is a far-infrared camera.

[0018] The object detection unit 120 calculates a person detection score for the video acquired by the video acquisition unit 110 using a detection model trained on various person forms. As examples of various person forms, a pedestrian, a cyclist, or a rider may be defined as a first form or a second form. Furthermore, as examples of detection models trained on various person forms, a pedestrian detection model, a cyclist detection model, or a rider detection model may be defined as a first form detection model or a second form detection model.

[0019] The object detection unit 120 detects a person in a first form or a person in a second form using a first form detection model trained with video of a person in a first form and a second form detection model trained with video of a person in a second form. Specifically, the object detection unit 120 performs a process of detecting people by calculating a pedestrian detection score, a cyclist detection score, and a rider detection score. Specifically, the object detection unit 120 calculates the pedestrian detection score using the pedestrian detection model, calculates the cyclist detection score using the cyclist detection model, and calculates the rider detection score using the rider detection model. The pedestrian detection model is a model trained with video of a pedestrian. The pedestrian detection score is a numerical representation of the similarity between an object detected in a score calculation area and the pedestrian detection model. The cyclist detection model is a model trained with video of a cyclist, a person riding a bicycle. The cyclist detection model is a model trained with video of a range that includes not only cyclists but also the bicycles the cyclists are riding. The cyclist detection score is a numerical representation of the similarity between an object detected in the score calculation area and the cyclist detection model. The rider detection model is a model trained on video of a rider, a person riding a motorcycle. The rider detection model is a model trained on video of an area that includes not only the rider but also the motorcycle the rider is riding. The rider detection score is a numerical representation of the similarity between an object detected in the score calculation area and the rider detection model.

[0020] The object detection unit 120 scans the detection window for each frame of the video captured by the video capture unit 110 and calculates a pedestrian detection score, a cyclist detection score, and a rider detection score for the detection target range defined by the detection window. That is, the object detection unit 120 performs detection processing for the detection target range defined by the detection window using a pedestrian detection model, a cyclist detection model, and a lidar detection model. The pedestrian detection score, cyclist detection score, and lidar detection model calculated by the object detection unit 120 are calculated as values ​​ranging from 0.0 to 1.0, for example. The higher the likelihood that a pedestrian, cyclist, or rider is included in the detection target range, the larger the detection score value. If the detection score using the pedestrian detection model for the detection target range is 0.8 or higher, the object detection unit 120 determines that a pedestrian is included in the detection target range. If the score using the cyclist detection model is 0.8 or higher, the object detection unit 120 determines that a cyclist is included in the detection target range. If the score using the lidar detection model is 0.8 or higher, the object detection unit 120 determines that a rider is included in the detection target range. If a pedestrian, cyclist, or rider is included in the detection range, it means that a pedestrian, a bicycle ridden by a person, or a motorcycle ridden by a person is captured within the detection range defined by the detection window in the video acquired by the video acquisition unit 110.

[0021] The determination unit 130 detects a person in the first form or a person in the second form based on the detection results of the person in the first form and the person in the second form by the object detection unit 120. Specifically, the determination unit 130 analyzes the image fluctuations in a predetermined portion of the detection target range for a detection target range in which the detection score by the first form detection model is equal to or greater than a first threshold and the detection score by the second form detection model is equal to or greater than a second threshold that is higher than the first threshold. Then, based on the analysis results, the determination unit 130 determines that a person in the first form has been detected in the detection target range if the fluctuations are large. In this embodiment, the determination unit 130 analyzes the image fluctuations in the lower portion of the detection target range.

[0022] The second threshold value used by the determination unit 130 to determine the form of a person is a numerical value indicating that the detection target within the detection target range is highly likely to be a person with the first form or a person with the second form, or that the detection target is clearly a person with the first form or a person with the second form. The second threshold value is set to, for example, 0.8 when the detection score is detected in the range of 0.0 to 1.0. In contrast, the first threshold value is a numerical value indicating that the detection target within the detection target range is not highly likely to be a person with the first form or a person with the second form, but is possible, but not clear. The first threshold value is set to, for example, 0.6 when the detection score is detected in the range of 0.0 to 1.0.

[0023] In this embodiment, the determination unit 130 performs processing to determine whether a detected person is a pedestrian or a cyclist based on the detection result of the object detection unit 120. Specifically, the determination unit 130 determines that a cyclist is included in a detection target range where the detection score by the cyclist detection model defined as the first form detection model is equal to or greater than the second threshold t2. The determination unit 130 also determines that a pedestrian is included in a detection target range where the detection score by the pedestrian detection model defined as the second form detection model is equal to or greater than the second threshold t2. Furthermore, when the detection score by the cyclist detection model is not equal to or greater than the second threshold t2, the determination unit 130 analyzes image fluctuations in the lower part of the detection target range where the detection score by the pedestrian detection model is equal to or greater than the first threshold t1, which is lower than the second threshold t2, and is equal to or greater than the second threshold t2. When the image fluctuations in the lower part of the detection target range are large, the determination unit 130 determines that a cyclist is included in the detection target range. If the image acquired by the image acquisition unit 110 indicates the heat distribution by brightness values, the determination unit 130 may analyze fluctuations in brightness values ​​in the lower part of the detection target range. The output control unit 140 outputs the determination result of the determination unit 130 to the output device 300.

[0024] The determination unit 130 analyzes fluctuations in the image at the bottom of the detection target range, and if fluctuations at the bottom of the detection target range occur alternately left and right, it determines that a cyclist has been detected in the detection target range. The determination unit 130 also analyzes fluctuations in the up and down direction of the bottom edge position of the image at the bottom of the detection target range, and if fluctuations in the bottom edge position occur alternately left and right at the bottom of the detection target range, it determines that a cyclist has been detected in the detection target range.

[0025] The image fluctuation analyzed by the determination unit 130 is, for example, a state in which high and low temperatures sufficient to detect a person alternate in the area constituting a person in the thermal image. The determination unit 130 particularly detects fluctuations in the area corresponding to the person's legs and the area corresponding to the person's arms. Hereinafter, the lower part of the detection target area refers to the area corresponding to the person's legs. The upper part of the detection target area refers to the area corresponding to the person's arms, not the top of the detection target area, but the area below the area corresponding to the person's head. The lower part of the detection target area may be above the vertical center of the detection target area, or the lower part of the detection target area may be below the vertical center of the detection target area. Furthermore, it may be determined that fluctuations are occurring alternately left and right when fluctuations occur alternately in positions symmetrical to the horizontal center of the detection target area.

[0026] Next, the flow of the object detection method executed by object detection device 100 according to the present disclosure will be described with reference to FIG. 2. FIG. 2 is a flowchart showing the flow of the object detection method according to the present disclosure. The object detection method shown in FIG. 2 is executed at all times when object detection system 400 is installed on a smart pole or the like. In other words, object detection device 100 always performs image capture using camera 200 and detects pedestrians and cyclists from the images captured by camera 200. Because camera 200 of object detection device 100 according to the present disclosure is a far-infrared camera, the object detection method may be executed at night or when the ambient illumination is low.

[0027] 2 is executed while the vehicle or other moving body is in operation when object detection system 400 is installed in the vehicle or other moving body. Because camera 200 of object detection device 100 according to the present disclosure is a far-infrared camera, the object detection method may be executed at night or when the ambient illumination is low while the vehicle or other moving body is in operation.

[0028] 2, first, image acquisition unit 110 acquires an image captured by camera 200 (step S101). Next, object detection unit 120 detects a person from the image acquired in step S101 (step S102). Specifically, object detection unit 120 detects a pedestrian or a cyclist from the acquired image using a pedestrian detection model trained on images of pedestrians and a cyclist detection model trained on images of people riding bicycles.

[0029] Next, the determination unit 130 determines whether the cyclist detection score for the detection target range in which detection was performed in step S102 is equal to or greater than a second threshold value t2 (step S103). In this case, the second threshold value t2 is a value that indicates that the detection target in the detection target range is likely to be a cyclist or is clearly a cyclist, and is set to, for example, 0.8 when the cyclist detection score is detected in the range of 0.0 to 1.0.

[0030] If the cyclist detection score is equal to or greater than the second threshold t2 (step S103: Yes), the determination unit 130 determines that the person in the detection target range, i.e., the detection target, is a cyclist (step S109). If the cyclist detection score is equal to or greater than the second threshold t2, the image within the detection target range exhibits characteristics of a cyclist, making it unlikely that the person will be mistaken for a pedestrian and determining that the person is a cyclist. If the cyclist detection score is less than the second threshold t2 (step S103: No), the determination unit 130 determines whether the cyclist detection score determined to be less than the second threshold t2 is equal to or greater than the first threshold t1 (step S104).

[0031] If the cyclist detection score is less than the first threshold t1 (step S104: No), the determination unit 130 determines whether the pedestrian detection score is equal to or greater than the second threshold t2 for the same detection target range as the detection target range determined in step S103 (step S105). If the pedestrian detection score is equal to or greater than the second threshold t2 (step S105: Yes), the determination unit 130 determines that the person in the detection target range, i.e., the detection target, is a pedestrian (step S106). The second threshold t2 in this case is also a numerical value indicating that the detection target in the detection target range is likely to be a pedestrian or is clearly a pedestrian, and is set to, for example, 0.8 when the pedestrian detection score is detected in the range from 0.0 to 1.0. If the pedestrian detection score is less than the second threshold t2 (step S105: No), the determination unit 130 determines that neither a pedestrian nor a cyclist is included in the detection target range and ends the process.

[0032] If the cyclist detection score is equal to or greater than the first threshold t1 (step S104: Yes), the determination unit 130 determines whether the pedestrian detection score is equal to or greater than the second threshold t2 for the same detection target range as the detection target range determined in step S103 (step S107). If the pedestrian detection score is less than the second threshold t2 (step S107: No), the determination unit 130 determines that the detection target is a cyclist (step S109). If the determination in step S107 is No, the cyclist detection score is lower than the second threshold t2, but the video within the detection target range exhibits characteristics of a cyclist to the extent that it is equal to or greater than the first threshold t1, and further the pedestrian detection score is less than the second threshold, so the detection target is determined to be a cyclist and not a pedestrian.

[0033] If the pedestrian detection score is equal to or greater than the second threshold value t2 (step S107: Yes), the determination unit 130 analyzes the image fluctuation in the lower part of the detection target range and determines whether or not there is a fluctuation (step S108). The analysis of the image fluctuation in the lower part of the detection target range will be described later. If there is a fluctuation in the image in the lower part of the detection target range (step S108: Yes), the determination unit 130 determines that the detection target is a cyclist (step S109). If there is no fluctuation in the image in the lower part of the detection target range (step S108: No), the determination unit 130 determines that the detection target is a pedestrian (step S106).

[0034] If the answer is Yes in step S107, the cyclist detection score is equal to or greater than the first threshold value t1, and the pedestrian detection score is equal to or greater than the second threshold value t2 for the same detection target range. In other words, even though the characteristics of a cyclist are present, the pedestrian detection score also indicates a high probability of being a pedestrian. Therefore, even if an actual cyclist is present, there is a possibility that he or she will be erroneously detected as a pedestrian. Therefore, based on the determination in step S108, it is determined whether the detection target is a cyclist or a pedestrian.

[0035] Next, the analysis of image fluctuations in the lower part of the detection target range performed in step S108 will be described in detail with reference to FIGS.

[0036] FIG. 3 shows an example of an image for which a person is to be detected. The left side of FIG. 3 shows an example of an image captured by camera 200. The right side of FIG. 3 is an enlarged view of the dashed area in the left side of FIG. 3. The image shown in FIG. 3 shows a cyclist facing away from the camera. The object detection unit 120 calculates a pedestrian detection score for the image captured by camera 200 using a pedestrian detection model, and calculates a cyclist detection score using the cyclist detection model. For example, in the detection target range A defined by the detection window shown in FIG. 3, if the cyclist detection score is less than the second threshold t2 and greater than or equal to the first threshold t1, and further, the pedestrian detection score is greater than or equal to the second threshold t2, the determination unit 130 analyzes the image fluctuations in the lower part B of the detection target range A. When pedaling a bicycle, people often move their legs up and down more significantly than when walking. Therefore, in a detection target range in which a cyclist is captured, the image fluctuations in the lower part of the detection target range are greater than in a detection target range in which a pedestrian is captured. Therefore, the determination unit 130 can determine whether the person included in the detection target range is a pedestrian or a cyclist by analyzing the fluctuation of the image in the lower part of the detection target range.

[0037] When pedaling a bicycle, a cyclist moves their legs up and down alternately. Therefore, in the detection target range in which the cyclist is captured, image fluctuations alternate at the lower left and right sides. Therefore, the determination unit 130 may determine that a person captured in the detection target range is a cyclist if fluctuations occur alternately at the lower left and right sides of the detection target range. Alternatively, the determination unit 130 may determine that a person captured in the detection target range is a cyclist if fluctuations occur alternately at the lower left and right sides at a predetermined interval at the lower left and right sides of the detection target range. For example, the determination unit 130 may determine that a person captured in the detection target range is a cyclist if fluctuations are repeated at a frequency of approximately 0.5 to 1.5 times per second. Furthermore, in situations where cyclists can travel a certain distance without pedaling, such as downhill, they may ride their bicycles with their feet on the pedals without moving for a while. In such cases, differences in image height may occur at the lower left and right sides of the detection target range in which the cyclist is captured. Therefore, the determination unit 130 may analyze the difference in height between the left and right sides at the bottom of the detection target range, and if there is a difference in height between the left and right sides at the bottom of the detection target range, determine that the person appearing in the detection target range is a cyclist.

[0038] When the image acquired by the image acquisition unit 110 indicates the heat distribution using brightness values, the determination unit 130 may analyze the brightness value fluctuations in the lower part of the detection target range to determine whether the fluctuations occur alternately between the left and right sides of the lower part of the detection target range. FIG. 4 shows an example of analyzing the brightness value fluctuations. The two images shown in FIG. 4 were captured at different times of the detection target range A containing the same person. In the image on the left of FIG. 4, the cyclist's left leg is captured in the lower left part C1 of the detection target range A, and the brightness value in region D1 is high. On the other hand, in the image on the right of FIG. 4, the cyclist's right leg is captured in the lower right part C2 of the detection target range A, and the brightness value in region D2 is high. The determination unit 130 may determine that a cyclist is captured in the detection target range A when the brightness value changes above a predetermined level alternately occur in the lower left and right parts C1 and C2.

[0039] The determination unit 130 may determine whether fluctuations alternate between the left and right sides of the lower part of the detection target range by analyzing vertical fluctuations in the bottom end position, i.e., the lowest end position, of the lower part of the detection target range. FIG. 5 shows an example of analyzing fluctuations in the lowest end position. The two images shown in FIG. 5 were captured at different times of the detection target range A containing the same person. In the image on the left of FIG. 5, the cyclist is in a state where his right leg is raised, so the lowest end position E2 in the lower right part C2 is higher than the lowest end position E1 in the lower left part C1. On the other hand, in the image on the right of FIG. 5, the cyclist is in a state where his left leg is raised, so the lowest end position E2 in the lower right part C2 is lower than the lowest end position E1 in the lower left part C1. The determination unit 130 may determine that a cyclist is present in the detection target range A when changes in the bottom end position of the left and right lower parts C1 and C2 alternately exceed a predetermined level.

[0040] In this way, the object detection device 100 can appropriately recognize pedestrians or people riding bicycles.

[0041] The object detection device 100 includes a processor, memory, and storage device (not shown). The storage device stores a computer program that implements the processing of the object detection method according to the present disclosure. The processor then loads the computer program from the storage device into the memory and executes the computer program. As a result, the processor realizes the functions of the image acquisition unit 110, object detection unit 120, determination unit 130, and output control unit 140.

[0042] Alternatively, each component of the object detection device 100 may be realized by dedicated hardware. Furthermore, some or all of the components of each device may be realized by general-purpose or dedicated circuits, processors, etc., or a combination of these. These may be configured by a single chip, or by multiple chips connected via a bus. Some or all of the components of each device may be realized by a combination of the above-mentioned circuits, etc., and programs. Furthermore, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), quantum processor (quantum computer control chip), etc., may be used as the processor.

[0043] Furthermore, when some or all of the components of object detection device 100 are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally or decentralized. For example, the information processing devices, circuits, etc. may be realized as a client-server system, a cloud computing system, or the like, in a form in which each is connected via a communication network. Furthermore, the functions of object detection device 100 may be provided in a SaaS (Software as a Service) format.

[0044] <Embodiment 2> In the first embodiment, a case where a cyclist is defined as the first form and a pedestrian is defined as the second form has been described. However, the definitions of the first form and the second form are not limited to this. In the second embodiment, a case where a cyclist is defined as the first form and a rider is defined as the second form will be described. Note that descriptions that overlap with the first embodiment will be omitted as appropriate.

[0045] In this embodiment, the determination unit 130 performs processing to determine whether a detected person is a rider or a cyclist based on the detection result of the object detection unit 120. Specifically, the determination unit 130 determines that a cyclist is included in a detection target range where the detection score using the cyclist detection model defined as the first form detection model is equal to or greater than the second threshold value t2. The determination unit 130 also determines that a rider is included in a detection target range where the detection score using the lidar detection model defined as the second form detection model is equal to or greater than the second threshold value t2. Furthermore, when the detection score using the cyclist detection model is not equal to or greater than the second threshold value t2, the determination unit 130 analyzes image fluctuations in the lower part of the detection target range where the detection score using the lidar detection model is equal to or greater than the first threshold value t1, which is lower than the second threshold value t2, and where the detection score using the lidar detection model is equal to or greater than the second threshold value t2. When the image fluctuations in the lower part of the detection target range are large, the determination unit 130 determines that a cyclist is included in the detection target range.

[0046] The determination unit 130 analyzes fluctuations in the image at the bottom of the detection target range, and if fluctuations at the bottom of the detection target range occur alternately left and right, it determines that a cyclist has been detected in the detection target range. The determination unit 130 also analyzes fluctuations in the up and down direction of the bottom edge position of the image at the bottom of the detection target range, and if fluctuations in the bottom edge position occur alternately left and right at the bottom of the detection target range, it determines that a cyclist has been detected in the detection target range.

[0047] Next, the flow of the object detection method executed by object detection device 100 according to the present disclosure will be described with reference to Fig. 6. In the object detection method shown in Fig. 6, first, image acquisition unit 110 acquires image captured by camera 200 (step S201). Next, object detection unit 120 detects people from the image acquired in step S201 (step S202). Specifically, object detection unit 120 detects riders or cyclists from the acquired image using a rider detection model trained on images of people riding riders and a cyclist detection model trained on images of people riding bicycles.

[0048] Next, the determination unit 130 determines whether the cyclist detection score for the detection target range in which detection was performed in step S202 is equal to or greater than a second threshold value t2 (step S203). In this case, the second threshold value t2 is a value that indicates that the detection target in the detection target range is likely to be a cyclist or is clearly a cyclist, and is set to, for example, 0.8 when the cyclist detection score is detected in the range of 0.0 to 1.0.

[0049] If the cyclist detection score is equal to or greater than the second threshold t2 (step S203: Yes), the determination unit 130 determines that the person in the detection target range, i.e., the detection target, is a cyclist (step S209). If the cyclist detection score is equal to or greater than the second threshold t2, the image within the detection target range shows characteristics of a cyclist, so there is little chance that the person will be mistaken for a rider, and the person is determined to be a cyclist. If the cyclist detection score is less than the second threshold t2 (step S203: No), the determination unit 130 determines whether the cyclist detection score determined to be less than the second threshold t2 is equal to or greater than the first threshold t1 (step S204).

[0050] If the cyclist detection score is less than the first threshold t1 (step S204: No), the determination unit 130 determines whether the rider detection score is equal to or greater than the second threshold t2 for the same detection target range as the detection target range determined in step S203 (step S205). If the rider detection score is equal to or greater than the second threshold t2 (step S205: Yes), the determination unit 130 determines that the person in the detection target range, i.e., the detection target, is a rider (step S206). The second threshold t2 in this case is also a numerical value indicating that the detection target in the detection target range is likely to be a rider or is clearly a rider, and is set to, for example, 0.8 when the rider detection score is detected in the range from 0.0 to 1.0. If the rider detection score is less than the second threshold t2 (step S205: No), the determination unit 130 determines that neither a rider nor a cyclist is included in the detection target range and ends the process.

[0051] If the cyclist detection score is greater than or equal to the first threshold t1 (step S204: Yes), the determination unit 130 determines whether the rider detection score is greater than or equal to the second threshold t2 for the same detection target range as the detection target range determined in step S203 (step S207). If the rider detection score is less than the second threshold t2 (step S207: No), the determination unit 130 determines that the detection target is a cyclist (step S209). If the determination in step S207 is No, the cyclist detection score is lower than the second threshold t2, but characteristics of a cyclist appear in the image within the detection target range to the extent that they are greater than or equal to the first threshold t1, and further the rider detection score is less than the second threshold, so the determination is made that the detection target is a cyclist and not a rider.

[0052] If the rider detection score is equal to or greater than the second threshold value t2 (step S207: Yes), the determination unit 130 analyzes the image fluctuations in the lower part of the detection target range and determines whether or not there is a fluctuation (step S208). The analysis of the image fluctuations in the lower part of the detection target range is performed in the same manner as in the first embodiment. If there is a fluctuation in the image in the lower part of the detection target range (step S208: Yes), the determination unit 130 determines that the detection target is a cyclist (step S209). If there is no fluctuation in the image in the lower part of the detection target range (step S208: No), the determination unit 130 determines that the detection target is a rider (step S206).

[0053] In this way, the object detection device 100 can appropriately recognize a person riding a motorcycle or a person riding a bicycle.

[0054] <Embodiment 3> In the third embodiment, a case will be described in which a pedestrian is defined as the first form, and a cyclist and a rider are defined as the second form. In this embodiment, the determination unit 130 performs processing to determine whether a detected person is a pedestrian, a rider, or a cyclist based on the detection result of the object detection unit 120. Specifically, the determination unit 130 determines that a pedestrian is included in a detection target range in which the detection score by the pedestrian detection model defined as the first form detection model is equal to or greater than the second threshold t2. The determination unit 130 also determines that a cyclist or a rider is included in a detection target range in which the detection score by the cyclist detection model or the rider detection model defined as the second form detection model is equal to or greater than the second threshold t2. Furthermore, when the detection score by the pedestrian detection model is not equal to or greater than the second threshold t2, the determination unit 130 analyzes image fluctuations in the upper part of the detection target range in which the detection score by the cyclist detection model or the rider detection model is equal to or greater than the second threshold t2. When the image fluctuation in the upper part of the detection target range is large, the determination unit 130 determines that the detection target range includes a pedestrian.When the image fluctuation in the upper part of the detection target range is small, the determination unit 130 determines that the detection target range includes a cyclist or rider.

[0055] If the determination unit 130 determines that a cyclist or rider is included in the detection target range, it analyzes fluctuations in the image at the bottom of the detection target range. The determination unit 130 analyzes the fluctuations in the image at the bottom of the detection target range, and if the fluctuations at the bottom of the detection target range occur alternately left and right, it determines that a cyclist has been detected in the detection target range. The determination unit 130 also analyzes fluctuations in the up and down direction of the bottom edge position of the image at the bottom of the detection target range, and if the fluctuations in the bottom edge position occur alternately left and right at the bottom of the detection target range, it determines that a cyclist has been detected in the detection target range.

[0056] Next, the flow of the object detection method executed by object detection device 100 according to the present disclosure will be described with reference to Fig. 7. In the object detection method shown in Fig. 7, first, image acquisition unit 110 acquires image captured by camera 200 (step S301). Next, object detection unit 120 detects people from the image acquired in step S301 (step S302). Specifically, object detection unit 120 detects pedestrians, riders, or cyclists from the acquired image using a pedestrian detection model trained on images of pedestrians, a rider detection model trained on images of people riding riders, and a cyclist detection model trained on images of people riding bicycles.

[0057] Next, the determination unit 130 determines whether the pedestrian detection score for the detection target range in which detection was performed in step S302 is equal to or greater than a second threshold value t2 (step S303). The second threshold value t2 in this case is a numerical value indicating that the detection target in the detection target range is likely to be a pedestrian or is clearly a pedestrian, and is set to, for example, 0.8 when the pedestrian detection score is detected in the range from 0.0 to 1.0.

[0058] If the pedestrian detection score is equal to or greater than the second threshold t2 (step S303: Yes), the determination unit 130 determines that the person captured in the detection target range, i.e., the detection target, is a pedestrian (step S311). If the pedestrian detection score is equal to or greater than the second threshold t2, the image within the detection target range shows characteristics of a pedestrian, making it unlikely that the person will be mistaken for a cyclist or rider, and therefore determining that the person is a pedestrian. If the pedestrian detection score is less than the second threshold t2 (step S303: No), the determination unit 130 determines whether the pedestrian detection score determined to be less than the second threshold t2 is equal to or greater than the first threshold t1 (step S304).

[0059] If the pedestrian detection score is less than the first threshold value t1 (step S304: No), the determination unit 130 determines whether the cyclist detection score or rider detection score is equal to or greater than the second threshold value t2 for the same detection target range as the detection target range determined in step S303 (step S305). If the cyclist detection score or rider detection score is equal to or greater than the second threshold value t2 (step S305: Yes), the determination unit 130 analyzes fluctuations in the image at the bottom of the detection target range and determines whether fluctuations exist (step S306). The analysis of fluctuations in the image at the bottom of the detection target range is performed in the same manner as in the first embodiment. If fluctuations exist at the bottom of the detection target range (step S306: Yes), the determination unit 130 determines that the detection target is a cyclist (step S307). If there is no fluctuation in the image at the bottom of the detection target range (step S306: No), the determination unit 130 determines that the detection target is a rider (step S308). If the cyclist detection score and the rider detection score are less than the second threshold value t2 (step S305: No), the determination unit 130 determines that none of a pedestrian, rider, or cyclist is included within the detection target range, and ends the process.

[0060] If the pedestrian detection score is equal to or greater than the first threshold value t1 (step S304: Yes), the determination unit 130 determines whether the cyclist detection score or rider detection score is equal to or greater than the second threshold value t2 for the same detection target range as the detection target range determined in step S303 (step S309). If the cyclist detection score and rider detection score are less than the second threshold value t2 (step S309: No), the determination unit 130 determines that the detection target is a pedestrian (step S311). If the determination in step S309 is No, the pedestrian detection score is lower than the second threshold value t2, but pedestrian characteristics appear in the image within the detection target range to the extent that they are equal to or greater than the first threshold value t1, and further the cyclist detection score and rider detection score are less than the second threshold value, so the detection target is determined to be a pedestrian and not a cyclist or rider.

[0061] If the cyclist detection score or rider detection score is greater than or equal to the second threshold t2 (step S309: Yes), the determination unit 130 analyzes fluctuations in the image in the upper part of the detection target range and determines whether fluctuations exist (step S310). If the determination in step S309 is Yes, the pedestrian detection score is greater than or equal to the first threshold t1, and for the same detection target range, the cyclist detection score or rider detection score is greater than or equal to the second threshold t2. In other words, even though characteristics of a cyclist or rider are present, the pedestrian detection score also indicates a high probability of being a pedestrian. Therefore, even if an actual cyclist or rider is present, there is a possibility that they will be erroneously detected as a pedestrian. Therefore, based on the determination in step S310, it is determined whether the detection target is a cyclist or rider, or a pedestrian.

[0062] The analysis of image fluctuations in the upper part of the detection target range performed in step S310 is performed in the same manner as the analysis of image fluctuations in the lower part of the detection target range described in embodiment 1. When walking, people tend to move their hands more widely than when riding a bicycle or motorcycle. Therefore, in a detection target range that includes a pedestrian, image fluctuations in the upper part of the detection target range are greater than in a detection target range that includes a cyclist or rider. Therefore, by analyzing image fluctuations in the upper part of the detection target range, the determination unit 130 can determine whether a person included in the detection target range is a pedestrian, a cyclist, or a rider.

[0063] If the image is fluctuating in the upper part of the detection target range (step S310: Yes), the determination unit 130 determines that the detection target is a pedestrian (step S311). If the image is not fluctuating in the upper part of the detection target range (step S310: No), the determination unit 130 determines that the detection target is a cyclist or rider, and proceeds to step S306.

[0064] In this way, the object detection device 100 can appropriately recognize a pedestrian, a person on a bicycle, or a person on a motorcycle.

[0065] <Embodiment 4> In the fourth embodiment, a modified example will be described in which a pedestrian is defined as the first form and a cyclist is defined as the second form. In this embodiment, the determination unit 130 performs processing to determine whether a detected person is a pedestrian or a cyclist based on the detection result of the object detection unit 120. Specifically, the determination unit 130 determines that a pedestrian is included in a detection target range in which the detection score by the pedestrian detection model defined as the first form detection model is equal to or greater than the second threshold t2. The determination unit 130 also determines that a cyclist is included in a detection target range in which the detection score by the cyclist detection model defined as the second form detection model is equal to or greater than the second threshold t2. Furthermore, when the detection score by the pedestrian detection model is not equal to or greater than the second threshold t2, the determination unit 130 analyzes fluctuations in the video in the detection target range in which the detection score by the cyclist detection model is equal to or greater than the first threshold t1, which is lower than the second threshold t2, and is equal to or greater than the second threshold t2. When the fluctuation location is in the lower part of the detection target range, the determination unit 130 determines that a cyclist is included in the detection target range. When the fluctuation location is in the upper part of the detection target range, the determination unit 130 determines that a pedestrian is included in the detection target range.

[0066] Next, the flow of the object detection method executed by object detection device 100 according to the present disclosure will be described with reference to Fig. 8. In the object detection method shown in Fig. 8, first, image acquisition unit 110 acquires image captured by camera 200 (step S401). Next, object detection unit 120 detects people from the image acquired in step S401 (step S402). Specifically, object detection unit 120 detects pedestrians or cyclists from the acquired image using a pedestrian detection model trained on images of pedestrians and a cyclist detection model trained on images of people riding bicycles.

[0067] Next, the determination unit 130 determines whether the pedestrian detection score for the detection target range in which detection was performed in step S402 is equal to or greater than a second threshold value t2 (step S403). The second threshold value t2 in this case is a numerical value indicating that the detection target in the detection target range is likely to be a pedestrian or is clearly a pedestrian, and is set to, for example, 0.8 when the pedestrian detection score is detected in the range from 0.0 to 1.0.

[0068] If the pedestrian detection score is equal to or greater than the second threshold t2 (step S403: Yes), the determination unit 130 determines that the person captured in the detection target range, i.e., the detection target, is a pedestrian (step S410). If the pedestrian detection score is equal to or greater than the second threshold t2, the image within the detection target range shows characteristics of a pedestrian, so the person is unlikely to be mistaken for a cyclist and is determined to be a pedestrian. If the pedestrian detection score is less than the second threshold t2 (step S403: No), the determination unit 130 determines whether the pedestrian detection score determined to be less than the second threshold t2 is equal to or greater than the first threshold t1 (step S404).

[0069] If the pedestrian detection score is less than the first threshold t1 (step S404: No), the determination unit 130 determines whether the cyclist detection score is equal to or greater than the second threshold t2 for the same detection target range as the detection target range determined in step S403 (step S405). If the cyclist detection score is equal to or greater than the second threshold t2 (step S405: Yes), the determination unit 130 determines that the person appearing in the detection target range, i.e., the detection target, is a cyclist (step S411). If the cyclist detection score is less than the second threshold t2 (step S405: No), the determination unit 130 determines that neither a pedestrian nor a cyclist is included in the detection target range, and ends the process.

[0070] If the pedestrian detection score is equal to or greater than the first threshold t1 (step S404: Yes), the determination unit 130 determines whether the cyclist detection score is equal to or greater than the second threshold t2 for the same detection target range as the detection target range determined in step S403 (step S406).If the cyclist detection score is less than the second threshold t2 (step S406: No), the determination unit 130 determines that neither a pedestrian nor a cyclist is included in the detection target range, and ends the process.

[0071] If the cyclist detection score is greater than or equal to the second threshold t2 (step S406: Yes), the determination unit 130 analyzes the image fluctuations in the detection target range and determines whether or not there is a fluctuation (step S407). The analysis of the image fluctuations in the detection target range in step S407 is performed in the same manner as the analysis of the image fluctuations in the detection target range described in embodiment 1. If the determination in step S407 is Yes, the pedestrian detection score is greater than or equal to the first threshold t1, and the cyclist detection score is greater than or equal to the second threshold t2 for the same detection target range. In other words, even though the characteristics of a cyclist are present, the pedestrian detection score also indicates a high possibility of a pedestrian. Therefore, even if an individual is actually a cyclist, there is a possibility that they will be erroneously detected as a pedestrian. Therefore, based on the determinations in steps S408 and S409, it is determined whether the detection target is a cyclist or a pedestrian.

[0072] When the determination unit 130 determines that there is a fluctuation in the image within the detection target range, in other words, when it detects a fluctuation in the image within the detection target range, it determines which part of the detection target range the fluctuation is in. Specifically, the determination unit 130 determines whether the fluctuation in the image detected in step S407 is located at the bottom or the top of the detection target range.

[0073] If the answer is Yes in step S407, the determination unit 130 determines whether the fluctuation location is in the lower part of the detection target range (step S408). If the fluctuation location is in the lower part of the detection target range (step S408: Yes), the determination unit 130 determines that the detection target is a cyclist (step S411). If the fluctuation location is not in the lower part of the detection target range (step S408: No), the determination unit 130 determines whether the fluctuation location is in the upper part of the detection target range (step S409). If the fluctuation location is in the upper part of the detection target range (step S409: Yes), the determination unit 130 determines that the detection target is a pedestrian (step S410). If the fluctuation location is not in the upper part of the detection target range (step S409: No), the determination unit 130 determines that neither a pedestrian nor a cyclist is included in the detection target range, and ends the process. If the answer is No in step S407, the determination unit 130 determines that the detection target is a cyclist (step S411).

[0074] In this way, the object detection device 100 can appropriately recognize pedestrians or people riding bicycles.

[0075] The present disclosure is not limited to the above-described embodiments and can be modified as appropriate without departing from the spirit and scope of the present disclosure. For example, in the fourth embodiment, a cyclist may be defined as the first form and a pedestrian as the second form. In this case, the processing of FIG. 8 is performed with the pedestrian and cyclist interchanged. The present disclosure may also be implemented by appropriately combining the embodiments and examples thereof. [Explanation of symbols]

[0076] 100 Object detection device 110 Video acquisition unit 120 Object detection unit 130 Judgment section 140 Output control section 200 cameras 300 Output Device 400 Object Detection System

Claims

1. an image acquisition unit that acquires an image captured by an infrared camera; an object detection unit that detects a person in the first form or a person in the second form from the acquired video using a first form detection model that has been trained with a video of a person in the first form and a second form detection model that has been trained with a video of a person in the second form; a determination unit that analyzes fluctuations in the video for a detection target range in which the detection score by the first form detection model is equal to or greater than a first threshold and the detection score by the second form detection model is equal to or greater than a second threshold that is higher than the first threshold, based on detection results of the first form and the second form in the acquired video, and determines whether the person in the detection target range is in the first form or the second form based on the location of large fluctuations; An object detection device comprising:

2. The combination of the first form and the first form detection model and the second form and the second form detection model is [1] Pedestrians and pedestrian detection models, and cyclists and cyclist detection models or [2] Cyclists and cyclist detection models, and pedestrians and pedestrian detection models and the determination unit determines that the person in the detection target range is a cyclist when the detected area where the variation is large is in the lower part of the detection target range, and determines that the person in the detection target range is a pedestrian when the detected area where the variation is large is in the upper part of the detection target range. The object detection device according to claim 1 .

3. the determination unit determines whether the person in the detection target range is in the first form or the second form based on a location in the detection target range where variations occur alternately left and right. The object detection device according to claim 1 .

4. the image acquisition unit acquires an image showing a heat distribution by a brightness value; the determination unit analyzes fluctuations in luminance values ​​in the detection target range. The object detection device according to claim 1 .

5. acquiring an image captured by an infrared camera; detecting a person of the first form or a person of the second form from the acquired video using a first form detection model trained with a video of a person of the first form and a second form detection model trained with a video of a person of the second form; a step of analyzing fluctuations in the image for a detection target range in which the detection score by the first form detection model is equal to or greater than a first threshold and the detection score by the second form detection model is equal to or greater than a second threshold that is higher than the first threshold, based on the detection results of the first form and the second form in the acquired image, and determining whether the person in the detection target range is in the first form or the second form based on the location of large fluctuations; The object detection method is performed by the object detection device.

Images