Object detection device and object detection method

Abstract

An object detection device (100) comprises: a radar signal processor (3) that detects, on the basis of a radar reception signal output from a radar device (1) for scanning a monitoring area, the distance, speed, and horizontal angle of an object present in the monitoring area, computes a first recognition score, which is an index indicating the likelihood of whether or not the detected object is a person, on the basis of the feature of the reception strength of the radar reception signal, and outputs the first recognition score in association with information pertaining to the position and speed of the object; a captured image processor (4) that recognizes the object present in the monitoring area on the basis of image data which is obtained from an image captured by a camera (2) for capturing an image of the monitoring area and computes and outputs position information pertaining to the recognized object and a second recognition score, which indicates the likelihood of whether or not the recognized object is a person; and a fusion processor (5) that determines, on the basis of the first and second recognition scores, whether or not the person who has been detected on the basis of the radar reception signal and the person who has been detected on the basis of the image data are the same person.

Description

Object detection device and object detection method

[0001] The present disclosure relates to an object detection device and an object detection method for detecting an object present in a monitoring area.

[0002] Patent Document 1 listed below discloses a surroundings monitoring system for a work machine that is mounted on a work machine such as a backhoe, a wheel loader, an asphalt finisher, etc. The technology in Patent Document 1 uses a camera to obtain detection data on people present at a work site where civil engineering work, construction work, etc. is being carried out.

[0003] International Publication No. 2018 / 084146

[0004] However, in addition to people, there are various other objects at work sites, such as traffic cones (registered trademark) and containers. Therefore, object detection devices that use detection data from cameras alone may mistakenly detect non-human objects as people. As a result, conventional technologies have had the problem of causing erroneous control or false alarms in operations such as vehicle control and warning notifications.

[0005] The present disclosure has been made in consideration of the above, and aims to provide an object detection device that can reduce the possibility of erroneous control or false alarms in operations such as vehicle control and alarm notification.

[0006] To solve the above-mentioned problems and achieve the object, the object detection device according to the present disclosure includes a radar signal processor, a captured image processor, and a fusion processor. The radar signal processor detects the distance, speed, and horizontal angle of an object present in a monitoring area based on a radar reception signal output from a radar device scanning the monitoring area, and calculates a first recognition score, which is an index indicating the likelihood of whether the detected object is a person, based on characteristics of the reception strength of the radar reception signal. The first recognition score is linked to information on the object's position and speed and outputs the first recognition score. The captured image processor recognizes objects present in the monitoring area based on image data obtained from images captured by a camera capturing the monitoring area, and calculates and outputs position information of the recognized object and a second recognition score indicating the likelihood of whether the recognized object is a person. The fusion processor determines whether the person detected by the radar reception signal and the person detected by the image data are the same person based on the first and second recognition scores.

[0007] The object detection device according to the present disclosure has the effect of reducing the possibility of erroneous control or false alarms in operations such as vehicle control and alarm notification.

[0008] A block diagram showing an example of the configuration of an object detection device according to an embodiment. A block diagram showing an example of the hardware configuration that realizes the functions of a radar signal processor, a captured image processor, and a fusion processor provided in the object detection device according to the embodiment. A first flow diagram showing the processing flow in the object detection device according to the embodiment. A second flow diagram showing the processing flow in the object detection device according to the embodiment.

[0009] Hereinafter, an object detection device and an object detection method according to embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0010] Embodiment. First, the configuration and functions of an object detection device according to an embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing an example configuration of an object detection device 100 according to an embodiment. In the following description, objects are divided into people and objects other than people. In this paper, assuming that the object detection device 100 is mounted on a work machine, objects other than people will be referred to as "obstacles" as appropriate. Examples of obstacles include traffic cones (registered trademark) and containers that may be present at a work site. Examples of work machines include excavators, wheel loaders, bulldozers, dump trucks, and asphalt finishers.

[0011] The object detection device 100 includes a radar signal processor 3, a captured image processor 4, and a fusion processor 5. Outside the object detection device 100, there are a radar device 1 and a camera 2. The radar device 1 is a radar device that scans a monitoring area, and the camera 2 is an imaging device that captures images of the monitoring area. When the monitoring area is a work site, the radar device 1, camera 2, and object detection device 100 are installed at the work site or mounted on a work machine used at the work site. When the radar device 1, camera 2, and object detection device 100 are mounted on a traveling vehicle, the object detection device 100 monitors the area in front of or behind the traveling vehicle as the monitoring area.

[0012] The radar device 1 emits electromagnetic waves toward an object and receives a reflected signal from the object in a monitoring area. The radar device 1 performs required reception processing and outputs the processed signal as a radar reception signal to a radar signal processor 3. When the radar device 1 is mounted on a vehicle, a frequency modulated continuous wave (FMCW) radar device or a fast chirp modulation (FCM) radar device is generally used. Furthermore, radar devices of these types can be configured using high-frequency semiconductor components, power semiconductor components, substrates, crystal devices, chip components, antennas, etc.

[0013] The radar signal processor 3 detects the distance, speed, and horizontal angle of an object based on the radar reception signal output from the radar device 1. The radar signal processor 3 also calculates a person recognition score, which is an index indicating the likelihood of whether the detected object is a person, based on the characteristics of the reception strength of the radar reception signal. Furthermore, the radar signal processor 3 links the person recognition score to information on the position and speed of the object and outputs it to the fusion processor 5. In this document, the person recognition score calculated by the radar signal processor 3 may be referred to as a "first recognition score."

[0014] To achieve the above functions, the radar signal processor 3 includes a distance detection unit 6, a speed detection unit 7, a horizontal angle detection unit 8, a reception intensity feature amount calculation unit 9, a person recognition unit 10, and a reception intensity feature amount database for person recognition 11. The radar signal processor 3 outputs radar detection data 12 to the fusion processor 5. As shown in FIG. 1 , the radar detection data 12 includes a position, a speed, and a person recognition score. The position and speed are generated by the distance detection unit 6, the speed detection unit 7, and the horizontal angle detection unit 8, and the person recognition score is generated by the reception intensity feature amount calculation unit 9, the person recognition unit 10, and the reception intensity feature amount database for person recognition 11. In this paper, the reception intensity feature amount database for person recognition 11 held by the radar signal processor 3 may be referred to as a "first database."

[0015] Fast Fourier Transformation (FFT) is generally used for the processing of the distance detection unit 6, the velocity detection unit 7, and the horizontal angle detection unit 8. In this case, the distance detection unit 6, the velocity detection unit 7, and the horizontal angle detection unit 8 each perform FFT processing on the distance direction, the velocity direction, and the horizontal angle direction to detect the distance, velocity, and horizontal angle of an object.

[0016] FFT can also be used for the processing of the reception intensity feature amount calculation unit 9. Specifically, the reception intensity feature amount calculation unit 9 performs FFT processing in the distance direction and the velocity direction on the reflected signal of the electromagnetic wave from the object, finds the peak value of the signal as the reflected signal intensity, and detects the amount of change over time in the reflected signal intensity as the reception intensity feature amount.

[0017] In the radar signal processor 3, a data group that collects data related to person reception intensity feature quantities is constructed as a person recognition reception intensity feature quantity database 11. The person reception intensity feature quantities are reception intensity feature quantities related to people that have been obtained in advance through experiments, learning processes, or the like. The person recognition unit 10 refers to the data stored in the person recognition reception intensity feature quantity database 11 and calculates a person recognition score related to the reception intensity feature quantities output from the reception intensity feature quantity calculation unit 9. The person recognition score is an index that indicates the likelihood of whether a detected object is a person. The person recognition score is calculated as a value in the range of 0 to 100%, for example, but is not limited to this example. The person recognition score may also be calculated as a value between 0 and 1.

[0018] The camera 2 is composed of components such as a lens, a holder, a CMOS (Complementary Metal Oxide Semiconductor) sensor, power semiconductor components, a crystal device, etc. The camera 2 captures an image of the monitoring area and outputs the captured image to the captured image processor 4.

[0019] The captured image processor 4 recognizes objects present in the monitoring area based on image data obtained from the captured image by the camera 2, calculates position information of the recognized object and a person recognition score indicating the likelihood that the recognized object is a person, and outputs the results to the fusion processor 5. In this paper, the person recognition score calculated by the captured image processor 4 may be referred to as a "second recognition score."

[0020] To achieve the above functions, the captured image processor 4 includes a person recognition unit 13, a position detection unit 14, and a deep learning feature database for person recognition 15. The captured image processor 4 outputs camera detection data 16 to the fusion processor 5. As shown in FIG. 1 , the camera detection data 16 includes a position and a person recognition score. The position is generated by the person recognition unit 13 and the position detection unit 14, and the person recognition score is generated by the person recognition unit 13 and the deep learning feature database for person recognition 15. In this paper, the deep learning feature database for person recognition 15 held by the captured image processor 4 may be referred to as the "second database."

[0021] In the captured image processor 4, a data group that collects data related to deep learning features for person recognition, which are features obtained by deep learning, is constructed as a deep learning feature database for person recognition 15. The person recognition unit 13 recognizes objects from image data obtained from images captured by the camera 2, and calculates a person recognition score for the recognized object by referring to data stored in the deep learning feature database for person recognition 15. The position detection unit 14 detects the position of the object based on the pixel coordinates of the object for which the person recognition score has been calculated. The person recognition score is an index that indicates the likelihood of whether the object recognized by the person recognition unit 13 is a person. The person recognition score is calculated as a value in the range of 0 to 100%, for example, but is not limited to this example. The person recognition score may also be calculated as a value between 0 and 1.

[0022] The object recognition process in the person recognition unit 13 can use, for example, network algorithms such as YOLO (You Only Look Once) and SSD (Single Shot multibox Detector). The person recognition unit 13 may also use a method other than deep learning. For example, learning processes such as machine learning and the HOG (Histograms of Oriented Gradients) method can also be used.

[0023] The fusion processor 5 determines whether the person detected based on the radar reception signal and the person detected based on image data obtained from the image captured by the camera 2 are the same person, based on the person recognition score calculated by the person recognition unit 10 and the person recognition score calculated by the captured image processor 4. To achieve this function, the fusion processor 5 includes an object identity determination unit 17 and a person determination unit 18.

[0024] The object identity determination unit 17 performs identity determination processing to determine whether or not the objects are the same object based on information about the position of the object detected by the radar device 1 and information about the position of the object detected by the camera 2. Through this processing, the data of the position, speed, and person recognition score detected based on the radar reception signal is linked to the data of the position and person recognition score detected based on the image data of the camera 2.

[0025] If the object identity determination unit 17 determines that the objects are the same in its identity determination process, the person determination unit 18 performs person determination processing using the person recognition score calculated by the radar signal processor 3 and the person recognition score calculated by the captured image processor 4. The person determination unit 18 generates fusion detection data 19 as a result of this person determination processing. As shown in FIG. 1 , the fusion detection data 19 includes position, speed, and recognized object type. The recognized object type is classified into people and non-people. The position and speed are information on the position and speed of the corresponding object for each recognized object type linked in the identity determination process of the object identity determination unit 17. The fusion detection data 19 is used as a control signal for vehicle control, warning notification, etc.

[0026] Although the fusion processor 5 determines whether a detected object is a person, the object to be determined may be replaced with any object other than a person, or any object other than a person may be added to the objects to be determined. For example, if a vehicle is added to the objects to be determined, the recognized object types can be classified into people, vehicles, and objects other than people and vehicles.

[0027] Next, a hardware configuration of the object detection device 100 according to the embodiment will be described. Fig. 2 is a block diagram showing an example of a hardware configuration that realizes the functions of the radar signal processor 3, the captured image processor 4, and the fusion processor 5 provided in the object detection device 100 according to the embodiment. As shown in Fig. 2, the radar signal processor 3, the captured image processor 4, and the fusion processor 5 can each include a processing circuit 90 and an interface 91 that inputs and outputs signals, and the processing circuit 90 can include a processor 92 that executes software, and a storage unit 93.

[0028] The processor 92 is configured to include at least one arithmetic means called a microprocessor, microcomputer, microcontroller, CPU (Central Processing Unit), MCU (Micro Control Unit), GPU (Graphics Processing Unit), or DSP (Digital Signal Processor). Examples of the storage unit 93 include non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically EPROM), as well as magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs (Digital Versatile Discs).

[0029] The storage unit 93 stores programs that execute the functions of the above-mentioned radar signal processor 3, captured image processor 4, and fusion processor 5. The processor 92 exchanges necessary information via the interface 91, executes the programs stored in the storage unit 93, and performs the above-mentioned processing by referring to the table data stored in the storage unit 93, i.e., the data in the person recognition reception intensity feature database 11 and the person recognition deep learning feature database 15 stored in the storage unit 93. The calculation results by the processor 92 can be held in the storage unit 93.

[0030] Next, a series of processing flows in object detection device 100 according to the embodiment will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a first flowchart showing the processing flow in object detection device 100 according to the embodiment. Fig. 4 is a second flowchart showing the processing flow in object detection device 100 according to the embodiment.

[0031] 3 , when the object detection process is started (step S1), the radar signal processor 3 outputs reception data acquired by the radar device 1 (step S2). In the radar signal processor 3, the distance detector 6, the speed detector 7, and the horizontal angle detector 8 detect the position and speed of the object (step S3). The reception intensity feature calculator 9 detects the amount of change over time in the reflection intensity of the electromagnetic wave (step S4). The person recognition unit 10 recognizes people by referring to the reception intensity feature database 11 for person recognition and calculates a person recognition score (step S5). The position, speed, and person recognition score obtained by these processes are stored in the storage unit 93 as radar detection data 12 (step S6).

[0032] The captured image processor 4 receives the captured image captured by the camera 2 (step S7). In the captured image processor 4, the person recognition unit 13 recognizes objects by referring to the deep learning feature database for person recognition 15 and calculates a person recognition score for the recognized object (step S8). The position detection unit 14 detects the position of the object for which the person recognition score has been calculated (step S9). The position and person recognition score obtained by these processes are stored in the storage unit 93 as camera detection data 16 (step S10).

[0033] The object identity determination unit 17 of the fusion processor 5 performs the above-described identity determination process (step S11). If it is determined in this identity determination process that the detected objects are not the same (step S12, No), the object identity determination unit 17 discards the radar detection data 12 and the camera detection data 16 (step S13) and proceeds to the detection process for the next frame (step S21). On the other hand, if it is determined that the detected objects are the same (step S12, Yes), the object identity determination unit 17 transfers the position, speed, and person recognition score data of the radar detection data 12 and the position and person recognition score data of the camera detection data 16 to the person determination unit 18 (step S14). This identity determination process links the position, speed, and person recognition score data detected by the radar signal processor 3 with the position and person recognition score data detected by the captured image processor 4.

[0034] The person determination unit 18 multiplies the person recognition scores of the radar device 1 and the camera 2 by a predetermined coefficient, adds the results, and compares them with a predetermined person determination threshold (step S15). Here, the person recognition score of the radar detection data 12 is defined as Sr, and the person recognition score of the camera detection data 16 is defined as Sc. Furthermore, the weighting coefficients of the person recognition scores Sr and Sc are defined as Wr and Wc, respectively. The person determination unit 18 calculates the person determination score S from these person recognition scores Sr and Sc and the weighting coefficients Wr and Wc using the following equation.

[0035] S=(Wr×Sr)+(Wc×Sc)

[0036] The person determination threshold is set to Th. The person determination unit 18 determines whether the person determination score S is equal to or greater than the person determination threshold Th (step S16). If the person determination score S is equal to or greater than the person determination threshold Th (step S16, Yes), the position and speed of the radar device 1 are stored as the position and speed of the fusion detection data 19, and "person" is stored as the recognized object type of the fusion detection data 19 (step S17). On the other hand, if the person determination score S is less than the person determination threshold Th (step S16, No), the position and speed of the radar device 1 are stored as the position and speed of the fusion detection data 19, and "obstacle" is stored as the recognized object type of the fusion detection data 19 (step S18).

[0037] After steps S17 and S18 are completed, the fusion processor 5 outputs the latest fusion detection data 19 (step S19), and vehicle control is performed using this fusion detection data 19 (step S20). The above processing completes one frame of object detection processing, and the process moves on to object detection processing for the next frame (step S21). In the object detection processing for the next frame, the above-described processing of steps S1 to S20 is repeated.

[0038] Conventional object detection devices use cameras, which can mistakenly detect various obstacles, such as traffic cones (registered trademark) and containers that may be present at worksites where civil engineering or construction work is being carried out, as people, resulting in erroneous control or false alarms in vehicle control, warning notifications, and other operations. In contrast, the object detection device according to the embodiment detects people using different detection principles, namely, a radar device and a camera, calculates a person recognition score, and then performs final person determination processing using a fusion processor, thereby reducing the possibility of mistakenly detecting obstacles as people. Therefore, the object detection device according to the embodiment can contribute to safe vehicle control operations at worksites.

[0039] As described above, the object detection device according to the embodiment includes a radar signal processor, a captured image processor, and a fusion processor. The radar signal processor detects the distance, speed, and horizontal angle of an object present in a monitoring area based on a radar reception signal output from a radar device scanning the monitoring area, and calculates a first recognition score, which is an index indicating the likelihood that the detected object is a person, based on characteristics of the reception strength of the radar reception signal. The first recognition score is linked to information on the object's position and speed and outputs the first recognition score. The captured image processor recognizes objects present in the monitoring area based on image data obtained from images captured by a camera capturing the monitoring area, and calculates and outputs position information of the recognized object and a second recognition score indicating the likelihood that the recognized object is a person. The fusion processor determines whether the person detected by the radar reception signal and the person detected by the image data are the same person based on the first and second recognition scores. The object detection device configured as described above can reduce the possibility of erroneously detecting objects other than people, such as traffic cones (registered trademark) and containers that may be present at a work site, as people. This makes it possible to reduce the possibility of erroneous control or false alarms in operations such as vehicle control and alarm notification.

[0040] Furthermore, an object detection method according to an embodiment can be a process including the following first to sixth steps. In the first step, the distance, speed, and horizontal angle of an object present in a monitoring area are detected based on a radar reception signal output from a radar device scanning the monitoring area. In the second step, a first recognition score, which is an index indicating the likelihood of whether the detected object is a person, is calculated based on characteristics of the reception strength of the radar reception signal. In the third step, the first recognition score is linked to information on the object's position and speed. In the fourth step, objects present in the monitoring area are recognized based on image data obtained from an image captured by a camera capturing the monitoring area. In the fifth step, position information of the recognized object and a second recognition score indicating the likelihood of whether the recognized object is a person are calculated. In the sixth step, based on the first and second recognition scores, it is determined whether the person detected by the radar reception signal and the person detected by the image data are the same person. Using an object detection method including these first to sixth steps can reduce the possibility of erroneously detecting objects other than people, such as traffic cones (registered trademark) or containers that may be present at a work site, as people. This makes it possible to reduce the possibility of erroneous control or false alarms in operations such as vehicle control and alarm notification.

[0041] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, and parts of the configurations may be omitted or modified without departing from the spirit of the invention.

[0042] 1 Radar device, 2 Camera, 3 Radar signal processor, 4 Captured image processor, 5 Fusion processor, 6 Distance detection unit, 7 Speed ​​detection unit, 8 Horizontal angle detection unit, 9 Reception intensity feature calculation unit, 10, 13 Person recognition unit, 11 Reception intensity feature database for person recognition, 12 Radar detection data, 14 Position detection unit, 15 Deep learning feature database for person recognition, 16 Camera detection data, 17 Object identity determination unit, 18 Person determination unit, 19 Fusion detection data, 90 Processing circuit, 91 Interface, 92 Processor, 93 Memory unit, 100 Object detection device.

Claims

1. A radar signal processor that detects the distance, speed, and horizontal angle of an object present in the monitoring area based on a radar reception signal output from a radar device that scans the monitoring area, calculates a first recognition score that is an index indicating the likelihood that the detected object is a person based on the characteristics of the reception intensity of the radar reception signal, and associates and outputs the first recognition score with the information on the position and speed of the object; a captured image processor that recognizes an object present in the monitoring area based on image data obtained from a captured image of the monitoring area, and calculates and outputs position information of the recognized object and a second recognition score that indicates the likelihood that the recognized object is a person; and a fusion processor that determines whether the person detected by the radar reception signal and the person detected by the image data are the same person based on the first and second recognition scores. An object detection device characterized by comprising:

2. The radar signal processor has a first database that collects data on reception intensity feature amounts of persons, and calculates the first recognition score by referring to the data stored in the first database. The captured image processor has a second database that collects data on deep learning feature amounts for person recognition, which are feature amounts obtained by deep learning, recognizes an object from the image data, and calculates the second recognition score by referring to the data stored in the second database for the recognized object. The object detection device according to claim 1, characterized in that:

3. The fusion processor includes an object identity determination unit that determines whether each object is the same object based on the information on the position of the object detected based on the radar reception signal and the information on the position of the object detected based on the image data obtained from the captured image of the camera; and a person determination unit that performs a person determination process using the first recognition score calculated by the radar signal processor and the second recognition score calculated by the captured image processor when a determination is made that they are the same in the identity determination process of the object identity determination unit. The object detection device according to claim 1 or 2, characterized in that:

4. A first step of detecting the distance, speed, and horizontal angle of an object present in the monitoring area based on a radar reception signal output from a radar device that scans the monitoring area; a second step of calculating a first recognition score, which is an index indicating the likelihood that the detected object is a person, based on the characteristics of the reception intensity of the radar reception signal; a third step of associating the first recognition score with the position and speed information of the object; a fourth step of recognizing an object present in the monitoring area based on image data obtained from a captured image of a camera that captures the monitoring area; a fifth step of calculating the position information of the recognized object and a second recognition score indicating the likelihood that the recognized object is a person; and a sixth step of determining whether the person detected by the radar reception signal and the person detected by the image data are the same person based on the first and second recognition scores. An object detection method characterized by including these steps.

Images