Monitoring device and monitoring method

Abstract

In one aspect, a monitoring device (100) has a tracking unit (13), a position information acquisition unit (15), a determination unit (17), and a sensor control unit (18). The tracking unit (13) tracks, on the basis of image data acquired from an image sensor, a plurality of objects reflected in the image data. The position information acquisition unit (15) acquires position information of the plurality of objects from a distance measurement device. The determination unit (17) determines the collision possibility of the plurality of objects on the basis of the position information and the tracking information of the plurality of objects. The sensor control unit (18) controls, on the basis of the determination result of the collision possibility, switching between a first mode for acquiring the position information at a prescribed resolution and a second mode for acquiring the position information at a resolution higher than the prescribed resolution.

Description

Monitoring device and monitoring method

[0001] The present invention relates to a monitoring device and a monitoring method.

[0002] Conventionally, there is a technology that uses a light detection and ranging (LiDAR) to acquire position information of surrounding objects, and there is also an event-based vision sensor (EVS) that acquires information on the movement of objects in the field of view based on changes in brightness.

[0003] US Patent Application Publication No. 2023 / 0005169 US Patent No. 9,164,511 US Patent No. 10,345,447 US Patent Application Publication No. 2023 / 0236321

[0004] For example, in automobile driving assistance, determining whether a collision with a vehicle or person ahead of the vehicle is occurring is necessary to determine whether the vehicle needs to take evasive action. In this case, for example, based on the position information of multiple objects acquired by LiDAR, it is possible to determine the possibility of a collision between the multiple objects. However, LiDAR generally has problems such as higher power consumption and lower resolution compared to other sensors.

[0005] In one aspect, the present invention aims to provide a monitoring device and a monitoring method that are capable of monitoring collisions with reduced power consumption and improved resolution.

[0006] In one aspect, the monitoring device disclosed herein includes a tracking unit, a position information acquisition unit, a determination unit, and a sensor control unit. The tracking unit tracks multiple objects captured in image data acquired from an image sensor based on the image data. The position information acquisition unit acquires position information of the multiple objects from a distance measurement device. The determination unit determines a possibility of a collision between the multiple objects based on the tracking information of the multiple objects and the position information. The sensor control unit controls switching between a first mode in which the position information is acquired at a predetermined resolution and a second mode in which the position information is acquired at a resolution higher than the predetermined resolution based on the result of the determination of the possibility of a collision.

[0007] According to one aspect of the monitoring device disclosed in the present application, it is possible to monitor collisions with reduced power consumption and improved resolution. As a result, according to one aspect of the monitoring device disclosed in the present application, for example, based on the result of determining the possibility of a collision, it is possible to switch between a first mode in which position information is acquired at a predetermined resolution with reduced power consumption and a second mode in which position information is acquired at a resolution higher than the predetermined resolution, i.e., high resolution.

[0008] FIG. 1 is a diagram illustrating an example of a monitoring system including a monitoring device according to a first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration related to the monitoring system according to the first embodiment. FIG. 3 is a functional block diagram illustrating an example of a functional configuration according to the first embodiment. FIG. 4 is a flowchart illustrating an example of a procedure for a mode change process in a monitoring process according to the first embodiment. FIG. 5 is a diagram illustrating an example of the detection range of a ranging device and the detection range of an image sensor (CIS) in a first mode, and two objects within the detection ranges according to the first embodiment. FIG. 6 is a diagram illustrating an example of the positional relationship between two objects when it is determined that the collision indicator is equal to or greater than a first threshold according to the first embodiment. FIG. 7 is a diagram illustrating an example of acquisition of position information in a second mode according to the first embodiment. FIG. 8 is a diagram illustrating an example of a case in which the distances from the vehicle (mobile object) to two objects are approximately the same during monitoring in the second mode according to the first embodiment. FIG. 9 is a diagram illustrating an example of a case in which one object is located near the mobile object and the other object is located far from the mobile object during monitoring in the second mode according to the first embodiment. Fig. 10 is a diagram showing an example of application of the first embodiment. Fig. 11 is a functional block diagram showing an example of a functional configuration according to a second embodiment. Fig. 12 is a diagram showing an example of application of the second embodiment. Fig. 13 is a diagram showing an example of application of a monitoring device installed in a street light according to a third embodiment.

[0009] Hereinafter, with reference to the accompanying drawings, embodiments of the monitoring device and monitoring method disclosed herein will be described in detail. Note that the following embodiments do not limit the disclosed technology. Furthermore, each embodiment can be appropriately combined within a range that does not cause contradictions in the processing content.

[0010] First Embodiment FIG. 1 is a diagram illustrating an example of a monitoring system 1 including a monitoring device 100 according to this embodiment. The monitoring system 1 illustrated in FIG. 1 is mounted on a mobile body 200 such as an automobile. The mobile body 200 on which the monitoring system 1 is mounted is a movable object. The mobile body 200 may be, for example, a vehicle, a flyable object (a manned airplane, an unmanned airplane (e.g., a UAV (Unmanned Aerial Vehicle) or a drone), a transport robot, or a ship. The mobile body 200 may be, for example, a mobile body that moves forward through human driving operation, or a mobile body that can move automatically (autonomously) without human driving operation. In this embodiment, a case where the mobile body 200 is a vehicle will be described as an example. Examples of vehicles include a two-wheeled vehicle, a three-wheeled vehicle, and a four-wheeled vehicle. In this embodiment, a case where the vehicle is a four-wheeled vehicle will be described as an example.

[0011] As shown in Fig. 1, the monitoring system 1 includes a monitoring device 100, image sensors 19A and 19B, and a distance measuring device 20. In Fig. 1, for ease of understanding, the monitoring system 1 is depicted superimposed on an image of a mobile object 200 viewed from above. However, in reality, the monitoring device 100 is mounted on a control board or the like mounted on the mobile object 200.

[0012] The image sensors 19A and 19B are provided, for example, outside and inside the vehicle, respectively, ahead of the moving body 200 in the traveling direction D. The number of image sensors 19A and 19B is not limited to two, and multiple sensors may be provided at positions other than ahead of the moving body 200 in the traveling direction D. For example, other image sensors different from the image sensors 19A and 19B may be installed at the rear of the moving body 200, on the side of the moving body 200, or in the opposite direction to the traveling direction D of the moving body 200. The other image sensors are, for example, imaging devices such as cameras.

[0013] Image sensors 19A, 19B and distance measuring device 20 are electrically connected to monitoring device 100 as shown in Fig. 1. Note that image sensors 19A, 19B and monitoring device 100 are not limited to being electrically directly connected as shown in Fig. 1, and may be connected wirelessly. Also, distance measuring device 20 and monitoring device 100 are not limited to being electrically directly connected as shown in Fig. 1, and may be connected wirelessly.

[0014] The image sensor 19A is realized by, for example, an event-based image sensor (hereinafter referred to as an EVS (Event-based Vision Sensor)). The image sensor 19A acquires information on the movement of an object in its field of view from changes in brightness. The installation location of the image sensor 19A ahead in the traveling direction D of the moving body 200 is not limited to the interior of the moving body 200 shown in FIG. 1 , but may be installed outside the moving body 200.

[0015] The image sensor 19B is realized by, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The CMOS image sensor may also be referred to as a CIS. The CIS corresponds to, for example, an RGB camera. Note that the image sensor 19B is not limited to a CIS and may be realized by another optical camera. The installation location of the image sensor 19B is not limited to the outside of the vehicle 200 shown in FIG. 1 , but may also be installed inside the vehicle 200.

[0016] The ranging device 20 is realized by, for example, a LiDAR (Light Detection and Ranging) sensor. The ranging device (LiDAR sensor) 20 acquires position information of objects in the vicinity of the moving body 200. Note that the ranging device 20 is not limited to LiDAR, and may be realized by radar or the like. Furthermore, the installation location of the ranging device 20 ahead in the traveling direction D of the moving body 200 is not limited to the interior of the moving body 200 shown in FIG. 1 , and may be installed outside the moving body 200.

[0017] Fig. 2 is a diagram showing an example of the hardware configuration of the monitoring system 1. As shown in Fig. 2, the monitoring system 1 includes a monitoring device 100, a distance measuring device 20, an image sensor 19A, an image sensor 19B, a drive control circuit (Driver control) 300, and an upper level ECU (Upper level ECU) 400. The monitoring system 1 may further include a memory for storing various data, a display device for displaying various data, and the like.

[0018] The drive control circuit 300 is a processing circuit that controls various drives related to the movement of the mobile object 200. The drive control circuit 300 is electrically connected to the monitoring device 100 via a communication interface 109. The drive control circuit 300 controls various drives in the mobile object 200 under the control of a higher-level ECU 400. The drive control circuit 300 corresponds to a subdomain of control in the mobile object 200. The drive control circuit 300 also controls various drives in the mobile object 200 in accordance with outputs from the monitoring device 100, which will be described later.

[0019] The upper ECU 400 controls the entire mobile object 200. In other words, the upper ECU 400 corresponds to the highest control unit that controls the control targets of the sub-domains, such as the drive control circuit 300. The upper ECU 400 also controls the systems of the various sub-domains in response to outputs from the monitoring device 100, which will be described later.

[0020] The monitoring device 100 includes an electronic control unit (hereinafter referred to as ECU) 101, a LiDAR control circuit (LiDAR Control) 103, an EVS image signal processing circuit (EVS ISP (Image Signal Processor)) 105, a CIS image signal processing circuit (CIS ISP) 107, and a communication interface (Communication Interface) 109. The ECU 101, the LiDAR Control 103, the EVS ISP 105, the CIS ISP 107, and the Communication Interface 109 are electrically connected via a bus. The ECU 101 processes various types of information and may therefore be referred to as an information processing device.

[0021] The LiDAR control circuit 103 is a circuit that controls the LiDAR, which is the distance measuring device 20. The LiDAR control circuit 103 controls the LiDAR sensor and the laser generator / actuator in accordance with control from the ECU 101. For example, the LiDAR control circuit 103 controls the laser generator / actuator to adjust the frequency at which a laser is generated per unit time and the range to which the laser is irradiated (hereinafter referred to as the LiDAR field of view) in accordance with the output from the ECU 101. At this time, the laser generator / actuator adjusts the frequency at which the laser is generated and the LiDAR field of view under the control of the LiDAR control circuit 103. These adjustments will be described later.

[0022] The LiDAR control circuit 103 also receives output from the LiDAR sensor in accordance with the timing of laser generation. The LiDAR control circuit 103 measures the distances to multiple objects in the LiDAR field of view in response to the signal received from the LiDAR sensor. As a result, the LiDAR control circuit 103 acquires position information of the multiple objects in the LiDAR field of view. The LiDAR control circuit 103 outputs the position information of the multiple objects to the ECU 101. Note that the LiDAR Control 103 may be mounted in the distance measuring device 20. As described above, the multiple objects are different from the object (moving body 200) on which the monitoring device 100 is mounted.

[0023] The EVS ISP 105 processes the image signal output from the image sensor 19A. Specifically, the EVS ISP 105 acquires changes in luminance in the field of view of the image sensor 19A (hereinafter referred to as the EVS field of view). That is, the EVS ISP 105 detects changes in luminance of each pixel in the image signal output from the image sensor 19A, and combines only the changed data with "coordinate" and "time information" to generate information indicating the movement of an object in the EVS field of view (hereinafter referred to as motion image information) as image data (hereinafter referred to as the EVS image). That is, in the EVS image, as described above, the greater the movement of an object, the larger the pixel value indicating the difference in luminance value. Based on this, the EVS ISP 105 generates the motion image information. The EVS ISP 105 may be mounted on the image sensor 19A.

[0024] The CIS ISP 107 generates, for example, an RGB image based on the signal output from the image sensor 19 B. The CIS ISP 107 may be mounted on the image sensor 19 B.

[0025] In the present embodiment, a functional configuration using the image sensor (EVS) 19A will be described below. FIG. 3 is a functional block diagram showing an example of the functional configuration in this embodiment. As shown in FIG. 3, the ECU 101 has an image acquisition unit 11, a tracking unit 13, a position information acquisition unit 15, a determination unit 17, and a sensor control unit 18. Functions performed by the image acquisition unit 11, the tracking unit 13, the position information acquisition unit 15, the determination unit 17, and the sensor control unit 18 are executed by a processing circuit in the ECU 101. Furthermore, functions performed by the vehicle control unit 350 in FIG. 3 are realized, for example, by a processing circuit mounted in the drive control circuit 300.

[0026] The image acquisition unit 11 acquires an image of the periphery of the moving object 200 from the image sensor 19A. In this embodiment, the image sensor 19A is, for example, an EVS. Specifically, the image acquisition unit 11 acquires motion image information as an EVS image via the EVS ISP 105. The image acquisition unit 11 may store the acquired motion image information (image data, EVS image) in a memory in the ECU 101.

[0027] The tracking unit 13 tracks the movements of multiple objects captured in image data serving as motion image information. That is, the tracking unit 13 tracks the movements of multiple objects captured in image data (EVS images) based on the image data acquired from the image sensor 19A. For example, the tracking unit 13 tracks the movements of the multiple objects by tracing the regions of the multiple objects in two adjacent EVS images in a time series using position information. The tracking unit 13 stores the tracking results of the multiple objects (hereinafter referred to as tracking information) in the memory of the ECU 101. The tracking information is, for example, information on the movement directions and position information of the multiple objects generated by tracking the multiple objects around the vehicle corresponding to the moving object 200. Note that the tracking information may also include information on the relative positional relationship between the vehicle corresponding to the moving object 200 and each of the multiple objects.

[0028] The position information acquisition unit 15 acquires position information of multiple objects around the vehicle from the distance measuring device 20 via the LiDAR control circuit 103. As described above, the distance measuring device 20 is, for example, a LiDAR. The position information acquired by the position information acquisition unit 15 includes, for example, the relative distance between the vehicle corresponding to the moving object 200 and other objects excluding the moving object 200, and information on the movement direction of the other objects. The frequency at which the position information acquisition unit 15 acquires the position information from the distance measuring device 20 via the LiDAR control circuit 103 is variable. In other words, the frequency at which the position information acquisition unit 15 acquires the position information can be changed as appropriate depending on the control of the LiDAR control circuit 103 by the sensor control unit 18. The position information acquisition unit 15 stores the acquired position information in a memory in the ECU 101.

[0029] The determination unit 17 determines the possibility of a collision between the multiple objects based on the tracking information obtained by tracking the multiple objects around the vehicle corresponding to the moving body 200 and the position information. For example, the determination unit 17 calculates an index indicating the possibility of a collision between the multiple objects (hereinafter referred to as a collision index) using a predetermined calculation formula (hereinafter referred to as a collision calculation formula) that uses the movement directions of the multiple objects in the tracking information and the positions of the multiple objects (or the relative positional relationships between the multiple objects) in the position information. The collision index corresponds to, for example, a numerical value that indicates the probability of a collision between the multiple objects.

[0030] In addition, instead of using a predetermined calculation formula, the judgment unit 17 may determine the collision index by comparing the movement directions of the multiple objects and the positions of the multiple objects with a correspondence table (hereinafter referred to as the collision correspondence table) that shows collision indexes for the movement directions of the multiple objects in the tracking information and the positions of the multiple objects (or the relative positional relationships of the multiple objects) in the position information.

[0031] Alternatively, the determination unit 17 may calculate a velocity vector of each of the plurality of objects from the tracking information and the position information, and determine the collision index by taking the calculated velocity vector into consideration. The collision calculation formula or the collision correspondence table is set in advance and stored in the memory of the ECU 101. The calculation (determination) of the collision index is not limited to the above method. For example, the collision index may be determined by scoring the time change of the relative positions (coordinates, number of pixels between the objects) of the plurality of objects in the image data in the position information and tracking information, and the position information of each object and the time change of the position information (velocity, acceleration, vector), etc., acquired in a first mode that acquires position information at a predetermined resolution with reduced power consumption compared to a second mode described below.

[0032] The determination unit 17 compares the collision index with a threshold value (hereinafter referred to as the collision prediction threshold value) for determining the possibility of a collision between multiple objects. The collision prediction threshold value is set in advance and stored in a memory in the ECU 101. The collision prediction threshold value may also be referred to as a first threshold value. If the collision index is equal to or greater than the collision prediction threshold value, the determination unit 17 determines that the possibility of a collision between multiple objects is high. In other words, if the collision index is equal to or greater than the collision prediction threshold value, the determination unit 17 determines that a collision between multiple objects is expected.

[0033] Furthermore, the determination unit 17 compares the collision index with a threshold value greater than the collision threshold value. The threshold value greater than the collision threshold value is a threshold value (hereinafter referred to as a collateral damage prediction threshold value) for determining the possibility of collateral damage due to a collision between objects other than the vehicle (mobile body 200). The collateral damage prediction threshold value is set in advance and stored in the memory of the ECU 101. The collateral damage prediction threshold value may also be referred to as a second threshold value. The second threshold value is a value greater than the first threshold value (second threshold value > first threshold value).

[0034] When the determination unit 17 determines that there is a possibility of a collision, the determination unit 17 outputs a predetermined warning. For example, when it is determined that the collision index is equal to or greater than the second threshold, the determination unit 17 outputs a predetermined warning from a speaker or a display device connected to the monitoring device 100. The predetermined warning is, for example, a voice, a warning sound, a light, a vibration of the steering wheel, a warning display, or the like that can be recognized by a user (driver) riding in the mobile object 200.

[0035] Furthermore, the determination unit 17, upon output of the predetermined warning, outputs a signal regarding an action to avoid being caught up in a collision between multiple objects. For example, when it is determined that the collision index is equal to or greater than the second threshold, the determination unit 17 controls the vehicle control unit 350 to provide information for avoiding the collateral damage (hereinafter referred to as avoidance information). The avoidance information includes position information and / or tracking information regarding multiple objects determined to be at risk of collision.

[0036] The avoidance information may include an avoidance direction and / or an avoidance position, etc. In this case, the determination unit 17 determines the avoidance direction and / or the avoidance position of the moving body 200 based on the position information and / or tracking information regarding the plurality of objects determined to have a possibility of collision. The avoidance direction and / or the avoidance position can be determined as appropriate using a predetermined calculation formula that inputs the position information and / or tracking information regarding the plurality of objects determined to have a possibility of collision.

[0037] The vehicle control unit 350 corresponds to the drive control circuit 300 and / or the higher-level ECU 400, and may be referred to as a control device. At this time, the vehicle control unit 350 controls the steering wheel, brake, accelerator, shift, etc. 205 to avoid being caught in the collision.

[0038] The sensor control unit 18 controls the frequency with which the position information acquisition unit 15 acquires position information based on the collision possibility information (determination result) output by the determination unit 17. For example, when the collision index indicating the collision possibility is equal to or greater than a first threshold, the sensor control unit 18 controls the distance measuring device 20 and the position information acquisition unit 15 to change the frequency of acquiring position information to a mode (second mode) in which the frequency is higher than normal (first mode). The first mode is a monitoring mode in which position information is acquired at a predetermined resolution. The second mode is a monitoring mode in which position information is acquired at a resolution higher than the predetermined resolution.

[0039] For example, the sensor control unit 18 controls switching between a first mode in which position information is acquired at a predetermined resolution and a second mode in which position information is acquired at a resolution higher than the predetermined resolution, based on the result of the collision possibility determination. For example, the sensor control unit 18 controls the frequency of acquiring position information from the distance measuring device 20 when switching between the first mode and the second mode. Furthermore, the sensor control unit 18 controls the resolution of the position information of multiple objects acquired by the distance measuring device 20 when switching between the first mode and the second mode.

[0040] For example, the second mode may be a mode in which distance measurement is performed more frequently than in the first mode, or a mode in which, in addition to the frequent distance measurement, position information is acquired at a higher resolution than in the first mode in a part of the range measured by the distance measuring device 20 (a region of interest). In this case, the sensor control unit 18 sets an area of ​​a predetermined size that includes multiple objects within the detection range (field of view) of the distance measuring device 20 as the region of interest, and controls the distance measuring device 20 and the position information acquisition unit 15 in the second mode so that the resolution of the position information acquired in the region of interest is higher than the resolution of the other regions of the detection range excluding the region of interest. In addition, the sensor control unit 18 controls the distance measuring device 20 and the position information acquisition unit 15 in the second mode so that the frequency of acquiring position information in the region of interest is higher than the frequency of acquiring position information in the other regions of the detection range excluding the region of interest.

[0041] Furthermore, when the collision index indicating the possibility of a collision becomes less than the first threshold, the sensor control unit 18 changes from the second mode to the first mode. Note that the sensor control unit 18 may control the frequency with which the image acquisition unit 11 acquires EVS images.

[0042] Some or all of the above-described units may be realized by software, for example, by causing an information processing device such as the ECU 101 to execute a program. Also, some or all of the above-described units may be realized by hardware such as an integrated circuit (IC), or may be realized by a combination of software and hardware.

[0043] The overall configuration and functions of the monitoring system 1 according to the first embodiment have been described above. With this configuration, the monitoring system 1 according to the first embodiment executes a process (hereinafter referred to as a mode change process) for changing the mode of acquiring position information by the distance measuring device 20 in accordance with the possibility of a collision between the other objects, during a monitoring process for monitoring a collision between the other objects other than the vehicle (mobile body 200). The procedure for the mode change process will be described below.

[0044] 4 is a flowchart showing an example of the procedure for the mode change process in the monitoring process. The procedure for the monitoring process including the mode change process will be described with reference to FIGS.

[0045] (Monitoring Process) (Step S1) When the engine of the mobile body 200 is started, the monitoring process is started in the first mode. Note that the start of the monitoring process in the first mode is not limited to the start of the engine of the mobile body 200, and may be, for example, when a user gets into the mobile body 200 or when a door of the mobile body 200 is unlocked. Furthermore, when a continuous monitoring process such as parking monitoring is performed on the mobile body 200, if the battery capacity of the mobile body 200 is equal to or greater than a predetermined value, the monitoring process in the first mode is started in response to the supply of power to the mobile body 200.

[0046] For example, in response to the start of the monitoring process, the LiDAR control circuit 103 operates the ranging device 20 in the first mode. As a result, the position information acquisition unit 15 acquires position information from the ranging device 20. In addition, the ECU 101 performs imaging using the image sensor 19A. As a result, the image acquisition unit 11 acquires an EVS image. At this time, the tracking unit 13 tracks multiple objects using the EVS image and the position information of the multiple objects to generate tracking information. The frequency of acquiring position information in the first mode is, for example, 10 fps (frames per second). At this time, the tracking information in the first mode is updated at 10 fps.

[0047] 5 is a diagram showing an example of the detection range DR of the distance measuring device 20 in the first mode, the detection range DR of the image sensor (CIS) 19A, and two objects OB1 and OB2 within the detection range DR. As shown in Fig. 5, the detection range of the distance measuring device 20 in the first mode and the detection range of the image sensor (CIS) 19A are set as substantially the same detection range DR. The arrows in Fig. 5 indicate the movement directions of the two objects OB1 and OB2.

[0048] (Step S2) The determination unit 17 determines (calculates) a collision index based on the tracking information and the position information. For example, the determination unit 17 determines the collision index by scoring the position information of multiple objects in the detection range and the time changes in the position information (velocity, acceleration, vector), etc.

[0049] (Step S3) The determination unit 17 compares the collision index with the first threshold. If the collision index is equal to or greater than the first threshold (Yes in step S3), the process proceeds to step S4. If the collision index is not equal to or greater than the first threshold (No in step S3), the process repeats step S1.

[0050] 6 is a diagram showing an example of the positional relationship between two objects OB1 and OB2 when it is determined that the collision index is equal to or greater than the first threshold. As shown in Fig. 6, the two objects OB1 and OB2 are closer to each other than in Fig. 5. At this time, the process of step S4 is executed.

[0051] (Step S4) The sensor control unit 18 switches the distance measurement mode of the distance measuring device 20 from the first mode to the second mode.

[0052] (Step S5) In the second mode, the position information acquisition unit 15 acquires position information of multiple objects from the distance measuring device 20. Furthermore, the image acquisition unit 11 acquires EVS images. Note that the frequency of acquisition of EVS images by the image acquisition unit 11 can be set arbitrarily. For example, the frequency of acquisition of EVS images may be the same as that described in step S1, or may be changed to the same frequency as the acquisition frequency of position information in the second mode. Next, the tracking unit 13 tracks the multiple objects using the EVS images and the position information of the multiple objects, and generates tracking information.

[0053] FIG. 7 is a diagram illustrating an example of acquiring position information in the second mode. As shown in FIG. 7 , for example, the frequency of acquiring position information in the second mode is higher than that in the first mode. Specifically, if the frequency of acquiring position information in the first mode is 10 fps, the frequency of acquiring position information in the second mode is 30 fps. As a result, the temporal resolution of the position information is higher in the second mode than in the first mode. Also, as shown in FIG. 7 , in the second mode, a region including multiple objects may be set as the region of interest (ROI). In this case, the number of laser scanning lines for acquiring position information in the region of interest shown in FIG. 7 is set to be greater than that of other regions of the detection range DR excluding the region of interest. As a result, the spatial resolution of the position information in the region of interest is higher in the second mode than in the first mode.

[0054] The number of lasers in the region of interest can be changed, for example, when the distance measuring device 20 is a specific LiDAR such as a MEMS (Micro Electro Mechanical Systems) type. In the second mode, at least one of the temporal resolution and the spatial resolution can be increased as described above compared to the first mode.

[0055] (Step S6) The determination unit 17 determines (calculates) a collision index based on the tracking information and the position information. The processing content in this step is the same as in step S2, so a description thereof will be omitted.

[0056] (Step S7) The determination unit 17 compares the collision index with the second threshold. If the collision index is equal to or greater than the second threshold (Yes in step S7), the process proceeds to step S8. If the collision index is not equal to or greater than the second threshold (No in step S7), the process proceeds to step S9.

[0057] 8 is a diagram illustrating an example of a case where, for example, the distances from the vehicle (moving body 200) to two objects OB1 and OB2 are approximately the same during monitoring in the second mode. In this case, there is a possibility that the two objects OB1 and OB2 may collide. In the case illustrated in FIG. 8, if it is determined that the collision index is equal to or greater than the second threshold, the process of step S8 is executed.

[0058] 9 is a diagram showing an example of a case where, for example, the distances from the vehicle (moving body 200) to two objects OB1 and OB2 are approximately the same during monitoring in the second mode. In this case, there is a possibility that the two objects OB1 and OB2 may collide. In the case shown in FIG. 8 , if it is determined that the collision index is equal to or greater than the second threshold, the process of step S8 is executed.

[0059] (Step S8) The determination unit 17 outputs a predetermined warning from a speaker or a display device (such as a display or monitor) mounted on the monitoring device 100 or the moving body 200. The warning is output to make the user aware of the possibility of a collision between the two objects OB1 and OB2. Note that the predetermined warning may be output by the higher-level ECU 400.

[0060] Furthermore, the determination unit 17 may generate avoidance information in response to the output of a predetermined warning, and output the generated avoidance information to the vehicle control unit 350. At this time, the vehicle control unit 350 controls the steering wheel, brakes, etc. 205 of the vehicle (mobile body 200) to take avoidance action such as stopping the mobile body 200 or taking a detour. Furthermore, the upper ECU 400 may control the drive recorder in response to the output of a predetermined warning, and record video using the image sensor 19B.

[0061] (Step S9) The determination unit 17 compares the collision index with the first threshold. If the collision index is less than the first threshold (Yes in step S9), the process proceeds to step S10. If the collision index is not less than the first threshold (No in step S9), the process from step S5 onwards is repeated.

[0062] (Step S10) If the monitoring process has not ended (No in step S10), the process of step S11 is executed. If the monitoring process has ended (Yes in step S10), the flow of this monitoring process ends. The end of the monitoring process is determined, for example, by the engine of the mobile object 200 being stopped, the user getting off the mobile object 200, or the boarding door of the mobile object 200 being locked. Furthermore, if continuous monitoring process such as parking monitoring is performed on the mobile object 200, the end of the monitoring process is determined, for example, when the battery capacity of the mobile object 200 falls below a predetermined value.

[0063] (Step S11) The sensor control unit 18 switches the distance measurement mode of the distance measuring device 20 from the second mode to the first mode, so that the monitoring device 100 monitors a plurality of objects using the distance measuring device 20 in the first mode.

[0064] 9 is a diagram showing an example of monitoring in the second mode, in which one object OB1 is located near the moving body 200 and the other object OB2 is located far from the moving body 200. In this case, the one object OB1 and the other object OB2 are distant from each other. In the case shown in FIG. 9, the possibility of a collision between the one object OB1 and the other object OB2 is low. If the collision index is less than the first threshold (Yes in step S9) and the monitoring process has not ended, the ranging mode of the ranging device 20 is switched from the second mode to the first mode. Then, the transition process to step S1 is repeated.

[0065] FIG. 10 is a diagram illustrating an example of application of this embodiment. As an example, FIG. 10 illustrates a situation in which a moving object 200 is moving in a single driving lane DL of a two-lane road. As shown in FIG. 10, the moving direction of the moving object 200 is D. As shown in FIG. 10, in the driving lane (left lane) DL in which the moving object 200 is traveling, a leading vehicle FV is moving in the left lane DL in the same direction FVD as the moving object 200, ahead of the moving object 200. Furthermore, an oncoming vehicle OV is traveling in an oncoming lane (right lane) OL opposite the driving lane DL across the center line CL. FIG. 10 illustrates that the oncoming vehicle OV has crossed the center line CL and is heading toward the leading vehicle FV in the direction of the arrow OVD.

[0066] That is, the situation shown in Fig. 10 indicates a situation in which the oncoming vehicle OV deviates from the center line CL, potentially resulting in a head-on collision. As shown in Fig. 10 , in the monitoring process, the possibility of a collision between the forward vehicle FV and the oncoming vehicle OV is determined by, for example, scoring (calculating a collision index) the time changes in the relative positions (coordinates, number of pixels between the objects) of multiple objects in the image data, and the position information of each object acquired in the first mode and its time changes (velocity, acceleration, vector), etc., and determining whether the collision index is equal to or greater than a first threshold. In the situation shown in Fig. 10 , if the collision index is determined to be equal to or greater than a second threshold, the determination unit 17 issues a warning to the moving body 200 to take evasive action so as to prevent the moving body 200 from becoming collateral damage in the collision between the forward vehicle FV and the oncoming vehicle OV.

[0067] 10 has been described regarding the possibility of a collision between a leading vehicle FV traveling in front of the moving body 200 and an oncoming vehicle OV traveling in the opposite lane on a road with one lane in each direction, but this is not limited to this. For example, when the moving body 200 is traveling on a road with two or more lanes in each direction, such as a highway or a bypass, the monitoring process works more effectively on a vehicle traveling in the opposite direction to the moving direction D of the moving body 200 (hereinafter referred to as a wrong-way vehicle) in the same manner as in FIG. 10 . Furthermore, this monitoring process is not limited to monitoring the wrong-way vehicle as a monitoring target. For example, this monitoring process can also monitor for a collision between a moving body (such as a person, a light vehicle such as a bicycle, or a vehicle) traveling on a sidewalk located to the side of the moving body 200 and the leading vehicle FV.

[0068] As described above, the monitoring device 100 according to the first embodiment includes the tracking unit 13 that tracks multiple objects captured in an EVS image acquired from the image sensor 19B based on the EVS image; the position information acquisition unit 15 that acquires position information of the multiple objects in the EVS image from the distance measuring device 20; the determination unit 17 that determines a possibility of a collision between the multiple objects based on the tracking information and position information of the multiple objects; and the sensor control unit 18 that controls switching between a first mode in which position information is acquired at a predetermined resolution and a second mode in which position information is acquired at a resolution higher than the predetermined resolution based on the determination result of the possibility of a collision. For example, in the monitoring device 100 according to the first embodiment, the sensor control unit 18 controls the frequency of acquiring position information from the distance measuring device 20 when switching between the modes. Furthermore, in the monitoring device 100 according to the first embodiment, the sensor control unit 18 may control the resolution of the position information of the multiple objects acquired by the distance measuring device 20 when switching between the modes.

[0069] Specifically, in the monitoring device 100 according to the first embodiment, an area of ​​a predetermined size that encompasses multiple objects within the detection range DR of the distance measuring device 20 is defined as the region of interest ROI, and in the second mode, the resolution of the position information acquired in the region of interest ROI is higher than the resolution of the other regions of the detection range DR excluding the region of interest ROI. In this case, in the second mode of the article processing of the monitoring device 100 according to the first embodiment, the frequency with which position information is acquired in the region of interest ROI is higher than the frequency with which position information is acquired in the other regions of the detection range DR excluding the region of interest ROI.

[0070] As a result, the monitoring device 100 of the first embodiment determines the possibility of a collision between multiple objects other than the vehicle (mobile body 200) and switches the mode of the distance measuring device 20 depending on the possibility of the collision, thereby reducing power consumption compared to constantly collecting position information at high resolution. That is, the monitoring device 100 of the first embodiment can collect position information of multiple objects with a high possibility of collision in a second mode with higher resolution than the first mode that is implemented when it is determined that a collision is highly likely.

[0071] Furthermore, the monitoring device 100 according to the first embodiment outputs a predetermined warning when the determination unit 17 determines that there is a possibility of a collision. Additionally, the monitoring device 100 according to the first embodiment outputs a signal related to an action to be taken to avoid being caught up in a collision between multiple objects, triggered by the output of the predetermined warning. As a result, the monitoring device 100 according to the first embodiment can collect position information of multiple objects with a high probability of collision in the high-resolution second mode when it is determined that there is a high probability of a collision. This reduces the power consumption of the ranging device 20 compared to constantly collecting position information at high resolution, and can efficiently determine whether or not evasive action is required to avoid a collision between multiple objects.

[0072] In the present embodiment, the image sensor 19A and the distance measuring device 20 are provided at a position in front of the moving body 200, but the present embodiment is not limited to this. For example, the image sensor 19A and the distance measuring device 20 may be provided on the side and / or rear of the moving body 200. When the image sensor 19A and the distance measuring device 200 are provided facing the side of the moving body 200, the monitoring process can monitor collisions between multiple objects on the side of the moving body 200. In this case, the monitoring process can monitor collisions between multiple objects, for example, when multiple objects join a path along the traveling direction of the moving body 200. Furthermore, when the image sensor 19A and the distance measuring device 200 are provided facing the rear of the moving body 200, the monitoring process can monitor collisions between multiple objects behind the moving body 200, for example, when the moving body 200 is reversing.

[0073] Second Embodiment In this embodiment, the monitoring process is performed further using image data (19B) acquired from an image sensor (CIS) 19B. That is, the image sensor in this embodiment includes an event-based vision sensor (EVS) and an optical camera (which may also be referred to as an RGB camera). Hereinafter, for ease of explanation, in this embodiment, the image sensor (EVS) 19A described in the first embodiment will be referred to as the first image sensor, and the image acquisition unit 11 described in the first embodiment will be referred to as the first image acquisition unit. Furthermore, in this embodiment, the image sensor (CIS) 19B described in the first embodiment will be referred to as the second image sensor.

[0074] The monitoring process in this embodiment is for determining the possibility of a collision between a stopped object (e.g., an oncoming vehicle, a vehicle ahead, a light vehicle such as a bicycle, a person, etc.) and an object approaching the stopped object (e.g., an oncoming vehicle, a vehicle ahead, a light vehicle such as a bicycle, a person, etc.) when the moving body 200 is stationary. Note that the monitoring process in this embodiment is not limited to the above example, and for example, on a road with two lanes on each side, when the relative speed between the moving body 200 and the vehicle ahead is zero, a collision between an oncoming vehicle traveling in the wrong direction and the vehicle ahead may be monitored.

[0075] FIG. 11 is a functional block diagram showing an example of a functional configuration according to this embodiment. In the functional configuration in FIG. 11 , functional configurations numbered the same as those in FIG. 3 will not be described unless necessary. As shown in FIG. 11 , the ECU 101 includes a first image acquisition unit 11, a tracking unit 13, a position information acquisition unit 15, a determination unit 17, a sensor control unit 18, a second image acquisition unit 21, and a recognition unit 23. The functions performed by the first image acquisition unit 11, the tracking unit 13, the position information acquisition unit 15, the determination unit 17, the sensor control unit 18, the second image acquisition unit 21, and the recognition unit 23 are executed by a processing circuit in the ECU 101.

[0076] The second image acquisition unit 21 acquires an image of the periphery of the moving object 200 from the second image sensor 19B. In this embodiment, the second image sensor 19B is, for example, a CIS (optical camera). Specifically, the second image acquisition unit 21 acquires an RGB image as image data via the CIS ISP 107. The second image acquisition unit 21 stores the acquired RGB image in a memory in the ECU 101.

[0077] The recognition unit 23 recognizes multiple objects in the RGB image based on image data (RGB image) generated by the optical camera 19B. For example, the recognition unit 23 reads out a trained model for image recognition stored in the memory of the monitoring device 100. The trained model is, for example, a semantic neural network. The semantic neural network may also be referred to as a semantic segmentation model. The semantic neural network is realized, for example, by a region-based convolutional neural network, and is a model devised for object detection applied to instance segmentation, etc. Note that the object recognition process in the RGB image is not limited to the semantic neural network, and various image recognition processes such as segmentation process may also be used.

[0078] The recognition unit 23 outputs the recognition results of the multiple objects in the RGB image to the determination unit 17. The recognition results of the multiple objects in the RGB image indicate, for example, image areas in the RGB image that are recognized as objects, regardless of the movement of the objects.

[0079] The determination unit 17 determines the likelihood of a collision between a moving object and a stationary object among the multiple objects based on tracking information obtained by tracking multiple objects around the vehicle corresponding to the moving object 200, position information, and image data generated by the optical camera 19B. In this embodiment, even for stationary objects that show little reaction (small changes in pixel values) in the EVS image, the determination unit 17 uses the recognition results of multiple objects in the RGB image, so that a collision index can be determined using a collision calculation formula or a collision correspondence table. That is, the determination unit 17 determines the likelihood of a collision between objects other than the host vehicle (moving object 200) not only for objects moving relative to the moving object 200, but also for stationary objects and objects having the same relative speed as the moving object 200. In other words, the determination unit 17 in this embodiment determines a collision index not only for objects moving relative to the moving object 200, but also for stationary objects and objects having the same relative speed as the moving object 200.

[0080] As a modification of this embodiment, instead of mounting the second image sensor 19B on the moving object 200, the determination unit 17 may store the generated tracking information in its own memory as past tracking information together with the generation time of the tracking information or the acquisition time of the EVS image. In this case, the determination unit 17 reads the past tracking information from its own memory and determines the possibility of a collision between a moving object and a stationary object among the multiple objects using the past tracking information and position information. In other words, the determination unit 17 may determine collision indicators for the multiple objects, regardless of whether they are stationary or not, based on the past tracking information and position information.

[0081] For example, when the host vehicle (mobile body 200) stops from a moving state, the determination unit 17 estimates the tracking of an object that was already stopped in the state before the host vehicle 200 stopped, using past tracking information. Specifically, the determination unit 17 updates the past tracking information to current tracking information by multiplying the velocity vector in the past tracking information by the time interval between EVS image acquisitions. The determination unit 17 determines a collision index based on the updated current tracking information and position information, as in the first embodiment.

[0082] The monitoring process in this embodiment differs from the first embodiment in the process of determining the collision indicator, as described above. The other process details in the terminal process are the same as those in the first embodiment, so a description thereof will be omitted.

[0083] FIG. 12 is a diagram showing an example of application of this embodiment when the host vehicle (mobile body 200) is stopped. As an example, FIG. 12 shows a situation in which the mobile body 200 traveling in the driving lane DL (left lane) of a two-lane road has stopped at a stop line just before an intersection. As shown in FIG. 12 , in the oncoming lane (right lane) OL opposite the driving lane DL across the center line CL, an oncoming vehicle OV is stopped at the stop line at the intersection. Also shown is a situation in which a truck TR enters the intersection from the left side of the intersection. FIG. 12 shows that the truck TR is proceeding along the arrow TRA. That is, in FIG. 12 , the truck TR is proceeding toward an oncoming vehicle OV stopped in the oncoming lane OL.

[0084] That is, the situation shown in Fig. 12 illustrates a situation in which a truck TR is approaching an oncoming vehicle OV stopped in the oncoming lane OL ahead of the moving body 200, potentially resulting in a collision. As shown in Fig. 12 , in the monitoring process of this embodiment, a collision index is determined using tracking information estimated based on past tracking information or tracking information generated using an RGB image, and position information. The determination unit 17 determines the possibility of a collision by comparing the collision index with a first threshold and a second threshold. In the situation shown in Fig. 12 , if it is determined that the collision index is equal to or greater than the second threshold, the determination unit 17 issues a warning to the moving body 200 to take evasive action so that the moving body 200 does not become involved in the collision between the forward vehicle FV and the truck TR.

[0085] As described above, in the monitoring device 100 according to the second embodiment, the image sensor includes an event-based vision sensor (EVS) 19A and an optical camera 19B, the image data is an EVS image, and the likelihood of a collision between a moving object and a stationary object among the plurality of objects is determined based on tracking information and position information of the plurality of objects and the image data generated by the optical camera 19B. Furthermore, the monitoring device 100 according to the second embodiment accumulates tracking information of the plurality of objects as past tracking information, and determines the likelihood of a collision between a moving object and a stationary object among the plurality of objects based on the past tracking information and position information.

[0086] As a result, the monitoring device 100 according to the second embodiment can determine the possibility of a collision between a plurality of objects other than the vehicle (moving body 200) even if the object is stationary, and switch the mode of the distance measuring device 20 depending on the possibility of the collision, thereby reducing power consumption compared to constantly collecting position information at high resolution. Other effects are the same as those of the first embodiment, and therefore will not be described here.

[0087] Third Embodiment In the first and second embodiments, the monitoring device 100 has been described as being mounted on a mobile object 200, but the mounting location of the monitoring device 100 is not limited to the mobile object 200. In this embodiment, the monitoring device 100 may be mounted (installed) on a street and / or a building, such as inside or outside a building. When the monitoring device 100 is installed on a street or outside a building, the monitoring target of the monitoring device 100 is, for example, multiple pedestrians, light vehicles such as bicycles, etc. When the monitoring device 100 is installed inside a building, the monitoring target of the monitoring device 100 is, for example, multiple pedestrians, etc. In the following, for the sake of concrete explanation, an example in which the monitoring device 100 is installed on a street will be described.

[0088] FIG. 13 is a diagram illustrating an example of a use of the monitoring device 100 installed at a street light SL. In FIG. 13 , pedestrian A is traveling in the same traveling direction DT. FIG. 13 also illustrates a situation in which pedestrian B approaches pedestrian A from behind and reaches out to grab the bag pedestrian A is carrying. At this time, if the collision indicator between pedestrian A and pedestrian B exceeds a first threshold, the monitoring device 100 acquires position information of pedestrians A and B in a second mode, switched from the first mode. Furthermore, if the collision indicator between pedestrians A and B exceeds a second threshold, i.e., if pedestrians A and B suddenly approach each other as shown in FIG. 13 , the monitoring device 100 outputs a warning to a server connected to the network, a security company, the police, or the like.

[0089] As described above, the monitoring device 100 according to this embodiment can be installed not only on the mobile object 200 but also on a structure such as a streetlight, and can determine, for example, the possibility of a collision between multiple people captured in the image. As a result, the monitoring device 100 according to this embodiment can detect, for example, the sudden approach of multiple pedestrians, as shown in Fig. 13, and detect and determine the possibility of a crime such as a snatch theft. Other effects are similar to those of the first and second embodiments, and therefore will not be described further.

[0090] When the technical idea of ​​the embodiment is realized in a monitoring method, the monitoring method tracks multiple objects captured in image data acquired from image sensor 19A based on the image data, acquires position information of the multiple objects from distance measuring device 20, determines a possibility of a collision between the multiple objects based on the tracking information of the multiple objects and the position information, and controls switching between a first mode in which position information is acquired at a predetermined resolution and a second mode in which position information is acquired at a resolution higher than the predetermined resolution based on the result of the collision possibility determination. The procedures and effects of the monitoring process executed by the monitoring method are similar to those of the first embodiment, and therefore description thereof will be omitted.

[0091] When the technical concept of this embodiment is realized by a surveillance program, the surveillance program causes a computer to track multiple objects captured in image data acquired from an image sensor 19A based on the image data, acquire position information of the multiple objects from a distance measuring device 20, determine the likelihood of a collision between the multiple objects based on the tracking information and the position information, and, based on the result of the collision likelihood determination, control switching between a first mode in which position information is acquired at a predetermined resolution and a second mode in which position information is acquired at a higher resolution than the predetermined resolution. For example, the surveillance process can be realized by installing the surveillance program from a non-volatile storage medium on various server devices (processing devices) and expanding the program in memory. In this case, a program capable of causing a computer to execute the method can also be stored and distributed on a storage medium such as a magnetic disk (e.g., a hard disk), an optical disk (e.g., a CD-ROM or DVD), or a semiconductor memory. The processing procedures and effects of the surveillance program are similar to those of the first embodiment, and therefore will not be described here.

[0092] The monitoring device 100 and the monitoring method having such a configuration can monitor collisions with reduced power consumption and improved resolution. As a result, one aspect of the monitoring device 100 and the monitoring method disclosed herein can switch between a first mode in which position information is acquired at a predetermined resolution with reduced power consumption and a second mode in which position information is acquired at a higher resolution than the predetermined resolution, i.e., at a higher resolution, based on the result of the collision possibility determination, for example.

[0093] Although the embodiments and modifications have been described above, the information processing device 101, information processing method, and information processing program disclosed herein are not limited to the above-described embodiments, and the components can be modified and embodied in each implementation stage without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments and modifications. For example, some components may be deleted from all of the components shown in the embodiments.

[0094] REFERENCE SIGNS LIST 1 Surveillance system 11 Image acquisition unit (first image acquisition unit) 13 Tracking unit 15 Position information acquisition unit 17 Determination unit 18 Sensor control unit 19A Image sensor (EVS: Event based Vision Sensor) 19B Image sensor (CMOS (Complementary Metal Oxide Semiconductor) image sensor) 20 Distance measuring device 21 Second image acquisition unit 23 Recognition unit 100 Surveillance device 101 Electronic control unit (Electronic Control Unit: ECU) 103 LiDAR (Light Detection And Ranging) control circuit (LiDAR Control) 105 EVS image signal processing circuit (EVS ISP (Image Signal Processor)) 107 CIS image signal processing circuit (CIS ISP) 109 Communication interface 200 Mobile object 205 Steering wheel, brake, accelerator, shift, etc. 300 Drive control circuit 350 Vehicle control unit 400 Upper level ECU

Claims

1. A monitoring device comprising: a tracking unit that tracks multiple objects captured in image data acquired from an image sensor based on the image data; a position information acquisition unit that acquires position information of the multiple objects from a distance measuring device; a judgment unit that judges the possibility of a collision between the multiple objects based on the tracking information of the multiple objects and the position information; and a sensor control unit that controls switching between a first mode that acquires the position information at a predetermined resolution and a second mode that acquires the position information at a resolution higher than the predetermined resolution based on the result of the judgment of the possibility of collision.

2. The monitoring device according to claim 1, wherein the sensor control unit controls the frequency of obtaining the position information from the distance measuring device during the switching.

3. The monitoring device according to claim 1, wherein the sensor control unit controls the resolution of the position information of a plurality of objects obtained by the distance measuring device during the switching.

4. A monitoring device as described in claim 1, wherein an area of ​​a predetermined size within the detection range of the distance measuring device that encompasses the plurality of objects is defined as an area of ​​interest, and in the second mode, the resolution of the position information obtained in the area of ​​interest is higher than the resolution of other areas within the detection range excluding the area of ​​interest.

5. The monitoring device according to claim 4, wherein in the second mode, the frequency with which the position information in the region of interest is acquired is higher than the frequency with which the position information in other regions of the detection range excluding the region of interest is acquired.

6. The monitoring device according to claim 1, wherein the determining unit outputs a predetermined warning when it determines that there is a possibility of a collision as a result of the determination.

7. The monitoring device according to claim 6, wherein the determination unit, upon output of the warning, outputs a signal regarding an action to be taken to avoid being caught in a collision between the plurality of objects.

8. The monitoring device of claim 1, wherein the image sensor has an event-based vision sensor (EVS) and an optical camera, the image data is an EVS image, and the determination unit determines the possibility of a collision between a moving object and a stationary object among the plurality of objects based on tracking information of the plurality of objects, the position information, and the image data generated by the optical camera.

9. The monitoring device of claim 1, wherein the judgment unit accumulates tracking information of the plurality of objects as past tracking information, and judges the possibility of a collision between moving objects and stationary objects among the plurality of objects based on the past tracking information and the position information.

10. A monitoring device according to any one of claims 1 to 9, wherein the plurality of objects are different from the object on which the monitoring device is mounted.

11. A monitoring method comprising: tracking a plurality of objects captured in image data acquired from an image sensor based on the image data; acquiring position information of the plurality of objects from a distance measuring device; determining a possibility of a collision between the plurality of objects based on the tracking information of the plurality of objects and the position information; and controlling switching between a first mode in which the position information is acquired at a predetermined resolution and a second mode in which the position information is acquired at a resolution higher than the predetermined resolution based on the result of the determination of the possibility of collision.

Images