Moving speed measuring device

Abstract

To suppress an increase in development cost and obtain practically sufficient accuracy.SOLUTION: A moving speed measuring device includes means of: recognizing and extracting an object from a two-dimensional image captured by a two-dimensional image sensor aligned with a distance image sensor; determining whether or not objects extracted from two-dimensional images captured at a first time point and a second time point are the same objects; identifying a point group of a region in a three-dimensional point group distance image corresponding to a region in the two-dimensional image of the object determined to be the same object and calculating the moving speed of the object on the basis of a distance to the identified point group and a time difference between the first time point and the second time point; enabling outputting the calculated moving speed.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a moving speed measuring device for measuring the moving speed of a moving object. [Background technology]

[0002] Conventionally, devices that detect moving objects (mobile bodies) and measure their direction of movement and speed of movement include those that use laser scanners (LiDAR) that emit and output laser light and capture multiple points where the laser light is reflected as measurement points to generate a three-dimensional point cloud range image, detect moving bodies from the three-dimensional point cloud range image generated by these laser scanners (LiDAR), and identify, calculate, and output their direction of movement and speed of movement (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-129446 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Document 1, extracting the moving object to be measured on a three-dimensional point cloud range image generated by a laser scanner (LiDAR) and identifying only the point cloud of that moving object does not currently provide sufficient extraction accuracy for practical use, even when extracting moving objects using deep learning.In addition, a huge number of three-dimensional point cloud range images corresponding to various objects must be generated as learning data to be used in this deep learning, and since creating (annotating) this learning data is very costly, there is a problem that the cost of developing the device becomes very high.

[0005] The present invention has been made in view of these problems, and has as its object to provide a moving speed measuring device that can obtain sufficient accuracy for practical use while suppressing increases in the cost of device development. [Means for solving the problem]

[0006] The moving speed measuring device of claim 1 The system has a distance image sensor (e.g., a laser scanner (LiDAR) 20) capable of generating a three-dimensional point cloud distance image including a plurality of point clouds with specified distances, and the system detects objects included in the three-dimensional point cloud distance images generated by the distance image sensor at different times. three dimensional A moving speed measuring device capable of calculating and outputting a moving speed, A two-dimensional image can be captured, and the position of the two-dimensional image can be aligned with a three-dimensional point cloud range image generated by the range image sensor. It is possible to capture two-dimensional images a two-dimensional image sensor (e.g., two-dimensional image sensor 21); Recognizing an object from a two-dimensional image captured by the two-dimensional image sensor The recognized object and the area of ​​the object An object recognition means to extract (for example, the part where the DNN executes the object detection process in Figure 8), an identical object determination means (for example, a part where a CPU executes a tracking process) that determines whether an object (for example, a vehicle) extracted by the object recognition means from two-dimensional images captured at a first time point (for example, an image capture timing one frame before) and a second time point (a new image capture timing) after the first time point is the same object; For an object determined to be the same object by the same object determination means, a point cloud included in a matching region on a three-dimensional point cloud range image that matches the region of the object extracted from the two-dimensional images captured at the first time point and the second time point by the object recognition means is determined to be the same object by the same object determination means. (e.g., point groups Q1, Q2) are extracted from the three-dimensional point cloud range images corresponding to the first time point and the second time point. And, extraction and determining a three-dimensional movement distance of the object based on the three-dimensional movement distance obtained from the point cloud at the first time point and the point cloud at the second time point and the time difference between the first time point and the second time point. three dimensional Calculate movement speed three dimensional A moving speed calculation means (for example, a part where a CPU executes the speed calculation process of FIG. 9 ); The aforementioned three dimensional Calculated by the moving speed calculation means three dimensionalan output means capable of outputting the travel speed (for example, a part that transmits speeding vehicle information including the travel speed calculated in the speed calculation process to the server computer 100); Equipped with The moving speed measuring device is fixedly installed at a position to the side of a path along which the object can move, the object recognition means is capable of simultaneously recognizing and extracting a plurality of objects, A plurality of objects extracted by the object recognition means Area Whether they overlap or not, the extracted objects are always three dimensional Calculate and output the movement speed, It is characterized by the following. This feature makes it possible to obtain sufficient accuracy for practical use while suppressing increases in the cost of device development.

[0007] The moving speed measurement device of claim 2 is the moving speed measurement device according to claim 1, an imaging timing of the two-dimensional image sensor and a generation timing of the three-dimensional point cloud range image by the range image sensor are synchronized; It is characterized by the following. According to this feature, the time lag between the two-dimensional image and the three-dimensional point cloud range image can be significantly reduced, thereby improving measurement accuracy.

[0009] Claim 3 The moving speed measurement device is a moving speed measurement device according to claim 1 or 2, The object is a vehicle traveling on a road that is the route, an upper limit speed setting means for setting an upper limit speed of the vehicle; The aforementioned three dimensional Calculated by the moving speed calculation means three dimensional a determination means for determining whether the moving speed exceeds the upper limit speed; Equipped with The output means is capable of outputting a two-dimensional image of the vehicle determined by the determination means to be exceeding the upper speed limit and time information when the two-dimensional image is captured by the two-dimensional image sensor. It is characterized by the following. According to this feature, it is possible to identify the vehicle that is exceeding the upper speed limit and the time information on when the upper speed limit was exceeded.

[0010] Claim 4 The moving speed measurement device is a moving speed measurement device according to claim 1 or 2, The aforementioned three dimensional The moving speed calculation means Match Identify all point clouds included in the area and use the distance data of all the identified point clouds three dimensional Calculate the movement speed, It is characterized by the following. According to this feature, by using a two-dimensional image and using a point cloud included in a corresponding area corresponding to an area extracted with high accuracy, it is possible to prevent unnecessary points from being included in these point clouds, and all point clouds that have been prevented from including unnecessary points can be used. three dimensional Since the moving speed is calculated, the accuracy of the calculated speed can be improved.

[0012] Furthermore, the present invention may have only the invention-specifying matters set forth in the claims of the present invention, or may have the invention-specifying matters set forth in the claims of the present invention as well as configurations other than the invention-specifying matters. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of an installation state of a moving speed measurement device in an embodiment. [Figure 2] 1A is a top view of the moving speed measurement device in the embodiment, and FIG. 1B is a cross-sectional view of the moving speed measurement device in the embodiment taken along the line AA. [Figure 3] 1 is a block diagram showing a hardware configuration of a moving speed measuring device according to an embodiment; [Figure 4] 3 is a diagram showing an example of the structure of data stored in a RAM 32 in the embodiment. FIG. [Figure 5] FIG. 10 is a diagram illustrating an example of the configuration of a tracking list in the embodiment. [Figure 6]FIG. 2 is a functional block diagram of the moving speed measurement device according to the embodiment. [Figure 7] FIG. 10 is a process flow diagram showing the flow of processing in speed measurement by the moving speed measurement device. [Figure 8] FIG. 10 is a process flow diagram showing the flow of processing in object detection processing. [Figure 9] FIG. 10 is a process flow diagram showing the flow of a velocity calculation process. [Figure 10] FIG. 10 is a schematic diagram showing the configuration of object information transmitted to a server computer. [Figure 11] 10 is an example of output of object information in a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A description will now be given of a moving speed measuring device according to the present invention based on examples. [Example]

[0015] Fig. 1 is a diagram showing the installation of a mobile speed measurement device 1 of this embodiment used for enforcing the speed limits on vehicles, which are moving bodies, and Fig. 2 is a top view and an AA cross-sectional view of the mobile speed measurement device 1 of this embodiment. Note that in this embodiment, an example of the use of the mobile speed measurement device of the present invention is shown, in which it is used for enforcing the speed limits on vehicles, which are moving bodies, but as will be described later, the use of the mobile speed measurement device is not limited to enforcing the speed limits on these vehicles.

[0016] As shown in Figures 1 and 2, the moving speed measuring device 1 of this embodiment is a rectangular box-shaped device, and is attached to a cylindrical support P erected on one side of the target location on the road R where speed enforcement is conducted, at an appropriate height position, for example, several meters above the ground, using a mounting bracket 3, so that a vehicle C passing through the target location, which is the specific space to be imaged, can be imaged and scanned while minimizing the presence of obstacles such as other vehicles C different from the vehicle C in question.

[0017] More specifically, the installation height of these moving speed measuring devices 1 is preferably set at a height that allows for good imaging and scanning of the distant vehicle C, including its license plate, even when there are successive vehicles C passing by with a narrow distance between them. Taking this into consideration, if the installation height is too low, for example, when vehicles C pass by in succession with a narrow distance between them, the license plate of the distant vehicle C will be hidden by the nearby vehicle C and will not be able to be imaged, making the tracking process described below impossible and preventing speed calculation.

[0018] Furthermore, in this embodiment, the moving speed measurement device 1 is installed in a position as shown in FIG. 1 so that vehicles C passing through a road R with one lane on each side can be imaged and scanned with almost no overlapping of individual vehicles C, but the present invention is not limited to this. For example, if only one lane of a road R with one lane on each side is to be imaged, the moving speed measurement device 1 can be installed in a position close to the lane and at a low height from the ground, or, if there is a facility that crosses over road R, such as a footbridge, the moving speed measurement device 1 can be installed in a position directly above the lane on the footbridge or the like.

[0019] As shown in Fig. 2, the moving speed measurement device 1 of this embodiment is configured such that a laser scanner (LiDAR) 20, which is a distance image sensor, a two-dimensional image sensor 21, and a processing device 22 connected to the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 and performing speed measurement processing as well as controlling the moving speed measurement device 1 are stored inside a rectangular, bottomed box-like housing 2 whose top surface serves as an openable / closable lid 2a. Note that 25 in Figs. 1 and 2 is a mobile communication antenna for wireless data communication with the outside, and is connected to a network interface (I / F) 37 of the processing device 22 via a connection cable (see Fig. 3).

[0020] The housing 2 is made of a lightweight, mechanically strong aluminum alloy and is composed of an open-close lid 2a that covers the top surface and a rectangular box-shaped storage section 2b with a bottom and an open top. The open-close lid 2a is fixed to the storage section 2b with a predetermined fixing jig (not shown). The aluminum alloy is just one example, and the material is not limited to this as long as it is lightweight and has high mechanical strength.

[0021] A window W is formed on one side of the storage section 2b, and is fitted with tempered glass 6 that has been strengthened to prevent breakage or scratches due to collisions with flying stones or the like from automobiles, and in the window W, the tempered glass 6 is firmly fixed to the storage section 2b so as not to move, so that a large change in the angle or distance between the tempered glass 6 and the laser scanner (LiDAR) 20 or the two-dimensional image sensor 21 does not adversely affect the scanning or imaging of the laser scanner (LiDAR) 20 or the two-dimensional image sensor 21. Note that, although this embodiment illustrates a form in which tempered glass 6 is used in the window W, the present invention is not limited to this, and if the travel speed measurement device 1 can be installed in a location where there is almost no risk of collisions with flying stones or the like, the tempered glass 6 may not be provided.

[0022] As shown in Figure 2(A), a rectangular base plate 4 made of a relatively thick metal plate and slightly smaller than the bottom surface of the storage section 2b is fixed at its four corners by four fixing bolts B1.

[0023] On this base plate 4, a processing device 22 is fixedly disposed on the rear side opposite the window W, and on the front side facing the window W, a laser scanner (LiDAR) 20 and a two-dimensional image sensor 21 are fixedly disposed side by side so that the distance between the sensor position of the laser scanner (LiDAR) 20 and the sensor position of the two-dimensional image sensor 21 and the surface of the tempered glass 6 is approximately the same, and the distance between the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 is minimized. By arranging them in this way, it is possible to reduce the difference in distance between each point in the imaging area and the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21, thereby preventing a decrease in accuracy due to these distance differences.

[0024] 2, the laser scanner (LiDAR) 20 is modularized into a relatively large cylindrical shape, and this module is fixed directly to the base plate 4 with fixing bolts B3, whereas the two-dimensional image sensor 21 is modularized into a relatively small rectangular box shape and is fixed to the base plate 4 via a clog-shaped height-adjustment pedestal 5. In other words, if the relatively small two-dimensional image sensor 21 were fixed directly to the base plate 4, the height of the sensor position of the laser scanner (LiDAR) 20 and the height of the sensor position of the two-dimensional image sensor 21 would differ greatly, which could have an adverse effect on the measurement accuracy. Therefore, just as the above-mentioned distance in the front-to-rear direction is made uniform between the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21, the height-adjustment pedestal 5 is used to fix the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 so that their height positions are also aligned at the same position. The height-adjusting pedestal 5 is made of metal and is fixed to the base plate 4 by a bolt B2, and the two-dimensional image sensor 21 is fixed to the top surface of the height-adjusting pedestal 5 by a bolt B4.

[0025] The laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 are connected to the processing device 22 via a connection cable. A battery box (not shown) is fixedly disposed on the underside of the housing 2, and power is supplied from a battery stored in the battery box to the processing device 22 via a power cable (not shown), and operating power is supplied from the processing device 22 to the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 via the connection cable. As described above, the movement speed measurement device 1 of this embodiment can be operated by replacing the battery in the battery box without opening the housing 2, but the present invention is not limited to this. For example, a rechargeable, small, high-output lithium ion battery may be disposed inside the housing 2 so that the lithium ion battery can be charged externally. Needless to say, when the movement speed measurement device 1 is to be operated continuously for a long period of time, power may be supplied by directly connecting to an external power source without providing a battery.

[0026] In this embodiment, the configuration in which the opening / closing lid portion 2a is provided on the top surface of the housing 2 not only facilitates access to the interior of the storage portion 2b and the storage of the base plate 4 on which the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 are mounted, but also prevents the mechanical strength of the side surfaces of the housing 2 from being high, which is preferable because it prevents the side surfaces from being deformed by a collision with an obstacle or the like, causing a shift in the relative positional relationship between the six tempered glass surfaces arranged in the window portion W and the laser scanner (LiDAR) 20 or the two-dimensional image sensor 21 and adversely affecting measurement accuracy. However, the present invention is not limited to this, and the opening / closing portions may be provided on the side surfaces of the housing 2. However, if the side surface on which the window portion W is formed is used as an opening / closing door, there is a possibility that the relative positional relationship between the six tempered glass surfaces and the laser scanner (LiDAR) 20 or the two-dimensional image sensor 21 will be shifted due to opening and closing, thereby reducing measurement accuracy. Therefore, it is preferable to use the side surface on which the window portion W is not formed as an opening / closing door.

[0027] Next, the hardware configuration of the moving speed measuring device 1 of this embodiment, particularly the configuration of the processing device 22, will be described with reference to FIG.

[0028] As shown in FIG. 3, the processing device 22 includes a central processing unit (CPU) 31 that performs various processes such as control processing of the moving speed measurement device 1 by calculation, an image processing unit (GPU) 38 that mainly performs calculation processing related to a deep learning neural network (DNN), a random access memory (RAM) 32 that is used as a work memory or the like in processing by the central processing unit (CPU) 31 and the image processing unit (GPU) 38, and various parameters related to processing programs and initial settings of various processes executed by the central processing unit (CPU) 31 and the image processing unit (GPU) 38. The computer 200 is a relatively small computer connected to a flash memory (F-MEM) 33 storing the above information, a real-time clock (RTC) 36 that outputs time information and generates system time, a laser scanner interface (I / F) 34 that inputs and outputs data to and from the laser scanner (LiDAR) 20, a camera interface (I / F) 35 that inputs and outputs data to and from the two-dimensional image sensor 21, and a network interface (I / F) 37 for mobile data communication with an external server computer 100 or a maintenance personal computer (PC) 200. In this embodiment, as will be described later, information such as the speed of a speeding vehicle C (speeding excess information) is transmitted to the server computer 100, so a display interface (I / F) for generating a display screen is not provided. However, for example, as shown in a modified example ( FIG. 11 ) described later, a display interface (I / F) may be provided in the case where information such as the speed of each vehicle C is displayed on a display device D connected to the processing device 22.

[0029] The laser scanner interface (I / F) 34 is capable of outputting an imaging timing signal (described later) to the laser scanner (LiDAR) 20, and has the function of receiving distance image data output from the laser scanner (LiDAR) 20 in response to the output of the imaging timing signal, and transferring the data to the central processing unit (CPU) 31 via the data bus 30.

[0030] In addition, the camera interface (I / F) 35 is capable of outputting an imaging timing signal, which will be described later, to the two-dimensional image sensor 21, and has the function of receiving image data output from the two-dimensional image sensor 21 in response to the output of the imaging timing signal, and transferring it to the central processing unit (CPU) 31 via the data bus 30.

[0031] As the laser scanner interface (I / F) 34 and the camera interface (I / F) 35, specifically, a well-known USB interface (I / F) used as an interface (I / F) with peripheral devices in a computer can be suitably used.

[0032] Furthermore, the network interface (I / F) 37 uses a mobile router capable of mobile data communication, which is high-speed data communication via wireless communication. This is preferable because it allows the mobile speed measurement device 1 to be quickly installed and operational without laying a communication cable when installing the device, but the present invention is not limited to this, and the network interface (I / F) 37 may also be capable of high-speed data communication using an optical cable or the like.

[0033] The two-dimensional image sensor 21 is capable of capturing two-dimensional images at a frame rate of 30 FPS (30 frames per second), which is the frame rate of a normal video, and outputting the image data. It has an external synchronization function that enables capturing images in response to an external signal input, and can be a CCD camera or the like that is capable of capturing known video images.

[0034] In addition, the laser scanner (LiDAR) 20 used is capable of generating distance images at a rate of 30 FPS (30 frames per second), which is the imaging period of the two-dimensional image sensor 21 mentioned above, or higher, and has an external synchronization function that generates distance images by scanning in response to signal input from outside.

[0035] In this embodiment, as will be described later, in order to improve the accuracy of speed measurement without performing complex processing such as image correction by substantially eliminating the time difference between capturing a two-dimensional image and scanning a distance image, distance images are generated in synchronization with the imaging timing of the two-dimensional image sensor 21. Therefore, as the distance image sensor, a laser scanner (LiDAR) 20 is used, which can generate distance images by scanning at the same high rate as the imaging cycle of the two-dimensional image sensor 21, 30 FPS (30 frames per second), and is not easily affected by sunlight in outdoor environments. However, the present invention is not limited to this, and as described above, a distance image sensor other than the laser scanner (LiDAR) 20, such as a depth camera using a ToF method, a stereo method, or a structured illumination method, may be used as long as it can generate distance images in synchronization with the imaging timing of the two-dimensional image sensor 21. Note that when the imaging cycle of the two-dimensional image sensor 21 used is a low frame rate, such as 5 FPS, due to the relatively slow moving speed of the object to be imaged, the above-mentioned depth camera may be more preferably used.

[0036] Furthermore, in this embodiment, as will be described later, the processing device 22 outputs an imaging timing signal to both the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21, thereby enabling the processing device 22 to control (synchronize) the timing of imaging and laser scanning. Therefore, an example is given in which both the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 have an external synchronization function, but the present invention is not limited to this. For example, if one of the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 is capable of outputting a timing signal that can identify the timing of imaging or laser scanning, it is possible to consider only the other, the laser scanner (LiDAR) 20 or the two-dimensional image sensor 21, as having an external synchronization function, and input a timing signal to synchronize the timing of imaging or laser scanning without the involvement of the processing device 22.

[0037] Next, the data stored in the RAM 32 will be briefly described with reference to Fig. 4. As shown in Fig. 4, the RAM 32 stores a processing program booted from the flash memory (F-MEM) 33, as well as two-dimensional image data with a system time attached, range image data with a system time attached, an object detection list in which objects extracted from the two-dimensional image data are registered, a detected vehicle list in which data related to vehicle C among objects detected from newly captured two-dimensional image data is registered, and a tracking list used in tracking processing to determine whether the newly detected vehicle C is the same vehicle C as the vehicle C captured at the previous imaging timing.

[0038] In addition, the two-dimensional image data and distance image data are stored for a predetermined number of frames (e.g., five frames) set in the initial settings based on parameters, and are sequentially erased once the time corresponding to the predetermined number of frames has elapsed.

[0039] The object detection list is list data in which all objects extracted from two-dimensional image data are registered, and the type of detected object, the detection area in which the object exists (the center coordinates of the area in the image, the size of the area described by the number of pixels in the X and Y directions), and mask data for making it possible to identify the area in which the object exists within that area are registered in association with a detection number individually assigned to each extracted object. Note that objects extracted from two-dimensional image data are not limited to vehicles C, but also include the license plates of individual vehicles C, people riding in vehicles C, etc.

[0040] The detected vehicle list is a list of objects registered in the object detection list that are classified as "vehicles." Corresponding to the detection number, the list includes time information (system time information assigned to the two-dimensional image data including the extracted vehicle), the detection area, mask data, vehicle number information read from the license plate extracted at the position corresponding to the vehicle C, and identified data for determining whether tracking has been performed using the tracking process.

[0041] As shown in Figure 5, the tracking list stores time information, movement speed, detection area, mask data, vehicle number information, and a non-update period described by the number of frames since the data stopped being updated, in association with an ID, which is identification information uniquely assigned to the vehicle C based on the fact that the same vehicle C is not registered in the tracking list. Note that the time information is system time information assigned to the 2D image data from which the corresponding detection area and mask data were created, and the movement speed is the movement speed of the vehicle C calculated from the previous captured and scanned 2D image and range image. Therefore, when vehicle C is first registered in the tracking list, the movement speed is not registered.

[0042] Next, the main functions of the processing device 22 will be described below with reference to Fig. 6. As shown in Fig. 6, the processing device 22 mainly has an imaging control function unit, an image input function unit, a distance image input function unit, an object detection function unit, a tracking processing function unit, a speed calculation function unit, and an overspeed determination function unit, which are enclosed by a dashed rectangular line.

[0043] The imaging control function unit has the function of outputting an imaging timing signal to the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 when it is time to capture an image. In this embodiment, the imaging timing signal is output when the counter value of the system clock reaches a value corresponding to a predetermined period of 33 milliseconds.

[0044] The image input function unit has the function of acquiring the system time from the system based on the time information from the RTC 36, and assigning the acquired system time to the two-dimensional image data output from the two-dimensional image sensor 21 as the imaging time.

[0045] The range image input function unit has a function of adding the system time acquired from the above-mentioned system to the three-dimensional point cloud range image data output from the laser scanner (LiDAR) 20 as the scan time.

[0046] The object detection function unit has the functions of processing two-dimensional image data to which the image capture time has been assigned by the image input function unit, extracting objects contained in the two-dimensional image, identifying the detection area of ​​each extracted object and creating a mask, creating an object detection list of the extracted objects, reading vehicle number information for objects among the extracted objects whose object type is "license plate", and creating a detected vehicle list for objects among the extracted objects whose object type is "vehicle", and in this embodiment, these functions are formed by a deep learning neural network (DNN).

[0047] The tracking processing function unit determines, based on the detected vehicle list created by the object detection function unit, for each vehicle C registered in the detected vehicle list, whether the same vehicle C is also registered in the tracking list, and if the same vehicle C is not present in the tracking list, adds information about that vehicle C to the tracking list and updates it.If the same vehicle C is present in the tracking list, it outputs the ID of that vehicle C and the vehicle data of that vehicle C (the detection area and mask data registered in the detected vehicle list in correspondence with the detection number of that vehicle) to the speed calculation function unit to obtain the moving speed of that vehicle C from the speed calculation function unit, and creates tracking information including the obtained moving speed, time information, vehicle data and vehicle number information about that vehicle C, and outputs the created tracking information to the speeding determination function unit.

[0048] The speed calculation function unit reads from RAM 32 the detected vehicle distance image data (new), which is new distance image data used to detect a new vehicle C in the object detection function unit, and the same vehicle distance image data (old), which is distance image data corresponding to the time information registered in association with the ID output from the tracking processing function unit, and identifies the new and old three-dimensional positions of the vehicle C determined to be identical, which are specified by the read new and old distance image data, the vehicle data (detection area, mask data) of the vehicle C determined to be identical output from the tracking processing function unit upon being determined to be identical, and the vehicle data (detection area, mask data) of the vehicle C determined to be identical registered in association with the ID in the tracking list, and has the function of calculating the movement speed of the vehicle C from the difference between the identified three-dimensional positions and the difference in time.

[0049] The speeding determination function unit has the function of determining whether the speed of the vehicle C exceeds the set upper speed limit based on the tracking information output from the tracking processing function unit, and if it does exceed the upper speed limit, creating speeding vehicle information including the ID of the vehicle C, time information, speed, and a vehicle image (two-dimensional image) of the vehicle cut out based on the detection area, and transmitting this information to the server computer 100.

[0050] Next, the speed measurement process executed by the processing device 22 will be described with reference to Fig. 7. The speed measurement process is started by the central processing unit (CPU) 31 when a processing program stored in the flash memory (F-MEM) 33 is automatically bootloaded upon startup of the processing device 22, and the processing program is stored in an executable state in the RAM 32.

[0051] First, in the speed measurement process, the central processing unit (CPU) 31 executes initialization processing (step S1). In this initialization processing, the operation of each piece of hardware included in the processing device 22 is checked, the connection of the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 is confirmed and initialized, and the communication connection with the server computer 100 is confirmed. In addition, various list data is initialized, an upper limit speed is set, and the like.

[0052] After the initialization process is completed, it is determined whether it is time to acquire image data (step S2), and if it is not time to acquire image data, it is determined whether there is a setting operation login by the maintenance PC 200 (step S20), and if there is no setting operation login, the process returns to step S2.

[0053] If a login for setting operation has been made (Yes in step S20), the process proceeds to step S21, where changes to the setting values ​​of various parameters, upper limit speed, etc. are accepted, and a setting change process is executed to update the corresponding data stored in the flash memory (F-MEM) 33, after which the process returns to the initialization process of step S1.

[0054] On the other hand, if it is the time to acquire image data, that is, if 33 milliseconds corresponding to one frame have elapsed since the previous acquisition, the process proceeds to step S3, where an imaging timing signal is output to the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 to acquire two-dimensional image data from the two-dimensional image sensor 21 and depth image data from the laser scanner (LiDAR) 20, and the system time at that time is assigned to each acquired image data as the imaging time and scan time. Note that step S3 is performed using the functions of the imaging control function unit, image input function unit, and depth image input function unit described above.

[0055] Then, the process proceeds to step S4, where the object detection process shown in Fig. 8 is executed. The object detection process is performed using the object detection function unit, and as described above in the description of the object detection function unit, the process is executed by a deep learning neural network (DNN) using the image processing circuit (GPU) 38.

[0056] As an object detection method for these deep learning neural networks (DNNs), YOLACT (Daniel Bolya Chong Zhou Fanyi Xiao Yong Jae Lee. YOLACT: Real-time Instance Segmentation. In ICCV, 2019), a method of instance segmentation that performs real-time object detection on a pixel-by-pixel basis in two-dimensional images, can be suitably used. In this embodiment, an example is shown in which YOLACT using a deep learning neural network (DNN) is used because it can perform object detection (extraction) in two-dimensional images at high speed and with high accuracy. However, the present invention is not limited to this. In cases where the movement speed of the object to be measured is relatively slow and the imaging period of the two-dimensional image is relatively long, and object detection (extraction) can be performed with sufficient accuracy using an object detection method that does not use a deep learning neural network (DNN) by spending a sufficiently long time on the object detection (extraction) process, a form that does not use a deep learning neural network (DNN) may be used, or even when a deep learning neural network (DNN) is used, a method other than the above-mentioned YOLACT may be used. The method for object detection (extraction) in these two-dimensional images may be selected appropriately based on the time available for object detection (extraction) processing based on the imaging period corresponding to the movement speed of the object whose speed is to be measured, and the required recognition accuracy.

[0057] In addition, in the deep learning neural network (DNN) of this embodiment, images of various vehicles (automobiles) are input as learning data and trained so that vehicles (automobiles) can be extracted as objects with high accuracy, and various license plates (color, with / without frame, name of Land Transport Bureau, etc.) are input as learning data and trained, thereby using a deep learning neural network (DNN) that particularly improves the accuracy of extracting vehicles (automobiles) and license plates.

[0058] In the object detection process of this embodiment, as shown in FIG. 8, first, all objects contained in the entire two-dimensional image acquired from the two-dimensional image sensor 21 in the above-mentioned step S3 are extracted (step S201).

[0059] Then, the type of each extracted object is identified, and a detection region (ROI) and mask are created for each object, which are then registered in an object detection list in association with the detection number assigned to each object.

[0060] Next, a detected object list C for object type "vehicle" and a detected object list P for object type "license plate" are created (step S203), and then the vehicles registered in the detected object list C are associated with the license plates registered in the detected object list P based on the detection area (step S204).

[0061] Specifically, a vehicle whose detection area is close to the detection area of ​​the license plate registered in the detected object list P is identified from the vehicles registered in the detected object list C, and the license plate is associated with the identified vehicle's license plate.

[0062] Then, the vehicle number information is read from the associated license plate. Specifically, the vehicle number information includes land transport, classification, kana, series, vehicle size, and vehicle type (step S205), and then the vehicle number information, which is the read license plate information, is added to and stored in the detected object list C (step S206).

[0063] In this embodiment, all information on the license plate is read as vehicle number information, but the present invention is not limited to this. The content to be read does not have to be all of this content, but only a part of this content, for example, only a "series" of four digit numbers, and the content of the vehicle number information to be read may be the minimum amount that allows for sufficient accuracy in determining that it is the same vehicle in the tracking process.

[0064] Next, proceed to step S207 to determine whether the association with license plates has been completed for all vehicles registered in the detected object list C. If the association has not been completed (No in step S207), return to step S204. If the association has been completed (Yes in step S207), create a detected vehicle list based on the detected object list C, in which the detection number, time information, detection area, mask data, vehicle number information, and identified data corresponding to unidentified data are associated and registered.

[0065] Once the object detection process in step S4 is completed, the processes in steps S5 to S19 are performed to determine whether each detected vehicle is identical to a vehicle registered in the tracking list, calculate the moving speed based on the difference in three-dimensional position and time between the detected vehicle and the vehicle determined to be identical, and determine whether the calculated moving speed is exceeding the upper limit speed.

[0066] First, in step S5, the vehicle that is first registered in the detected vehicle list is identified, and the identified data corresponding to that vehicle is updated to "identified." Then, the process proceeds to step S6, where the distance between the center position of the detection area of ​​the identified vehicle and the center position of the detection area of ​​each vehicle registered in the tracking list is calculated.

[0067] Then, it is determined whether or not there is a vehicle in the tracking list whose distance difference calculated in S6 is equal to or less than a threshold value (step S7).

[0068] If there are no vehicles in the tracking list whose calculated distance difference is less than the threshold (No in step S7), the process proceeds to step S14. On the other hand, if there are vehicles in the tracking list whose calculated distance difference is less than the threshold, the process proceeds to step S8, where it is determined whether there are any vehicles with matching vehicle number information among the vehicles whose distance difference is less than the threshold.

[0069] If there is no vehicle with matching vehicle number information (No in step S8), the process proceeds to step S14, where the vehicle identified in step S5 or step S19 described below is determined to be a new vehicle that has not yet been registered in the tracking list, and a new ID is assigned to the vehicle by adding 1 to the lowest ID registered in the tracking list, and the vehicle is added to the tracking list, and the process proceeds to step S14.

[0070] If a vehicle with matching vehicle number information exists (Yes in step S8), the ID in the tracking list of the vehicle with matching vehicle number information and the detected vehicle data of the vehicle identified in step S5 or step S19 are identified, and the speed calculation process is then carried out.

[0071] In this embodiment, the moving speeds of vehicles passing through two lanes are measured simultaneously, and therefore, due to vehicles passing each other, it is possible that the difference in distance calculated for different vehicles will be below a threshold value. Therefore, rather than determining that the vehicles are identical simply because the difference in calculated distance is below a threshold value, the vehicle identification determination is made with high accuracy by further determining whether the vehicle number information matches. However, the present invention is not limited to this, and for example, when measuring the speed of only vehicles passing through one lane, it is also possible to determine that the vehicles are identical using only one of these conditions.

[0072] Furthermore, in this embodiment, as described above, vehicle number information is read and used to determine identity. This vehicle number information is preferable because it is information specific to the vehicle and can significantly increase the accuracy of the identity determination, but the present invention is not limited to this. For example, instead of vehicle number information, information such as the color and size of the vehicle may be used as characteristic information, and identity determination may be made based on whether or not the characteristic information matches.

[0073] The functions of steps S5 to S9 and step S13 are mainly performed by using a tracking processing function unit.

[0074] In step S10, the speed calculation process consisting of steps S301 to S308 shown in Figure 9 is executed to calculate the moving speed of the vehicle determined to be the same vehicle from the new three-dimensional position of the vehicle, the old three-dimensional position at the time of updating the tracking list, and the time difference between them.

[0075] Specifically, as shown in FIG. 9, first, distance image data including the detected vehicle (rear distance image) and distance image data including the same vehicle (front distance image) are read from the RAM 32 (step S301).

[0076] Next, from the detection area and mask data contained in the detected vehicle data of the vehicle identified in step S9 and the distance image data (rear distance image), which is newly stored distance image data read from RAM 32, a point group (Q1) existing within the area of ​​the detected vehicle in the distance image data (rear distance image) is extracted (step S302).

[0077] Furthermore, from the detection area and mask data registered in the tracking list in association with the ID of the vehicle determined to be the same vehicle identified in step S9, and the distance image data (future distance image) read from RAM 32, a point cloud (Q2) existing within the area of ​​the vehicle determined to be the same in the distance image data (future distance image) is extracted (step S303).

[0078] Then, representative points q1 and q2 are determined for each of the point group Q1 extracted in step S302 and the point group Q2 extracted in step S303 (step S304). In this embodiment, the centers of gravity of the extracted point groups are used as representative points, but the present invention is not limited to this, and these representative points may be selected appropriately depending on the shape of the target object for which velocity measurement is performed, etc.

[0079] Next, the three-dimensional movement speed of the object is calculated from the difference (three-dimensional movement distance) between the three-dimensional coordinates of the representative points q1 and q2 determined in step S304 and the time difference (specifically, 33 milliseconds) between the acquisition times of each distance image.

[0080] Then, the process proceeds to step S306 to determine whether or not calculation is impossible, specifically, whether the data is incomplete and calculation is impossible, and if calculation is impossible (No in step S306), the fact that calculation is impossible is associated with the ID and temporarily stored (step S308), whereas if calculation is not impossible (Yes in step S306), the calculated movement speed information is associated with the ID and temporarily stored (step S309). Note that as the direction of vehicle movement differs depending on the lane, the calculated movement speed can be not only positive speed but also negative speed, so the absolute value of the obtained value is used as the movement speed.

[0081] Next, returning to FIG. 7, it is determined whether the travel speed calculated in step S10 exceeds a preset upper limit speed (step S11). If it does not exceed the upper limit speed (No in step S11), the process proceeds to step S15. If it exceeds the upper limit speed (Yes in step S11), the process proceeds to step S12, where the speeding vehicle information transmission setting process is executed.

[0082] In this speeding vehicle information transmission setting process, speeding vehicle information is created in a predetermined format, including the ID of the vehicle whose speed has been determined to exceed the upper speed limit, the speed, time information, and a two-dimensional image of the detection area, and settings are made to transmit the created speeding vehicle information to the server computer 100 (for example, the transmission data is set in a transmission queue register). Each piece of data that has been set for transmission is transmitted sequentially by the operating system running in the processing device 22.

[0083] Next, proceed to step S13, where the data of the vehicle determined to be the same in the tracking list is overwritten and updated with the data registered in the detected vehicle list of the vehicle identified in step S5 or step S19, and then proceed to step S15.

[0084] In step S15, it is determined whether all vehicles registered in the detected vehicle list have been identified based on the identified data in the detected vehicle list.If all vehicles have been identified (Yes in step S15), vehicles that have not been updated for a certain period of time due to being determined to be the same vehicle are deleted from the tracking list (step S16), and the process returns to step S2.

[0085] On the other hand, if not all vehicles have been identified (No in step S15), that is, if there are any vehicles among the vehicles extracted in the object detection process in step S4 for which a determination as to whether they are the same vehicle or not has not yet been made, the next vehicle registered in the detected vehicle list is identified, and the identified data corresponding to that vehicle is updated to "identified" (step S19), and the process returns to step S6. In this way, in step S19, vehicles for which a determination as to whether they are the same vehicle or not is made are sequentially identified, and tracking processes and the like are performed for all vehicles registered in the detected vehicle list, and for vehicles determined to be the same, the traveling speed is calculated and a determination is made as to whether it exceeds the upper speed limit. If it does, speeding vehicle information is sent to server computer 100.

[0086] In this embodiment, the speeding vehicle information sent to the server computer 100 includes an ID, time information, the speed of the vehicle, and a two-dimensional cutout image of the detection area of ​​the vehicle.By storing this speeding vehicle information in the server computer 100, it is possible to access the server computer 100 and obtain information on vehicles that have exceeded the speed limit, such as that shown in Figure 10, which can be used for speed enforcement.

[0087] As described above, according to the moving speed measuring device 1 of this embodiment, the area of ​​the moving vehicle is extracted using a two-dimensional image aligned with the distance image, and the new and old three-dimensional positions of the moving vehicle are identified from the point cloud of the distance image corresponding to the extracted area of ​​the vehicle, and the moving speed is calculated.Therefore, compared to when only the distance image is used to extract the moving vehicle in the distance image, the vehicle can be extracted with high accuracy and quickly enough to be in real time, so that speed measurement can be performed with sufficient accuracy and speed for practical use.

[0088] In particular, since the images used by the deep learning neural network (DNN) to extract vehicles are two-dimensional images rather than range images, existing two-dimensional images of various vehicles can be used as training data for the deep learning of the deep learning neural network (DNN).This means that there is no need to prepare a huge number of three-dimensional point cloud range images for various vehicles, and there is no need to incur significant costs for creating training data (annotation), which helps to reduce the increase in costs of device development.

[0089] Furthermore, according to the moving speed measurement device 1 of this embodiment, tracking determination of whether a newly extracted vehicle is the same as which vehicle extracted one frame earlier can be performed using two-dimensional images rather than distance images, so the accuracy of these tracking determinations can be significantly improved compared to when distance images are used, and the time required for these tracking determinations can also be significantly reduced, making it possible to measure speed with sufficient accuracy and speed for practical use.

[0090] Furthermore, according to the movement speed measuring device 1 of this embodiment, the processing device 22 outputs an imaging timing signal to the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21, thereby synchronizing the imaging timing of the two-dimensional image and the scanning timing of the distance image. This significantly reduces the time difference between the imaging time of the two-dimensional image and the scanning time of the distance image, thereby preventing errors from occurring in the measured speed due to these time differences, thereby improving measurement accuracy.

[0091] Furthermore, according to the movement speed measuring device 1 of this embodiment, mask data in which the vehicle area is accurately extracted by object detection using a two-dimensional image is used to identify all point groups Q1, Q2 in the area in the distance image corresponding to the vehicle area identified by the mask data, and the three-dimensional position of the center of gravity calculated using the identified point groups Q1, Q2 is determined as representative points q1, q2 to calculate the movement speed.Therefore, compared to when object detection is performed using a distance image to extract the vehicle area, the vehicle area can be extracted with higher accuracy, and it is possible to prevent unnecessary point groups that do not correspond to the vehicle from being included in the point groups Q1, Q2 corresponding to the vehicle area, thereby improving the accuracy of the speed calculation.

[0092] Furthermore, according to the moving speed measuring device 1 of this embodiment, the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 are stored in a fixed state within the storage section 2b of the bottomed box-shaped housing 2, so that when the moving speed measuring device 1 is transported and installed at the measurement location, it is possible to prevent the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 from coming into contact with an obstacle or the like, causing misalignment and preventing good measurements from being taken.

[0093] Although the embodiments of the present invention have been described above with reference to the drawings, the specific configuration is not limited to these embodiments, and the present invention also includes modifications and additions that do not deviate from the gist of the present invention.

[0094] For example, in the above embodiment, an example is given of a form in which speeding vehicle information is created and transmitted to the server computer 100, but the present invention is not limited to this. A display device D connected by a cable to the processing device 22 in the housing 2 may be installed, for example, at a monitoring location on the side of the road R, and the display device D may display a two-dimensional image at that time, as shown in FIG. 11, and may also display and output in real time a display indicating the detection region (ROI), the ID of each vehicle, and the calculated moving speed of each vehicle included in the displayed two-dimensional image.

[0095] In addition, in the above embodiment, an example was given of creating and transmitting speeding vehicle information every frame (33 milliseconds), but the present invention is not limited to this. The average value of the moving speed contained in the tracking information of multiple frames may be compared with the upper limit speed, and if the average value of the moving speed exceeds the upper limit speed, speeding vehicle information may be created and transmitted to the server computer 100.

[0096] Furthermore, in the above embodiment, an example is given of a form in which an imaging timing signal is output at all imaging timings and scan timings, but the present invention is not limited to this, and these imaging timing signals may be output, for example, only at the first imaging timing and scan timing, and thereafter the laser scanner (LiDAR) 20 and the two-dimensional image sensor 21 may capture images and scan at a predetermined cycle, for example, every 33 milliseconds, or an imaging timing signal may be output at every predetermined number of imaging timings and scan timings (for example, 10 times).

[0097] Furthermore, while the above embodiment illustrates a configuration in which the timing of capturing a two-dimensional image and the timing of scanning a distance image are synchronized, the present invention is not limited to this. For example, the timing of capturing a two-dimensional image and the timing of scanning a distance image may be performed independently of each other without being synchronized, and the two-dimensional image at the scanning timing of the distance image may be image-generated or image-corrected based on two-dimensional images captured before and after the scanning timing, and the vehicle may be extracted using the image-generated or image-corrected two-dimensional image at the scanning timing of the distance image. However, if the timing of capturing a two-dimensional image and the timing of scanning a distance image are not synchronized in this manner, not only will the processing load due to image generation and image correction be incurred and processing time will be required for these image generation and image correction, but because two-dimensional images captured before and after the scanning timing are required, image generation or image correction can only be performed after the later capturing timing, making real-time vehicle extraction (object detection) impossible. Therefore, this is suitable for applications where real-time speed measurement is not required.

[0098] Furthermore, in the above embodiment, an example is given of the use of the mobile speed measurement device 1 for speed enforcement of a vehicle, which is a moving body, but the present invention is not limited to this, and the use of these mobile speed measurement devices 1 may be for any type of moving body, for example, it may be used to measure the moving speed of an ADR or the like that moves at a relatively slow speed within a specific area such as a factory, or it may be used to measure the moving speed of a drone or the like that moves through space rather than a moving body that moves on the ground. [Explanation of symbols]

[0099] 1 Movement speed measuring device 2. Case 4 base plates 5 Height-adjustable base 6. Tempered glass 20 Laser scanner (LiDAR) 21 Two-dimensional image sensor 22 Processing equipment 25 Antenna 31 CPU 32 RAM 33 Flash Memory 34 Laser scanner interface (I / F) 35 Camera Interface (I / F) 36 RTC 37 Network Interface (I / F) 38 GPU 100 server computers 200 Maintenance PC

Claims

1. A movement speed measuring device has a distance image sensor capable of generating a three-dimensional point cloud distance image including a plurality of point clouds with specified distances, and is capable of calculating and outputting the three-dimensional movement speed of an object included in the three-dimensional point cloud distance images generated by the distance image sensor at different times, a two-dimensional image sensor capable of capturing two-dimensional images, the two-dimensional images being aligned with the three-dimensional point cloud range image generated by the range image sensor; an object recognition means for recognizing an object from the two-dimensional image captured by the two-dimensional image sensor and extracting the recognized object and its area; an identical object determination means for determining whether or not objects extracted by the object recognition means from two-dimensional images captured at a first time point and a second time point after the first time point are the same object; a three-dimensional movement velocity calculation means for extracting, from the three-dimensional point cloud range images corresponding to the first and second time points, point clouds included in a matching region on a three-dimensional point cloud range image that matches a region of the object extracted from the two-dimensional images taken at the first and second time points by the object recognition means for the object determined to be the same object by the same object determination means, and calculating a three-dimensional movement velocity of the object based on a three-dimensional movement distance obtained from the extracted point clouds at the first and second time points and a time difference between the first and second time points; an output means capable of outputting the three-dimensional movement velocity calculated by the three-dimensional movement velocity calculation means; Equipped with The moving speed measuring device is fixedly installed at a position to the side of a path along which the object can move, the object recognition means is capable of simultaneously recognizing and extracting a plurality of objects, always calculating and outputting the three-dimensional movement speed of each of the extracted objects, regardless of whether the areas of the multiple objects extracted by the object recognition means are overlapping or not; A moving speed measuring device characterized by:

2. an imaging timing of the two-dimensional image sensor and a generation timing of the three-dimensional point cloud range image by the range image sensor are synchronized; 2. The moving speed measuring device according to claim 1.

3. The object is a vehicle traveling on a road that is the route, an upper limit speed setting means for setting an upper limit speed of the vehicle; a determination means for determining whether or not the three-dimensional movement speed calculated by the three-dimensional movement speed calculation means exceeds the upper limit speed; Equipped with The output means is capable of outputting a two-dimensional image of the vehicle determined by the determination means to be exceeding the upper speed limit and time information when the two-dimensional image is captured by the two-dimensional image sensor.

3. The moving speed measuring device according to claim 1 or 2.

4. the three-dimensional movement speed calculation means identifies all point groups included in the matched region, and calculates the three-dimensional movement speed using distance data of all the identified point groups; 3. The moving speed measuring device according to claim 1 or 2.

Images