Traffic flow measurement system and traffic flow measurement method

Abstract

To provide a traffic flow measurement system for presenting results of traffic flow analysis to a user, capable of allowing the user to intuitively grasp the changing state of the moving body while viewing sensor images as the detection results of a sensor targeting a measurement area.SOLUTION: A traffic flow measurement server is configured to generate a behavior image of visualized time series data representing the changing situation of states (position, velocity, and acceleration) of the moving body based on results of traffic flow analysis processing, in particular, a trajectory line 411, a velocity line 412, and an acceleration line 413, and to generate a time-series display screen 401 in which the behavior image is superimposed on sensor images (a camera image 403, a lidar intensity image 404, and a lidar point cloud image 405) based on the sensor detection results, so as to display the screen on a user terminal.SELECTED DRAWING: Figure 24

Description

[Technical Field]

[0001] The present invention relates to a traffic flow measurement system and a traffic flow measurement method that measure traffic flow at a target point using sensors such as cameras and lidars. [Background technology]

[0002] Traffic flow measurements are carried out to understand the traffic conditions at target points such as intersections, with the aim of improving road traffic safety and smoothness. In this traffic flow measurement, it is desirable to obtain detailed and highly accurate traffic flow data on the status of moving objects such as vehicles and pedestrians without requiring a large number of personnel.

[0003] In response to such demands, a technology has been known in the past that uses cameras as sensors for detecting objects within a measurement area, as well as LIDAR, which has recently been attracting attention in the field of autonomous driving, to acquire the trajectory of a moving object in three-dimensional space (see Patent Document 1). This technology involves processing to associate images captured by the camera with 3D point cloud data acquired by the LIDAR. It also involves processing to integrate 3D point cloud data acquired by multiple LIDARs installed at different locations. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-113645 Summary of the Invention [Problem to be solved by the invention]

[0005] Conventional traffic flow measurement using sensors such as cameras and lidars makes it possible to perform traffic flow analysis by obtaining various information, such as the trajectory of moving objects and the speed and acceleration of moving objects, based on detailed and highly accurate information about moving objects and road components.

[0006] On the other hand, when presenting the results of traffic flow analysis to a user, it is desirable to present the user with sensor images as the detection results of sensors targeting the measurement area, and also to enable the user to intuitively grasp the transition of the state of moving objects while viewing the sensor images. However, conventional technologies do not take such a need into consideration at all.

[0007] Therefore, the main object of the present invention is to provide a traffic flow measurement system and a traffic flow measurement method that, when presenting the results of traffic flow analysis to a user, allows the user to intuitively grasp the changes in the state of moving objects while viewing sensor images as the detection results of sensors targeting the measurement area. [Means for solving the problem]

[0008] The traffic flow measurement system of the present invention comprises a first sensor that acquires two-dimensional detection results for a traffic flow measurement area, a second sensor that acquires three-dimensional detection results for the measurement area, a server device that is connected to the first and second sensors and executes a traffic flow analysis process based on the detection results of the first and second sensors, and a terminal device that is connected to the server device via a network and displays the results of the traffic flow analysis process, and the server device generates a behavior image that associates time-series data of the position, speed, and acceleration of the mobile object as an image representing the behavior of the mobile object, and visualizes the behavior image based on the results of the traffic flow analysis process, and The first and second sensors A traffic flow viewing screen is generated in which the traffic flow is superimposed on the sensor image based on the detection results, and the traffic flow viewing screen is transmitted to the terminal device.

[0009] Furthermore, a traffic flow measurement method of the present invention is a traffic flow measurement system comprising a first sensor that acquires two-dimensional detection results for a traffic flow measurement area, a second sensor that acquires three-dimensional detection results for the measurement area, a server device that is connected to the first and second sensors and executes a traffic flow analysis process based on the detection results of the first and second sensors, and a terminal device that is connected to the server device via a network and displays the results of the traffic flow analysis process, wherein the server device generates a behavior image that associates time-series data of the position, speed, and acceleration of the mobile object as an image representing the behavior of the mobile object, and visualizes the behavior image based on the results of the traffic flow analysis process, The first and second sensors A traffic flow viewing screen is generated in which the traffic flow is superimposed on the sensor image based on the detection results, and the traffic flow viewing screen is transmitted to the terminal device. [Effects of the Invention]

[0010] According to the present invention, As an image showing the behavior of a moving object, time series data of the moving object's position, speed, and acceleration are associated. The visualized behavior image is superimposed on the sensor image based on the sensor detection results. This allows the user to view the results of the traffic flow analysis while viewing the sensor image as the sensor detection results for the measurement area. In addition to the change in the position of a moving object, the change in the speed and acceleration of the moving object can be easily detected. It can be grasped. [Brief explanation of the drawings]

[0011] [Figure 1] Overall configuration diagram of a traffic flow measurement system according to this embodiment [Figure 2] Block diagram showing the general configuration of the traffic flow measurement server [Figure 3] An explanatory diagram showing the contents of traffic flow data generated by the traffic flow measurement server. [Figure 4] An explanatory diagram showing the transition status of the screen displayed on the user terminal. [Figure 5] FIG. 10 is an explanatory diagram showing a main menu screen displayed on a user terminal. [Figure 6] FIG. 10 is an explanatory diagram showing a submenu screen displayed on a user terminal. [Figure 7]FIG. 10 is an explanatory diagram showing a submenu screen displayed on a user terminal. [Figure 8] FIG. 10 is an explanatory diagram showing a basic adjustment screen displayed on a user terminal. [Figure 9] FIG. 10 is an explanatory diagram showing a basic adjustment screen displayed on a user terminal. [Figure 10] FIG. 10 is an explanatory diagram showing a basic adjustment screen displayed on a user terminal. [Figure 11] FIG. 10 is an explanatory diagram showing a basic adjustment screen displayed on a user terminal. [Figure 12] FIG. 10 is an explanatory diagram showing an alignment screen displayed on a user terminal; [Figure 13] FIG. 10 is an explanatory diagram showing an alignment screen displayed on a user terminal; [Figure 14] FIG. 10 is an explanatory diagram showing an alignment screen displayed on a user terminal; [Figure 15] FIG. 10 is an explanatory diagram showing an alignment screen displayed on a user terminal; [Figure 16] FIG. 10 is an explanatory diagram showing an installation confirmation screen displayed on a user terminal. [Figure 17] FIG. 10 is an explanatory diagram showing an installation confirmation screen displayed on a user terminal. [Figure 18] FIG. 10 is an explanatory diagram showing an installation confirmation screen displayed on a user terminal. [Figure 19] FIG. 10 is an explanatory diagram showing an installation confirmation screen displayed on a user terminal. [Figure 20] FIG. 10 is an explanatory diagram showing another example of an installation confirmation screen displayed on a user terminal. [Figure 21] FIG. 10 is an explanatory diagram showing a sensor data recording screen displayed on a user terminal. [Figure 22] An explanatory diagram showing a sensor data analysis screen displayed on a user terminal. [Figure 23] An explanatory diagram showing a sensor data analysis screen displayed on a user terminal. [Figure 24] FIG. 10 is an explanatory diagram showing a time series display screen displayed on a user terminal. [Figure 25] FIG. 10 is an explanatory diagram showing a time series display screen displayed on a user terminal. [Figure 26] FIG. 10 is an explanatory diagram showing the main part of a time series display screen displayed on a user terminal. [Figure 27] FIG. 10 is an explanatory diagram showing a scenario specification screen displayed on a user terminal. [Figure 28] FIG. 10 is an explanatory diagram showing a scenario specification screen displayed on a user terminal. [Figure 29] FIG. 10 is an explanatory diagram showing a statistical information specification screen displayed on a user terminal. [Figure 30] FIG. 10 is an explanatory diagram showing a specified event viewing screen displayed on a user terminal. [Figure 31] FIG. 10 is an explanatory diagram showing a specified event viewing screen displayed on a user terminal. [Figure 32] FIG. 10 is an explanatory diagram showing a tracking mode screen displayed on a user terminal. [Figure 33] FIG. 10 is an explanatory diagram showing a tracking mode screen displayed on a user terminal. [Figure 34] FIG. 10 is an explanatory diagram showing an extended viewing mode screen displayed on a user terminal. [Figure 35] FIG. 10 is an explanatory diagram showing an extended viewing mode screen displayed on a user terminal. [Figure 36] Flow diagram showing the procedure for sensor installation adjustment processing performed by the traffic flow measurement server [Figure 37] A flowchart showing the procedure for processing related to traffic flow data generation performed by the traffic flow measurement server. [Figure 38] A flowchart showing the procedure for processing related to viewing traffic flow data performed by the traffic flow measurement server. DETAILED DESCRIPTION OF THE INVENTION

[0012] The first invention made to solve the above problems is a traffic flow measurement system comprising a first sensor that acquires two-dimensional detection results for a traffic flow measurement area, a second sensor that acquires three-dimensional detection results for the measurement area, a server device that is connected to the first and second sensors and executes a traffic flow analysis process based on the detection results of the first and second sensors, and a terminal device that is connected to the server device via a network and displays the results of the traffic flow analysis process, wherein the server device generates a behavior image that associates time-series data of the position, speed, and acceleration of the mobile body as an image representing the behavior of the mobile body, and visualizes the behavior image based on the result of the traffic flow analysis process, The first and second sensors A traffic flow viewing screen is generated in which the traffic flow is superimposed on the sensor image based on the detection results, and the traffic flow viewing screen is transmitted to the terminal device.

[0013] According to this, As an image showing the behavior of a moving object, time series data of the moving object's position, speed, and acceleration are associated. The visualized behavior image is superimposed on the sensor image based on the sensor detection results. This allows the user to view the results of the traffic flow analysis while viewing the sensor image as the sensor detection results for the measurement area. In addition to the change in the position of a moving object, the change in the speed and acceleration of the moving object can be easily detected. It can be grasped.

[0016] Also, Second In the invention, the behavior image includes a label image that expresses, in characters, at least one of the ID, speed, and acceleration of the moving object.

[0017] This allows the user to specifically know the ID, speed, and acceleration of the moving object.

[0018] Also, Third The invention is the behavior image includes a trajectory image visualizing time series data representing a change in the position of the moving object, a velocity image visualizing time series data representing a change in the velocity of the moving object, and an acceleration image visualizing time series data representing a change in the acceleration of the moving object; The velocity image and the acceleration image are configured such that a trajectory point representing the position of the moving body on the trajectory image is used as the origin, and the distance from the trajectory point to a velocity point on the velocity image at the same time represents the velocity, and the distance from the trajectory point to an acceleration point on the acceleration image at the same time represents the acceleration.

[0019] This allows the user to intuitively grasp the changes in the position of the moving object as well as the changes in the speed and acceleration. In addition, in a fourth aspect of the present invention, the behavior image has a trajectory image that visualizes time series data that represents a change in the position of the moving object, and the time series data that represents a change in the speed of the moving object and the time series data that represents a change in the acceleration of the moving object are expressed by attributes of the trajectory image. The composition is as follows. This allows the user to intuitively grasp the changes in speed and acceleration from the color density and thickness of the trajectory image of the moving object.

[0020] In addition, a fifth invention is configured such that, in response to a user's operation to change the viewpoint on the traffic flow viewing screen, the server device generates the sensor image from the changed viewpoint from the three-dimensional detection results and transmits the generated sensor image to the terminal device.

[0021] This allows the user to view the sensor image and the behavior image superimposed on the sensor image while changing the viewpoint, thereby enabling the user to observe the behavior of the moving object from various directions.

[0022] A sixth aspect of the present invention is a traffic flow measurement system including a first sensor that acquires two-dimensional detection results for a traffic flow measurement area, a second sensor that acquires three-dimensional detection results for the measurement area, a server device that is connected to the first and second sensors and executes a traffic flow analysis process based on the detection results of the first and second sensors, and a terminal device that is connected to the server device via a network and displays the results of the traffic flow analysis process, wherein the server device generates a behavior image that associates time-series data of the position, speed, and acceleration of the mobile body as an image representing the behavior of the mobile body, and visualizes the behavior image based on the result of the traffic flow analysis process, The first and second sensors A traffic flow viewing screen is generated in which the traffic flow is superimposed on the sensor image based on the detection results, and the traffic flow viewing screen is transmitted to the terminal device.

[0023] According to this, as in the first invention, when presenting the results of traffic flow analysis to the user, the user can view the sensor image as the detection result of the sensor targeting the measurement area, and In addition to the change in the position of a moving object, the change in the speed and acceleration of the moving object can be easily detected. It can be grasped.

[0024] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0025] FIG. 1 is a diagram showing the overall configuration of a traffic flow measurement system according to this embodiment.

[0026] This system measures traffic flow in a measurement area. The system includes a camera 1 (first sensor), a lidar 2 (second sensor), a traffic flow measurement server 3 (server device), a user terminal 4 (terminal device), and a management terminal 5. The camera 1 and the lidar 2 are connected to the traffic flow measurement server 3 via a first network N1. The user terminal 4 and the management terminal are connected to the traffic flow measurement server 3 via a second network N2.

[0027] The camera 1 captures an image of the measurement area and acquires a camera image as a two-dimensional detection result (two-dimensional information) of the measurement area. The camera 1 includes a visible light imaging element and can acquire a color image.

[0028] LIDAR 2 detects objects in a measurement area and acquires 3D point cloud data as three-dimensional detection results (three-dimensional information) for the measurement area. LIDAR 2 emits laser light and detects reflected light from objects to acquire three-dimensional information. Note that a three-dimensional sensor other than LIDAR 2 may also be used.

[0029] The traffic flow measurement server 3 acquires camera images from the camera 1 and 3D point cloud data from the lidar 2, and performs traffic flow analysis processing for the measurement area based on the camera images and the 3D point cloud data. In addition, the traffic flow measurement server 3 performs processing to assist the user in easily adjusting the sensor installation status when a new sensor (camera 1 and lidar 2) is installed or when a sensor is replaced.

[0030] The user terminal 4 is configured as a tablet terminal or the like. The user terminal 4 displays a screen for setting and viewing information sent from the traffic flow measurement server 3, and this screen allows the user to adjust the installation status of the sensor, view the results of the traffic flow analysis processing, and so on.

[0031] The management terminal is configured with a PC, etc. The management terminal displays a management screen sent from the traffic flow measurement server 3, and the administrator can use this management screen to perform management tasks such as setting the conditions for processing performed by the traffic flow measurement server 3.

[0032] Camera 1 and LIDAR 2 are equipped with the function of receiving satellite signals from a satellite positioning system (such as GPS), and update the time information in Camera 1 and LIDAR 2 using the time information contained in the satellite signals. Camera 1 and LIDAR 2 add the time synchronized with the satellite signals as the detection time to the detection results (camera images, 3D point cloud data) and send them to the traffic flow measurement server 3. The traffic flow measurement server 3 synchronizes the detection results of LIDAR 2 and Camera 1 based on the detection time. If Camera 1 and LIDAR 2 do not have the function of receiving satellite signals, they may synchronize the time via the first network.

[0033] Next, we will explain the general configuration of the traffic flow measurement server 3. Figure 2 is a block diagram showing the general configuration of the traffic flow measurement server 3.

[0034] The traffic flow measurement server 3 includes a communication unit 11, a storage unit 12, and a processor 13.

[0035] The communication unit 11 communicates with the camera 1 and the rider 2 via the first network, and also communicates with the user terminal 4 and the management terminal via the second network.

[0036] The memory unit 12 stores programs executed by the processor 13, etc. The memory unit 12 also stores camera images acquired from the camera 1 and 3D point cloud data acquired from the LIDAR 2. The memory unit 12 also stores traffic flow data generated by the processor 13. The memory unit 12 also stores CG images (simulation images) of measurement points. The memory unit 12 also stores sensor installation information acquired in each process of basic adjustment, alignment, and installation confirmation. The sensor installation information includes information on the detection angles of the sensors (camera 1, LIDAR 2), information on the positional relationship between the camera images and 3D point cloud data, information on the mutual positional relationship between the 3D point cloud data from multiple LIDARs 2, etc.

[0037] The processor 13 performs various processes by executing programs stored in the memory. In this embodiment, the processor 13 performs a sensor data synchronization process P1, a sensor data integration process P2, a LIDAR image generation process P3, a sensor installation support process P4, a traffic flow data generation process P5, an event detection process P6, an event extraction process P7, a statistical process P8, a risk determination process P9, and a traffic flow data presentation process P10.

[0038] In the sensor data synchronization process P1, the processor 13 associates the camera images acquired from each camera 1 and the 3D point cloud data acquired from each LIDAR 2 based on the detection time. In this embodiment, the camera 1 adds the time included in the received satellite signal as the detection time to the camera image and transmits it to the traffic flow measurement server 3. Furthermore, the LIDAR 2 adds the time included in the received satellite signal as the detection time to the 3D point cloud data and transmits it to the traffic flow measurement server 3.

[0039] In the sensor data integration process P2, the processor 13 integrates (combines) multiple pieces of 3D point cloud data obtained by multiple LIDARs 2 installed at multiple locations.

[0040] In the LIDAR image generation process P3, the processor 13 generates a LIDAR intensity image with the sensor installation point as the viewpoint based on the 3D point cloud data from the LIDAR 2. The processor 13 also generates a LIDAR point cloud image with the viewpoint specified by the user based on the 3D point cloud data. In this embodiment, the 3D point cloud data is displayed as a LIDAR point cloud image using a 3D viewer on the user terminal 4, and the user can change the viewpoint by performing an operation to change the viewpoint of the 3D viewer, for example, by dragging the cursor up, down, left, or right on the displayed LIDAR point cloud image.

[0041] In the sensor installation support process P4, the processor 13 performs a process to support the user's adjustment work performed when installing the sensor (camera 1, LIDAR 2) in accordance with the user's operation on the user terminal 4. The sensor installation support process P4 includes a sensor adjustment support process P21, a positioning process P22, and an installation confirmation support process P23.

[0042] In the sensor adjustment support process P21, the processor 13 performs a process to support the user's operation to adjust the installation state of the sensors (camera 1, LIDAR 2). Specifically, the processor 13 displays a camera image and a LIDAR intensity image generated from the 3D point cloud data on the user terminal 4, and controls the imaging angle (angle of view) of the sensors (camera 1, LIDAR 2) in accordance with the user's adjustment operation.

[0043] In the registration process P22, the processor 13 estimates the relative positional relationship between the installation points of each sensor (camera 1, LIDAR 2) and associates the coordinates of the detection results for each of the multiple sensors. Specifically, the processor 13 associates the coordinates on the camera image with the coordinates of the 3D point cloud data. The processor 13 also corrects the positional deviation of the point cloud data from multiple LIDARs 2 installed at different points.

[0044] In the installation confirmation support process P23, the processor 13 places a virtual object of the moving body in a three-dimensional space including the 3D point cloud data of the LIDAR 2 in response to a user operation on the user terminal 4, and superimposes the virtual object of the moving body on the camera image and the LIDAR intensity image based on the positional relationship. Then, the processor 13 determines whether or not there is a missing part in the virtual object of the moving body superimposed on the camera image and the LIDAR intensity image, i.e., whether or not the virtual object of the moving body extends beyond the display range of the camera image and the LIDAR intensity image.

[0045] In the traffic flow data generation process P5, the processor 13 generates traffic flow data (see FIG. 3) representing the traffic conditions in the measurement area based on the sensor data (camera images, 3D point cloud data). The traffic flow data generation process P5 includes a sensor data recording process P31 and a sensor data analysis process P32 (traffic flow analysis process).

[0046] In the sensor data recording process P31, the processor 13 stores the camera images acquired from each camera 1 and the 3D point cloud data acquired from each rider 2 in the memory unit 12 in accordance with the user's instructions on the user terminal 4.

[0047] In the sensor data analysis process P32 (traffic flow analysis process), the processor 13 generates traffic flow data based on the sensor data (camera images, 3D point cloud data) collected in the sensor data recording process P31. The sensor data analysis process P32 includes a moving object detection process P33, a moving object ID management process P34, and a road component detection process P35.

[0048] In the moving object detection process P33, the processor 13 detects moving objects from the camera image captured by the camera 1 in an identifiable manner. Specifically, the processor 13 detects buses, trucks, trailers, passenger cars, motorcycles, bicycles, pedestrians, and the like. The processor 13 also detects moving objects from 3D point cloud data that integrates 3D point cloud data captured by multiple riders 2. The processor 13 also acquires location information for the detected moving objects and assigns a moving object ID to the detected moving objects. The moving object detection process P33 can use an image recognition engine (machine learning model) built using machine learning such as deep learning.

[0049] In the mobile object ID management process P34, the processor 13 changes the mobile object IDs assigned to the mobile objects when the mobile objects are detected from each camera image and the mobile object IDs assigned to the mobile objects when the mobile objects are detected from the 3D point cloud data so that the same mobile object ID is assigned to the same mobile object. In this embodiment, the user can specify a priority sensor when changing the mobile object ID. In this case, the mobile object IDs assigned to the mobile objects for sensors other than the priority sensor are changed to the mobile object ID assigned to the mobile object for the priority sensor.

[0050] In the road component detection process P35, the processor 13 detects road components from the 3D point cloud data in an identifiable manner. Specifically, by segmentation (area division), the processor 13 detects areas of road components, i.e., features such as sidewalks, curbs, and curb rails, and road markings such as white lines and stop lines. The road component detection process P35 can use an image recognition engine (machine learning model) built by machine learning such as deep learning.

[0051] In the event detection process P6, the processor 13 detects an event that corresponds to a predetermined scenario (event type) based on the traffic flow data resulting from the sensor data analysis process P32 (traffic flow analysis process). Specifically, based on the trajectory of each moving object, the processor 13 detects events that correspond to scenarios such as rear-end collisions, right-turn collisions, left-turn collisions, wrong-way driving, and tailgating. The results of the event detection process are stored in an event database.

[0052] In the event extraction process P7, the processor 13 extracts events that correspond to the scenario specified by the user from the events stored in the event database. In this embodiment, the user can directly specify a scenario at the user terminal 4, or can present statistical information to the user and allow the user to specify a scenario within the statistical information.

[0053] In the statistical processing P8, the processor 13 performs statistical processing based on the traffic flow data to generate statistical information, such as statistical information on the occurrence frequency of events corresponding to each of a plurality of scenarios.

[0054] In the risk determination process P9, processor 13 acquires information about the traffic environment at a target point, specifically, the positional relationship between the mobile object and road components, based on the traffic flow data, and determines the risk of the traffic environment at the target point. In the risk determination process P9, an index for evaluating the risk of the traffic environment at each point is created in advance based on the statistical information acquired by statistical processing for each point, and the risk is determined from the situation of the mobile object at the target point based on the index for evaluating the risk.

[0055] In the traffic flow data presentation process P10, the processor 13 presents traffic flow data to the user by displaying various screens on the user terminal 4. The traffic flow data presentation process P10 includes a time series display process P41, a specified event display process P42, and ancillary information display process P43.

[0056] In the time-series display process P41, the processor 13 visualizes information (status information) representing the behavior (status change) of the moving object using graphics and characters based on the traffic flow data and displays it on the screen. In this embodiment, as information representing the behavior of the moving object, behavior images (trajectory image, speed image, acceleration image) that visualize time-series data representing changes in the position, speed, and acceleration of the moving object are superimposed and displayed on the sensor images (camera image, LIDAR point cloud image, LIDAR intensity image).

[0057] In the specified event display process P42, the processor 13 displays sensor images (camera images, LIDAR point cloud images, etc.) related to events that meet the conditions specified by the user on the user terminal 4. In this embodiment, the user can specify a scenario (event type) as the extraction condition, and sensor images related to events that meet the scenario specified by the user are displayed on the user terminal 4.

[0058] In the supplementary information display process P43, the processor 13 displays information related to the traffic environment in the measurement area, such as the status of road components such as white lines and sidewalks, and the type of moving object (passenger car, truck, etc.) as supplementary information on the screen simultaneously with the moving object. Specifically, an object specified by the user is highlighted on the sensor image (camera image, LIDAR point cloud image, LIDAR intensity image) so that it can be identified.

[0059] Next, a description will be given of the traffic flow data generated by the traffic flow measurement server 3. Fig. 3 is an explanatory diagram showing the contents of the traffic flow data.

[0060] The traffic flow measurement server 3 generates traffic flow data (tracking data) for each moving object (trajectory ID). Each row in the table shown in Figure 3 represents unit data for each time using a timestamp, and this unit data is generated sequentially in chronological order. The example shown in Figure 3 relates to a moving object with a trajectory ID of "1." The target moving object is a passenger car with an attribute of "0," and is traveling in the x direction.

[0061] Traffic flow data includes a timestamp (year / month / date, hour / minute / second), a trajectory ID, relative coordinates (x, y, z), attributes, vehicle size (width, length, height), lane, distance to white line (left white line, right white line), and type of white line. The timestamp, trajectory ID, and relative coordinates (location information) are the main data, and the rest are additional data.

[0062] The trajectory ID is assigned to the trajectory of a moving object and serves as information for identifying the moving object. The relative coordinates (x, y, z) represent the position of the moving object at each time. The attribute represents the type of moving object, for example, 0 for a passenger car, 1 for a large vehicle, 2 for a motorcycle, and 3 for unknown. The driving lane is represented by the numbers 1, 2, from left to right. The type of white line, for example, is 0 for a solid line and 1 for a dashed line.

[0063] The traffic flow data may also include absolute coordinates (latitude, longitude, altitude above sea level), the direction of travel of the moving object (angle), road alignment (road curvature, road longitudinal gradient, road lateral gradient), road coordinates (Lx, Ly, dLx, dLy), speed, acceleration (direction of travel, lateral), road width, number of lanes, road type (1: intercity expressway, urban expressway, national highway, 2: main line, merging, branch, ramp), type of lane marker (left side of vehicle, right side of vehicle), time to collision with the vehicle ahead, attributes of the vehicle ahead (passenger car, large vehicle, motorcycle, unknown), relative speed and lane of surrounding vehicles, etc. Furthermore, the traffic flow data may also include information regarding the tracking frame of the moving object based on image recognition.

[0064] Next, we will explain the screens displayed on the user terminal 4. Fig. 4 is an explanatory diagram showing the transition status of the screens displayed on the user terminal 4. Fig. 5 is an explanatory diagram showing the main menu screen displayed on the user terminal 4. Figs. 6 and 7 are explanatory diagrams showing submenu screens displayed on the user terminal 4.

[0065] A main menu screen 101 shown in Fig. 5 has a sensor installation adjustment button 102, a traffic flow data generation button 103, a traffic flow data viewing button 104, and an options button 105. When the user operates the sensor installation adjustment button 102, the screen transitions to a sub-menu screen related to sensor installation adjustment shown in Fig. 6(A). When the user operates the traffic flow data generation button 103, the screen transitions to a sub-menu screen related to traffic flow data generation shown in Fig. 6(B). When the user operates the traffic flow data viewing button 104, the screen transitions to a sub-menu screen related to traffic flow data viewing shown in Fig. 7(A). When the user operates the options button 105, the screen transitions to a sub-menu screen related to options shown in Fig. 7(B).

[0066] 6(A) is provided with a basic adjustment button 112, an alignment button 113, and an installation confirmation button 114. When the user operates the basic adjustment button 112, the screen transitions to a basic adjustment screen 201 (see FIG. 8). When the user operates the alignment button 113, the screen transitions to an alignment screen 231 (see FIG. 12). When the user operates the installation confirmation button 114, the screen transitions to an installation confirmation screen 261 (see FIG. 16).

[0067] 6(B), a submenu screen 121 related to traffic flow data generation is provided with a sensor data recording button 122 and a sensor data analysis button 123. When the user operates the sensor data recording button 122, the screen transitions to a sensor data recording screen 301 (see FIG. 21). When the user operates the sensor data analysis button 123, the screen transitions to a sensor data analysis screen 311 (see FIG. 22).

[0068] The submenu screen 131 for viewing traffic flow data shown in Fig. 7(A) has a time series display button, a scenario designation button, and a statistical information designation button. When the user operates the time series display button, the screen transitions to a time series display screen 401 (see Fig. 24). When the user operates the scenario designation button, the screen transitions to a scenario designation screen 431 (see Fig. 27). When the user operates the statistical information designation button, the screen transitions to a statistical information designation screen 461 (see Fig. 29). Furthermore, when the user performs a predetermined operation on the scenario designation screen 431 or the statistical information designation screen 461, the screen transitions to a designated event viewing screen 471 (see Fig. 30).

[0069] 7(B) has a tracking mode button 142 and an extended browsing mode button 143. When the user operates the tracking mode button 142, the screen transitions to a tracking mode screen 501 (see FIG. 32). When the user operates the extended browsing mode button 143, the screen transitions to an extended browsing mode screen 531 (see FIG. 34).

[0070] As shown in Fig. 4, the basic adjustment screen 201 (see Fig. 8) and the positioning screen 231 (see Fig. 12) are appropriately referred to as installation adjustment screens. Also, the time series display screen 401 (see Fig. 24), the scenario specification screen 431 (see Fig. 27), the statistical information specification screen 461 (see Fig. 29), the specified event viewing screen 471 (see Fig. 30), and the extended viewing mode screen 531 (see Fig. 34) are appropriately referred to as traffic flow viewing screens.

[0071] Furthermore, as shown in Fig. 8 and other figures, each screen transitioned to from each of the sub-menu screens 111, 121, 131, and 141 (see Figs. 6 and 7) has tabs 161 corresponding to each sub-menu item (basic adjustment, alignment, installation confirmation, etc.) and a menu button 162. When the user operates tab 161, the screen transitions to a screen relating to each corresponding sub-menu item. When the user operates menu button 162, the screen returns to the main menu screen 101 (see Fig. 5).

[0072] Furthermore, each screen transitioned to from each of the sub-menu screens 111, 121, 131, and 141 (see FIGS. 6 and 7) has a measurement point designation section 163, as shown in FIG. 8 etc. In the measurement point designation section 163, the user can designate a measurement point by operating a pull-down menu.

[0073] Next, the basic adjustment screen 201 displayed on the user terminal 4 will be described. Fig. 8 is an explanatory diagram showing the basic adjustment screen 201 in a state where CG images are not displayed during camera adjustment. Fig. 9 is an explanatory diagram showing the basic adjustment screen 201 in a state where CG images are displayed during camera adjustment. Fig. 10 is an explanatory diagram showing the basic adjustment screen 201 in a state where CG images are not displayed during rider adjustment. Fig. 11 is an explanatory diagram showing the basic adjustment screen 201 in a state where CG images are displayed during rider adjustment.

[0074] On the user terminal 4, when the user operates the sensor installation adjustment button 102 on the main menu screen 101 (see FIG. 5), a submenu screen 111 (see FIG. 6(A)) is displayed, and when the user operates the basic adjustment button 112, the basic adjustment screen 201 shown in FIG. 8 is displayed.

[0075] The basic adjustment screen 201 shown in Fig. 8 is in a state where CG images are not displayed (initial state) during camera adjustment. The basic adjustment screen 201 is provided with a floor plan display section 202. The floor plan display section 202 displays a plan view 203 and a side view 204 that show the installation status of the sensors (camera 1 and LIDAR 2) at the sensor installation point. The user can perform basic adjustment work while visually checking the plan view 203 and the side view 204 and confirming the installation status of camera 1 and LIDAR 2.

[0076] Plan view 203 depicts the state of cameras 1 and lidar 2 installed around the measurement area as seen from above. Side view 204 depicts the state of cameras 1 and lidar 2 installed around the measurement area as seen from the side. In this example, camera 1 and lidar 2 #1 and camera 1 and lidar 2 #2 are installed facing each other across the intersection.

[0077] Here, if the camera 1 and the LIDAR 2 are equipped with an IMU (Inertial Measurement Unit), the traffic flow measurement server 3 can obtain the actual detection direction of the LIDAR 2 based on the output information of the IMU. As a result, the traffic flow measurement server 3 displays the plan view 203 and the side view 204 so that the depicted orientations of the camera 1 and the LIDAR 2 change in conjunction with changes in the actual detection direction of the camera 1 and the LIDAR 2. On the other hand, if the camera 1 and the LIDAR 2 are not equipped with an IMU, the traffic flow measurement server 3 does not know the actual detection direction of the camera 1. Therefore, the orientations of the camera 1 and the LIDAR 2 depicted in the plan view 203 and the side view 204 differ from their actual orientations.

[0078] The basic adjustment screen 201 also has a sensor switching unit 205. The sensor switching unit 205 has a camera adjustment button 206 and a rider adjustment button 207. When the user operates the camera adjustment button 206, the camera adjustment mode is entered, and the basic adjustment screen 201 for camera adjustment shown in Fig. 8 is displayed. On the other hand, when the user operates the rider adjustment button 207, the rider adjustment mode is entered, and the basic adjustment screen 201 for rider adjustment shown in Fig. 10 is displayed.

[0079] 8, the basic adjustment screen 201 for camera adjustment has a sensor image display section 211. The sensor image display section 211 displays a camera image 212 as a sensor image (image detected by the sensor). In this example, two cameras 1 are installed, and therefore two camera images 212 captured by each camera 1 are displayed.

[0080] In the sensor image display section 211, a sensor angle operation section 213 is superimposed on the camera image 212. In the sensor angle operation section 213, an operation for changing the shooting angle (angle of view) of the camera 1 as a sensor in a specified direction can be performed, specifically, panning (horizontal direction) and tilting (vertical direction) can be performed. This allows the user to adjust the angle of the camera 1 while visually viewing the camera image 212.

[0081] 8 also has a CG image designation section 217 and a CG image display button 218. In the CG image designation section 217, the user can input the name of a measurement point and issue a search command. This loads a CG image file storing a CG image for camera adjustment related to the measurement point, and the file name of the CG image file is displayed in the CG image designation section 217. Next, when the user operates the CG image display button 218, the screen transitions to the basic adjustment screen 201 shown in FIG. 9.

[0082] 9 shows a basic adjustment screen 201 in a CG image display state when adjusting camera 1. In this case, a CG image display section 221 is provided on the basic adjustment screen 201. The CG image display section 221 displays a CG image 222 corresponding to the camera image 212 (sensor image) displayed on the sensor image display section 211. In this example, since two cameras 1 are installed, two CG images 222 corresponding to the two camera images 212 captured by each camera 1 are displayed.

[0083] Here, the CG image 222 (simulation image) is a CG reproduction of a camera image taken of the measurement area with the camera 1 adjusted to an appropriate angle, and is created in advance using CG. The CG image 222 serves as a model when adjusting the angle (angle of view) of the camera 1.

[0084] The user can visually compare the camera image 212 (actual image captured by camera 1) displayed on the sensor image display unit 211 with the CG image 222, and adjust the angle of camera 1 using the sensor angle operation unit 213 so that the two images are in a similar state, thereby setting camera 1 to the optimal angle.

[0085] The basic adjustment screen 201 shown in FIG. 10 is in a state where the CG image is not displayed during LIDAR adjustment. In this case, on the basic adjustment screen 201, a LIDAR intensity image 215 is displayed as a sensor image (image detected by the sensor) in the sensor image display section 211. In this example, two LIDARs 2 are installed, and therefore two LIDAR intensity images 215 detected by each LIDAR 2 are displayed. The LIDAR intensity image 215 is an image that represents the reflection intensity in the 3D point cloud data acquired by the LIDAR 2 as brightness.

[0086] 10, the basic adjustment screen 201 for LIDAR adjustment displays a sensor angle operation unit 213 superimposed on a LIDAR intensity image 215. The sensor angle operation unit 213 allows the user to perform an operation to change the detection angle (angle of view) of the LIDAR 2 as a sensor in a specified direction, specifically, a pan (horizontal direction) and tilt (vertical direction) operation. This allows the user to adjust the angle of the LIDAR 2 while visually observing the LIDAR intensity image 215.

[0087] In the basic adjustment screen 201 shown in FIG. 10, similarly to the basic adjustment screen 201 (see FIG. 9), the user inputs the name of a measurement point in the CG image designation section 217 and instructs a search, whereby a CG image file storing a CG image for rider adjustment relating to the measurement point is read out, and when the user then operates the CG image display button 218, the screen transitions to the basic adjustment screen 201 in a CG image display state for rider adjustment as shown in FIG. 11.

[0088] 11 shows the basic adjustment screen 201 in a CG image display state during LIDAR adjustment. In this case, on the basic adjustment screen 201, a CG image 225 corresponding to the LIDAR intensity image 215 displayed on the sensor image display unit 211 is displayed on the CG image display unit 221. In this example, two LIDARs 2 are installed, and therefore two CG images 225 corresponding to the two LIDAR intensity images 215 detected by each LIDAR 2 are displayed.

[0089] Here, the CG image 225 (simulation image) is a CG reproduction of a LIDAR intensity image when the measurement area is detected by the LIDAR 2 adjusted to an appropriate angle, and is created in advance using CG. The CG image 225 serves as a model when adjusting the angle (angle of view) of the LIDAR 2.

[0090] The user can visually compare the LIDAR intensity image 215 (actual image detected by LIDAR 2) displayed on the sensor image display unit 211 with the CG image 225, and adjust the angle of LIDAR 2 using the sensor angle operation unit 213 so that the two are in a similar state, thereby setting the LIDAR 2 to the optimal angle.

[0091] In this way, on the basic adjustment screen 201, the user can adjust the angle (angle of view) of the sensors (camera 1 and LIDAR 2) by visually viewing the camera image 212. Furthermore, the user can adjust the angle of the sensor by referring to CG images 222, 225, which are CG reproductions of the sensor image when the measurement area is detected by the sensor adjusted to the appropriate angle. This allows the user to easily adjust the installation state of the sensor when installing or replacing a new sensor. Note that although the sensor switching unit 205 is configured to switch between the camera adjustment mode and the LIDAR adjustment mode, the camera adjustment mode and the LIDAR adjustment mode may also be switched based on the user's operation of selecting a sensor (camera, LIDAR) displayed on the plan view 203 or side view 204 of the floor plan display unit 202.

[0092] Next, the alignment screen 231 displayed on the user terminal 4 will be described. Fig. 12 is an explanatory diagram showing the alignment screen 231 in an initial state. Fig. 13 is an explanatory diagram showing the alignment screen 231 when the alignment result is correct. Fig. 14 is an explanatory diagram showing the alignment screen 231 when the alignment result is an error. Fig. 15 is an explanatory diagram showing the alignment screen 231 during manual alignment.

[0093] On the user terminal 4, when the user operates the sensor installation adjustment button 102 on the main menu screen 101 (see Figure 5), a submenu screen 111 (see Figure 6(A)) is displayed, and when the user operates the alignment button 113, an alignment screen 231 shown in Figure 12 is displayed.

[0094] 12, similar to the basic adjustment screen 201 (see FIG. 8), a floor plan display section 202 is provided. The floor plan display section 202 displays a plan view 203 and a side view 204 showing the installation status of the camera 1 and the rider 2.

[0095] The alignment screen 231 also has a sensor image display section 232. The user can specify a target measurement point in the measurement point specification section 163. As a result, a camera image 233, a LIDAR intensity image 234, and a LIDAR point cloud image 235 related to the specified measurement point are displayed on the sensor image display section 232. The displayed images may be real-time images or stored images.

[0096] Furthermore, the positioning screen 231 is provided with a positioning button 237. When the user operates the positioning button 237, the process proceeds to an automatic positioning process, the positioning process is executed in the traffic flow measurement server 3, and the screen transitions to the positioning screen 231 shown in FIG. 13 when the positioning is completed.

[0097] In the registration process, the correspondence between the cameras and 3D point cloud data from two cameras 1 and two LIDARs 2 installed at two locations is estimated, and the 3D point cloud data from each LIDAR 2 is integrated based on the estimation results. At this time, the relative positional relationship of one of the two 3D point cloud data with respect to the other is corrected as necessary. Specifically, one of the two 3D point cloud data is moved or rotated with respect to the other.

[0098] On the alignment screen 231 shown in FIG. 13 when alignment is completed, the integrated LIDAR point cloud image 241 is displayed in the sensor image display section 232.

[0099] Furthermore, on the alignment screen 231 upon completion of alignment, a line 242 connecting corresponding portions in each sensor image (the two camera images 233 and the two LIDAR intensity images 234) is displayed in the sensor image display section 232. This allows the user to confirm the mutual correspondence between the sensor images.

[0100] The user visually checks whether the alignment is sufficient by looking at the integrated LIDAR point cloud image 241. If the alignment is insufficient, problems such as double images of moving objects appearing in the integrated LIDAR point cloud image 241 will occur, as shown in Figure 14.

[0101] A manual alignment confirmation section 243 is displayed on the alignment screen 231 when alignment is completed, as shown in Fig. 14. The manual alignment confirmation section 243 is provided with a Yes button 244 and a No button 245. If a problem occurs in the integrated LIDAR point cloud image 241, the user operates the Yes button 244. This causes the screen to transition to the alignment screen 231 at the time of manual alignment, as shown in Fig. 15.

[0102] A manual alignment screen 231 shown in FIG. 15 displays a manual alignment operation section 251 and a re-alignment button 252.

[0103] The manual alignment operation unit 251 is operated by the user to correct the relative positional relationship of the 3D point cloud data from the two LIDARs 2. The manual alignment operation unit 251 is provided with a movement operation unit 253 and a rotation operation unit 254. The movement operation unit 253 allows one of the 3D point cloud data from the two LIDARs 2 to be moved in a specified direction (up, down, left, right, forward, backward) relative to the other. The rotation operation unit 254 allows one of the 3D point cloud data from the two LIDARs 2 to be rotated in a specified direction (roll, pitch, yaw) relative to the other. In this case, it is preferable to make it possible to operate the movement or rotation alignment by selecting one of the LIDAR intensity images 234.

[0104] The user visually checks the integrated LIDAR point cloud image 241 and performs the necessary operations on the manual alignment operation unit 251. The display area for the LIDAR point cloud image 241 has a 3D viewer function, and by operating to move the viewpoint, the LIDAR point cloud image 241 can be displayed from any viewpoint. This allows the user to confirm whether the relative positional deviation of the 3D point cloud data from the two LIDARs 2 has been sufficiently improved by the manual alignment operation.

[0105] When the user confirms that the relative positional deviation of the 3D point cloud data from the two LIDARs 2 has been sufficiently improved, he or she operates the re-alignment button 252. This causes the traffic flow measurement server 3 to execute the alignment process again, and the screen transitions to the alignment screen 231 shown in FIG. 13, which is displayed when alignment is complete.

[0106] In this way, the alignment screen 231 displays the integrated result after correcting the misalignment between the two 3D point cloud data from the two LIDARs 2 installed at two locations, allowing the user to easily confirm that the alignment of the 3D point cloud data has been performed appropriately. Furthermore, if the misalignment between the two 3D point cloud data is too large to allow for proper automatic alignment, the user can manually correct the misalignment between the two 3D point cloud data in response to an operation by the user, and by performing the alignment process again, the alignment of the 3D point cloud data can be properly completed.

[0107] Next, the installation confirmation screen 261 displayed on the user terminal 4 will be described. Fig. 16 is an explanatory diagram showing the installation confirmation screen 261 in an initial state. Fig. 17 is an explanatory diagram showing the installation confirmation screen 261 when a virtual object is selected. Fig. 18 is an explanatory diagram showing the installation confirmation screen 261 when a virtual object is superimposed and displayed. Fig. 19 is an explanatory diagram showing the installation confirmation screen 261 in an error state when a virtual object is superimposed and displayed.

[0108] On the user terminal 4, when the user operates the sensor installation adjustment button 102 on the main menu screen 101 (see FIG. 5), a submenu screen 111 (see FIG. 6(A)) is displayed, and when the user operates the installation confirmation button 114, an installation confirmation screen 261 shown in FIG. 16 is displayed.

[0109] 16 includes a sensor image display section 262. The sensor image display section 262 displays a camera image 263, a LIDAR intensity image 264, and a LIDAR point cloud image 265.

[0110] 16 shows an installation confirmation screen 261 in which sensors (camera 1 and lidar 2) are installed at two points on either side of an intersection. In this case, camera images 263 taken by the two cameras 1 installed at the two points, lidar intensity images 264 taken by the two lidar 2 installed at the two points, and a lidar point cloud image 265 based on 3D point cloud data obtained by integrating the 3D point cloud data from the two lidars 2 are displayed.

[0111] Furthermore, the installation confirmation screen 261 is provided with a virtual object designation section 267. The virtual object designation section 267 is provided with buttons 268 for designating a large bus, a motorbike, a pedestrian, a passenger car, and a trailer as virtual moving objects.

[0112] 17, when the user operates button 268 to specify a virtual object of a moving body, an image 271 of the virtual object of the specified moving body appears on the lidar point cloud image 265. At this time, the traffic flow measurement server 3 performs a process of placing the virtual object of the specified moving body in a three-dimensional space including the three-dimensional point cloud data, and generates a lidar point cloud image 265 in which the three-dimensional space including the virtual object of the moving body and the point cloud of the three-dimensional point cloud data is viewed from the specified viewpoint.

[0113] The display area of ​​the lidar point cloud image 265 has a 3D viewer function, and the user can display the lidar point cloud image 265 from any viewpoint by performing an operation to move the viewpoint (free viewpoint display). In addition, the position and angle of the virtual object of the moving body can be adjusted by operating the image 271 of the virtual object that appears on the lidar point cloud image 265. This allows the virtual object of the moving body to be placed in an appropriate state for the three-dimensional point cloud data. Specifically, by adjusting the position and angle of the virtual object while operating the 3D viewer, the image 271 of the virtual object of the moving body is placed on the road in an appropriate state.

[0114] The installation confirmation screen 261 also has a virtual object superimposition button 273, an OK button 274, and a reinstallation setting button 275. When the user confirms that the positional relationship between the virtual object of the moving body and the 3D point cloud data has been appropriately adjusted based on the arrangement state of the virtual object image 271 on the LIDAR point cloud image 265, the user operates the virtual object superimposition button 273.

[0115] 18 , when the user operates virtual object superimposition button 273, a virtual object image 277 corresponding to virtual object image 271 of the moving object arranged on LIDAR point cloud image 265 is superimposed on camera image 263 in sensor image display unit 262, and a similar virtual object image 278 is superimposed on LIDAR intensity image 264. The user visually checks whether virtual object image 277 of the moving object is displayed appropriately on camera image 263, and also checks whether virtual object image 278 of the moving object is displayed appropriately on LIDAR intensity image 264.

[0116] At this time, traffic flow measurement server 3 superimposes and displays virtual object images 277 and 278 of the moving body on camera image 263 and LIDAR intensity image 264, respectively, based on the positional relationship between the 3D point cloud data and the virtual object and the correspondence relationship between the camera image acquired by the alignment process and the 3D point cloud data. Furthermore, in camera image 263 and LIDAR intensity image 264, virtual object images 277 and 278 of the moving body are displayed in a state where they have been deformed into shapes corresponding to camera image 263 and LIDAR intensity image 264, respectively.

[0117] Here, if the user confirms that the virtual object images 277, 278 of the moving body are not displayed appropriately on the camera image 263 or the LIDAR intensity image 264, respectively, the user can again perform the operation to adjust the position and angle of the virtual object image 271 of the moving body on the LIDAR point cloud image 265. Next, the user can again operate the virtual object superimposition button 273 to check whether the virtual object images 277, 278 of the moving body are displayed appropriately on the camera image 263 and the LIDAR intensity image 264. Here, if the user confirms that the virtual object images 277, 278 of the moving body are displayed appropriately on the camera image 263 and the LIDAR intensity image 264, the user operates the OK button 274.

[0118] In this example, by again operating the virtual object overlay button 273, the images 277 and 278 of the virtual objects on the camera image 263 and the LIDAR intensity image 264 are updated, but the images 277 and 278 of the virtual objects on the camera image 263 and the LIDAR intensity image 264 may also be updated in real time in accordance with adjustments to the position and angle of the virtual object image 271 on the LIDAR point cloud image 265.

[0119] Here, the traffic flow measurement server 3 determines for each camera image 263 whether or not the image 277 of the virtual object of the moving body extends beyond the display range of the camera image 263, and also determines whether or not the image 278 of the virtual object of the moving body extends beyond the display range of the lidar intensity image 264.

[0120] 19, for example, if the image 277 of the virtual object of the moving body protrudes from the display range of the camera image 263, the display frame of the camera image 263 is highlighted to notify the user. Specifically, a frame image 281 of a predetermined color (for example, red) is displayed in the display frame of the camera image 263. Note that if the image 278 of the virtual object of the moving body protrudes from the display range of the LIDAR intensity image 264, the display frame of the LIDAR intensity image 264 is highlighted, similar to the case of the camera image 263.

[0121] In this case, the user can again perform the operation to adjust the position and angle of the image 271 of the virtual object of the moving body on the LIDAR point cloud image 265, but if readjustment on the LIDAR point cloud image 265 is not sufficient, the user operates the reset button 275. This returns the user to the basic adjustment process in the sensor installation adjustment, and transitions to the basic adjustment screen 201 (see FIG. 8).

[0122] In this example, when the virtual object images 277, 278 of the moving body extend beyond the display range of the sensor images (camera image 263 and LIDAR intensity image 264), the display frame of the sensor image is displayed in a predetermined color (for example, red) to highlight the target sensor image, but highlighting of the sensor image is not limited to changing the color of the display frame in this way. For example, highlighting of the sensor image may be achieved by flashing the display frame of the sensor image or by changing the line type (dashed line, dotted line, etc.) of the display frame of the sensor image.

[0123] In this way, on the installation confirmation screen 261, images 277 and 278 of the virtual object of the moving body are superimposed on the sensor images (camera image 263, LIDAR intensity image 264) as the virtual object of the moving body is placed in the 3D space including the 3D point cloud data. This allows the user to easily check whether the sensors are set to a state where they can properly detect a moving body that appears in the measurement area when adjusting the installation state of the sensors (camera 1, LIDAR 2).

[0124] In addition, if the images 277, 278 of the virtual object of the moving body extend beyond the display range of the sensor image (camera image 263, LIDAR intensity image 264), the level of warning given to the user may be changed based on the degree of extension and the priority of the sensor image in which the extension is detected.

[0125] Next, a description will be given of another example of the installation confirmation screen 261 displayed on the user terminal 4. Fig. 20 is an explanatory diagram showing another example of the installation confirmation screen 261.

[0126] In the example shown in Figure 16, multiple sensors (camera 1 and lidar 2) are installed so as to detect moving objects in the measurement area from opposite sides. Specifically, sensors are installed at two points facing each other across an intersection (measurement area), and the sensors at the two points detect the same location from different directions.

[0127] On the other hand, in this example, multiple sensors (camera 1 and lidar 2) are installed so that their measurement areas are adjacent and partially overlap. Specifically, the measurement area is a wide intersection and its surrounding area, and sensors are installed at four points around the intersection. The sensors at each point mainly detect the center of the intersection, and although their measurement areas are adjacent and partially overlap, they are installed so that each of the multiple roads connected to the intersection is included in the detection area, so the detection areas of the sensors at each point are significantly different.

[0128] In this example, the installation confirmation screen 261 displays four camera images 263 and one LIDAR point cloud image 265 in the sensor image display section 262. The four camera images 263 were taken by four cameras 1 installed at four locations. The one LIDAR point cloud image 265 was generated from 3D point cloud data that integrated the 3D point cloud data from four LIDARs 2 installed at the four locations.

[0129] In this example, as in the example shown in Figure 17, when the user selects a virtual object of a moving body in the virtual object designation unit 267, an image 271 of the virtual object of the specified moving body appears on the rider point cloud image 265, and when the user operates the virtual object superimposition button 273, an image 277 of the virtual object of the moving body is superimposed on the camera image 263.

[0130] 19, in this example, if the image 277 of the virtual object of the moving body extends beyond the display range of the camera image 263, the camera image 263 is highlighted. If the problem cannot be resolved by readjusting the position and angle of the virtual object of the moving body in the LIDAR point cloud image 265, operating the reset setting button 275 returns to the basic adjustment process in the sensor installation adjustment.

[0131] In this example, even if the sensors (camera 1 and lidar 2) at each location are installed so that their measurement areas are adjacent and partially overlap, but the detection areas are significantly offset, the user can easily check whether the sensors are set up to properly detect moving objects that appear within the measurement area.

[0132] Next, a description will be given of the sensor data recording screen 301 displayed on the user terminal 4. FIG.

[0133] On the user terminal 4, when the user operates the traffic flow data generation button 103 on the main menu screen 101 (see Figure 5), a submenu screen 121 (see Figure 6(B)) is displayed, and when the user operates the sensor data recording button 122, the sensor data recording screen 301 shown in Figure 21 is displayed.

[0134] The sensor data recording screen 301 has a sensor image display section 302. The sensor image display section 302 displays a camera image 303 and a LIDAR intensity image 304. In this example, since camera 1 and LIDAR 2 are installed at two locations, two camera images 303, which are the detection results of each camera 1, and two LIDAR intensity images 304, which are the detection results of each LIDAR 2, are displayed.

[0135] On the sensor data recording screen 301, the user can specify a measurement point to be the target of sensor data recording in the measurement point specification unit 163. As a result, a camera image 303 and a LIDAR intensity image 304 taken by a camera and LIDAR installed at the specified measurement point are displayed in the sensor image display unit 302. Note that in the measurement point specification unit 163, the user can select a measurement point from pre-registered measurement points by operating a pull-down menu, but in the case of an unregistered measurement point, the user can register the measurement area by inputting the name of the measurement point in the measurement point specification unit 163.

[0136] The sensor data recording screen 301 also has a start recording button 305 and an end recording button 306. When the user operates the start recording button 305, the traffic flow measurement server 3 starts the sensor data recording process. In the sensor data recording process, the camera images transmitted from the camera 1 are stored in the storage unit 12. In addition, the lidar point cloud data transmitted from the lidar 2 is stored in the storage unit 12. When the user operates the end recording button 306, the traffic flow measurement server 3 ends the recording process of the traffic flow data.

[0137] The sensor data recording process may be performed until a measurement time specified by the user has elapsed by the user setting a timer, or may be performed from a start time to an end time specified by the user by the user setting a schedule in advance.

[0138] Next, we will explain the sensor data analysis screen 311 displayed on the user terminal 4. Fig. 22 is an explanatory diagram showing the sensor data analysis screen 311 in an initial state. Fig. 23 is an explanatory diagram showing the sensor data analysis screen 311 when the sensor data analysis process has started.

[0139] On the user terminal 4, when the user operates the traffic flow data generation button 103 on the main menu screen 101 (see Figure 5), a submenu screen 121 (see Figure 6(B)) is displayed, and when the user operates the sensor data analysis button 123, the sensor data analysis screen 311 shown in Figure 22 is displayed.

[0140] 22 includes a sensor image display section 312. A camera image 313 and a LIDAR intensity image 314 are displayed on the sensor image display section 312. In this example, since camera 1 and LIDAR 2 are installed at two locations, two camera images 313, which are the detection results of each camera 1, and two LIDAR intensity images 314, which are generated from the 3D point cloud data, which are the detection results of each LIDAR 2, are displayed.

[0141] On the sensor data analysis screen 311, the user can specify a measurement point to be the target of sensor data analysis in the measurement point specification section 163. As a result, a camera image and 3D point cloud data related to the specified measurement point are read out, and a camera image 313 and a LIDAR intensity image 314 are displayed on the sensor image display section 312.

[0142] The sensor data analysis screen 311 is also provided with an analysis start button 316 and an analysis end button 317. When the user operates the analysis start button 316, the traffic flow measurement server 3 starts the sensor data analysis process (traffic flow analysis process).

[0143] The sensor data analysis process reads out the camera images and LIDAR point cloud data stored in the storage unit 12, and performs processes such as detecting moving objects from the camera images and LIDAR point cloud data, and extracting events (traffic accidents, etc.) that correspond to a predetermined scenario from the traffic flow data. When the user operates the analysis end button 317, the traffic flow measurement server 3 ends the sensor data analysis process.

[0144] 23, when the sensor data analysis process starts, a LIDAR point cloud image 315 is displayed on the sensor image display unit 312 in addition to a camera image 313 and a LIDAR intensity image 314. The camera image 313, the LIDAR intensity image 314, and the LIDAR point cloud image 315 can be displayed as moving images. The LIDAR point cloud image 315 is generated from 3D point cloud data that combines the 3D point cloud data of each LIDAR 2 installed at two locations.

[0145] At this time, on the sensor image display unit 312, a tracking frame of the moving object detected from the camera image 313 is displayed on the camera image 313. In addition, a tracking frame of the moving object detected from the 3D point cloud data is displayed on the LIDAR intensity image 314. In addition, a tracking frame of the moving object detected from the 3D point cloud data is displayed on the LIDAR point cloud image 315.

[0146] Next, we will explain the time series display screen 401 displayed on the user terminal 4. Figures 24 and 25 are explanatory diagrams showing the time series display screen 401. Figure 26 is an explanatory diagram showing a trajectory line 407, a speed line 408, and an acceleration line 409 displayed on the time series display screen 401.

[0147] On the user terminal 4, when the user operates the traffic flow data viewing button 104 on the main menu screen 101 (see Figure 5), a submenu screen 131 (see Figure 7(A)) is displayed, and when the user operates the time series display button 132, the time series display screen 401 shown in Figure 24 is displayed.

[0148] The time series display screen 401 has a sensor image display section 402. The sensor image display section 402 displays a camera image 403, a LIDAR intensity image 404, and a LIDAR point cloud image 405. The display area for the LIDAR point cloud image 405 has a 3D viewer function, and the LIDAR point cloud image 405 can be displayed from any viewpoint by the user moving the viewpoint. Figures 24 and 25 show examples of when the viewpoint of the LIDAR point cloud image 405 is changed.

[0149] The sensor image display unit 402 superimposes a trajectory line 411, a speed line 412, and an acceleration line 413 on the camera image 403, the LIDAR intensity image 404, and the LIDAR point cloud image 405 as a behavior image that visualizes time-series data that represents the behavior (change in state) of the moving object. The trajectory line 411 (trajectory image) visualizes time-series data that represents changes in the position of the moving object. The speed line 412 (velocity image) visualizes time-series data that represents changes in the speed of the moving object. The acceleration line 413 (acceleration image) visualizes time-series data that represents changes in the acceleration of the moving object.

[0150] 26, trajectory points 414 representing the position of the moving object at the display time (currently displayed time) are drawn on trajectory line 411. Speed ​​points 415 representing the speed of the moving object at the display time are drawn on speed line 412. Acceleration points 416 representing the acceleration of the moving object at the display time are drawn on acceleration line 413. The display positions of trajectory points 414, speed points 415, and acceleration points 416 change as the display time progresses.

[0151] Furthermore, if trajectory point 414, which indicates the position of the moving object at the display time, is taken as the origin and the direction of travel is taken as the first coordinate axis, second and third coordinate axes perpendicular to the first coordinate axis represent the magnitude (absolute value) of velocity and the magnitude (absolute value) of acceleration, respectively. The distance in the velocity axis direction from trajectory point 414 as the origin to velocity point 415 represents the magnitude (absolute value) of velocity. The distance in the acceleration axis direction from trajectory point 414 as the origin to acceleration point 416 represents the magnitude (absolute value) of acceleration.

[0152] Therefore, a trajectory line 411 connects trajectory points 414 at each time and represents changes in the position of the moving object. A speed line 412 connects speed points 415 at each time and represents changes in the speed of the moving object. An acceleration line 413 connects acceleration points 416 at each time and represents changes in the acceleration of the moving object.

[0153] 24 and 25, in the sensor image display unit 402, an ID label 417 (label image) with a moving object ID written thereon is displayed near the trajectory line 411. A speed label 418 (label image) with a speed (absolute value) written thereon is displayed near the speed line 412. An acceleration label 419 (label image) with an acceleration (absolute value) written thereon is displayed near the acceleration line 413.

[0154] Furthermore, in the sensor image display unit 402, a tracking frame of a moving object detected from the camera image 403 is displayed on the camera image 403. Furthermore, a tracking frame of a moving object detected from the 3D point cloud data is displayed on the LIDAR intensity image 404. Furthermore, a tracking frame of a moving object detected from the 3D point cloud data is displayed on the LIDAR point cloud image 405.

[0155] The time series display screen 401 also has a next frame button 421 and a previous frame button 422. When the user operates the next frame button 421, the camera image 403, the LIDAR intensity image 404, and the LIDAR point cloud image 405 switch to the next frame, i.e., the image from the next time. When the user operates the previous frame button 422, the camera image 403, the LIDAR intensity image 404, and the LIDAR point cloud image 405 switch to the previous frame, i.e., the image from the previous time.

[0156] The method of expressing the trajectory (position), speed, and acceleration of a moving object is not limited to the example shown in the figure. For example, the speed and acceleration can be expressed by the attributes of the trajectory line. Specifically, the color density and thickness of the trajectory line may represent the speed and acceleration.

[0157] In this way, the time series display screen 401 visualizes and displays changes in the state of the moving object over time. Specifically, behavior images that visualize the position, speed, and acceleration, specifically, a trajectory line 411, a speed line 412, and an acceleration line 413, are superimposed on sensor images (camera image 403, LIDAR intensity image 404, and LIDAR point cloud image 405). This allows the user to intuitively grasp changes in the state (position, speed, and acceleration) of the moving object. Note that, on a display selection screen (not shown), any image may be selected from the types of behavior images (trajectory line 411, speed line 412, acceleration line 413) and the types of label images (ID label 417, speed label 418, acceleration label 419) to be displayed.

[0158] Next, a description will be given of the scenario specification screen 431 displayed on the user terminal 4. Fig. 27 is an explanatory diagram showing the scenario specification screen 431. Fig. 28 is an explanatory diagram showing the scenario specification screen 431 in an extraction condition addition state.

[0159] On the user terminal 4, when the user operates the traffic flow data viewing button 104 on the main menu screen 101 (see Figure 5), a submenu screen 131 (see Figure 7(A)) is displayed, and when the user operates the scenario specification button 133, the scenario specification screen 431 shown in Figure 27 is displayed.

[0160] The scenario specification screen 431 is provided with an extraction condition selection section 432 and a summary diagram display section 433. In the extraction condition selection section 432, the user can select a scenario (event type) as an extraction condition (narrowing condition) by operating a pull-down menu. In this example, the user can select scenarios such as rear-end collision, right-turn collision, left-turn collision, wrong-way driving, and tailgating. When the user selects a scenario, a summary diagram 434 related to the selected scenario is displayed in the summary diagram display section 433. The summary diagram 434 specifically shows the situation of the scenario.

[0161] If the selected scenario has multiple patterns, a summary diagram 434 for each pattern is displayed. In the example shown in FIG. 27, the first pattern is a collision between a right-turning vehicle and a straight-moving vehicle, and the second pattern is a collision between a right-turning vehicle and a straight-moving motorcycle. The user can select a pattern by operating the summary diagram 434.

[0162] The scenario specification screen 431 also has an extraction display button 436. When the user operates the extraction display button 436 after selecting a scenario as an extraction condition in the extraction condition selection section 432, extraction processing is executed and the screen transitions to a specified event viewing screen 471 (see FIG. 30) that displays the extraction results. In the extraction processing, events that correspond to the scenario selected by the user are extracted from the events (traffic accidents, etc.) detected by traffic flow analysis (event detection).

[0163] Here, on the scenario specification screen 431, when the user selects a scenario as an extraction condition in the extraction condition selection section 432, an extraction condition addition specification section 441 (dialog box) is displayed. The extraction condition addition specification section 441 is provided with a Yes button 442 and a No button 443. When the user operates the Yes button 442, the screen transitions to the scenario specification screen 431 in an extraction condition addition state shown in FIG.

[0164] 28, in addition to an extraction condition selection section 432 and a summary diagram display section 433 related to the original extraction conditions, an extraction condition selection section 445 and a summary diagram display section 446 related to the additional extraction conditions are displayed. This allows the events to be extracted to be narrowed down by combining scenarios.

[0165] In this way, on the scenario specification screen 431, the user can specify a scenario (event type) that he or she is interested in, and thereby events (traffic accidents, etc.) that correspond to the scenario can be extracted.

[0166] The scenario may be, but is not limited to, a traffic accident such as a rear-end collision, a right-turn collision, or a left-turn collision, or a violation of traffic rules or dangerous driving other than a traffic accident such as wrong-way driving or tailgating. The content of the scenario may also be set by the user.

[0167] Next, a description will be given of the statistical information specification screen 461 displayed on the user terminal 4. FIG.

[0168] On the user terminal 4, when the user operates the traffic flow data viewing button 104 on the main menu screen 101 (see Figure 5), a submenu screen 131 (see Figure 7(A)) is displayed, and when the user operates the statistical information specification button 134, the statistical information specification screen 461 shown in Figure 29 is displayed.

[0169] The statistical information specification screen 461 is provided with a first statistical information display section 462 (graph display section) and a second statistical information display section 463 (summary display section). In the first statistical information display section 462, the number of events (frequency) corresponding to each scenario is displayed as a bar graph for each scenario as statistical information. In the second statistical information display section 463, the number of events (frequency) corresponding to a combination of scenarios is displayed as a summary table as statistical information.

[0170] In the first statistical information display section 462, the user can select one scenario by manipulating the bar graph for each scenario. In the second statistical information display section 463, the user can select a combination of scenarios by manipulating one cell in the summary table.

[0171] The statistical information specification screen 461 is also provided with an extraction display button 464. When the user selects one scenario in the first statistical information display section 462 and then operates the extraction display button 464, a process of extracting an event that corresponds to the selected scenario is performed, and the screen transitions to a specified event viewing screen 471 (see FIG. 30) that displays the extraction results. When the user selects a combination of scenarios in the second statistical information display section 463 and then operates the extraction display button 464, a process of extracting an event that corresponds to the selected combination of scenarios is performed, and the screen transitions to a specified event viewing screen 471 (see FIG. 30) that displays the extraction results.

[0172] In this way, on the statistical information specification screen 461, the user can check the status (frequency) of events that correspond to the scenario using statistical information (graphs and summary tables), then select the scenario they wish to view from the statistical information and extract events (traffic accidents, etc.) that correspond to that scenario.

[0173] In addition, when the user selects a scenario by operating a pull-down menu, as in the extraction condition selection sections 432, 445 on the scenario specification screen 431 shown in Figures 27 and 28, the statistical information (graphs and summary tables) in the statistical information display sections 462, 463 on the statistical information specification screen 461 shown in Figure 29 may be displayed in a state limited to the scenario selected by the user.

[0174] Next, a description will be given of the designated event viewing screen 471 displayed on the user terminal 4. Figures 30 and 31 are explanatory diagrams showing the designated event viewing screen 471.

[0175] On the user terminal 4, when the user specifies a scenario on the scenario specification screen 431 (see Figures 27 and 28) or the statistical information specification screen 461 (see Figure 29) and instructs extraction and display, the specified event viewing screen 471 shown in Figure 30 is displayed.

[0176] The specified event viewing screen 471 is provided with an entire image display section 472, a detailed image display section 473, a first detailing button 474, and a second detailing button 475.

[0177] A LIDAR point cloud image 476 showing the overall situation of the event is displayed as an overall image in the overall image display section 472. The LIDAR point cloud image 476 is generated from 3D point cloud data acquired by the LIDAR 2 with a viewpoint set above the measurement area.

[0178] In the whole image display section 472, moving objects related to an event corresponding to a scenario specified by the user on the scenario specification screen 431 (see FIGS. 27 and 28) or the statistical information specification screen 461 (see FIG. 29) are highlighted. In the example shown in FIG. 30, tracking frames are displayed on two vehicles related to a traffic accident (right-turn collision) as a specific event.

[0179] The detailed image display section 473 displays, as detailed images, enlarged lidar point cloud images 477 and 478 so that the details of the events corresponding to the scenario specified by the user can be grasped.

[0180] Here, when the user operates the first detailing button 474, a lidar point cloud image 477 from the driver's viewpoint is displayed as a detailed image from a first viewpoint. Here, the driver is the person who drives a vehicle as a moving body related to the event of interest. Furthermore, when the user operates the second detailing button 475, a lidar point cloud image 478 (orthoimage) with a viewpoint set above the measurement area is displayed as a detailed image from a second viewpoint.

[0181] In this way, on the specified event viewing screen 471, by specifying a scenario (event type) that the user is interested in on the scenario specification screen 431 (see FIGS. 27 and 28) or the statistical information specification screen 461 (see FIG. 29), it is possible to view sensor images (lidar point cloud images 477, 478) that show events (traffic accidents, etc.) that correspond to that scenario. This allows the user to limit the view to a specific scenario and check in detail the situation when an event that corresponds to that scenario occurred.

[0182] In this example, the user can select either rider point cloud image 476, in which the viewpoint is set to the driver, or rider point cloud image 477, in which the viewpoint is set to the sky above the measurement area, but the display frame for the rider point cloud image may also have the function of a 3D viewer, allowing the user to display a rider point cloud image from any viewpoint.

[0183] In addition, LIDAR point cloud images 476, 477 can be generated from any viewpoint from the 3D point cloud data from LIDAR 2, but images from any viewpoint can also be generated by using multi-view stereo technology to generate dense point clouds from multiple camera images from multiple cameras 1.

[0184] Next, we will explain the tracking mode screen 501 displayed on the user terminal 4. Fig. 32 is an explanatory diagram showing the tracking mode screen 501 in the multiple location installation mode. Fig. 33 is an explanatory diagram showing the tracking mode screen 501 in the single location installation mode.

[0185] On the user terminal 4, when the user operates the tracking mode button 142 on the submenu screen 141 (see Figure 7(B)), which is displayed when the user operates the options button 105 on the main menu screen 101 (see Figure 5), the tracking mode screen 501 shown in Figure 32 is displayed.

[0186] The tracking mode screen 501 has a mode selection section 502. The mode selection section 502 has a multi-location installation mode button 503 and a single-location installation mode button 504. When the user operates the multi-location installation mode button 503, the tracking mode screen 501 for the multi-location installation mode shown in FIG. 32 is displayed. When the user operates the single-location installation mode button 504, the screen transitions to the tracking mode screen 501 for the single-location installation mode shown in FIG. 33. Here, the multi-location installation mode is when camera 1 and lidar 2 are installed at multiple locations targeting a common measurement area. The single-location installation mode is when camera 1 and lidar 2 are installed at a single location.

[0187] The tracking mode screen 501 has a moving object image display section 505. The moving object image display section 505 displays an image 506 of a moving object detected from a camera image and an image 507 of a moving object detected from 3D point cloud data. The image 506 of a moving object detected from a camera image is an image region including the moving object extracted from the camera image. The image 507 of a moving object detected from the 3D point cloud data is an image region including the moving object extracted from a LIDAR point cloud image generated from the 3D point cloud data. Note that if the moving object image 507 includes multiple moving objects, it is preferable to draw a frame image surrounding the target moving object on the image 507.

[0188] 32, an image 506 of a moving object detected from camera images is displayed for each camera 1. In this example, two cameras 1 are installed, so two images 506 of the moving object are displayed. Also, since the moving object is detected from 3D point cloud data that integrates multiple 3D point cloud data from multiple lidars 2 installed at multiple locations, one image 507 of the moving object detected from the 3D point cloud data is displayed.

[0189] On the other hand, the tracking mode screen 501 in the single location installation mode shown in FIG. 33 displays one image 506 of a moving object detected from a camera image and one image 507 of a moving object detected from 3D point cloud data.

[0190] Furthermore, the moving object ID assigned to the moving object when it is detected from the camera image and the moving object ID assigned to the moving object when it is detected from the 3D point cloud data are displayed on the moving object image display unit 505. Since the detection of the moving object from each camera image and 3D point cloud data and the assignment of the moving object ID are performed individually, the moving object IDs corresponding to each camera 1 and lidar 2 are different even for the same moving object.

[0191] The tracking mode screen 501 also has a camera priority button 511, a lidar priority button 512, and a settings button 513. When the user operates the camera priority button 511, the traffic flow measurement server 3 changes the IDs of moving objects by prioritizing the IDs assigned to moving objects detected in camera images. When the user operates the lidar priority button 512, the traffic flow measurement server 3 changes the IDs of moving objects by prioritizing the IDs assigned to moving objects detected in lidar point cloud data. Note that camera images and lidar point cloud data have advantages and disadvantages depending on the detection scene, such as the conditions of the measurement area and the weather. For example, the user may specify that a sensor that is expected to have higher accuracy in moving object detection be prioritized.

[0192] When the moving object ID is changed, the moving object image display unit 505 updates the moving object IDs assigned to the moving objects detected in the camera image and the LIDAR point cloud image, and the same moving object ID is displayed for the same moving object. Here, when the user confirms that the moving object ID has been changed appropriately, he or she operates the setting button 513. This confirms the moving object ID.

[0193] In this way, on the tracking mode screen 501, the user can select the sensor to be prioritized when performing the process of changing the IDs assigned to moving objects detected from multiple detection results (camera images, 3D point cloud data) by multiple sensors (camera 1, lidar 2) so that a common moving object ID is assigned to the same moving object.

[0194] Next, a description will be given of the extended browsing mode screen 531 displayed on the user terminal 4. Fig. 34 is an explanatory diagram showing the extended browsing mode screen 531 in the viewer mode. Fig. 35 is an explanatory diagram showing the extended browsing mode screen 531 in the risk determination mode.

[0195] On the user terminal 4, when the user operates the option button 105 on the main menu screen 101 (see FIG. 5), the submenu screen 141 (see FIG. 7(B)) is displayed, and when the user operates the extended viewing mode button 143, the extended viewing mode screen 531 shown in FIG. 34 is displayed.

[0196] The extended viewing mode screen 531 has a sensor image display section 532. The sensor image display section 532 displays a camera image 533 and a LIDAR point cloud image 534. In this example, cameras 1 are installed at two locations, and therefore two camera images 533 captured by each camera 1 are displayed. The LIDAR point cloud image 534 was generated by setting a viewpoint above the measurement area based on 3D point cloud data that combines 3D point cloud data captured by LIDARs 2 installed at two locations.

[0197] The extended viewing mode screen 531 is also provided with a mode designation section 541, a road component designation section 542, a traveling object designation section 543, and an automatic driving designation section 544.

[0198] Mode designation section 541 is provided with viewer button 551 and danger judgment button 552. When the user operates viewer button 551, extended viewing mode screen 531 (see FIG. 34) for viewer mode is displayed. When the user operates danger judgment button 552, extended viewing mode screen 531 (see FIG. 35) for danger judgment mode is displayed.

[0199] The road component designation unit 542 has a button 553 for selecting road components (land features and road surface marks). In this example, by operating the button 553, a white line, a stop line, a curb, a crosswalk, a guardrail, and a sidewalk can be selected as road accessories. The selected road components are highlighted on the camera image 533 and the LIDAR point cloud image 534. Specifically, an area image 561 (ancillary image) drawn in a predetermined color or pattern is transparently superimposed on the area of ​​the target road component in the camera image 533 and the LIDAR point cloud image 534. In this example, the areas of the stop line, the crosswalk, and the sidewalk are each highlighted. The area images 561 of the road components are drawn in a color or pattern set for each type of road component. For example, the area image 561 of the crosswalk is drawn in blue, and the area image 561 of the sidewalk is drawn in red. This allows the user to easily distinguish the types of road components. Note that multiple road components can be selected.

[0200] The moving object designation unit 543 is provided with a button 554 for selecting a moving object (moving body). In this example, by operating the button 554, a moving object can be selected from a passenger car, a truck, a motorcycle, a bicycle, a bus, and a pedestrian. The selected moving object is highlighted on the rider point cloud image 534. Specifically, a region image 562 (ancillary image) drawn in a predetermined color or pattern is transparently superimposed on the region of the target moving object in the rider point cloud image 534. The region image 562 of the moving object is drawn in a color or pattern set for each type of moving object. For example, the region image 562 of a passenger car is drawn in light blue, and the region image 562 of a truck is drawn in yellow. This allows the user to easily identify the type of moving object. Note that multiple moving objects can be selected.

[0201] The autonomous driving designation unit 544 is provided with buttons 555, 556 for selecting whether or not the vehicle is autonomous. When the user operates the on button 555, the autonomous driving vehicle is highlighted on the lidar point cloud image 534, and an autonomous driving label 563 (accessory image) with the words "autonomous driving" written on it is displayed. When the user operates the off button 556, the autonomous driving vehicle is not highlighted on the lidar point cloud image 534.

[0202] Furthermore, on the extended viewing mode screen 531, when a road component is selected on the lidar point cloud image 534, specifically, when an area image 561 of the road component or an area image 562 of the traveling object superimposed on the lidar point cloud image 534 is operated, a positional relationship label 564 (ancillary image) containing information regarding the positional relationship between the traveling object and the road component is displayed. In the example shown in Fig. 34, when the user selects a truck and a crosswalk on the lidar point cloud image 534, a positional relationship label 564 containing the distance between the truck and the crosswalk is displayed.

[0203] 35, the extended viewing mode screen of the risk determination mode is provided with a risk level display section 565. The risk level related to the traffic environment at the target point is displayed in the risk level display section 565. At this time, the traffic flow measurement server 3 determines the risk level related to the traffic environment at the target point based on information related to the traffic environment at the target point, specifically, the positional relationship between the moving body (traveling object) and road components.

[0204] In this way, on the extended viewing mode screen 531, the area of ​​the moving object or road component specified by the user is highlighted in the LIDAR point cloud image 534, allowing the user to easily grasp the relative positional relationship between the moving object and the road component. Also, on the extended viewing mode screen 531, information (distance, etc.) regarding the positional relationship between the moving object and the road component and information regarding the degree of danger regarding the traffic environment at the target point are displayed, allowing the user to easily recognize the danger of the moving object. This makes it possible to consider measures necessary to reduce traffic accidents by improving the road structure, such as installing guardrails at high-risk points.

[0205] Next, the procedure for processing related to sensor installation adjustment performed by the traffic flow measurement server 3 will be described. Figure 36 is a flow diagram showing the procedure for processing related to sensor installation adjustment. Here, the user sequentially selects submenu items on a submenu screen 111 (see Figure 6(A)) related to sensor installation adjustment displayed on the user terminal 4, thereby sequentially performing the processes related to basic adjustment, alignment, and installation confirmation as described below. Note that prior to this flow, an operator installs the sensors (camera 1 and lidar 2) at a predetermined location. This determines the position of the sensor, and the sensor orientation (angle of view) is adjusted in the sensor installation adjustment process.

[0206] The traffic flow measurement server 3 first proceeds to the basic adjustment process, reads out a CG image file in response to user operations on the basic adjustment screen 201 (see Figures 8 to 11) displayed on the user terminal 4, and transmits the CG image file to the user terminal 4 to display the CG image on the user terminal 4 (ST101).

[0207] Next, the traffic flow measurement server 3 controls the angles (pan and tilt) of the sensors (camera 1, LIDAR 2) in response to the user's operation on the basic adjustment screen 201 (see FIGS. 8 to 11) displayed on the user terminal 4 (ST102). At this time, the traffic flow measurement server 3 transmits the sensor images (camera image, LIDAR intensity image) transmitted from the sensors to the user terminal 4 and causes the sensor images to be displayed on the user terminal 4.

[0208] Next, the traffic flow measurement server 3 proceeds to the alignment process, and performs alignment processing to correct positional deviations in the 3D point cloud data between the cameras 1 and the LIDAR 2 obtained by multiple cameras 1 and LIDAR 2 installed at different locations, in response to user operations on the alignment screen 231 (see FIGS. 12 to 15) displayed on the user terminal 4. Then, the traffic flow measurement server 3 transmits a LIDAR point cloud image generated from the integrated 3D point cloud data after alignment to the user terminal 4, and causes the LIDAR point cloud image to be displayed on the user terminal 4 (ST103).

[0209] In addition, in ST103, if the traffic flow measurement server 3 cannot properly correct the positional shift of the 3D point cloud data from multiple LIDARs 2 through automatic alignment, it can correct the positional shift of the 3D point cloud data from multiple LIDARs 2 through manual alignment in accordance with the user's operation.

[0210] Next, the traffic flow measurement server 3 proceeds to the installation confirmation step, and generates a lidar point cloud image including the virtual object of the moving body by placing the virtual object of the moving body in the three-dimensional space including the three-dimensional point cloud data in accordance with the user's operation on the installation confirmation screen 261 (see FIGS. 16 to 20) displayed on the user terminal 4. Then, the traffic flow measurement server 3 transmits the lidar point cloud image including the virtual object of the moving body to the user terminal 4 and causes the user terminal 4 to display the lidar point cloud image (ST104).

[0211] Next, the traffic flow measurement server 3 generates a camera image and a LIDAR intensity image including the virtual object of the moving body in response to a user operation on the installation confirmation screen 261 (see FIGS. 16 to 20) displayed on the user terminal 4. Then, the traffic flow measurement server 3 transmits the camera image and the LIDAR intensity image including the virtual object of the moving body to the user terminal 4 and causes the user terminal 4 to display the camera image and the LIDAR intensity image (ST105).

[0212] In ST105, if there is a defect in the display state of the virtual object of the moving object in the camera image and the LIDAR intensity image, the processes of ST104 and ST105 are repeated to adjust the display state of the virtual object of the moving object.

[0213] Next, the traffic flow measurement server 3 stores the sensor installation information acquired in the basic adjustment, alignment, and installation confirmation processes in the storage unit 12 (ST106). The sensor installation information includes information about the angle of the sensor (camera 1, LIDAR 2), information about the positional relationship between the camera image and the 3D point cloud data, information about the mutual positional relationship between the 3D point cloud data from multiple LIDAR 2, etc.

[0214] Next, we will explain the processing procedure for generating traffic flow data, which is performed by the traffic flow measurement server 3. Figure 37 is a flow diagram showing the processing procedure for generating traffic flow data. Here, the user sequentially selects submenu items on the submenu screen 121 (see Figure 6(B)) related to traffic flow data generation displayed on the user terminal 4, and the following processes for data recording and data analysis are sequentially performed.

[0215] The traffic flow measurement server 3 first proceeds to the data recording process and receives camera images from the camera 1 (ST201). The traffic flow measurement server 3 also receives 3D point cloud data from the lidar 2 (ST202).

[0216] Next, the traffic flow measurement server 3 synchronizes the camera image and the 3D point cloud data based on the time information added to the camera image received from the camera 1 and the time information added to the 3D point cloud data received from the LIDAR 2 (data synchronization process) (ST203). The time information is obtained from satellite signals by the camera 1 and the LIDAR 2. If the camera 1 and the LIDAR 2 do not have the function to receive satellite signals, they may obtain the time information via a local network and perform synchronization.

[0217] Next, the traffic flow measurement server 3 stores the synchronized camera images and 3D point cloud data in the storage unit 12 (ST204).

[0218] Next, the traffic flow measurement server 3 proceeds to the sensor data analysis process, analyzes the camera images and 3D point cloud data, and generates traffic flow data (ST205). The sensor data analysis process involves detecting moving objects and road components from the camera images and LIDAR point cloud data.

[0219] Next, the traffic flow measurement server 3 stores the traffic flow data generated by the sensor data analysis process in the storage unit 12 (ST206). The traffic flow data includes a timestamp (year / month / date, hour / minute / second), a trajectory ID (information for identifying a moving object), and relative coordinates (location information).

[0220] Next, there will be explained the procedure of the process related to browsing traffic flow data, which is performed by the traffic flow measurement server 3. Fig. 38 is a flowchart showing the procedure of the process related to browsing traffic flow data.

[0221] The traffic flow measurement server 3 first determines which item the user has selected on the submenu screen 131 (see FIG. 7(A)) related to browsing traffic flow data displayed on the user terminal 4 (ST301).

[0222] Here, if the user selects time series display ("Time series display" in ST301), the traffic flow measurement server 3 transitions the user terminal 4 to the time series display screen 401 (see FIG. 24) (ST302). Then, when the user specifies a measurement point (measurement area) on the time series display screen 401, the traffic flow measurement server 3 extracts traffic flow data corresponding to the specified measurement point (ST303). Next, the traffic flow measurement server 3 starts a viewer on the time series display screen 401 to display the traffic flow data in time series (ST304). At this time, the traffic flow data displayed includes the trajectory of the moving object and changes in speed and acceleration, along with sensor images (camera images, LIDAR intensity images, and LIDAR point cloud images).

[0223] On the other hand, if the user selects scenario specification ("Specify scenario" in ST301), the traffic flow measurement server 3 transitions the user terminal 4 to the scenario specification screen 431 (see FIG. 27) (ST305). Then, when the user directly specifies a scenario on the scenario specification screen 431, the traffic flow measurement server 3 extracts sensor images related to events that correspond to the specified scenario (ST306).

[0224] Next, the traffic flow measurement server 3 transitions the user terminal 4 to the specified event viewing screen 471 (see FIG. 30) (ST307). Next, the traffic flow measurement server 3 starts a viewer for viewing the sensor image on the specified event viewing screen 471, and displays the sensor image on the specified event viewing screen 471 (ST308). At this time, a lidar point cloud image related to the event corresponding to the specified scenario is displayed as the sensor image.

[0225] Furthermore, if the user selects statistical information specification ("Specify statistical information" in ST301), the traffic flow measurement server 3 transitions the user terminal 4 to a statistical information specification screen 461 (see FIG. 29) (ST309). Then, when the user specifies a scenario from the statistical information on the statistical information specification screen 461, the traffic flow measurement server 3 extracts sensor images related to events that correspond to the specified scenario (ST310). Next, the traffic flow measurement server 3 performs the processes of ST307 and ST308.

[0226] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0227] The traffic flow measurement system and traffic flow measurement method of the present invention have the effect of allowing the user to intuitively grasp changes in the state of moving objects while viewing sensor images as the detection results of sensors targeting the measurement area when presenting the results of traffic flow analysis to the user, and are useful as traffic flow measurement systems and traffic flow measurement methods that measure traffic flow at target points using sensors such as cameras and lidars. [Explanation of symbols]

[0228] 1 camera (first sensor) 2. Lidar (second sensor) 3 Traffic flow measurement server (server device) 4. User terminal (terminal device) 5 Management terminal

Claims

1. a first sensor for acquiring two-dimensional detection results for a traffic flow measurement area; a second sensor that acquires a three-dimensional detection result for the measurement area; a server device connected to the first and second sensors and configured to execute a traffic flow analysis process based on the detection results of the first and second sensors; a terminal device connected to the server device via a network and displaying the results of the traffic flow analysis processing; A traffic flow measurement system comprising: The server device A traffic flow measurement system characterized by generating a behavior image that visualizes the behavior of a moving object by associating time series data of the moving object's position, speed, and acceleration based on the results of a traffic flow analysis process, generating a traffic flow viewing screen in which the behavior image is superimposed on a sensor image based on the detection results of the first and second sensors, and transmitting the traffic flow viewing screen to the terminal device.

2. 2. The traffic flow measurement system according to claim 1, wherein the behavior image includes a label image that expresses, in characters, at least one of an ID, a speed, and an acceleration of a moving object.

3. The traffic flow measurement system of claim 1, wherein the behavior image comprises a trajectory image that visualizes time series data that represent changes in the position of the moving body, a speed image that visualizes time series data that represent changes in the speed of the moving body, and an acceleration image that visualizes time series data that represent changes in the acceleration of the moving body, and the speed image and the acceleration image have a trajectory point that represents the position of the moving body on the trajectory image as their origin, and the distance from the trajectory point to a speed point on the speed image at the same time represents the speed, and the distance from the trajectory point to an acceleration point on the acceleration image at the same time represents the acceleration.

4. The traffic flow measurement system according to claim 1, characterized in that the behavior image has a trajectory image that visualizes time series data representing changes in the position of a moving body, and the time series data representing changes in the speed of the moving body and the time series data representing changes in the acceleration of the moving body are expressed by attributes of the trajectory image.

5. The server device The traffic flow measurement system according to claim 1, characterized in that, in response to a user's operation to change the viewpoint on the traffic flow viewing screen, the sensor image from the changed viewpoint is generated from the three-dimensional detection results and transmitted to the terminal device.

6. a first sensor for acquiring two-dimensional detection results for a traffic flow measurement area; a second sensor that acquires a three-dimensional detection result for the measurement area; a server device connected to the first and second sensors and configured to execute a traffic flow analysis process based on the detection results of the first and second sensors; a terminal device connected to the server device via a network and displaying the results of the traffic flow analysis processing; In a traffic flow measurement system comprising: The server device: A traffic flow measurement method characterized by generating a behavior image that visualizes the behavior of a moving object by associating time series data of the moving object's position, speed, and acceleration based on the results of a traffic flow analysis process, generating a traffic flow viewing screen in which the behavior image is superimposed on a sensor image based on the detection results of the first and second sensors, and transmitting the traffic flow viewing screen to the terminal device.

Images