Traffic flow measurement system

Abstract

To enable a user to limit a specific scenario and confirm in detail a situation when an event corresponding to a scenario occurs.SOLUTION: A traffic flow measurement server detects an event corresponding to one of a plurality of scenarios set in advance on the basis of the results of traffic flow analysis processing, accumulates information on the event, extracts the event corresponding to the scenario specified by the user from the accumulated events according to the user's operation on a user terminal, generates a screen displaying a sensor image related to the event, and displays the screen on the user terminal.SELECTED DRAWING: Figure 27

Description

Technical Field

[0001] The present invention relates to traffic flow measurement for measuring the traffic flow at a target location using sensors such as cameras and lidars. system

Background Art

[0002] For the purpose of enhancing the safety and smoothness of road traffic, traffic flow measurement is performed to grasp the traffic situation at target locations such as intersections. In this traffic flow measurement, it is desired to obtain detailed and highly accurate traffic flow data regarding the situation of moving objects such as vehicles and pedestrians without requiring a large amount of manpower.

[0003] In response to such demands, conventionally, as a sensor for detecting an object within a measurement area, in addition to a camera, a technique for acquiring the trajectory of a moving object in a three-dimensional space using a lidar that has attracted attention in recent years in autonomous driving and the like is known (see Patent Document 1). In this technique, a process of associating a captured image acquired by a camera with 3D point cloud data acquired by a lidar is performed. Further, a process of integrating 3D point cloud data obtained by lidars installed at a plurality of different locations is performed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In traffic flow measurement using sensors such as cameras and lidars as in the conventional technology, based on detailed and highly accurate information regarding moving objects and road structures, in addition to the trajectory of the moving object, traffic flow analysis for acquiring various information such as the speed and acceleration of the moving object becomes possible. Further, based on the trajectory of the moving object and the like, it is also possible to detect notable events such as traffic accidents. ​

[0006] On the other hand, by classifying various events occurring at a target location into predetermined scenarios (event types), the traffic situation at that location can be efficiently examined. In this case, it is advisable to classify events detected from traffic flow data acquired through traffic flow analysis targeting various locations and periods into predetermined scenarios and then store them. Furthermore, users often have a desire to examine in detail the circumstances when events corresponding to specific scenarios of their interest occur. However, conventional technologies do not take such requests into consideration at all.

[0007] Therefore, the present invention provides a traffic flow measurement that allows users to check in detail the situation when an event corresponding to a specific scenario occurs. system The primary purpose is to provide. [Means for solving the problem]

[0008] The traffic flow measurement system of the present invention comprises: a first sensor that acquires two-dimensional detection results for a traffic flow measurement area; a second sensor that acquires three-dimensional detection results for the measurement area; a server device connected to the first and second sensors that acquires sensor images based on the detection results of the first and second sensors and performs traffic flow analysis processing based on the detection results of the first and second sensors; and a terminal device connected to the server device via a network that displays the sensor images and the results of the traffic flow analysis processing, wherein the server device detects an event that corresponds to one of a plurality of pre-set scenarios based on the results of the traffic flow analysis processing and stores information about that event. The system displays a statistical information selection screen to the terminal device, which includes a graph or summary table of statistical information regarding the frequency of occurrence of events corresponding to each of several scenarios. The system extracts information on events corresponding to the specified scenario based on the user's operation on the statistical information graph or summary table, generates a traffic flow viewing screen that displays the sensor images related to the extracted events, and transmits the traffic flow viewing screen to the terminal device. This will be the structure. [Effects of the Invention]

[0010] According to the present invention, The user checks the status (frequency, etc.) of events corresponding to a scenario using statistical information (graphs, summary tables, etc.), selects the scenario they want to view from the statistical information, and then displays the specific status of events corresponding to that scenario using sensor images. It can be confirmed. [Brief explanation of the drawing]

[0011] [Figure 1] Overall configuration diagram of the traffic flow measurement system according to this embodiment [Figure 2] Block diagram showing the schematic configuration of the traffic flow measurement server [Figure 3] Explanatory diagram showing the content of the traffic flow data generated by the traffic flow measurement server [Figure 4] Explanatory diagram showing the transition status of the screen displayed on the user terminal [Figure 5] Explanatory diagram showing the main menu screen displayed on the user terminal [Figure 6] Explanatory diagram showing the sub-menu screen displayed on the user terminal [Figure 7] Explanatory diagram showing the sub-menu screen displayed on the user terminal [Figure 8] Explanatory diagram showing the basic adjustment screen displayed on the user terminal [Figure 9] Explanatory diagram showing the basic adjustment screen displayed on the user terminal [Figure 10] Explanatory diagram showing the basic adjustment screen displayed on the user terminal [Figure 11] Explanatory diagram showing the basic adjustment screen displayed on the user terminal [Figure 12] Explanatory diagram showing the alignment screen displayed on the user terminal [Figure 13] Explanatory diagram showing the alignment screen displayed on the user terminal [Figure 14] Explanatory diagram showing the alignment screen displayed on the user terminal [Figure 15] Explanatory diagram showing the alignment screen displayed on the user terminal<000008^ [Figure 16] Explanatory diagram showing the installation confirmation screen displayed on the user terminal [Figure 17] Explanatory diagram showing the installation confirmation screen displayed on the user terminal [Figure 18] Explanatory diagram showing the installation confirmation screen displayed on the user terminal ' [Figure 19] Explanatory diagram showing the installation confirmation screen displayed on the user terminal [Figure 20] Explanatory diagram showing another example of the installation confirmation screen displayed on the user terminal [Figure 21] Explanatory drawing showing a sensor data recording screen displayed on a user terminal [Figure 22] Explanatory drawing showing a sensor data analysis screen displayed on a user terminal [Figure 23] Explanatory drawing showing a sensor data analysis screen displayed on a user terminal [Figure 24] Explanatory drawing showing a time series display screen displayed on a user terminal [Figure 25] Explanatory drawing showing a time series display screen displayed on a user terminal [Figure 26] Explanatory drawing showing a main part of a time series display screen displayed on a user terminal [Figure 27] Explanatory drawing showing a scenario specification screen displayed on a user terminal [Figure 28] Explanatory drawing showing a scenario specification screen displayed on a user terminal [Figure 29] Explanatory drawing showing a statistical information specification screen displayed on a user terminal [Figure 30] Explanatory drawing showing a specified event browsing screen displayed on a user terminal [Figure 31] Explanatory drawing showing a specified event browsing screen displayed on a user terminal [Figure 32] Explanatory drawing showing a tracking mode screen displayed on a user terminal [Figure 33] Explanatory drawing showing a tracking mode screen displayed on a user terminal [Figure 34] Explanatory drawing showing an extended browsing mode screen displayed on a user terminal [Figure 35] Explanatory drawing showing an extended browsing mode screen displayed on a user terminal [Figure 36] Flow chart showing the procedure of the process related to sensor installation adjustment performed by the traffic flow measurement server [Figure 37] Flow chart showing the procedure of the process related to traffic flow data generation performed by the traffic flow measurement server [Figure 38] Flow chart showing the procedure of the process related to traffic flow data browsing performed by the traffic flow measurement server

Mode for Carrying Out the Invention

[0014] Also, First The invention relates to a traffic flow measurement system comprising: a first sensor that acquires two-dimensional detection results for a traffic flow measurement area; a second sensor that acquires three-dimensional detection results for the measurement area; a server device connected to the first and second sensors that acquires sensor images based on the detection results of the first and second sensors and performs traffic flow analysis processing based on the detection results of the first and second sensors; and a terminal device connected to the server device via a network that displays the sensor images and the results of the traffic flow analysis processing, wherein the server device detects events that fall under any of a plurality of pre-set scenarios based on the results of the traffic flow analysis processing, stores information on those events, displays a statistical information specification screen to the terminal device that includes a graph or summary table of statistical information regarding the frequency of occurrence of events corresponding to each of the plurality of scenarios, extracts information on events corresponding to the specified scenario based on user operation on the graph or summary table of statistical information, generates a traffic flow viewing screen that displays the sensor images related to the extracted events, and displays the traffic flow viewing screen to the terminal device. Configuration to send Let's assume that.

[0015] According to this, users can check the status (frequency, etc.) of events corresponding to a scenario using statistical information (graphs, summary tables, etc.), select the scenario they want to view from the statistical information, and then check the specific status of events corresponding to that scenario using sensor images.

[0016] Also, Second The invention is configured such that the server device displays the sensor image from a viewpoint specified by the user on the traffic flow viewing screen in response to user operation on the terminal device.

[0017] According to this, users can view sensor images from their desired viewpoint and understand the situation of the event in question.

[0018] Also, The thirdThe invention is configured such that, in response to user operation on the terminal device, the server device displays on the traffic flow viewing screen either an image with the viewpoint set to the driver of a vehicle, which is a moving object related to the target event, or an image with the viewpoint set to the sky above the measurement area where the target event occurred.

[0019] According to this, users can view sensor images from their desired viewpoint and understand the situation of the event in question.

[0022] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0023] Figure 1 is an overall diagram of the traffic flow measurement system according to this embodiment.

[0024] This system measures traffic flow in a measurement area. The system comprises a camera 1 (first sensor), a lidar 2 (second sensor), a traffic flow measurement server 3 (server device), a user terminal 4 (terminal device), and a management terminal 5. Camera 1 and lidar 2 are connected to the traffic flow measurement server 3 via a first network N1. User terminal 4 and the management terminal are connected to the traffic flow measurement server 3 via a second network N2.

[0025] Camera 1 photographs the measurement area and acquires a camera image as a two-dimensional detection result (two-dimensional information) targeting the measurement area. Camera 1 is equipped with a visible light image sensor and can acquire color images.

[0026] LiDAR (LiDAR 2) detects objects in a measurement area and acquires 3D point cloud data as a 3D detection result (3D information) for the measurement area. LiDAR 2 acquires 3D information by irradiating objects with laser light and detecting the reflected light. Note that other 3D sensors besides LiDAR 2 may also be used.

[0027] The traffic flow measurement server 3 acquires camera images from camera 1 and 3D point cloud data from lidar 2, and performs traffic flow analysis processing for the measurement area based on the camera images and 3D point cloud data. In addition, the traffic flow measurement server 3 performs processing to assist users in easily adjusting the sensor installation status when new sensors (camera 1 and lidar 2) are installed or when sensors are replaced.

[0028] User terminal 4 consists of a tablet device or the like. User terminal 4 displays screens related to settings and viewing transmitted from the traffic flow measurement server 3. Through these screens, the user can perform tasks such as adjusting the sensor installation status and viewing the results of traffic flow analysis processing.

[0029] The management terminal consists of a PC or similar device. The management terminal displays a management screen transmitted from the traffic flow measurement server 3, and this management screen allows the administrator to perform management tasks such as setting conditions for the processing performed by the traffic flow measurement server 3.

[0030] Camera 1 and LiDAR 2 are equipped with the ability to receive satellite signals from a satellite positioning system (such as GPS), and update the time information within Camera 1 and LiDAR 2 using the time information contained in the satellite signals. Camera 1 and LiDAR 2 add the time synchronized with the satellite signal as the detection time to the detection results (camera image, 3D point cloud data) and transmit them to the traffic flow measurement server 3. The traffic flow measurement server 3 synchronizes the detection results of LiDAR 2 and Camera 1 based on the detection time. If Camera 1 and LiDAR 2 do not have the ability to receive satellite signals, time synchronization may be performed via the first network.

[0031] Next, we will describe the general configuration of the traffic flow measurement server 3. Figure 2 is a block diagram showing the general configuration of the traffic flow measurement server 3.

[0032] The traffic flow measurement server 3 comprises a communication unit 11, a storage unit 12, and a processor 13.

[0033] The communication unit 11 communicates with the camera 1 and the lidar 2 via the first network. The communication unit 11 also communicates with the user terminal 4 and the management terminal via the second network.

[0034] The memory unit 12 stores programs executed by the processor 13. It also stores camera images acquired from camera 1 and 3D point cloud data acquired from lidar 2. Furthermore, it stores traffic flow data generated by the processor 13. The memory unit 12 also stores CG images (simulation images) of the measurement points. Finally, it stores sensor installation information acquired during the basic adjustment, alignment, and installation verification processes. This sensor installation information includes information regarding the detection angles of the sensors (camera 1, lidar 2), the positional relationship between camera images and 3D point cloud data, and the relative positional relationships between 3D point cloud data from multiple lidar 2s.

[0035] The processor 13 performs various processes by executing programs stored in memory. In this embodiment, the processor 13 performs sensor data synchronization processing P1, sensor data integration processing P2, LiDAR image generation processing P3, sensor installation support processing P4, traffic flow data generation processing P5, event detection processing P6, event extraction processing P7, statistical processing P8, risk determination processing P9, and traffic flow data presentation processing P10, etc.

[0036] In the sensor data synchronization process P1, the processor 13 associates the camera images acquired from each camera 1 and the 3D point cloud data acquired from each LiDAR 2 based on the detection time. In this embodiment, in camera 1, the time included in the received satellite signal is added to the camera image as the detection time and transmitted to the traffic flow measurement server 3. Similarly, in LiDAR 2, the time included in the received satellite signal is added to the 3D point cloud data as the detection time and transmitted to the traffic flow measurement server 3.

[0037] In the sensor data integration process P2, the processor 13 integrates (combines) multiple 3D point cloud data from multiple LiDAR 2 sensors installed at multiple locations.

[0038] In the LiDAR image generation process P3, the processor 13 generates a LiDAR intensity image with the sensor installation location as the viewpoint, based on the 3D point cloud data from LiDAR 2. The processor 13 also generates a LiDAR point cloud image with a viewpoint specified by the user, based on the 3D point cloud data. In this embodiment, the 3D point cloud data is displayed as a LiDAR point cloud image using a 3D viewer on the user terminal 4, and the viewpoint can be changed by performing viewpoint change operations on the 3D viewer, for example, by dragging the cursor up, down, left, or right on the displayed LiDAR point cloud image.

[0039] In the sensor installation support process P4, the processor 13 performs processing to support the user's adjustment work when installing the sensors (camera 1, lidar 2) in response to user operations on the user terminal 4. The sensor installation support process P4 includes the sensor adjustment support process P21, the positioning process P22, and the installation confirmation support process P23.

[0040] In the sensor adjustment support process P21, the processor 13 performs processing to support the user's operation to adjust the installation state of the sensors (camera 1, lidar 2). Specifically, the processor 13 displays the camera image and the lidar intensity image generated from 3D point cloud data on the user terminal 4, and controls the shooting angle (field of view) of the sensors (camera 1, lidar 2) according to the user's adjustment operation.

[0041] In the alignment process P22, the processor 13 estimates the relative positional relationship of the installation locations of each sensor (camera 1, lidar 2) and associates the coordinates of the detection results for each of the multiple sensors. Specifically, it associates the coordinates on the camera image with the coordinates in the 3D point cloud data. The processor 13 also corrects for positional shifts in the point cloud data from multiple lidar 2s installed at different locations.

[0042] In the installation confirmation support process P23, the processor 13 places a virtual object of the moving object in a three-dimensional space containing 3D point cloud data from LiDAR 2, in response to user operations on the user terminal 4, and superimposes the virtual object of the moving object onto the camera image and LiDAR intensity image based on their positional relationships. The processor 13 then determines whether or not there are any gaps in the virtual object of the moving object superimposed on the camera image and LiDAR intensity image, that is, whether or not the virtual object of the moving object extends beyond the display range of the camera image and LiDAR intensity image.

[0043] In the traffic flow data generation process P5, the processor 13 generates traffic flow data (see Figure 3) that represents the traffic conditions in the measurement area based on sensor data (camera images, 3D point cloud data). The traffic flow data generation process P5 includes sensor data recording process P31 and sensor data analysis process P32 (traffic flow analysis process).

[0044] In the sensor data recording process P31, the processor 13 stores the camera images acquired from each camera 1 and the 3D point cloud data acquired from each LiDAR 2 in the storage unit 12, in accordance with the user's instructions on the user terminal 4.

[0045] In the sensor data analysis process P32 (traffic flow analysis process), the processor 13 generates traffic flow data based on the sensor data (camera images, 3D point cloud data) collected in the sensor data recording process P31. The sensor data analysis process P32 includes the moving object detection process P33, the moving object ID management process P34, and the road structure detection process P35.

[0046] In the moving object detection process P33, the processor 13 detects moving objects from the camera image obtained by the camera 1 in an identifiable manner. Specifically, it detects buses, trucks, trailers, passenger cars, motorcycles, bicycles, and pedestrians. The processor 13 also detects moving objects from 3D point cloud data obtained by integrating 3D point cloud data from multiple LiDAR 2s. Furthermore, the processor 13 acquires the location information of the detected moving objects and assigns a moving object ID to the detected moving objects. The moving object detection process P33 can use an image recognition engine (machine learning model) built using machine learning such as deep learning.

[0047] In the mobile object ID management process P34, the processor 13 replaces the mobile object IDs assigned to mobile objects when they are detected from each camera image and when they are detected from 3D point cloud data, so that the same mobile object ID is assigned to the same mobile object. In this embodiment, the user can specify a preferred sensor when replacing the mobile object IDs. In this case, the mobile object IDs assigned to mobile objects with respect to sensors other than the preferred sensor are changed to the mobile object IDs assigned to mobile objects with respect to the preferred sensor.

[0048] In the road structure detection process P35, the processor 13 detects road structures from 3D point cloud data in an identifiable manner. Specifically, it detects areas of road structures, i.e., features such as sidewalks, curbs, and guardrails, as well as road markings such as white lines and stop lines, through segmentation (region division). The road structure detection process P35 can utilize an image recognition engine (machine learning model) built using machine learning such as deep learning.

[0049] In event detection processing P6, the processor 13 detects events that correspond to predetermined scenarios (event types) based on traffic flow data obtained as a result of sensor data analysis processing P32 (traffic flow analysis processing). Specifically, it detects events that correspond to scenarios such as rear-end collisions, right-turn collisions, left-turn collisions, wrong-way driving, and aggressive driving, based on the trajectories of each moving object. The results of the event detection processing are stored in the event database.

[0050] In the event extraction process P7, the processor 13 extracts events from the events stored in the event database that correspond to the scenario specified by the user. In this embodiment, the user can directly specify a scenario at the user terminal 4, or the user can specify a scenario within the statistical information presented to the user.

[0051] In statistical processing P8, processor 13 performs statistical processing based on traffic flow data and generates statistical information. For example, statistical information is generated regarding the frequency of occurrence of events corresponding to each of multiple scenarios.

[0052] In the risk assessment process P9, the processor 13 acquires information about the traffic environment at the target location, specifically the positional relationship between moving objects and road components, based on traffic flow data, and determines the risk level of the traffic environment at the target location. In the risk assessment process P9, an index for evaluating the risk of the traffic environment at each location is created in advance based on statistical information obtained from statistical processing for each location, and the risk level is determined from the condition of moving objects at the target location based on this index for evaluating risk.

[0053] In the traffic flow data presentation process P10, the processor 13 presents traffic flow data to the user by displaying various screens on the user terminal 4. The traffic flow data presentation process P10 includes the time-series display process P41, the specified event display process P42, and the supplementary information display process P43.

[0054] In the time-series display processing P41, the processor 13 visualizes information representing the behavior (state changes) of a moving object (state information) using graphics and characters and displays it on the screen based on traffic flow data. In this embodiment, as information representing the behavior of a moving object, behavior images (trajectory image, velocity image, acceleration image) that visualize time-series data representing the changes in the position, velocity, and acceleration of the moving object are superimposed on sensor images (camera image, lidar point cloud image, lidar intensity image).

[0055] In the specified event display process P42, the processor 13 displays sensor images (camera images, lidar point cloud images, etc.) related to events that meet the conditions specified by the user on the user terminal 4. In this embodiment, the user can specify a scenario (event type) as an extraction condition, and sensor images related to events that match the scenario specified by the user are displayed on the user terminal 4.

[0056] In the supplementary information display process P43, the processor 13 displays information related to the traffic environment of the measurement area, such as the condition of road components like white lines and sidewalks, and the type of moving object (passenger car, truck, etc.), as supplementary information on the screen simultaneously with the moving object. Specifically, it highlights and displays on the sensor image (camera image, lidar point cloud image, lidar intensity image) an object specified by the user so that it can be identified.

[0057] Next, we will explain the traffic flow data generated by the traffic flow measurement server 3. Figure 3 is an explanatory diagram showing the contents of the traffic flow data.

[0058] In the traffic flow measurement server 3, traffic flow data (tracking data) is generated for each moving object (trajectory ID). Each row in the table shown in Figure 3 represents the unit data for each time point, and this unit data is generated sequentially in chronological order. The example shown in Figure 3 concerns a moving object with trajectory ID "1". The target moving object is a passenger car with attribute "0" and is moving in the x direction.

[0059] Traffic flow data includes a timestamp (year, month, day, hour, minute, second), track ID, relative coordinates (x, y, z), attributes, vehicle size (width, length, height), driving lane, distance to white lines (left white line, right white line), and type of white line. The timestamp, track ID, and relative coordinates (location information) are the main data, while the others are supplementary data.

[0060] The trajectory ID is assigned to the trajectory of a moving object and serves as information to identify the object. The relative coordinates (x, y, z) represent the position of the moving object at each point in time. The attribute represents the type of moving object; for example, a passenger car is 0, a large vehicle is 1, a motorcycle is 2, and an unknown object is 3. The lanes are represented by numbers 1, 2, etc., from left to right. The type of white line is represented by numbers such as solid line is 0 and dashed line is 1.

[0061] Furthermore, the traffic flow data may include absolute coordinates (latitude, longitude, altitude), direction of travel of the moving object (angle), road alignment (road curvature, road longitudinal gradient, road lateral gradient), road coordinates (Lx, Ly, dLx, dLy), speed, acceleration (direction of travel, lateral direction), road width, number of lanes, road type (1: intercity expressway, urban expressway, national highway, 2: main road, merge, junction, ramp), lane marker type (left side of vehicle, right side of vehicle), collision margin with the vehicle ahead, attributes of the vehicle ahead (passenger car, heavy vehicle, motorcycle, unknown), relative speed of surrounding vehicles, and driving lane. In addition, information regarding the tracking frame of the moving object by image recognition may be included in the traffic flow data.

[0062] Next, we will explain the screens displayed on user terminal 4. Figure 4 is an explanatory diagram showing the transitions between screens displayed on user terminal 4. Figure 5 is an explanatory diagram showing the main menu screen displayed on user terminal 4. Figures 6 and 7 are explanatory diagrams showing the submenu screens displayed on user terminal 4.

[0063] The main menu screen 101 shown in Figure 5 includes a button 102 for sensor installation adjustment, a button 103 for traffic flow data generation, a button 104 for viewing traffic flow data, and an options button 105. When the user operates the sensor installation adjustment button 102, the screen transitions to the submenu screen related to sensor installation adjustment shown in Figure 6(A). When the user operates the traffic flow data generation button 103, the screen transitions to the submenu screen related to traffic flow data generation shown in Figure 6(B). When the user operates the traffic flow data viewing button 104, the screen transitions to the submenu screen related to viewing traffic flow data shown in Figure 7(A). When the user operates the options button 105, the screen transitions to the submenu screen related to options shown in Figure 7(B).

[0064] The submenu screen 111 for sensor installation and adjustment shown in Figure 6(A) includes a basic adjustment button 112, a positioning button 113, and an installation confirmation button 114. When the user operates the basic adjustment button 112, the screen transitions to the basic adjustment screen 201 (see Figure 8). When the user operates the positioning button 113, the screen transitions to the positioning screen 231 (see Figure 12). When the user operates the installation confirmation button 114, the screen transitions to the installation confirmation screen 261 (see Figure 16).

[0065] The submenu screen 121 for traffic flow data generation shown in Figure 6(B) includes a button 122 for recording sensor data and a button 123 for analyzing sensor data. When the user operates the button 122 for recording sensor data, the screen transitions to the sensor data recording screen 301 (see Figure 21). When the user operates the button 123 for analyzing sensor data, the screen transitions to the sensor data analysis screen 311 (see Figure 22).

[0066] The submenu screen 131 for viewing traffic flow data, shown in Figure 7(A), is equipped with buttons for time-series display, scenario selection, and statistical information selection. When the user operates the time-series display button, the screen transitions to the time-series display screen 401 (see Figure 24). When the user operates the scenario selection button, the screen transitions to the scenario selection screen 431 (see Figure 27). When the user operates the statistical information selection button, the screen transitions to the statistical information selection screen 461 (see Figure 29). Furthermore, when the user performs a predetermined operation on the scenario selection screen 431 or the statistical information selection screen 461, the screen transitions to the specified event viewing screen 471 (see Figure 30).

[0067] The submenu screen 141 for the options shown in Figure 7(B) includes a tracking mode button 142 and an extended browsing mode button 143. When the user operates the tracking mode button 142, the screen transitions to the tracking mode screen 501 (see Figure 32). When the user operates the extended browsing mode button 143, the screen transitions to the extended browsing mode screen 531 (see Figure 34).

[0068] As shown in Figure 4, the basic adjustment screen 201 (see Figure 8) and the alignment screen 231 (see Figure 12) are referred to as the installation adjustment screen as appropriate. In addition, the time series display screen 401 (see Figure 24), scenario selection screen 431 (see Figure 27), statistical information selection screen 461 (see Figure 29), specified event viewing screen 471 (see Figure 30), and extended viewing mode screen 531 (see Figure 34) are referred to as the traffic flow viewing screen as appropriate.

[0069] Furthermore, each screen accessed from each submenu screen 111, 121, 131, 141 (see Figures 6 and 7) is provided with tabs 161 corresponding to each submenu item (such as basic adjustment, alignment, and installation confirmation), and menu buttons 162, as shown in Figure 8. When the user operates a tab 161, the screen transitions to the corresponding submenu item. When the user operates a menu button 162, the screen returns to the main menu screen 101 (see Figure 5).

[0070] Furthermore, each screen accessed from each submenu screen 111, 121, 131, 141 (see Figures 6 and 7) is provided with a measurement point specification section 163, as shown in Figure 8. The measurement point specification section 163 allows the user to specify a measurement point by operating a pull-down menu.

[0071] Next, we will explain the basic adjustment screen 201 displayed on the user terminal 4. Figure 8 is an explanatory diagram showing the basic adjustment screen 201 with the CG image hidden during camera adjustment. Figure 9 is an explanatory diagram showing the basic adjustment screen 201 with the CG image displayed during camera adjustment. Figure 10 is an explanatory diagram showing the basic adjustment screen 201 with the CG image hidden during LiDAR adjustment. Figure 11 is an explanatory diagram showing the basic adjustment screen 201 with the CG image displayed during LiDAR adjustment.

[0072] On the user terminal 4, when the user operates the sensor installation adjustment button 102 on the main menu screen 101 (see Figure 5), a submenu screen 111 (see Figure 6(A)) is displayed. When the user operates the basic adjustment button 112, the basic adjustment screen 201 shown in Figure 8 is displayed.

[0073] The basic adjustment screen 201 shown in Figure 8 is the initial state when the CG image is not displayed during camera adjustment. The basic adjustment screen 201 is equipped with a layout display unit 202. The layout display unit 202 displays a plan view 203 and a side view 204 that represent the installation status of the sensors (camera 1 and lidar 2) at the sensor installation location. The user can perform the basic adjustment work while visually checking the installation status of camera 1 and lidar 2 by looking at the plan view 203 and the side view 204.

[0074] Plan view 203 shows the camera 1 and lidar 2 installed around the measurement area, viewed from above. Side view 204 shows the camera 1 and lidar 2 installed around the measurement area, viewed from the side. In this example, camera 1 and lidar 2 of #1 and camera 1 and lidar 2 of #2 are installed facing each other across an intersection.

[0075] Here, if Camera 1 and LiDAR 2 are equipped with IMUs (Inertial Measurement Units), the traffic flow measurement server 3 can obtain the actual detection direction of LiDAR 2 based on the output information of the IMUs. As a result, the traffic flow measurement server 3 displays the top view 203 and the side view 204 so that the orientation of the drawn Camera 1 and LiDAR 2 changes in conjunction with the change in the actual detection direction of Camera 1 and LiDAR 2. On the other hand, if Camera 1 and LiDAR 2 are not equipped with IMUs, the actual detection direction of Camera 1 is unknown to the traffic flow measurement server 3. Therefore, the orientation of Camera 1 and LiDAR 2 drawn in the top view 203 and the side view 204 will differ from the actual orientation.

[0076] Furthermore, the basic adjustment screen 201 is equipped with a sensor switching unit 205. The sensor switching unit 205 is equipped with a camera adjustment button 206 and a lidar adjustment button 207. When the user operates the camera adjustment button 206, the system enters camera adjustment mode, and the basic adjustment screen 201 for camera adjustment shown in Figure 8 is displayed. On the other hand, when the user operates the lidar adjustment button 207, the system enters lidar adjustment mode, and the basic adjustment screen 201 for lidar adjustment shown in Figure 10 is displayed.

[0077] Furthermore, the basic adjustment screen 201 shown in Figure 8 during camera adjustment is equipped with a sensor image display unit 211. The sensor image display unit 211 displays the camera image 212 as the sensor image (detection image from the sensor). In this example, since two cameras 1 are installed, two camera images 212 captured by each camera 1 are displayed.

[0078] In the sensor image display unit 211, the sensor angle operation unit 213 is superimposed on the camera image 212. The sensor angle operation unit 213 allows the user to change the shooting angle (field of view) of the camera 1, which acts as a sensor, in a specified direction, specifically by performing pan (horizontal) and tilt (vertical) operations. This allows the user to adjust the angle of the camera 1 while visually observing the camera image 212.

[0079] Furthermore, the basic adjustment screen 201 shown in Figure 8 is equipped with a CG image selection unit 217 and a CG image display button 218. The CG image selection unit 217 allows the user to input the name of a measurement point and initiate a search. This loads a CG image file containing CG images for camera adjustment related to the measurement point, and the file name of the CG image file is displayed in the CG image selection unit 217. Next, when the user operates the CG image display button 218, the system transitions to the basic adjustment screen 201 shown in Figure 9.

[0080] The basic adjustment screen 201 shown in Figure 9 represents the state when the CG image is displayed during camera 1 adjustment. In this case, the basic adjustment screen 201 is provided with a CG image display unit 221. The CG image display unit 221 displays a CG image 222 corresponding to the camera image 212 (sensor image) displayed on the sensor image display unit 211. In this example, since two cameras 1 are installed, two CG images 222 corresponding to the two camera images 212 captured by each camera 1 are displayed.

[0081] Here, CG image 222 (simulation image) is a CG reproduction of the camera image when the measurement area is photographed with camera 1 adjusted to the appropriate angle, and is created in advance using CG. CG image 222 serves as a model when adjusting the angle (field of view) of camera 1.

[0082] The user can visually compare the camera image 212 (actual image captured by camera 1) displayed on the sensor image display unit 211 with the CG image 222, and adjust the angle of camera 1 using the sensor angle operation unit 213 so that both images appear similar, thereby setting camera 1 to the optimal angle.

[0083] The basic adjustment screen 201 shown in Figure 10 represents the state when the CG image is not displayed during LiDAR adjustment. In this case, the sensor image display unit 211 on the basic adjustment screen 201 displays the LiDAR intensity image 215 as the sensor image (sensor detection image). In this example, since two LiDAR 2s are installed, two LiDAR intensity images 215 detected by each LiDAR 2 are displayed. The LiDAR intensity image 215 is an image that represents the reflectance intensity in the 3D point cloud data acquired by LiDAR 2 as brightness.

[0084] In the basic adjustment screen 201 shown in Figure 10, the sensor angle control unit 213 is superimposed on the lidar intensity image 215. The sensor angle control unit 213 allows the user to change the detection angle (field of view) of the lidar 2 sensor in a specified direction, specifically by panning (horizontal) and tilting (vertical). This allows the user to adjust the angle of the lidar 2 while visually observing the lidar intensity image 215.

[0085] In the basic adjustment screen 201 shown in Figure 10, similar to the basic adjustment screen 201 (see Figure 9), the user enters the name of the measurement point in the CG image selection unit 217 and instructs it to search. This retrieves a CG image file containing CG images for Rider adjustment related to the measurement point. Then, when the user operates the CG image display button 218, the system transitions to the basic adjustment screen 201 with the CG image display state during Rider adjustment, as shown in Figure 11.

[0086] The basic adjustment screen 201 shown in Figure 11 represents the CG image display state during lidar adjustment. In this case, the CG image display unit 221 on the basic adjustment screen 201 displays a CG image 225 corresponding to the lidar intensity image 215 displayed on the sensor image display unit 211. In this example, since two lidars 2 are installed, two CG images 225 corresponding to the two lidar intensity images 215 detected by each lidar 2 are displayed.

[0087] Here, CG image 225 (simulation image) is a CG reproduction of the lidar intensity image when the measurement area is detected by lidar 2 adjusted to the appropriate angle, and is created in advance using CG. CG image 225 serves as a model when adjusting the angle (field of view) of lidar 2.

[0088] The user can visually compare the LiDAR intensity image 215 (the actual detection image by LiDAR 2) displayed on the sensor image display unit 211 with the CG image 225, and adjust the angle of LiDAR 2 using the sensor angle operation unit 213 so that both images appear in the same state, thereby setting LiDAR 2 to the optimal angle.

[0089] In this way, the basic adjustment screen 201 allows the user to adjust the angle (field of view) of the sensors (camera 1 and lidar 2) by visually observing the camera image 212. Furthermore, the user can adjust the sensor angle by referring to CG images 222 and 225, which are CG reproductions of the sensor image when the measurement area is detected by the sensor adjusted to the appropriate angle. This makes it easy for the user to adjust the sensor installation status when installing or replacing sensors. In this configuration, the camera adjustment mode and lidar adjustment mode are switched by the sensor switching unit 205, but the camera adjustment mode and lidar adjustment mode may also be switched based on the user's operation of selecting the sensors (camera, lidar) displayed on the plan view 203 or side view 204 of the layout display unit 202.

[0090] Next, we will explain the alignment screen 231 displayed on the user terminal 4. Figure 12 is an explanatory diagram showing the alignment screen 231 in its initial state. Figure 13 is an explanatory diagram showing the alignment screen 231 when the alignment result is correct. Figure 14 is an explanatory diagram showing the alignment screen 231 when the alignment result is an error. Figure 15 is an explanatory diagram showing the alignment screen 231 during manual alignment.

[0091] On the user terminal 4, when the user operates the sensor installation adjustment button 102 on the main menu screen 101 (see Figure 5), a submenu screen 111 (see Figure 6(A)) is displayed. When the user operates the alignment button 113, the alignment screen 231 shown in Figure 12 is displayed.

[0092] The alignment screen 231 shown in Figure 12 is equipped with a layout display unit 202, similar to the basic adjustment screen 201 (see Figure 8). The layout display unit 202 displays a plan view 203 and a side view 204 that represent the installation status of the camera 1 and lidar 2.

[0093] Furthermore, the alignment screen 231 is equipped with a sensor image display unit 232. The user can specify the target measurement point in the measurement point specification unit 163. As a result, the camera image 233, lidar intensity image 234, and lidar point cloud image 235 related to the specified measurement point are displayed in the sensor image display unit 232. The displayed images may be real-time images or stored images.

[0094] Furthermore, the alignment screen 231 is equipped with an alignment button 237. When the user operates the alignment button 237, the process proceeds to automatic alignment, the traffic flow measurement server 3 executes the alignment process, and the system transitions to the alignment screen 231 shown in Figure 13 when alignment is complete.

[0095] In the alignment process, the correspondence between the 3D point cloud data from two cameras 1 and two LiDAR 2 installed at two locations is estimated, and based on the estimation results, the 3D point cloud data from each LiDAR 2 is integrated. At this time, the relative positional relationship of one of the two 3D point cloud data sets to the other is corrected as needed. Specifically, one of the two 3D point cloud data sets is moved or rotated relative to the other.

[0096] In the alignment screen 231 shown in Figure 13, the integrated LiDAR point cloud image 241 is displayed on the sensor image display unit 232 when alignment is complete.

[0097] Furthermore, on the alignment screen 231 when alignment is complete, lines 242 connecting corresponding parts in each sensor image (two camera images 233 and two lidar intensity images 234) are displayed on the sensor image display unit 232. This allows the user to confirm the correspondence between each sensor image.

[0098] The user visually inspects the integrated lidar point cloud image 241 to confirm whether the alignment is sufficient. If the alignment is insufficient, problems such as the moving object appearing as a double image in the integrated lidar point cloud image 241 will occur, as shown in Figure 14.

[0099] The alignment screen 231 shown in Figure 14, which is displayed when alignment is complete, shows the manual alignment confirmation unit 243. The manual alignment confirmation unit 243 is equipped with a "Yes" button 244 and a "No" button 245. If the user finds a problem in the integrated LiDAR point cloud image 241, they press the "Yes" button 244. This transitions to the manual alignment screen 231 shown in Figure 15.

[0100] The alignment screen 231 shown in Figure 15 displays the manual alignment operation unit 251 and the realignment button 252.

[0101] The manual alignment unit 251 is operated by the user to correct the relative positional relationship of 3D point cloud data from two LiDAR units 2. The manual alignment unit 251 is equipped with a movement unit 253 and a rotation unit 254. The movement unit 253 allows the user to move one of the 3D point cloud data from the two LiDAR units 2 in a specified direction (up, down, left, right, forward, backward). The rotation unit 254 allows the user to rotate one of the 3D point cloud data from the two LiDAR units 2 in a specified direction (roll, pitch, yaw). In this case, it is preferable to enable the movement or rotation alignment by selecting one of the LiDAR intensity images 234.

[0102] The user visually inspects the integrated LiDAR point cloud image 241 and performs the necessary operations using the manual alignment unit 251. The display area of ​​the LiDAR point cloud image 241 has a 3D viewer function, and by moving the viewpoint, the LiDAR point cloud image 241 can be displayed from any viewpoint. This allows the user to confirm whether the relative positional misalignment of the 3D point cloud data from the two LiDARs 2 has been sufficiently improved through manual alignment operations.

[0103] Once the user confirms that the relative positional misalignment of the 3D point cloud data from the two LiDAR 2s has been sufficiently corrected, they operate the realignment button 252. This causes the traffic flow measurement server 3 to perform the alignment process again, and the system transitions to the alignment screen 231 shown in Figure 13, which indicates that the alignment is complete.

[0104] In this way, the alignment screen 231 displays the result of correcting and integrating the positional misalignment of two 3D point cloud data sets from two LiDAR 2 units installed at two locations, allowing the user to easily confirm that the 3D point cloud data has been properly aligned. Furthermore, if the positional misalignment of the two 3D point cloud data sets is too large for automatic proper alignment, the user can manually correct the misalignment of the two 3D point cloud data sets according to their input, and by performing the alignment process again, the 3D point cloud data alignment can be properly completed.

[0105] Next, we will explain the installation confirmation screen 261 displayed on the user terminal 4. Figure 16 is an explanatory diagram showing the installation confirmation screen 261 in its initial state. Figure 17 is an explanatory diagram showing the installation confirmation screen 261 when a virtual object is selected. Figure 18 is an explanatory diagram showing the installation confirmation screen 261 when virtual objects are superimposed. Figure 19 is an explanatory diagram showing the installation confirmation screen 261 in an error state when virtual objects are superimposed.

[0106] On the user terminal 4, when the user operates the sensor installation adjustment button 102 on the main menu screen 101 (see Figure 5), a submenu screen 111 (see Figure 6(A)) is displayed. When the user operates the installation confirmation button 114, the installation confirmation screen 261 shown in Figure 16 is displayed.

[0107] The installation confirmation screen 261 shown in Figure 16 is equipped with a sensor image display unit 262. The sensor image display unit 262 displays a camera image 263, a LiDAR intensity image 264, and a LiDAR point cloud image 265.

[0108] Here, the installation confirmation screen 261 shown in Figure 16 represents the case where sensors (camera 1 and lidar 2) are installed at two locations on opposite sides of an intersection. In this case, camera images 263 from the two cameras 1 installed at the two locations, lidar intensity images 264 from the two lidar 2 installed at the two locations, and a lidar point cloud image 265 based on 3D point cloud data integrated from the 3D point cloud data of the two lidar 2 are displayed.

[0109] Furthermore, the installation confirmation screen 261 is provided with a virtual object designation section 267. The virtual object designation section 267 is provided with buttons 268 for designating large buses, motorcycles, pedestrians, passenger cars, and trailers as virtual objects for moving objects.

[0110] As shown in Figure 17, when a user operates button 268 to specify a virtual object of a moving body, an image 271 of the specified virtual object of the moving body appears on the LiDAR point cloud image 265. At this time, the traffic flow measurement server 3 performs a process to place the specified virtual object of the moving body in a three-dimensional space containing three-dimensional point cloud data, and a LiDAR point cloud image 265 is generated, which is a three-dimensional space containing the virtual object of the moving body and the point cloud of the three-dimensional point cloud data, as viewed from the specified viewpoint.

[0111] The display area for the LiDAR point cloud image 265 has a 3D viewer function, allowing the user to move the viewpoint and display the LiDAR point cloud image 265 from any viewpoint (free viewpoint display). Furthermore, the position and angle of the virtual object of the moving body can be adjusted by manipulating the image 271 of the virtual object that appears on the LiDAR point cloud image 265. This allows the virtual object of the moving body to be positioned appropriately on the 3D point cloud data. Specifically, by adjusting the position and angle of the virtual object while operating the 3D viewer, the image 271 of the virtual object of the moving body can be positioned appropriately on the road.

[0112] Furthermore, the installation confirmation screen 261 is equipped with a button 273 for overlaying virtual objects, an OK button 274, and a button 275 for resetting settings. The user confirms that the positional relationship between the virtual object of the moving object and the 3D point cloud data has been appropriately adjusted based on the placement state of the virtual object image 271 on the LiDAR point cloud image 265, and then operates the button 273 for overlaying virtual objects.

[0113] As shown in Figure 18, when the user operates the virtual object superimposition button 273, the sensor image display unit 262 superimposes an image 277 of a virtual object corresponding to an image 271 of a virtual object of a moving object placed on the LiDAR point cloud image 265 onto the camera image 263, and a similar image 278 of the virtual object is superimposed onto the LiDAR intensity image 264. The user visually checks whether the image 277 of the virtual object of the moving object is displayed correctly on the camera image 263, and whether the image 278 of the virtual object of the moving object is displayed correctly on the LiDAR intensity image 264.

[0114] At this time, the traffic flow measurement server 3 superimposes images 277 and 278 of the moving virtual object onto the camera image 263 and the lidar intensity image 264, respectively, based on the positional relationship between the 3D point cloud data and the virtual object, and the correspondence between the camera image acquired by the alignment process and the 3D point cloud data. In addition, the camera image 263 and the lidar intensity image 264 display images 277 and 278 of the moving virtual object in a deformed state corresponding to the shape of the camera image 263 and the lidar intensity image 264, respectively.

[0115] Here, if the user confirms that the images 277 and 278 of the moving virtual object are not displayed correctly on the camera image 263 or the lidar intensity image 264, respectively, they can adjust the position and angle of the image 271 of the moving virtual object on the lidar point cloud image 265 again. Next, the user can check again whether the images 277 and 278 of the moving virtual object are displayed correctly on the camera image 263 and the lidar intensity image 264 by operating the virtual object superimposition button 273 again. Here, if the user confirms that the images 277 and 278 of the moving virtual object are displayed correctly on the camera image 263 and the lidar intensity image 264, they operate the OK button 274.

[0116] In this example, pressing the virtual object superimposition button 273 again updates the images 277 and 278 of the virtual object on the camera image 263 and the lidar intensity image 264. However, the images 277 and 278 of the virtual object on the camera image 263 and the lidar intensity image 264 may be updated in real time in response to adjustments to the position and angle of the image 271 of the virtual object on the lidar point cloud image 265.

[0117] Here, the traffic flow measurement server 3 determines for each camera image 263 whether the image 277 of the virtual object of the moving body extends beyond the display range of the camera image 263, and also determines whether the image 278 of the virtual object of the moving body extends beyond the display range of the LiDAR intensity image 264.

[0118] In this case, for example, as shown in Figure 19, if the image 277 of the virtual object of the moving object extends beyond the display range of the camera image 263, the display frame of the camera image 263 is highlighted as a notification to the user. Specifically, a frame image 281 of a predetermined color (e.g., red) is displayed within the display frame of the camera image 263. If the image 278 of the virtual object of the moving object extends beyond the display range of the LiDAR intensity image 264, the display frame of the LiDAR intensity image 264 is highlighted, similar to the case of the camera image 263.

[0119] In this case, the user can adjust the position and angle of the virtual object image 271 of the moving object on the lidar point cloud image 265 again. However, if this readjustment on the lidar point cloud image 265 is insufficient, the user operates the reset setting button 275. This returns the user to the basic adjustment process in sensor installation adjustment and transitions to the basic adjustment screen 201 (see Figure 8).

[0120] In this example, if the images 277 and 278 of the moving virtual object extend beyond the display range of the sensor image (camera image 263 and lidar intensity image 264), the display frame of the sensor image is highlighted in a predetermined color (e.g., red). However, highlighting of the sensor image is not limited to changing the color of the display frame. For example, the display frame of the sensor image may be made to blink, or the line type (dashed line, dotted line, etc.) of the display frame of the sensor image may be changed.

[0121] In this way, on the installation confirmation screen 261, as the virtual object of the moving body is placed in a 3D space containing 3D point cloud data, images 277 and 278 of the virtual object of the moving body are superimposed on the sensor images (camera image 263, lidar intensity image 264). This allows the user to easily check whether the sensors (camera 1, lidar 2) are set to properly detect moving bodies that appear within the measurement area after adjusting the installation state of the sensors.

[0122] Furthermore, if the images 277 and 278 of the moving virtual object extend beyond the display range of the sensor images (camera image 263, lidar intensity image 264), the level of warning to the user may be changed based on the degree of the extension and the priority of the sensor image in which the extension was detected.

[0123] Next, we will describe another example of the installation confirmation screen 261 displayed on the user terminal 4. Figure 20 is an explanatory diagram showing another example of the installation confirmation screen 261.

[0124] The example shown in Figure 16 involves multiple sensors (camera 1 and lidar 2) being installed to detect moving objects in the measurement area from opposite sides. Specifically, sensors are installed at two points opposite each other across an intersection (measurement area), and the sensors at the two points detect the same location from different directions.

[0125] On the other hand, in this example, multiple sensors (camera 1 and lidar 2) are installed so that their respective measurement areas are adjacent and partially overlap. Specifically, the measurement area is a wide intersection and its surroundings, with sensors installed at four points around the intersection. Each sensor at each point mainly detects the center of the intersection, and although their respective measurement areas are adjacent and partially overlap, they are installed so that each of the multiple roads connected to the intersection is included in the detection area. As a result, the detection areas of the sensors at each point are significantly shifted.

[0126] In the installation confirmation screen 261 in this example, the sensor image display unit 262 displays four camera images 263 and one LiDAR point cloud image 265. The four camera images 263 were captured by four cameras 1 installed at four locations. The single LiDAR point cloud image 265 was generated from 3D point cloud data compiled from 3D point cloud data from four LiDARs 2 installed at four locations.

[0127] In this example, as in the example shown in Figure 17, when the user selects a virtual object for a moving object in the virtual object selection unit 267, the image 271 of the selected virtual object for the moving object appears on the lidar point cloud image 265. When the user operates the virtual object superimposition button 273, the image 277 of the virtual object for the moving object is superimposed and displayed on the camera image 263.

[0128] In this example, as in the example shown in Figure 19, if the image 277 of the virtual object of the moving body extends beyond the display range of the camera image 263, the camera image 263 is highlighted. Furthermore, if readjusting the position and angle of the virtual object of the moving body in the lidar point cloud image 265 is insufficient, operating the reset setting button 275 returns the user to the basic adjustment process in sensor installation adjustment.

[0129] In this example, even if the sensors at each location (camera 1 and lidar 2) are installed with adjacent and partially overlapping measurement areas, but with significantly different detection areas, the user can easily verify whether the sensors are set up to properly detect moving objects appearing within the measurement area.

[0130] Next, we will explain the sensor data recording screen 301 displayed on the user terminal 4. Figure 21 is an explanatory diagram showing the sensor data recording screen 301.

[0131] On the user terminal 4, when the user operates the traffic flow data generation button 103 on the main menu screen 101 (see Figure 5), the submenu screen 121 (see Figure 6(B)) is displayed, and the user operates the sensor data recording button 122, the sensor data recording screen 301 shown in Figure 21 is displayed.

[0132] The sensor data recording screen 301 is provided with a sensor image display unit 302. The sensor image display unit 302 displays the camera image 303 and the lidar intensity image 304. In this example, since camera 1 and lidar 2 are installed at two locations, two camera images 303, which are the detection results of camera 1, and two lidar intensity images 304, which are the detection results of lidar 2, are displayed.

[0133] On the sensor data recording screen 301, the user can specify the measurement points to be recorded using the measurement point specification unit 163. As a result, camera images 303 and lidar intensity images 304 from cameras and lidar installed at the specified measurement points are displayed on the sensor image display unit 302. In the measurement point specification unit 163, the user can select a measurement point from a list of pre-registered measurement points by using a pull-down menu. If the measurement point is not registered, the user can enter the name of the measurement point into the measurement point specification unit 163 to register the measurement area.

[0134] The sensor data recording screen 301 is also provided with a recording start button 305 and a recording end button 306. When the user operates the recording start button 305, the traffic flow measurement server 3 starts the sensor data recording process. During the sensor data recording process, camera images transmitted from camera 1 are stored in the storage unit 12. Lidar point cloud data transmitted from Lidar 2 is also stored in the storage unit 12. When the user operates the recording end button 306, the traffic flow measurement server 3 terminates the traffic flow data recording process.

[0135] Alternatively, the system may be configured so that sensor data recording is performed until a user-specified measurement time has elapsed, after the user has set a timer. Furthermore, the system may be configured so that sensor data recording is performed from a user-specified start time to an end time, after the user has set a schedule in advance.

[0136] Next, we will explain the sensor data analysis screen 311 displayed on the user terminal 4. Figure 22 is an explanatory diagram showing the sensor data analysis screen 311 in its initial state. Figure 23 is an explanatory diagram showing the sensor data analysis screen 311 when the sensor data analysis process has started.

[0137] On the user terminal 4, when the user operates the traffic flow data generation button 103 on the main menu screen 101 (see Figure 5), a submenu screen 121 (see Figure 6(B)) is displayed. When the user operates the sensor data analysis button 123, the sensor data analysis screen 311 shown in Figure 22 is displayed.

[0138] The sensor data analysis screen 311 shown in Figure 22 is equipped with a sensor image display unit 312. The sensor image display unit 312 displays camera images 313 and lidar intensity images 314. In this example, since cameras 1 and lidars 2 are installed at two locations, two camera images 313, which are the detection results of each camera 1, and two lidar intensity images 314, which are the detection results of each lidar 2 and are generated from 3D point cloud data, are displayed.

[0139] On the sensor data analysis screen 311, the user can specify the measurement points to be analyzed using the measurement point specification unit 163. This reads out camera images and 3D point cloud data related to the specified measurement points, and the camera images 313 and lidar intensity images 314 are displayed on the sensor image display unit 312.

[0140] Furthermore, the sensor data analysis screen 311 is equipped with a button 316 to start analysis and a button 317 to end analysis. When the user operates the start analysis button 316, the traffic flow measurement server 3 starts the sensor data analysis process (traffic flow analysis process).

[0141] In the sensor data analysis process, camera images and LiDAR point cloud data stored in the memory unit 12 are read out, and processes such as detecting moving objects from the camera images and LiDAR point cloud data, and extracting events (such as traffic accidents) that correspond to predetermined scenarios from the traffic flow data are performed. When the user operates the analysis end button 317, the traffic flow measurement server 3 terminates the sensor data analysis process.

[0142] As shown in Figure 23, when the sensor data analysis process starts, the sensor image display unit 312 displays the camera image 313, the lidar intensity image 314, and the lidar point cloud image 315. The camera image 313, lidar intensity image 314, and lidar point cloud image 315 can be displayed as a video. The lidar point cloud image 315 is generated from 3D point cloud data that integrates the 3D point cloud data of each lidar 2 installed at two locations.

[0143] At this time, the sensor image display unit 312 displays the tracking frame of the moving object detected from the camera image 313 on the camera image 313. In addition, the tracking frame of the moving object detected from the 3D point cloud data is displayed on the LiDAR intensity image 314. Furthermore, the tracking frame of the moving object detected from the 3D point cloud data is displayed on the LiDAR point cloud image 315.

[0144] Next, the time-series display screen 401 displayed on the user terminal 4 will be described. Figures 24 and 25 are explanatory diagrams showing the time-series display screen 401. Figure 26 is an explanatory diagram showing the trajectory line 407, velocity line 408, and acceleration line 409 displayed on the time-series display screen 401.

[0145] On the user terminal 4, when the user operates the traffic flow data viewing button 104 on the main menu screen 101 (see Figure 5), a submenu screen 131 (see Figure 7(A)) is displayed. When the user operates the time series display button 132, the time series display screen 401 shown in Figure 24 is displayed.

[0146] The time-series display screen 401 is equipped with a sensor image display unit 402. The sensor image display unit 402 displays the camera image 403, the LiDAR intensity image 404, and the LiDAR point cloud image 405. The display area of ​​the LiDAR point cloud image 405 has a 3D viewer function, and the LiDAR point cloud image 405 can be displayed from any viewpoint by the user moving the viewpoint. Figures 24 and 25 show examples of when the viewpoint of the LiDAR point cloud image 405 is changed.

[0147] In the sensor image display unit 402, the trajectory line 411, velocity line 412, and acceleration line 413 are superimposed on the camera image 403, lidar intensity image 404, and lidar point cloud image 405 as a behavior image that visualizes time-series data representing the behavior (state change) of the moving object. The trajectory line 411 (trajectory image) visualizes time-series data representing the change in the position of the moving object. The velocity line 412 (velocity image) visualizes time-series data representing the change in the velocity of the moving object. The acceleration line 413 (acceleration image) visualizes time-series data representing the change in the acceleration of the moving object.

[0148] Here, as shown in Figure 26, a trajectory point 414 representing the position of the moving object at the displayed time (the currently displayed time) is drawn on the trajectory line 411. A velocity point 415 representing the velocity of the moving object at the displayed time is drawn on the velocity line 412. An acceleration point 416 representing the acceleration of the moving object at the displayed time is drawn on the acceleration line 413. The display positions of the trajectory point 414, velocity point 415, and acceleration point 416 change as the displayed time progresses.

[0149] Furthermore, if the trajectory point 414, which represents the position of the moving object at the displayed time, is taken as the origin, and the direction of travel is taken as the first coordinate axis, then the second and third coordinate axes, which are orthogonal to the first coordinate axis, represent the magnitude (absolute value) of velocity and the magnitude (absolute value) of acceleration, respectively. The distance in the direction of velocity from the trajectory point 414 (the origin) to the velocity point 415 represents the magnitude (absolute value) of velocity. The distance in the direction of acceleration from the trajectory point 414 (the origin) to the acceleration point 416 represents the magnitude (absolute value) of acceleration.

[0150] Therefore, the trajectory line 411 connects the trajectory points 414 at each time point and represents the change in the position of the moving object. The velocity line 412 connects the velocity points 415 at each time point and represents the change in the velocity of the moving object. The acceleration line 413 connects the acceleration points 416 at each time point and represents the change in the acceleration of the moving object.

[0151] Furthermore, as shown in Figures 24 and 25, the sensor image display unit 402 displays an ID label 417 (label image) containing the moving object ID near the trajectory line 411. A velocity label 418 (label image) containing the velocity (absolute value) is displayed near the velocity line 412. An acceleration label 419 (label image) containing the acceleration (absolute value) is displayed near the acceleration line 413.

[0152] Furthermore, the sensor image display unit 402 displays the tracking frame of the moving object detected from the camera image 403 on the camera image 403. In addition, the tracking frame of the moving object detected from the 3D point cloud data is displayed on the LiDAR intensity image 404. Furthermore, the tracking frame of the moving object detected from the 3D point cloud data is displayed on the LiDAR point cloud image 405.

[0153] Furthermore, the time-series display screen 401 is provided with a button 421 for the next frame and a button 422 for the previous frame. When the user operates the button 421 for the next frame, the camera image 403, the lidar intensity image 404, and the lidar point cloud image 405 switch to the next frame, i.e., the image from the next time. When the user operates the button 422 for the previous frame, the camera image 403, the lidar intensity image 404, and the lidar point cloud image 405 switch to the previous frame, i.e., the image from the previous time.

[0154] Note that the methods for representing the trajectory (position), velocity, and acceleration of a moving object are not limited to the examples shown. For example, velocity and acceleration can be represented by attributes of the trajectory line. Specifically, the intensity and thickness of the color of the trajectory line may represent velocity and acceleration.

[0155] In this way, the time-series display screen 401 visualizes and displays the changes in the state of the moving object over time. Specifically, behavioral images that visualize the position, velocity, and acceleration, specifically the trajectory line 411, velocity line 412, and acceleration line 413, are superimposed on the sensor images (camera image 403, lidar intensity image 404, and lidar point cloud image 405). This allows the user to intuitively grasp the changes in the state of the moving object (position, velocity, and acceleration). Note that the user may also select and display any image from the types of behavioral images (trajectory line 411, velocity line 412, acceleration line 413) and label images (ID label 417, velocity label 418, acceleration label 419) from a display selection screen (not shown).

[0156] Next, we will explain the scenario selection screen 431 displayed on the user terminal 4. Figure 27 is an explanatory diagram showing the scenario selection screen 431. Figure 28 is an explanatory diagram showing the scenario selection screen 431 in the state where extraction conditions have been added.

[0157] On the user terminal 4, when the user operates the traffic flow data viewing button 104 on the main menu screen 101 (see Figure 5), a submenu screen 131 (see Figure 7(A)) is displayed. When the user operates the scenario selection button 133, the scenario selection screen 431 shown in Figure 27 is displayed.

[0158] The scenario selection screen 431 includes an extraction condition selection unit 432 and an overview diagram display unit 433. In the extraction condition selection unit 432, the user can select a scenario (event type) as an extraction condition (narrowing condition) by operating a pull-down menu. In this example, the user can select scenarios such as rear-end collision, right-turn collision, left-turn collision, wrong-way driving, and road rage. When the user selects a scenario, an overview diagram 434 related to the selected scenario is displayed in the overview diagram display unit 433. The overview diagram 434 specifically represents the situation of the scenario.

[0159] If there are multiple patterns for the selected scenario, an overview diagram 434 for each pattern will be displayed. In the example shown in Figure 27, the first pattern is a collision between a right-turning vehicle and a straight-going vehicle, and the second pattern is a collision between a right-turning vehicle and a straight-going motorcycle. The user can select a pattern by manipulating the overview diagram 434.

[0160] Furthermore, the scenario selection screen 431 is provided with an extraction display button 436. When a user selects a scenario as an extraction condition in the extraction condition selection unit 432 and then operates the extraction display button 436, the extraction process is executed, and the system transitions to the specified event viewing screen 471 (see Figure 30), which displays the extraction results. In the extraction process, events (such as traffic accidents) detected by traffic flow analysis (event detection) that correspond to the scenario selected by the user are extracted.

[0161] Here, on the scenario selection screen 431, when the user selects a scenario as an extraction condition in the extraction condition selection unit 432, the extraction condition addition specification unit 441 (dialog box) is displayed. The extraction condition addition specification unit 441 is provided with a "Yes" button 442 and a "No" button 443. When the user operates the "Yes" button 442, the screen transitions to the scenario selection screen 431 in the state where the extraction condition has been added, as shown in Figure 28.

[0162] As shown in Figure 28, the scenario specification screen 431 in the state where extraction conditions have been added displays the extraction condition selection unit 432 and overview diagram display unit 433 related to the initial extraction conditions, as well as the extraction condition selection unit 445 and overview diagram display unit 446 related to the additional extraction conditions. This allows the events to be extracted to be narrowed down by combining scenarios.

[0163] Thus, on the scenario selection screen 431, the user can specify the scenario (type of event) they are interested in, and this allows the system to extract events (such as traffic accidents) that correspond to that scenario.

[0164] The scenarios may include, but are not limited to, traffic accidents such as rear-end collisions, right-turn collisions, and left-turn collisions, as well as other traffic rule violations and dangerous driving such as driving the wrong way or road rage. Furthermore, users may set the content of the scenarios.

[0165] Next, we will explain the statistics information specification screen 461 displayed on the user terminal 4. Figure 29 is an explanatory diagram showing the statistics information specification screen 461.

[0166] On the user terminal 4, when the user operates the button 104 for viewing traffic flow data on the main menu screen 101 (see Figure 5), a submenu screen 131 (see Figure 7(A)) is displayed. When the user operates the button 134 for specifying statistical information, the statistical information specification screen 461 shown in Figure 29 is displayed.

[0167] The statistical information specification screen 461 is provided with a first statistical information display unit 462 (graph display unit) and a second statistical information display unit 463 (summary table display unit). The first statistical information display unit 462 displays the number (frequency) of events corresponding to each scenario as statistical information in a bar graph for each scenario. The second statistical information display unit 463 displays the number (frequency) of events corresponding to the combination of scenarios as statistical information in a summary table.

[0168] In the first statistical information display unit 462, the user can select one scenario by manipulating a bar graph for each scenario. In the second statistical information display unit 463, the user can select a combination of scenarios by manipulating a single cell in the summary table.

[0169] Furthermore, the statistical information selection screen 461 is provided with an extract display button 464. When a user selects a scenario in the first statistical information display unit 462 and then operates the extract display button 464, a process is performed to extract events corresponding to the selected scenario, and the system transitions to the specified event viewing screen 471 (see Figure 30) which displays the extraction results. Also, when a user selects a combination of scenarios in the second statistical information display unit 463 and then operates the extract display button 464, a process is performed to extract events corresponding to the selected combination of scenarios, and the system transitions to the specified event viewing screen 471 (see Figure 30) which displays the extraction results.

[0170] Thus, on the statistical information selection screen 461, users can check the status (frequency) of events corresponding to a scenario using statistical information (graphs and summary tables), select the scenario they want to view from the statistical information, and extract events (such as traffic accidents) that correspond to that scenario.

[0171] Furthermore, as shown in Figures 27 and 28, when a user selects a scenario using a pull-down menu, the statistical information (graphs and summary tables) in the statistical information display units 462 and 463 of the statistical information specification screen 461 shown in Figure 29 may be displayed in a way that is limited to the scenario selected by the user.

[0172] Next, we will explain the specified event viewing screen 471 displayed on the user terminal 4. Figures 30 and 31 are explanatory diagrams showing the specified event viewing screen 471.

[0173] On user terminal 4, when the user specifies a scenario on the scenario specification screen 431 (see Figures 27 and 28) or the statistics information specification screen 461 (see Figure 29) and instructs the extraction and display, the specified event viewing screen 471 shown in Figure 30 is displayed.

[0174] The specified event viewing screen 471 is provided with an overall image display section 472, a detailed image display section 473, a first detail button 474, and a second detail button 475.

[0175] The overall image display unit 472 displays a lidar point cloud image 476 as an overall image, showing the overall situation of the event. The lidar point cloud image 476 is generated from 3D point cloud data from lidar 2, with the viewpoint set above the measurement area.

[0176] In the overall image display unit 472, moving objects related to events corresponding to the scenario specified by the user on the scenario selection screen 431 (see Figures 27 and 28) or the statistics information selection screen 461 (see Figure 29) are highlighted. In the example shown in Figure 30, tracking frames are displayed around two vehicles related to a traffic accident (right-turn collision) as a specific event.

[0177] The detailed image display unit 473 displays enlarged LiDAR point cloud images 477 and 478 as detailed images, allowing the user to understand the details of the events corresponding to the scenario specified by the user.

[0178] When the user operates the first detail button 474, a LiDAR point cloud image 477 from the driver's perspective is displayed as a detailed image from the first viewpoint. Here, the driver is the person operating the vehicle, which is a moving object related to the event being studied. When the user operates the second detail button 475, a LiDAR point cloud image 478 (orthophoto) with the viewpoint set above the measurement area is displayed as a detailed image from the second viewpoint.

[0179] In this way, on the specified event viewing screen 471, by specifying a scenario (event type) of interest to the user on the scenario specification screen 431 (see Figures 27 and 28) or the statistical information specification screen 461 (see Figure 29), the user can view sensor images (LiDAR point cloud images 477, 478) showing events (such as traffic accidents) corresponding to that scenario. This allows the user to check in detail the circumstances when an event corresponding to a specific scenario occurs.

[0180] In this example, the user can choose between a LiDAR point cloud image 476 with the driver's viewpoint set and a LiDAR point cloud image 477 with the viewpoint set above the measurement area. However, the display frame for the LiDAR point cloud image may also have 3D viewer functionality, allowing the user to display a LiDAR point cloud image from any viewpoint.

[0181] Furthermore, while it is possible to generate LiDAR point cloud images 476 and 477 from any viewpoint using 3D point cloud data from LiDAR 2, it is also possible to generate images from any viewpoint by generating a dense point cloud from multiple camera images from multiple cameras 1 using multi-view stereo technology.

[0182] Next, we will explain the tracking mode screen 501 displayed on the user terminal 4. Figure 32 is an explanatory diagram showing the tracking mode screen 501 in the multiple-location installation mode. Figure 33 is an explanatory diagram showing the tracking mode screen 501 in the single-location installation mode.

[0183] On the user terminal 4, when the user operates the option button 105 on the main menu screen 101 (see Figure 5), the submenu screen 141 (see Figure 7(B)) is displayed. When the user operates the tracking mode button 142, the tracking mode screen 501 shown in Figure 32 is displayed.

[0184] The tracking mode screen 501 is provided with a mode selection unit 502. The mode selection unit 502 is provided with a button 503 for the multiple-location installation mode and a button 504 for the single-location installation mode. When the user operates the button 503 for the multiple-location installation mode, the tracking mode screen 501 for the multiple-location installation mode shown in Figure 32 is displayed. When the user operates the button 504 for the single-location installation mode, the system transitions to the tracking mode screen 501 for the single-location installation mode shown in Figure 33. Here, the multiple-location installation mode is when the camera 1 and lidar 2 are installed at multiple locations targeting a common measurement area. The single-location installation mode is when the camera 1 and lidar 2 are installed at one location.

[0185] The tracking mode screen 501 is provided with a moving object image display unit 505. The moving object image display unit 505 displays an image 506 of the moving object detected from the camera image and an image 507 of the moving object detected from the 3D point cloud data. The image 506 of the moving object detected from the camera image is extracted from the image region containing the moving object from the camera image. The image 507 of the moving object detected from the 3D point cloud data is extracted from the image region containing the moving object from the LiDAR point cloud image generated from the 3D point cloud data. If the image 507 of the moving object contains multiple moving objects, it is preferable to draw a frame image surrounding the target moving object on the image 507.

[0186] In the tracking mode screen 501 of the multiple-location installation mode shown in Figure 32, the image 506 of the moving object detected from the camera image is displayed for each camera 1. In this example, since two cameras 1 are installed, two images 506 of the moving object are displayed. In addition, since the moving object is detected from 3D point cloud data that integrates multiple 3D point cloud data from multiple LiDAR 2 installed at multiple locations, one image 507 of the moving object detected from the 3D point cloud data is displayed.

[0187] On the other hand, in the tracking mode screen 501 of the single-location installation mode shown in Figure 33, one image 506 of the moving object detected from the camera image and one image 507 of the moving object detected from the 3D point cloud data are displayed.

[0188] Furthermore, the mobile object image display unit 505 displays the mobile object ID assigned to the mobile object when it is detected from the camera image, and the mobile object ID assigned to the mobile object when it is detected from the 3D point cloud data. Since the detection of mobile objects and the assignment of mobile object IDs from each camera image and 3D point cloud data are performed individually, the mobile object IDs corresponding to each camera 1 and LiDAR 2 will be different even for the same mobile object.

[0189] Furthermore, the tracking mode screen 501 is equipped with a camera priority button 511, a lidar priority button 512, and a settings button 513. When the user operates the camera priority button 511, the traffic flow measurement server 3 prioritizes the IDs assigned to moving objects detected by camera images and reassigns the IDs of the moving objects. When the user operates the lidar priority button 512, the traffic flow measurement server 3 prioritizes the IDs assigned to moving objects detected by lidar point cloud data and reassigns the IDs of the moving objects. Note that camera images and lidar point cloud data have advantages and disadvantages depending on the detection scene, such as the conditions of the measurement area and weather. For example, the user may specify that the sensor expected to have higher accuracy in detecting moving objects be prioritized.

[0190] When the mobile object ID is changed, the mobile object image display unit 505 updates the mobile object IDs assigned to the mobile objects detected in both the camera image and the LiDAR point cloud image, and the same mobile object ID is displayed for the same mobile object. At this point, the user confirms that the mobile object ID has been changed correctly and operates the setting button 513. This confirms the mobile object ID.

[0191] In this way, on the tracking mode screen 501, when reassigning the mobile object IDs assigned to mobile objects detected from multiple detection results (camera images, 3D point cloud data) from multiple sensors (camera 1, lidar 2) so that a common mobile object ID is assigned to the same mobile object, the user can select the sensor to prioritize.

[0192] Next, we will describe the extended browsing mode screen 531 displayed on the user terminal 4. Figure 34 is an explanatory diagram showing the extended browsing mode screen 531 in viewer mode. Figure 35 is an explanatory diagram showing the extended browsing mode screen 531 in danger judgment mode.

[0193] On the user terminal 4, when the user operates the options button 105 on the main menu screen 101 (see Figure 5), the submenu screen 141 (see Figure 7(B)) is displayed. When the user operates the extended browsing mode button 143, the extended browsing mode screen 531 shown in Figure 34 is displayed.

[0194] The extended viewing mode screen 531 is provided with a sensor image display unit 532. The sensor image display unit 532 displays a camera image 533 and a lidar point cloud image 534. In this example, since cameras 1 are installed at two locations, two camera images 533 taken by each camera 1 are displayed. The lidar point cloud image 534 is generated by setting a viewpoint above the measurement area based on 3D point cloud data obtained by integrating 3D point cloud data from lidars 2 installed at two locations.

[0195] Furthermore, the extended viewing mode screen 531 is provided with a mode selection unit 541, a road structure selection unit 542, a driving object selection unit 543, and an automatic driving selection unit 544.

[0196] The mode selection unit 541 is equipped with a viewer button 551 and a danger detection button 552. When the user operates the viewer button 551, the extended viewing mode screen 531 of the viewer mode (see Figure 34) is displayed. When the user operates the danger detection button 552, the extended viewing mode screen 531 of the danger detection mode (see Figure 35) is displayed.

[0197] The road structure designation section 542 is provided with a button 553 for selecting road structures (ground features and road markings). In this example, by operating the button 553, the user can select white lines, stop lines, curbs, pedestrian crossings, guardrails, and sidewalks as road ancillary structures. The selected road structures are highlighted on the camera image 533 and the LiDAR point cloud image 534. Specifically, a region image 561 (ancillary image) drawn in a predetermined color or pattern is superimposed transparently over the area of ​​the target road structure in the camera image 533 and the LiDAR point cloud image 534. In this example, the areas of the stop line, pedestrian crossing, and sidewalk are highlighted. The region images 561 of the road structures are drawn in colors or patterns set for each type of road structure. For example, the region image 561 of a pedestrian crossing is drawn in blue, and the region image 561 of a sidewalk is drawn in red. This allows the user to easily identify the type of road structure. Multiple road structures can be selected.

[0198] The driving object designation unit 543 is provided with a button 554 for selecting a driving object (moving object). In this example, by operating the button 554, a passenger car, truck, motorcycle, bicycle, bus, and pedestrian can be selected as driving objects. The selected driving object is highlighted on the LiDAR point cloud image 534. Specifically, a region image 562 (ancillary image) drawn with a predetermined color or pattern is superimposed transparently on the area of ​​the target driving object in the LiDAR point cloud image 534. The region image 562 of the driving object is drawn with a color or pattern set for each type of driving object. For example, the region image 562 of a passenger car is drawn in light blue, and the region image 562 of a truck is drawn in yellow. This allows the user to easily identify the type of driving object. Multiple driving objects can be selected.

[0199] The autonomous driving designation unit 544 is equipped with buttons 555 and 556 for selecting whether or not to use an autonomous vehicle. When the user operates the ON button 555, the autonomous vehicle is highlighted on the LiDAR point cloud image 534, and an autonomous driving label 563 (attached image) with the words "autonomous driving" is displayed. When the user operates the OFF button 556, the autonomous vehicle is not highlighted on the LiDAR point cloud image 534.

[0200] Furthermore, in the extended viewing mode screen 531, when a road structure is selected on the LiDAR point cloud image 534, specifically when the region image 561 of the road structure or the region image 562 of the driving object superimposed on the LiDAR point cloud image 534 is manipulated, a positional relationship label 564 (accompanying image) containing information about the positional relationship between the driving object and the road structure is displayed. In the example shown in Figure 34, when the user selects a truck and a pedestrian crossing on the LiDAR point cloud image 534, a positional relationship label 564 containing the distance between the truck and the pedestrian crossing is displayed.

[0201] Furthermore, the extended viewing mode screen of the hazard determination mode shown in Figure 35 is provided with a hazard level display unit 565. The hazard level display unit 565 displays the hazard level related to the traffic environment at the target location. At this time, the traffic flow measurement server 3 determines the hazard level related to the traffic environment at the target location based on information about the traffic environment at the target location, specifically, the positional relationship between moving objects (driving objects) and road components.

[0202] In the extended viewing mode screen 531, the areas of moving objects and road structures specified by the user are highlighted in the LiDAR point cloud image 534, allowing the user to easily understand the relative positional relationship between the moving object and the road structure. Furthermore, the extended viewing mode screen 531 displays information regarding the positional relationship between the moving object and the road structure (such as distance) and information regarding the risk level of the traffic environment at the target location, allowing the user to easily recognize the risk posed by the moving object. This enables consideration of necessary measures to reduce traffic accidents by improving road structures, such as installing guardrails at high-risk locations.

[0203] Next, the procedure for sensor installation and adjustment performed on the traffic flow measurement server 3 will be explained. Figure 36 is a flowchart showing the procedure for sensor installation and adjustment. Here, the user sequentially selects items from the submenu screen 111 (see Figure 6(A)) related to sensor installation and adjustment displayed on the user terminal 4, thereby sequentially performing the following processes: basic adjustment, positioning, and installation confirmation. Prior to this flow, the operator installs the sensors (camera 1 and lidar 2) at the designated locations. This determines the position of the sensors, and in the sensor installation and adjustment process, the orientation (field of view) of the sensors is adjusted.

[0204] The traffic flow measurement server 3 first proceeds to the basic adjustment process, and in response to the user's operation on the basic adjustment screen 201 (see Figures 8 to 11) displayed on the user terminal 4, reads the CG image file, sends the CG image file to the user terminal 4, and displays the CG image on the user terminal 4 (ST101).

[0205] Next, the traffic flow measurement server 3 controls the angle (pan and tilt) of the sensors (camera 1, lidar 2) in response to user operations on the basic adjustment screen 201 (see Figures 8 to 11) displayed on the user terminal 4 (ST102). At this time, the sensor images (camera images, lidar intensity images) transmitted from the sensors are sent to the user terminal 4 and displayed on the user terminal 4.

[0206] Next, the traffic flow measurement server 3 proceeds to the alignment process and performs alignment processing to correct the positional misalignment of 3D point cloud data between multiple cameras 1 and LiDAR 2 installed at different locations, in response to user operations on the alignment screen 231 (see Figures 12 to 15) displayed on the user terminal 4. Then, the traffic flow measurement server 3 sends the LiDAR point cloud image generated from the integrated 3D point cloud data after alignment to the user terminal 4 and displays the LiDAR point cloud image on the user terminal 4 (ST103).

[0207] Furthermore, in ST103, if the traffic flow measurement server 3 cannot adequately correct the positional misalignment of 3D point cloud data from multiple LiDAR 2s through automatic alignment, it can correct the positional misalignment of 3D point cloud data from multiple LiDAR 2s according to user operation through manual alignment.

[0208] Next, the traffic flow measurement server 3 proceeds to the installation confirmation process. In response to the user's operation on the installation confirmation screen 261 (see Figures 16 to 20) displayed on the user terminal 4, it places a virtual object of the moving body in a three-dimensional space containing three-dimensional point cloud data, and generates a LiDAR point cloud image including the virtual object of the moving body. The traffic flow measurement server 3 then transmits the LiDAR point cloud image including the virtual object of the moving body to the user terminal 4, and displays the LiDAR point cloud image on the user terminal 4 (ST104).

[0209] Next, the traffic flow measurement server 3 generates camera images and LiDAR intensity images, including virtual objects of moving objects, in response to user operations on the installation confirmation screen 261 (see Figures 16 to 20) displayed on the user terminal 4. Then, the traffic flow measurement server 3 transmits the camera images and LiDAR intensity images, including virtual objects of moving objects, to the user terminal 4 for display (ST105).

[0210] Furthermore, in ST105, if there are any defects in the display state of the virtual object of the moving object in the camera image and the lidar intensity image, the processes of ST104 and ST105 are repeated in order to adjust the display state of the virtual object of the moving object.

[0211] Next, the traffic flow measurement server 3 stores the sensor installation information acquired during the basic adjustment, alignment, and installation confirmation processes in the storage unit 12 (ST106). The sensor installation information includes information about the angles of the sensors (camera 1, lidar 2), information about the positional relationship between the camera image and the 3D point cloud data, and information about the relative positional relationships between the 3D point cloud data from multiple lidar 2s.

[0212] Next, we will explain the procedure for generating traffic flow data performed on the traffic flow measurement server 3. Figure 37 is a flowchart showing the procedure for generating traffic flow data. Here, the user sequentially selects items from the submenu screen 121 (see Figure 6(B)) related to traffic flow data generation displayed on the user terminal 4, and the following processes related to data recording and data analysis are carried out sequentially.

[0213] The traffic flow measurement server 3 first proceeds to the data recording process and receives camera images from camera 1 (ST201). The traffic flow measurement server 3 also receives 3D point cloud data from lidar 2 (ST202).

[0214] Next, the traffic flow measurement server 3 synchronizes the camera image and the 3D point cloud data based on the time information attached to the camera image received from camera 1 and the time information attached to the 3D point cloud data received from lidar 2 (data synchronization process) (ST203). The time information is obtained from satellite signals by camera 1 and lidar 2. If the system does not have the capability to receive satellite signals, the time information may be obtained and synchronized via the local network.

[0215] Next, the traffic flow measurement server 3 stores the synchronized camera images and 3D point cloud data in the storage unit 12 (ST204).

[0216] Next, the traffic flow measurement server 3 proceeds to the sensor data analysis process, where it analyzes camera images and 3D point cloud data to generate traffic flow data (ST205). The sensor data analysis process includes detecting moving objects and road components from camera images and lidar point cloud data.

[0217] Next, the traffic flow measurement server 3 stores the traffic flow data generated by the sensor data analysis process in the storage unit 12 (ST206). The traffic flow data includes a timestamp (year, month, day, hour, minute, second), a trajectory ID (information identifying the moving object), and relative coordinates (location information).

[0218] Next, we will explain the procedure for processing traffic flow data viewing on the traffic flow measurement server 3. Figure 38 is a flowchart showing the procedure for processing traffic flow data viewing.

[0219] The traffic flow measurement server 3 first determines which item the user has selected on the submenu screen 131 (see Figure 7(A)) related to viewing traffic flow data displayed on the user terminal 4 (ST301).

[0220] If the user selects the time-series display (in ST301, "Time-series display"), the traffic flow measurement server 3 transitions the user terminal 4 to the time-series display screen 401 (see Figure 24) (ST302). Then, on the time-series display screen 401, when the user specifies a measurement point (measurement area), the traffic flow measurement server 3 extracts the traffic flow data corresponding to the specified measurement point (ST303). Next, on the time-series display screen 401, the traffic flow measurement server 3 launches a viewer and displays the traffic flow data in time series (ST304). At this time, the traffic flow data displays sensor images (camera images, lidar intensity images, and lidar point cloud images) along with the trajectory, speed, and acceleration changes of moving objects.

[0221] On the other hand, if the user selects a scenario (in ST301, "Scenario Specification"), the traffic flow measurement server 3 transitions the user terminal 4 to the scenario specification screen 431 (see Figure 27) (ST305). Then, when the user directly specifies a scenario on the scenario specification screen 431, the traffic flow measurement server 3 extracts sensor images related to the events corresponding to the specified scenario (ST306).

[0222] Next, the traffic flow measurement server 3 transitions the user terminal 4 to the specified event viewing screen 471 (see Figure 30) (ST307). Next, the traffic flow measurement server 3 launches a viewer for viewing sensor images on the specified event viewing screen 471 and displays the sensor images on the specified event viewing screen 471 (ST308). At this time, the sensor images displayed are lidar point cloud images related to the events corresponding to the specified scenario.

[0223] Furthermore, if the user selects to specify statistical information (in ST301, "Specify statistical information"), the traffic flow measurement server 3 transitions the user terminal 4 to the statistical information specification screen 461 (see Figure 29) (ST309). Then, on the statistical information specification screen 461, when the user selects a scenario from the statistical information, the traffic flow measurement server 3 extracts sensor images related to the events corresponding to the selected scenario (ST310). Next, the traffic flow measurement server 3 performs the processes in ST307 and ST308.

[0224] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments. [Industrial applicability]

[0225] The traffic flow measurement system and traffic flow measurement method according to the present invention have the effect of allowing users to confirm in detail the situation when an event corresponding to a specific scenario occurs, and are useful as a traffic flow measurement system and traffic flow measurement method that measures traffic flow at a target location using sensors such as cameras and lidar. [Explanation of symbols]

[0226] 1. Camera (first sensor) 2. Rider (Second Sensor) 3. Traffic flow measurement server (server device) 4. User terminal (terminal device) 5 Management terminal

Claims

1. A first sensor that acquires two-dimensional detection results targeting a traffic flow measurement area, A second sensor that acquires three-dimensional detection results targeting the aforementioned measurement area, A server device connected to the first and second sensors, which acquires sensor images based on the detection results of the first and second sensors and performs traffic flow analysis processing based on the detection results of the first and second sensors, A terminal device connected to this server device via a network, which displays the sensor images and the results of the traffic flow analysis processing, A traffic flow measurement system comprising: The server device is Based on the results of the traffic flow analysis process, an event corresponding to one of several pre-set scenarios is detected, and information about that event is stored. The terminal device displays a statistical information specification screen containing a graph or summary table of statistical information regarding the frequency of occurrence of events corresponding to each of the multiple scenarios. Information on events corresponding to a scenario specified by the user's operation on the graph or summary table of the aforementioned statistical information is extracted. A traffic flow measurement system characterized by generating a traffic flow viewing screen that displays the sensor images related to the extracted events, and transmitting the traffic flow viewing screen to the terminal device.

2. The server device is The traffic flow measurement system according to claim 1, characterized in that, in response to user operation on the terminal device, the sensor image from a viewpoint specified by the user is displayed on the traffic flow viewing screen.

3. The server device is The traffic flow measurement system according to claim 2, characterized in that, in response to user operation on the terminal device, the system displays on the traffic flow viewing screen either an image in which the viewpoint is set to that of the driver of a vehicle, which is a moving object related to the target event, or an image in which the viewpoint is set to that of the sky above the measurement area where the target event occurred, as the sensor image.

Images