Data processing device, data processing method, and data processing program

Abstract

To provide a data processing method capable of generating image data in an initial state without obstacles from multiple image data including obstacles.SOLUTION: A data processing method for processing image data on a computer includes: a data storage step S10 for storing, in a storage unit, first image data obtained by capturing an imaging target and second image data obtained by capturing an imaging target of the first image data at different times; a data extraction step S60 for extracting partial data of an area corresponding to a data update area in the second image data as update data at a data extraction unit when the data update area for updating the data in the first image data is specified; and a data replacement step S80 for replacing the partial data of the data update area in the first image data with the update data at the data replacement unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a data processing device, a data processing method, and a data processing program. [Background technology]

[0002] In recent years, cooperative roadside units called smart poles equipped with various cameras, sensors, communication devices, etc. have begun to be installed as traffic infrastructure devices in order to realize safe driving assistance that supports drivers and automated driving of vehicles, such as unmanned autonomous driving. These cooperative roadside units are installed, for example, at intersections (crossings) without traffic lights, and are capable of communicating (transmitting and receiving) information (data) between traffic monitoring systems and vehicles.

[0003] In addition, in traffic monitoring systems, 3D-LiDAR, a sensor that uses laser light, is used to detect the presence or absence of objects (e.g., pedestrians, obstacles, and hazards) on roads that are vehicle routes. 3D-LiDAR generates 3D point cloud data that indicates the surrounding situation by detecting the distance and direction (azimuth) to a reflective object. Note that methods for identifying objects from this 3D point cloud data and detecting the presence or absence and movement of objects are described in, for example, Patent Document 1 and Patent Document 2.

[0004] The presence or absence of an object is detected by comparing two image data and analyzing the difference (difference), but image data in a state where there are no obstacles in the detection area (hereinafter referred to as the "initial state") is required as the comparison subject. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2019-536035 [Patent Document 2] JP 2022-104523 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, when the detection area is a parking lot as described in Patent Document 2, it is easy to obtain initial image data when the area is newly constructed or when usage is restricted; however, when the detection area is an existing road or intersection or other location with heavy traffic (vehicles and pedestrians), it can be difficult to obtain initial image data not only during the day but also at night.

[0007] Therefore, the present invention has been made in consideration of the above problems, and has an object to provide a data processing device, a data processing method, and a data processing program capable of generating image data of an initial state without obstacles from multiple image data in which obstacles are present. [Means for solving the problem]

[0008] (1) One aspect of the present invention is a data processing device that processes image data, comprising: a memory unit that stores first image data obtained by capturing an image of an object and second image data obtained by capturing an image of the object of the first image data at a different time; a data extraction unit that, when a data update area in the first image data for data update is specified, extracts partial data of an area in the second image data that corresponds to the data update area as update data; and a data replacement unit that replaces the partial data of the data update area in the first image data with the update data. (2) In the aspect (1) above, a superimposition display unit may be provided that, when the second image data is displayed on a display unit as a display image, superimposes a frame image indicating the outer edge of the data update area on the display image. (3) In the above aspect (1) or (2), the imaging target may be an area around an intersection of a road or passage, and the data update area may be an area of ​​the first image data that includes an obstacle located on the road or passage. (4) In any one of the above aspects (1) to (3), the image data may be in a 3D point cloud data format. (5) Another aspect of the present invention is a data processing method for processing image data in a computer, the method including: a data storage step of storing in a storage unit first image data obtained by capturing an image of an object and second image data obtained by capturing an image of the object of the first image data at a different time; a data extraction step of, when a data update area in the first image data where data updating is to be performed is specified, extracting, as update data, partial data of an area in the second image data corresponding to the data update area; and a data replacement step of replacing, in the first image data, the partial data of the data update area with the update data in a data replacement unit. (6) Another aspect of the present invention is a data processing program for processing image data, which causes a computer to execute the following steps: a data storage step of storing first image data obtained by capturing an image of an object and second image data obtained by capturing an image of the object of the first image data at a different time; a data extraction step of, when a data update area in the first image data for data update is specified, extracting, as update data, partial data of an area in the second image data corresponding to the data update area; and a data replacement step of replacing the partial data of the data update area in the first image data with the update data. Effect of the Invention

[0009] According to the present invention, it is possible to provide a data processing device, a data processing method, and a data processing program capable of generating image data of an initial state where no obstacles are present from a plurality of image data in which obstacles are present. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing a traffic monitoring system. [Diagram 2] FIG. 2 is a flow diagram illustrating a data processing method according to an embodiment of the present invention. [Figure 3A] FIG. 4 is a schematic diagram showing first image data. [Figure 3B]FIG. 4 is a schematic diagram showing second image data. [Figure 3C] FIG. 13 is a schematic diagram showing first image data in which a data update area is specified. [Figure 3D] FIG. 13 is a schematic diagram showing second image data on which a frame image is superimposed. [Figure 3E] FIG. 2 is a schematic diagram showing first image data on which update data is superimposed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the same elements will be denoted by the same reference numerals throughout the description of the embodiment.

[0012] FIG. 1 is a schematic diagram showing a traffic monitoring system 10 . The traffic monitoring system 10 includes a cooperative roadside unit 2 and a road-to-vehicle cooperative base station 3. Note that objects (obstacles) 6 on a road 4 include, for example, parked vehicles 1, pedestrians, structures, and intrusion control devices (road cones, guide posts, barricades), etc.

[0013] The vehicle 1 is equipped with a vehicle position detection device that detects the position of the vehicle itself, a wireless communication device capable of communicating with the cooperative roadside device 2, and the like. The vehicle position detection device includes an antenna that receives radio waves from the Global Navigation Satellite System (GNSS), and a processing unit (CPU) that detects the vehicle position from the received radio waves, and is connected to or incorporated in a drive recorder that records the driving route, a navigation system that provides guidance on the driving route to the destination, or the like, and the vehicle position is utilized.

[0014] The vehicle 1 is also equipped with a driving assistance device that assists in the automatic or semi-automatic driving of the vehicle 1. The driving assistance device performs some or all of the following: drive and braking control to control the starting and stopping of the vehicle 1 and the moving speed, and steering control to control the steering angle for right and left turns, enabling the vehicle 1 to travel automatically (autonomously) or semi-automatically (including manual driving by the passenger himself / herself) along a travel route from a departure point to a destination. At this time, the vehicle 1 may receive data (e.g., traffic information) from the cooperative roadside unit 2.

[0015] The cooperative roadside unit 2 includes a communication device and an imaging device 20. This cooperative roadside unit 2 may be installed independently near a road 4 or an intersection 5, or may be installed on an existing structure such as a utility pole or a building.

[0016] The imaging device 20 is installed at a predetermined height of the cooperative roadside unit 2 so as to overlook the road 4 and intersection 5 to be imaged. In this embodiment, the imaging device 20 is a measurement device such as a mechanical scan (mechanical rotation) type or solid-state type 3D-LiDAR, but may be another three-dimensional measurement device.

[0017] 3D-LiDAR can measure distance (position) based on the time it takes for the reflected light of the transmitted (irradiated) laser light to be received. For example, it has a measurement capability of a maximum distance of approximately 50m to 250m, a horizontal field of view (angle) of 90 degrees to 360 degrees, and a vertical field of view (angle) of 30 degrees to 100 degrees.

[0018] Then, image data LAS in a 3D point cloud data format (extensions LAS, PCD, PLY, PCAP, LAZ, etc.) is generated by scanning the imaging target (surroundings) using the imaging device 20 (3D-LiDAR). The image data LAS is converted into a 2D image format (for example, color separation according to distance) by a drawing application so that it can be displayed (visualized) on a monitor (including the display unit 150 described later), and is output, saved, and displayed as display image data in a point cloud data format with an extension such as BMP or JPG.

[0019] The road-to-vehicle coordination base station 3 is composed of a monitoring server, and includes a processing unit, a storage unit, a communication unit, an input unit, a display unit, an output unit, etc. In addition, the road-to-vehicle coordination base station 3 transmits and receives various data to and from the cooperative roadside unit 2.

[0020] The road-to-vehicle coordination base station 3 is further connected to the data processing device 100 by wire or wirelessly, and is capable of transmitting and receiving data. The data processing device 100 may also function as the road-to-vehicle coordination base station 3.

[0021] The data processing device 100 is configured from a personal computer or the like, and includes a processing unit 110, a storage unit 120, a communication unit 130, an input unit 140, a display unit 150, and the like.

[0022] The processing unit 110 is composed of a CPU, and also functions as a data extraction unit, a data replacement unit, a superimposition display unit, a data analysis unit, and the like, which will be described later.

[0023] For example, programs (applications) executed by the processing unit 110 and data (files) referenced by the processing unit 110 are stored in the storage unit 120. Examples of the programs include a file operation program and a data processing program described later.

[0024] Examples of the data include map data including road information, image data LAS described later, etc. The map data may be stored in a storage unit of a server (for example, a road-vehicle cooperation base station 3) other than the data processing device 100. In this case, various data may be received from the other server and temporarily stored in the storage unit 120 (as a cache).

[0025] The communication unit 130 transmits and receives data to and from an external device such as the road-to-vehicle coordination base station 3 in a wired or wireless manner.

[0026] The input unit 140 is configured with, for example, a mouse and a keyboard, and is used by an operator to give instructions to the data processing device 100 (processing unit 110).

[0027] The display unit 150 is configured with a monitor (display) and displays various data, operation screens, and display images as necessary. The data processing device 100 may also include a printing unit configured with a printer that prints various data, display images, and the like as necessary.

[0028] The data processing device 100 stores each of the following steps as a data processing program in the storage unit 120, and executes the data processing program to realize the function of each step in the processing unit 110, etc. A data processing method of the data processing device 100 will be described below. Fig. 2 is a flow diagram showing a data processing method according to an embodiment of the present invention. Fig. 3A is a schematic diagram showing the first image data LAS1, Fig. 3B is a schematic diagram showing the second image data LAS2, Fig. 3C is a schematic diagram showing the first image data LAS1 with a data update region RN specified, Fig. 3D is a schematic diagram showing the second image data LAS2 with a frame image FR superimposed, and Fig. 3E is a schematic diagram showing the first image data LAS1 with update data UD superimposed. It can also be said that Fig. 3E shows the first image data LAS1 in an initial state after data update.

[0029] First, in an imaging step (not shown) in the external processing of the data processing device 100, the imaging device 20 scans and images the road 4 and the periphery of the intersection 5 to be imaged at a plurality of times, and acquires and stores image data LAS, which is 3D point cloud data. At this time, an object 6 (obstacle) that becomes an obstacle may be present on the road 4.

[0030] Then, one of these image data LAS is stored as provisional initial image data (first image data) LAS1.

[0031] Next, the process moves to internal processing in the data processing device 100. In the data storage step S10, first image data LAS1 obtained by capturing an image of an object and second image data LAS2 obtained by capturing the same object as the first image data LAS1 from the same direction at a different time are stored (see FIGS. 3A and 3B). However, the number of second image data LAS2 is not limited to one, and may be multiple as long as they are captured at different times.

[0032] In the image display step S20, the first image data LAS1 used as an image in an initial state in the road-to-vehicle cooperative base station 3 (traffic monitoring system 10) is displayed on the display unit 150. The display image displayed on the display unit 150 at this time is in a point cloud data format obtained by converting the three-dimensional image data LAS into a two-dimensional image format.

[0033] In the area designation step S30, the operator uses the input unit 140 to designate a data update area RN in the first image data LAS1 where data is to be updated (see FIG. 3C). This data update area RN should be designated so as to include an obstacle 6 on the road 4. At this time, the designation may be performed while referring to a photograph (data) taken by a camera at the same time as the first image data LAS1 is acquired.

[0034] In the second image display step S40, the second image data LAS2 selected from one or more second image data LAS2 is displayed on the display unit 150. The selection of the second image data LAS2 may be performed by an operator or automatically by the processing unit 110.

[0035] Simultaneously with or around the time of the second image display step S40, a superimposed display step S50 is executed manually or automatically. In the superimposed display step S50, a frame image FR corresponding to the outer edge of the specified data update region RN is superimposed on the second image data LAS2 displayed on the display unit 150 (see FIG. 3D).

[0036] After the superimposed display step S50, an extraction confirmation step S55 may be added in which the operator performs a confirmation operation as necessary, and if there is no problem with the position of the data update area RN ("YES"), the next data extraction step S60 is executed. At this time, if there is a problem with the position of the data update area RN ("NO"), the process may be returned to the area designation step S30 and looped to be executed again. Alternatively, the process may be returned to the second image display step S40 and second image data LAS different from the previous one may be selected.

[0037] In the data extraction step S60, partial data of the area corresponding to the data update area RN designated in the area designation step S30 is extracted as update data UD from the second image data LAS2.

[0038] In a second superimposed display step S65, the update data UD is superimposed and displayed on the data update region RN of the first image data LAS1 displayed on the display unit 150 (see FIG. 3E).

[0039] In the replacement confirmation step S70, similarly to after the superimposed display step S50, the operator performs a confirmation operation as necessary, and if there is no problem with the update data UD of the data update area RN ("YES"), the next data replacement step S80 may be executed. At this time, if there is a problem with the update data UD ("NO"), the process may return to the area designation step S30 or the second display step S40 and loop to execute again. Note that the process may be terminated once without looping.

[0040] In the data replacement step S80, partial data in the data update region RN in the first image data LAS1 is replaced with the update data UD.

[0041] Through these steps, the obstacles 6 on the road 4 that existed in the first image data LAS1 are removed, the update of the first image data LAS1 is completed, and the initial state image data LAS can be generated and obtained.

[0042] Thereafter, the road-to-vehicle coordination base station 3 acquires the image data LAS of the initial state, and also acquires the image data LAS of the current location from the imaging device 20, thereby analyzing the difference (difference) between the two. Using this result, the road-to-vehicle coordination base station 3 can easily determine that a vehicle 1, a pedestrian 6, or an obstacle 6 has entered or appeared on the road 4 and intersection 5 that are the subject of monitoring. Then, the road-to-vehicle coordination base station 3 can notify the terminal devices of the vehicles 1 and pedestrians 6 of the traffic conditions via the cooperative roadside unit 2.

[0043] Incidentally, when obstacles 6 exist in a plurality of places on the road 4 in the first image data LAS1, the above-mentioned area designation step S30 to data replacement step S80 may be repeated until all obstacles 6 are removed. However, in the area designation step S30, a plurality of data update areas RN may be designated, and the program may be modified so that the data of the plurality of data update areas RN is updated collectively.

[0044] As described above, the data processing device 100 according to the embodiment of the present invention is a data processing device 100 that processes image data LAS, and includes a memory unit 120 that stores first image data LAS1 obtained by imaging an imaging target and second image data LAS2 obtained by imaging the imaging target of the first image data LAS1 at a different time, a data extraction unit (processing unit) 110 that, when a data update area RN in the first image data LAS1 where data updating is to be performed is specified, extracts partial data of an area corresponding to the data update area RN in the second image data LAS2 as update data UD, and a data replacement unit (processing unit) 110 that replaces the partial data of the data update area RN in the first image data LAS1 with the update data UD.

[0045] This makes it possible to generate image data LAS of an initial state without an obstacle 6 from a plurality of image data LAS1, LAS2 in which an obstacle 6 exists on the road 4. Therefore, even if the image data LAS1, LAS2 are captured during the day when traffic is heavy, the obstacle 6 on the road 4 can be removed, and image data LAS of the initial state can be generated and acquired. In addition, when a new cooperative roadside unit 2 is installed near an intersection 5, on-site adjustment can be easily performed in a short time, which leads to a reduction in the burden on workers. Furthermore, even if the environment near the intersection 5 changes due to a change in the shape of the road 4 or a new construction, or the construction or demolition of a building, it is not necessary to specify a data update area RN for the entire imaging range of the imaging device 20 (i.e., it is not necessary to generate image data LAS of the initial state from scratch), and it is possible to specify a data update area RN only for a specific range that has been changed.

[0046] The data processing device 100 of the embodiment includes a superimposition display unit (processing unit) 110 that, when displaying the second image data LAS2 as a display image on the display unit 150, superimposes a frame image FR indicating an area corresponding to the data update area RN on the display image. This makes it easy to confirm which part of the second image data LAS2 has been designated as the data update area RN.

[0047] The imaging target in the embodiment is the periphery of the road 4 or the intersection 5, and the data update region RN is the region of the first image data LAS1 that includes the obstacle 6 located on the road 4. This makes it possible to generate and acquire image data LAS of the initial state of the periphery of the road 4 or the intersection 5.

[0048] The image data LAS of the embodiment is in the form of 3D point cloud data, which can be easily generated by a three-dimensional measuring device such as a 3D-LiDAR.

[0049] The data processing method according to the embodiment of the present invention is a data processing method for processing image data LAS in a computer, and includes a data storage step S10 for storing in a storage unit 120 first image data LAS1 obtained by imaging an imaging target and second image data LAS2 obtained by imaging the imaging target of the first image data LAS1 at a different time, a data extraction step S60 for extracting, when a data update region RN for performing data update in the first image data LAS1 is specified, partial data of a region corresponding to the data update region RN in the second image data LAS2 as update data UD in a data extraction unit (processing unit) 110, and a data replacement step S80 for replacing the partial data of the data update region RN in the first image data LAS1 with the update data UD in a data replacement unit (processing unit) 110. This provides the same effects as the data processing device 100.

[0050] The data processing program according to the embodiment of the present invention is a data processing program for processing image data LAS, and causes a computer to execute a storage step S10 for storing first image data LAS1 obtained by capturing an image of an object and second image data LAS2 obtained by capturing the image of the object of the first image data LAS1 at a different time, a data extraction step S60 for extracting partial data of an area corresponding to the data update area RN of the second image data LAS2 as update data UD when a data update area RN for data update is specified in the first image data LAS1, and a data replacement step S80 for replacing the partial data of the data update area RN in the first image data LAS1 with the update data UD. This provides the same effects as the data processing device 100.

[0051] Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0052] (Modification) In the above embodiment, the operator sets the data update area RN and the second image data LAS2 via the input unit 140, but the data analysis unit of the processing unit 110 or another system may learn the shapes and dimensions of various objects in the image data LAS (or the displayed image) in advance so that the data update area RN and the second image data LAS2 can be set automatically, or these may be used in combination.

[0053] In the above embodiment, the image data LAS in the 3D point cloud data format is updated with the update data UD in the 3D point cloud data format, but the 2D image data converted for a 2D image may be updated.

[0054] In the above embodiment, the road 4 and intersection 5 along which the vehicle 1 travels are shown, but the intersections may also be passageways and intersections in a parking lot or a warehouse along which manned and unmanned guided vehicles travel. [Explanation of symbols]

[0055] 1 Vehicle, 2 Cooperative roadside unit, 3 Road-to-vehicle cooperative base station, 4 Road, 5 Intersection, 6 Obstacle (object) 10 Traffic Monitoring System 20 Imaging device 100 Data processing device 110 Processing unit, 120 Memory unit, 130 Communication unit, 140 Input unit, 150 Display unit LAS1, LAS2 image data, RN data update area, UD update data, FR frame image

Claims

1. A data processing device for processing image data, comprising: a storage unit that stores first image data obtained by capturing an image of an imaging target and second image data obtained by capturing an image of the imaging target of the first image data at a different time; a data extraction unit that extracts, when a data update area in which data is to be updated is designated in the first image data, partial data of an area corresponding to the data update area in the second image data as update data; a data replacement unit that replaces partial data of the data update area in the first image data with the update data.

23. A data processing device comprising:

2. a superimposition display unit that, when displaying the second image data as a display image on a display unit, displays a frame image indicating an outer edge of the data update area superimposed on the display image, 2. The data processing device according to claim 1.

3. The imaging target is a vicinity of an intersection of a road or a passage, the data update area is an area of ​​the first image data that includes an obstacle located on the road or the passage; 3. The data processing device according to claim 1, wherein the first and second inputs are connected to the first and second inputs.

4. The image data is in a 3D point cloud data format.

3. The data processing device according to claim 1, wherein the first and second inputs are connected to the first and second inputs.

5. A data processing method for processing image data in a computer, comprising the steps of: a data storage step of storing in a storage unit first image data obtained by capturing an image of an imaging target and second image data obtained by capturing an image of the imaging target of the first image data at a different time; a data extraction step of extracting, when a data update area in which data is to be updated is designated in the first image data, partial data of an area corresponding to the data update area in the second image data as update data by a data extraction unit; a data replacement step of replacing partial data of the data update area in the first image data with the update data by a data replacement unit, 23. A data processing method comprising:

6. A data processing program for processing image data, a data storage step of storing first image data obtained by capturing an image of an imaging target and second image data obtained by capturing an image of the imaging target of the first image data at a different time; a data extraction step of extracting, when a data update area in which data is to be updated is specified in the first image data, partial data of an area corresponding to the data update area in the second image data as update data; a data replacement step of replacing partial data in the data update area in the first image data with the update data. A data processing program comprising:

Images