Image processing device, image processing method, and computer program

Abstract

To upgrade freshness of map data and an information quantity.SOLUTION: An image processing device includes a receiving unit that receives plural shot images taken along with the passage of time by an imaging device mounted in a vehicle, a position and posture estimation unit that estimates, from the plural shot images, plural three-dimensional positions and postures of the imaging device which are observed along with the passage of time, a bird's-eye view image creation unit that creates plural bird's-eye view images, which are observed along with the passage of time, from the plural shot images, and an image road map creation unit that creates an image road map by joining the plural bird's-eye view images along with the passage of time on the basis of the plural three-dimensional positions and postures.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a computer program. [Background technology]

[0002] Non-Patent Document 2 discloses map data created from images captured from the sky. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "Google Maps", Internet<URL:https: / / www.google.co.jp / intl / ja / maps / about / #! / > Summary of the Invention [Problem to be solved by the invention]

[0004] However, the map data described in Non-Patent Document 2 uses images captured from the sky, making it difficult to update frequently. Also, there is no map data for places that cannot be seen from the sky, such as inside tunnels.

[0005] The present invention has been made in consideration of the above circumstances, and has an object to improve the freshness and amount of information of map data. [Means for solving the problem]

[0006] One aspect of the present invention is an image processing device that includes a receiving unit that receives a plurality of captured images taken over time by an imaging device mounted on a vehicle, a position and attitude estimation unit that estimates a plurality of three-dimensional positions and attitudes of the imaging device over time from the plurality of captured images, a bird's-eye view image creation unit that creates a plurality of bird's-eye view images over time from the plurality of captured images, and an image road map creation unit that creates an image road map by connecting the plurality of bird's-eye view images over time based on the plurality of three-dimensional positions and attitudes.

[0007] In one aspect of the present invention, in the image processing device described above, the bird's-eye view image creation section creates the plurality of bird's-eye view images from which areas in which a vehicle is captured are deleted.

[0008] One aspect of the present invention is an image processing device as described above, wherein the receiving unit further receives vehicle speed information indicating the speed of the vehicle corresponding to the plurality of captured images, and the bird's-eye view image creation unit changes the size of the bird's-eye view image in the direction of travel of the vehicle in accordance with the speed of the vehicle.

[0009] One aspect of the present invention is an image processing device as described above, in which the image road map creation unit overlays a temporally earlier bird's-eye view image on a subsequent bird's-eye view image at a portion where the subsequent bird's-eye view image is joined.

[0010] One aspect of the present invention is the image processing device described above, further comprising a drawing unit that draws a trajectory of the vehicle on the image road map.

[0011] One aspect of the present invention is an image processing device as described above, further comprising an object detection distance estimation unit that detects objects within the captured image and estimates the distance of the detected object from the vehicle, and the drawing unit further draws a trajectory of the object on the image road map based on the distance of the object from the vehicle.

[0012] One aspect of the present invention is an image processing method executed by an image processing device, the image processing method including: a receiving step of receiving a plurality of captured images taken over time by an imaging device mounted on a vehicle; a position and attitude estimation step of estimating a plurality of three-dimensional positions and attitudes of the imaging device over time from the plurality of captured images; a bird's-eye view image creation step of creating a plurality of bird's-eye view images over time from the plurality of captured images; and an image road map creation step of creating an image road map by connecting the plurality of bird's-eye view images over time based on the plurality of three-dimensional positions and attitudes.

[0013] One aspect of the present invention is a computer program for causing a computer to execute the following steps: a receiving step of receiving a plurality of captured images taken over time by an imaging device mounted on a vehicle; a position and attitude estimation step of estimating a plurality of three-dimensional positions and attitudes of the imaging device over time from the plurality of captured images; a bird's-eye view image creation step of creating a plurality of bird's-eye view images over time from the plurality of captured images; and an image road map creation step of creating an image road map by connecting the plurality of bird's-eye view images over time based on the plurality of three-dimensional positions and attitudes. Effect of the Invention

[0014] According to the present invention, it is possible to obtain an effect of improving the freshness and amount of information of map data. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram showing an example of the configuration of an image processing device according to an embodiment; [Diagram 2] 1 is a block diagram showing an example of the overall configuration of an image processing system according to an embodiment; [Diagram 3] FIG. 1 is a diagram showing an example of a procedure of an image processing method according to an embodiment. [Figure 4] FIG. 1 is a diagram for explaining an image processing method according to an embodiment. [Diagram 5]FIG. 1 is a diagram for explaining an image processing method according to an embodiment. [Figure 6] FIG. 1 is a diagram for explaining an image processing method according to an embodiment. [Figure 7] FIG. 1 is a diagram for explaining an image processing method according to an embodiment. [Figure 8] FIG. 1 is a diagram for explaining an image processing method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment described below does not unduly limit the contents of the present invention described in the claims, and all of the configurations described in the embodiment are not necessarily essential as a means for solving the problem of the present invention.

[0017] Fig. 1 is a block diagram showing an example of the configuration of an image processing apparatus according to an embodiment of the present invention, Fig. 2 is a block diagram showing an example of the overall configuration of an image processing system according to an embodiment of the present invention.

[0018] In FIG. 1, the image processing device 1 includes a control unit 10, a communication unit 12, and a storage unit .

[0019] The control unit 10 is a CPU (Central Processing Unit), and executes various functions by calling up and executing programs stored in the storage unit 14 .

[0020] The communication unit 12 communicates with an external device via a communication network NW such as the Internet, etc. For example, the communication unit 12 communicates with a captured image database 30 via the communication network NW.

[0021] The storage unit 14 stores various data such as programs executed by the control unit 10 and data used by the control unit 10. The storage unit 14 is configured from a hard disk device, a magneto-optical disk device, a non-volatile memory such as a flash memory, a read-only recording medium such as a CD-ROM, a volatile memory such as a dynamic random access memory (DRAM), or a combination of these.

[0022] The storage unit 14 stores a pictorial road map creation program 141 .

[0023] 2, a vehicle 20 such as an automobile includes an imaging device 201. The imaging device 201 is provided in the vehicle 20 so as to capture an image of the outside of the vehicle in the traveling direction of the vehicle 20. The imaging device 201 may be, for example, a drive recorder. The imaging device 201 transmits an image captured by itself together with its own identification information (referred to as an imaging device ID) via a communication network NW to a captured image database 30. The captured image database 30 stores the captured images received from the imaging device 201 in association with the imaging device ID. The captured image database 30 stores a plurality of captured images (hereinafter referred to as time-series captured images) in association with one imaging device ID along the flow of time.

[0024] The time-series captured images may be a plurality of captured images captured continuously by the imaging device 201, or a plurality of captured images obtained by thinning out the captured images at regular intervals from a plurality of captured images captured continuously by the imaging device 201.

[0025] The imaging device 201 (e.g., a drive recorder) may store the latest time-series captured images for a certain period of time captured by itself in its own storage device. In this case, the imaging device 201 may or may not transmit the captured images captured by itself to the captured image database 30.

[0026] 1, the control unit 10 includes, as its functions, a receiving unit 101, a position and orientation estimation unit 102, a bird's-eye view image creation unit 103, an image road map creation unit 104, a drawing unit 105, and an object detection distance estimation unit 106. These functions of the control unit 10 are realized by the CPU executing an image road map creation program 141 stored in the storage unit 14.

[0027] The receiving unit 101 receives time-series captured images captured by an imaging device 201 mounted on one vehicle 20 (hereinafter referred to as an imaging vehicle 20). For example, the receiving unit 101 receives time-series captured images associated with one imaging device ID from the captured image database 30 via the communication network NW by the communication unit 12. For example, the receiving unit 101 receives time-series captured images directly from the imaging device 201 (e.g., a drive recorder) via the communication network NW by the communication unit 12. Note that time-series captured images copied from the imaging device 201 (e.g., a drive recorder) to a recording medium such as a USB memory may be transmitted to the image processing device 1, and the receiving unit 101 may receive the time-series captured images.

[0028] The position and orientation estimation unit 102 estimates, from the time-series captured images received by the receiving unit 101, a plurality of three-dimensional positions and orientations over time of the imaging device 201 that captured the time-series captured images.

[0029] The bird's-eye view image generating unit 103 generates a plurality of bird's-eye view images in accordance with the flow of time from the time-series captured images received by the receiving unit 101 .

[0030] The image road map creation unit 104 creates an image road map by connecting together multiple bird's-eye view images created by the bird's-eye view image creation unit 103 in time sequence based on the multiple three-dimensional positions and orientations estimated by the position and orientation estimation unit 102. The image road map is an image map showing the roads on which the imaging vehicle 20 has traveled in images, and is an image of the roads viewed from above like an aerial photograph.

[0031] The drawing unit 105 draws the trajectory of the imaging vehicle 20 on the image road map created by the image road map creation unit 104 .

[0032] The object detection distance estimation unit 106 detects an object in the time-series captured images received by the receiving unit 101, and estimates the distance of the detected object from the capturing vehicle 20. The drawing unit 105 further draws the trajectory of the object on the image road map created by the image road map creation unit 104, based on the distance of the object from the capturing vehicle 20 estimated by the object detection distance estimation unit 106.

[0033] Next, image processing executed by the image processing device 1 according to this embodiment will be described. Fig. 3 is a diagram showing an example of the procedure of the image processing method according to this embodiment. Note that subscripts and superscripts, for example, "a" may be written as "_a".

[0034] (Step S1) The receiving unit 101 receives time-series captured images captured by the imaging device 201 (eg, a drive recorder) mounted on the imaging vehicle 20.

[0035] (Step S2) The position and orientation estimation unit 102 estimates a plurality of three-dimensional positions and orientations along the flow of time for the imaging device 201 that captured the time-series captured images from the time-series captured images received by the receiving unit 101. In estimating the position and orientation of the imaging device 201, the position and orientation estimation unit 102 performs self-position and orientation estimation for the time-series captured images to estimate the three-dimensional position and orientation of the imaging device 201. As self-position and orientation estimation techniques, SfM (Structure from Motion) and Visual-SLAM (Visual Simultaneous Localization And Mapping) are known. In this embodiment, either SfM or Visual-SLAM may be used. In general, it is known that SfM prioritizes accuracy and is non-real-time, while Visual-SLAM is real-time. Therefore, it is only necessary to determine whether to use SfM or Visual-SLAM depending on the application of this embodiment. The scale of the position of the estimation result may be determined based on the vehicle speed of the image capturing vehicle 20. The pitch angle may be corrected by estimating the vanishing point. Regarding the vehicle speed of the image capturing vehicle 20, vehicle speed information indicating the vehicle speed detected by a vehicle speed sensor (e.g., a vehicle speed sensor of a drive recorder) provided in the image capturing vehicle 20 may be associated with the time-series captured images. In this case, the receiving unit 101 receives the vehicle speed information together with the time-series captured images.

[0036] (Step S3) The bird's-eye view image creating unit 103 creates a plurality of bird's-eye view images along the flow of time from the time-series captured images received by the receiving unit 101. A bird's-eye view image creating method will be described below.

[0037] The relationship between a point (p_x, p_y, p_z) in the camera coordinate system of the imaging device 201 and the coordinates (i, j) of one captured image I of the time-series captured images is expressed by the following equations (1) and (2) using a camera internal matrix K and a pitch angle θ. The camera internal matrix K is set in advance in the image processing device 1. The pitch angle θ corresponds to the orientation of the imaging device 201 estimated by the position and orientation estimation unit 102.

[0038]

number

[0039]

number

[0040] In addition, if the bird's-eye view conversion area is set to p_xmax meters to the left and right and p_zmax meters forward, and the size of the bird's-eye view image is set to W x H pixels, the coordinates (u, v) of bird's-eye view image B corresponding to point (p_x, p_y, p_z) are expressed by the following equation (3).

[0041]

number

[0042] The bird's-eye view image creation unit 103 creates a bird's-eye view image B from the above formulas (1), (2), and (1) so that "the coordinates (i, j) of the captured image I=the coordinates (u, v) of the bird's-eye view image B." As a result, as exemplified in Fig. 4, one captured image I of the time-series captured images in Fig. 4(a) is converted into the bird's-eye view image B in Fig. 4(b).

[0043] In addition, in the bird's-eye view conversion region, the size in the traveling direction of the imaging vehicle 20 may be changed according to the vehicle speed of the imaging vehicle 20. The larger the vehicle speed of the imaging vehicle 20, the larger the size is. Regarding the vehicle speed of the imaging vehicle 20, vehicle speed information indicating the vehicle speed detected by a vehicle speed sensor (e.g., a vehicle speed sensor of a drive recorder) provided in the imaging vehicle 20 may be associated with the time-series captured images. In this case, the receiving unit 101 receives the vehicle speed information together with the time-series captured images.

[0044] Depending on how the bird's-eye view conversion area is specified, the hood 210 of the vehicle 20 or a vehicle ahead may be reflected in the bird's-eye view image B as exemplified in FIG. 5(a). Reflection of vehicles and other objects other than the road can cause a decrease in accuracy of an image road map created by stitching together a plurality of bird's-eye view images. For this reason, as exemplified in FIG. 5, an image is cropped so as to delete the area in which the vehicle is reflected from the created bird's-eye view image B. This creates a bird's-eye view image B1 in which the area in which the vehicle is reflected is deleted from the bird's-eye view image B as exemplified in FIG. 5(b). The vehicle 20 is not reflected in the bird's-eye view image B1, and only the road is reflected.

[0045] The area in which the vehicle is captured may be determined by performing image recognition on the bird's-eye view image B. Alternatively, the area in which the vehicle is captured may be determined to be a certain size in the traveling direction of the capturing vehicle 20 from the end of the bird's-eye view image B on the capturing vehicle side.

[0046] (Step S4) The image road map creation unit 104 creates an image road map by connecting together in time the multiple bird's-eye view images created by the bird's-eye view image creation unit 103 based on the multiple three-dimensional positions and orientations estimated by the position and orientation estimation unit 102. The image road map creation method will be described below.

[0047] The image road map creation unit 104 stitches together multiple (T) bird's-eye view images B_(1), B_(2), ..., B_(T) in order from "t=T" to "t=1" based on the multiple three-dimensional positions and orientations estimated by the position and orientation estimation unit 102. B_(T) is a later image in time than B_(1). The image road map creation unit 104 overlays the earlier bird's-eye view image B_(t) on the later bird's-eye view image B_(t+1) at the stitching portion with the later bird's-eye view image B_(t+1).

[0048] Here, consider superimposing the t-th frame bird's-eye view image B(t) on the image road map R. If the coordinates on the image road map corresponding to the coordinates (0,0) of the image capturing vehicle 20 are (c_x, c_y) and the position of the image capturing device 201 (image capturing vehicle 20) estimated by the position and orientation estimation unit 102 is (x(t), y(t)), the coordinates (X(t), Y(t)) on the image road map corresponding to the position (x(t), y(t)) of the image capturing vehicle 20 are expressed by the following equation (4).

[0049]

number

[0050] The image road map creation unit 104 rotates the bird's-eye view image B(t) around the coordinates (W / 2, H) of the position of the imaging device 201 estimated by the position and orientation estimation unit 102, and joins it to the bird's-eye view image B(t-1). Specifically, the bird's-eye view image B(t) and the image road map R satisfy the following formula (5). γ_(t) is the yaw angle of the imaging device 201, and corresponds to the orientation of the imaging device 201 estimated by the position and orientation estimation unit 102.

[0051]

number

[0052] The image road map creation unit 104 uses equation (5) to connect multiple (T) bird's-eye view images B_(1), B_(2), ..., B_(T) in order from "t=T" to "t=1" to create an image road map R as shown in Figure 6.

[0053] (Step S5) The object detection distance estimation unit 106 detects an object in the time-series captured images received by the receiving unit 101, and estimates the distance of the detected object from the image capturing vehicle 20. An example of a method for estimating the distance of the object will be described. First, the object detection distance estimation unit 106 obtains the coordinates of the bounding box of the detection target by object detection. Here, it is assumed that the above formulas (1) and (2) hold between a point (p_x, p_y, p_z) in the camera coordinate system of the image capturing device 201 and the coordinates (i, j) of one captured image I of the time-series captured images, and that the bottom end of the bounding box of the detection target is in contact with the road. As a result, the object detection distance estimation unit 106 obtains the front-rear distance d_z and the left-right distance d_x from the image capturing vehicle 20 to the object to be detected from the height of the image capturing device 201 and the coordinates of the bounding box of the detection target. As a distance estimation method other than the above, the distance to the object to be detected may be obtained using distance estimation by deep learning, 3D object detection, or disparity estimation by a stereo camera.

[0054] (Step S6) The drawing unit 105 draws the trajectory of the imaging vehicle 20 on the image road map R created by the image road map creation unit 104. The trajectory of the imaging vehicle 20 is expressed by the above formula (4). FIG. 7 shows an example of drawing the trajectory of the imaging vehicle 20 on the image road map R. In the drawing example of FIG. 7(a), the imaging vehicle 20 traveling along the lanes of the road is shown by drawing the trajectory Q of the imaging vehicle 20 on the image road map R. In the drawing example of FIG. 7(b), the imaging vehicle 20 traveling while changing lanes of the road is shown by drawing the trajectory Q1 of the imaging vehicle 20 on the image road map R1. As exemplified in FIGS. 7(a) and (b), the traveling situation such as the traveling position and lane changes of the imaging vehicle 20 can be clearly visualized by drawing the trajectory of the imaging vehicle 20 on the image road map.

[0055] The drawing unit 105 further draws the trajectory of the object on the image road map created by the image road map creation unit 104, based on the distance of the object from the image capturing vehicle 20 estimated by the object detection distance estimation unit 106. The trajectory (X_o, Y_o) of the object is calculated by the following formula (6) using the front-rear distance d_z and the left-right distance d_x from the image capturing vehicle 20 to the object to be detected calculated by the object detection distance estimation unit 106.

[0056]

number

[0057] The drawing unit 105 may draw a trajectory for each type of object by distinguishing the trajectory. For example, the drawing unit 105 may color-code the trajectory for each type of object. Examples of the object types include trucks, ordinary cars, pedestrians, bicycles, and motorcycles. The object detection distance estimation unit 106 determines the type of the object to be detected and notifies the drawing unit 105 of the determination result.

[0058] Fig. 8 shows an example of the detection result of an object outside the captured vehicle 20 and the trajectory of the detected object. Fig. 8(a) shows the result of object detection using bounding boxes by the object detection distance estimation unit 106. In the example of Fig. 8(a), a truck is detected by bounding box Box1, a bicycle is detected by bounding box Box2, and a pedestrian is detected by bounding box Box3 from a captured image I1.

[0059] FIG. 8(b) shows an example of drawing the trajectories of the image capturing vehicle 20 and the objects by the drawing unit 105. In the example of FIG. 8(b), the trajectory Q2 of the image capturing vehicle 20, the trajectory Q3 of the truck based on the detection result of FIG. 8(a), the trajectory Q4 of the bicycle, and the trajectory Q5 of the pedestrian are drawn on the image road map R, distinguished by color or the like. As exemplified in FIG. 8(b), the trajectories of the objects are color-coded for each type of object, so that the traffic situation when the image capturing vehicle 20 is traveling can be clearly shown. This can contribute to detailed analysis of the traffic situation, for example, when a near miss occurs or a traffic accident occurs.

[0060] As described above, according to this embodiment, a pictorial road map can be created from images captured by the imaging device 201 mounted on the imaging vehicle 20 that actually travels on the road. This makes it easy to update the pictorial road map more frequently, so that the freshness of the pictorial road map can be improved. In addition, a pictorial road map can be created even in places that cannot be seen from the sky, such as inside a tunnel, so that the amount of information in the pictorial road map can be improved.

[0061] In addition, the image processing device 1 may receive time-series captured images for a certain period of time captured by the imaging device 201 of the imaging vehicle 20 via the receiving unit 101, for example, via the captured image database 30, and create an image road map from the time-series captured images by offline processing.

[0062] Alternatively, the image processing device 1 may directly receive the time-series captured images captured by the imaging device 201 of the imaging vehicle 20 from the imaging device 201 via the receiving unit 101, and create a pictorial road map from the time-series captured images by online processing. In this case, the pictorial road map can be updated in real time.

[0063] This embodiment can be applied to various fields.

[0064] For example, this embodiment may be applied to the creation of a dynamic map used in an autonomous vehicle. In this case, a pictorial road map with more information may be created by creating a pictorial road map using a large number of time-series captured images taken by the imaging devices 201 mounted on a large number of imaging vehicles 20, and the created pictorial road map may be applied to the dynamic map.

[0065] For example, the image road map and trajectory information created in this embodiment may be applied to a system that distributes traffic environment information to vehicles that will travel in the future on the same road as the image capture vehicle 20. Examples of traffic environment information include traffic congestion information (traffic congestion conditions, reasons for traffic congestion, etc.), estimated cruising distance information for EVs (electric vehicles), road maintenance (construction) information, and evacuation guidance information in the event of a disaster.

[0066] For example, the image road map and trajectory information created in this embodiment may be applied to a system that automatically generates a traffic accident scene inspection report. This contributes to reducing the labor and time required for on-site investigation of a traffic accident, and also makes it possible to reproduce the situation of the traffic accident based on the actual captured images, thereby improving the accuracy of the on-site inspection report.

[0067] For example, the pictorial road map and trajectory information created in this embodiment may be applied to a system for automatically calculating the degree of fault at an insurance company that handles traffic accidents.

[0068] For example, the pictorial road map and trajectory information created in this embodiment may be applied to a system for analyzing the causes of traffic accidents.

[0069] For example, the pictorial road map and trajectory information created in this embodiment may be applied to a system that provides information on EV charging stations and available parking spaces.

[0070] Each function of the image processing device 1 may be realized by the image processing device 1 including computer hardware such as a CPU and a memory, and the CPU executing a computer program stored in the memory. The image processing device 1 may be configured using a general-purpose computer device, or may be configured as a dedicated hardware device. For example, the image processing device 1 may be configured using a server computer connected to a communication network such as the Internet. Each function of the image processing device 1 may be realized by cloud computing. The image processing device 1 may be realized by a single computer, or may be realized by distributing the functions of the image processing device 1 among multiple computers. The image processing device 1 may be configured to open a website using, for example, a WWW system.

[0071] In addition, a computer program for implementing the functions of the image processing device 1 described above may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Note that the "computer system" referred to here may include hardware such as an OS and peripheral devices. In addition, if a WWW system is used, the "computer system" also includes the homepage providing environment (or display environment). In addition, the term "computer-readable recording medium" refers to writable non-volatile memory such as a flexible disk, an optical magnetic disk, a ROM, or a flash memory, a portable medium such as a DVD (Digital Versatile Disc), or a storage device such as a hard disk built into a computer system.

[0072] Furthermore, the term "computer-readable recording medium" includes devices that retain a program for a certain period of time, such as volatile memory (e.g., DRAM (Dynamic Random Access Memory)) within a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. The program may be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by a transmission wave in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for implementing some of the functions described above, or may be a so-called differential file (differential program) that can implement the functions described above in combination with a program already recorded in the computer system.

[0073] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0074] 1...image processing device, 10...control unit, 12...communication unit, 14...storage unit, 101...receiving unit, 102...position and orientation estimation unit, 103...bird's-eye view image creation unit, 104...image road map creation unit, 105...drawing unit, 106...object detection distance estimation unit, 141...image road map creation program, 20...vehicle, 201...imaging device, 30...imaging image database, NW...communication network

Claims

1. A receiving unit that receives a plurality of captured images taken over time by an imaging device mounted on one vehicle; a position and orientation estimation unit that estimates a plurality of three-dimensional positions and orientations of the imaging device over time from the plurality of captured images; a bird's-eye view image creation unit that creates a plurality of bird's-eye view images along a time flow from the plurality of captured images; a pictorial road map creation unit that creates a pictorial road map by joining together the plurality of bird's-eye view images in a time sequence based on the plurality of three-dimensional positions and orientations; An image processing device comprising:

2. the bird's-eye view image creation unit creates the plurality of bird's-eye view images from which areas in which vehicles are captured are deleted; The image processing device according to claim 1 .

3. The receiving unit further receives vehicle speed information indicating a speed of the vehicle corresponding to the plurality of captured images, the bird's-eye view image creation unit changes a size of the bird's-eye view image in a traveling direction of the vehicle in accordance with a speed of the vehicle; The image processing device according to claim 1 .

4. the image road map creation unit superimposes a temporally earlier bird's-eye view image on a subsequent bird's-eye view image at a joint between the bird's-eye view image and the subsequent bird's-eye view image; The image processing device according to claim 1 .

5. A drawing unit for drawing a trajectory of the vehicle on the image road map. The image processing device according to claim 1 .

6. An object detection distance estimation unit that detects an object in the captured image and estimates a distance of the detected object from the vehicle, The drawing unit further draws a trajectory of the object on the image road map based on a distance of the object from the vehicle. The image processing device according to claim 5 .

7. An image processing method executed by an image processing device, comprising: A receiving step of receiving a plurality of captured images taken over time by an imaging device mounted on one vehicle; a position and orientation estimation step of estimating a plurality of three-dimensional positions and orientations of the imaging device over time from the plurality of captured images; a bird's-eye view image creating step of creating a plurality of bird's-eye view images along a time flow from the plurality of captured images; a pictorial road map creation step of creating a pictorial road map by connecting the plurality of bird's-eye view images in a time sequence based on the plurality of three-dimensional positions and orientations; An image processing method comprising:

8. On the computer, A receiving step of receiving a plurality of captured images taken over time by an imaging device mounted on one vehicle; a position and orientation estimation step of estimating a plurality of three-dimensional positions and orientations of the imaging device over time from the plurality of captured images; a bird's-eye view image creating step of creating a plurality of bird's-eye view images along a time flow from the plurality of captured images; a pictorial road map creation step of creating a pictorial road map by connecting the plurality of bird's-eye view images in a time sequence based on the plurality of three-dimensional positions and orientations; A computer program for executing the above.

Images