Object recognition device and object recognition method

Abstract

To precisely estimate the shape of an object.SOLUTION: An object recognition device receives multiple images obtained by photographing an object from different viewpoints, selects a reference image from the multiple images, selects pair images corresponding to the reference image from the multiple images, disposes an element point on a linear section of the object in the reference image, calculates a first sight line corresponding to the disposed element point, calculates an epipolar line on the pair images corresponding to the calculated first sight line, calculates multiple candidate correspondence point candidates corresponding to one element point on one calculated epipolar line, calculates second multiple sight lines corresponding to each of the multiple, calculated correspondence point candidates, calculates the intersection between the first sight line, and each of the multiple second sight lines, estimates the shape of the object on the basis of a three-dimensional point group of the calculated intersection, and outputs the estimated shape of the object.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an object recognition device and an object recognition method. [Background technology]

[0002] Mobile Mapping Systems (MMS) are known, which are equipped with measuring devices such as 3D laser scanners and GPS on vehicles and collect 3D shapes of the terrain and structures around the road in the form of 3D point cloud data while driving. However, since MMS are expensive, there is a demand for devices that can recognize the shapes of objects using camera images that are less expensive than MMS.

[0003] Patent Document 1 discloses a measuring device that measures the shape of an object using a camera image. The measuring device described in Patent Document 1 includes an edge line extraction unit that sets an image processing area near a reference contour image obtained by projecting a reference contour of the object onto the imaging plane of a first camera and the imaging plane of a second camera, and extracts edge lines from the object image in the area, an edge point generation unit that generates edge points that are intersections of the edge lines and epipolar lines, a corresponding point selection unit that selects a pair of edge points consisting of a first edge point and a second edge point that are generated using the same epipolar plane and correspond to the same parts of the reference contour from among the multiple edge points, and a three-dimensional point calculation unit that calculates three-dimensional points on the contour of the object based on the camera line of sight that passes through the pair of edge points. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-061687 A Summary of the Invention [Problem to be solved by the invention]

[0005] Estimating the shape of linear parts of objects such as overhead wires, pedestrian bridge girders, overpass girders, the bottom ends of traffic lights, and signboards located directly above roads can be used to measure the height at which vehicles can travel, which is useful for road management, etc. Furthermore, estimating the shape of linear parts of objects such as the edges of roofs of houses, etc. can be used to measure the distance from a road to a house, which is useful for laying service lines, etc. For this reason, there is a demand for an object recognition device and an object recognition method that can accurately estimate the shapes of various linear parts.

[0006] The technology described in Patent Document 1 aims to measure the three-dimensional contour of an object in a scene where a robot is working, and assumes that the object will be photographed from a position where it is easy to see. It does not assume a case where multiple linear parts (e.g., overhead lines) are lined up, and the spacing or order of the linear parts changes or crosses occur depending on the viewing angle. For this reason, the technology described in Patent Document 1 may not be able to accurately estimate the shape of an object from an image captured of a scene where multiple linear parts are lined up.

[0007] An object of the present invention is to estimate the shape of an object with high accuracy. [Means for solving the problem]

[0008] An object recognition device according to one aspect of the present invention includes an image receiving unit that receives a plurality of images of an object taken from different viewpoints, a reference image selecting unit that selects a reference image from the plurality of images, a paired image selecting unit that selects a paired image corresponding to the reference image from the plurality of images, an element point arranging unit that arranges element points on a linear portion of the object in the reference image, a first gaze calculation unit that calculates a first gaze corresponding to the element points arranged by the element point arranging unit, an epipolar line calculation unit that calculates an epipolar line on the paired images corresponding to the first gaze calculated by the first gaze calculation unit, and the corresponding point candidate calculation unit calculating a plurality of corresponding point candidates corresponding to one of the element points on one of the epipolar lines calculated by the polar line calculation unit; a second line of sight calculation unit calculating a plurality of second lines of sight corresponding to each of the plurality of corresponding point candidates calculated by the corresponding point candidate calculation unit; a three-dimensional intersection calculation unit calculating an intersection between the first line of sight and each of the plurality of second lines of sight; an object shape estimation unit estimating a shape of the object based on a three-dimensional point cloud of the intersections calculated by the three-dimensional intersection calculation unit; and an object shape output unit outputting the shape of the object estimated by the object shape estimation unit. Effect of the Invention

[0009] According to the present invention, the shape of an object can be estimated with high accuracy. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating a hardware configuration of an object recognition system 10 including an object recognition device 100 according to a first embodiment. [Diagram 2] FIG. 2 is a perspective view illustrating an example in which a plurality of images are captured by an on-vehicle camera device 32 provided on a vehicle 31 traveling on a road 33. As shown in FIG. [Figure 3A] FIG. 3A is an explanatory diagram of the epipolar surface ES. [Figure 3B] FIG. 3B is an explanatory diagram of epipolar points Ea, Eb and epipolar line EL. [Figure 4]FIG. 4 is an explanatory diagram of a three-dimensional reconstruction of the overhead line 36. [Diagram 5] FIG. 5 is a block diagram illustrating a functional configuration of the object recognition device 100 according to the first embodiment. [Figure 6] FIG. 6 is an explanatory diagram of a method for arranging element points Xa(i). [Figure 7] FIG. 7 is a diagram showing an example of the flow of processing executed by the object recognition device 100 according to the first embodiment. [Figure 8] FIG. 8 is an explanatory diagram of a method for estimating the shape of the overhead line 36. In FIG. [Figure 9] FIG. 9 is a flowchart showing an example of the flow of processing executed by the object recognition device 100 according to the second embodiment. [Figure 10] FIG. 10 is a flowchart showing details of the paired image selection process (S220) by the paired image selection unit 113 according to the second embodiment. [Figure 11] FIG. 11 is a schematic diagram showing the optical axis Oa of the camera Ca and a quadrangular pyramid that defines the angle of view range of the camera Ca. [Figure 12] FIG. 12 is a flowchart showing an example of the flow of processing executed by the object recognition device 100 according to the third embodiment. [Figure 13] FIG. 13 is a flowchart showing details of the paired image selection process (S320) by the paired image selection unit 113 according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] First Embodiment An object recognition device according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 8.

[0012] 1 is a block diagram showing a schematic hardware configuration of an object recognition system 10 including an object recognition device 100 according to a first embodiment. The object recognition system 10 includes, for example, an on-vehicle camera device 32, an on-vehicle controller 23, and a communication device 24 mounted on a vehicle 31, and the object recognition device 100, a display device 4, an input device 5, and a communication device 6 installed in a management facility. The management facility is a facility that manages the road environment, and collects data detected by a plurality of vehicles 31.

[0013] The vehicle-mounted camera device 32 has a first camera 2a and a second camera 2b. The first camera 2a and the second camera 2b are attached, for example, at the front and rear of the vehicle 31 in the traveling direction, and capture a predetermined range including the field of view in front or behind the vehicle. In this embodiment, the camera 2 is described as capturing an image in front of the vehicle. In the following description, the first camera 2a and the second camera 2b may be collectively referred to as the camera 2. The camera 2 is an imaging device equipped with an imaging element 21 such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) and a DSP (Digital Signal Processor) 22. The first camera 2a and the second camera 2b are attached so that their angle of view ranges overlap. It is difficult to estimate the camera parameters by SfM only from the image in front of the vehicle. When SfM is applied, it is better to configure the camera 2 to capture the surroundings of the vehicle including the sides of the vehicle.

[0014] The first camera 2a and the second camera 2b each generate an image of the front of the vehicle and output it to the in-vehicle controller 23. The in-vehicle controller 23 stores data associating the image of the front of the vehicle, the time when the image was captured, and the position and orientation of the vehicle 31 when the image was captured, for each predetermined sampling time. The position and orientation of the vehicle 31 are obtained by a well-known GNSS sensor or inertial measurement unit. Hereinafter, the data log stored in the in-vehicle controller 23 is also referred to as time history data.

[0015] The in-vehicle controller 23 transmits the time history data to the object recognition device 100 via the communication device 24 at a predetermined timing (for example, the timing when the vehicle is stopped after the ignition is turned off). The object recognition device 100 acquires an image of the front of the vehicle via the communication device 6. The communication devices 6 and 24 have a communication interface that enables two-way communication via a communication line NT of the wide area network.

[0016] The object recognition device 100 is a device that recognizes various objects, and is configured, for example, by a server operated by an administrator who manages roads and the like at a management facility. The object recognition device 100 detects an object present in front of the vehicle 31 based on a pair of vehicle front images. The objects include, for example, overhead wires, footbridges, overpasses, traffic lights, signs, and houses. The object recognition device 100 according to this embodiment estimates the shape of a linear part that is a straight or curved part of an object. An overhead wire is itself a linear object, and the entire overhead wire corresponds to the linear part. In a footbridge and an overpass, for example, a bridge girder corresponds to the linear part. In a traffic light and a sign, for example, the lower end portion corresponds to the linear part. In a house, for example, the edge of the roof (such as an eaves) corresponds to the linear part.

[0017] The object recognition device 100 estimates the shapes of various linear parts and outputs the estimation results. This allows the administrator to efficiently manage roads and plan the placement of overhead lines. As described above, there are various types of linear parts, but in this embodiment, the function of the object recognition device 100 will be described in detail using an example in which the linear part is an overhead line.

[0018] The object recognition device 100 calculates the three-dimensional coordinates of a detected object (such as an overhead line) and outputs the calculation result to the display device 4. The display device 4 is an output device that outputs the calculation result as an image, such as a liquid crystal display device. The display device 4 displays the object (such as an overhead line) on a display screen based on the calculation result of the object recognition device 100. The input device 5 outputs a signal according to an operation by an administrator (user) to the object recognition device 100. The input device 5 is, for example, a touch sensor provided on the display screen of the display device 4. The input device 5 may be a keyboard, a mouse, or the like.

[0019] The object recognition device 100 is composed of a computer including a processing device 101 such as a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP), a non-volatile memory 102 such as a read only memory (ROM), a flash memory, or a hard disk drive, a volatile memory 103 called a random access memory (RAM), an input / output interface 104, and other peripheral circuits. These hardware components work together to operate software and realize a plurality of functions. The object recognition device 100 may be composed of one computer or a plurality of computers. The processing device 101 may be an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like.

[0020] The non-volatile memory 102 stores programs capable of executing various calculations, as well as thresholds and function models used in the various calculations. That is, the non-volatile memory 102 is a storage medium (storage device) capable of reading programs that realize the functions of this embodiment. The volatile memory 103 is a storage medium (storage device) that temporarily stores the results of calculations performed by the processing device 101 and signals input from the input / output interface 104. The processing device 101 is a calculation device that loads the programs stored in the non-volatile memory 102 into the volatile memory 103 and executes calculations, and performs predetermined calculation processing on data taken from the input / output interface 104, the non-volatile memory 102, and the volatile memory 103 in accordance with the programs.

[0021] The object recognition device 100 is connected to each of a display device 4, an input device 5, and a communication device 6. An input section of the input / output interface 104 converts signals input from various devices (the input device 5, the communication device 6, etc.) into data that can be calculated by the processing device 101. In addition, an output section of the input / output interface 104 generates an output signal according to the calculation result in the processing device 101, and outputs the signal to the various devices (the display device 4, the communication device 6, etc.).

[0022] The first camera 2a includes an imaging element 21a (21) and a DSP 22a (22). Similarly, the second camera 2b includes an imaging element 21b (21) and a DSP 22b (22). The imaging element 21 includes photodiodes arranged in a grid pattern. The imaging element 21 outputs an imaging signal capturing an image of a subject to the DSP 22. The DSP 22 converts the imaging signal input from the imaging element 21 into an image signal. In this conversion process, the DSP 22 performs image quality correction such as contrast correction, gamma correction, and edge correction. The first camera 2a outputs an image signal generated by the DSP 22a to the in-vehicle controller 23. The second camera 2b outputs an image signal generated by the DSP 22b to the in-vehicle controller 23.

[0023] 2, an example will be described in which a plurality of images are acquired by an on-vehicle camera device 32 provided on a vehicle 31 traveling on a road 33. For example, the on-vehicle camera device 32 is mounted on the vehicle (automobile) 31, and the vehicle 31 travels while capturing images of a predetermined range 35 in front of the vehicle 31. The first camera 2a and the second camera 2b of the on-vehicle camera device 32 output images of the area in front of the vehicle 31 captured at a predetermined time interval (sampling time) to the on-vehicle controller 23.

[0024] The images captured by the vehicle-mounted camera device 32 include objects such as utility poles 34, houses (not shown), signs (not shown), and overhead lines 36. The overhead lines 36 include electric wires and communication lines installed in the air via utility poles 34, and service lines wired from the utility poles 34 to each house. Such information on the layout of the overhead lines 36 is useful when performing maintenance on the electric wires, communication lines, etc. Furthermore, if the overhead lines 36 are loose, they will hinder the passage of large vehicles. For this reason, there is also a demand for data on the positions of the overhead lines 36, their height from the road 33, etc.

[0025] SfM (Structure from Motion) is known as a technique for estimating the camera posture from multi-view camera images. Also, MVS (Multi-View Stereo) is known as a technique for generating a three-dimensional point cloud from multi-view camera images. However, in these techniques (SfM, MVS), it is difficult to perform correspondence by feature matching between images when recognizing a linear object such as an overhead wire 36. Although a method such as PatchMatch that complements the correspondence relationship in the vicinity of a predetermined position on a linear object may be applied, the accuracy is not sufficient. In this embodiment, an object recognition device 100 and an object recognition method are provided that enable restoration of a linear object (such as an overhead wire 36) that is difficult to perform feature matching with SfM or MVS from an image to three-dimensional data (three-dimensional restoration).

[0026] 3A, 3B, and 4, an outline of a process in which the object recognition device 100 restores three-dimensional data of the overhead wire 36 from an image including the overhead wire 36 will be described. Fig. 3A is an explanatory diagram of an epipolar plane ES, and Fig. 3B is an explanatory diagram of epipolar points Ea, Eb and an epipolar line EL.

[0027] As shown in Fig. 3A, the plane formed by the first camera 2a, the second camera 2b, and the gaze point X is called the epipolar plane ES. As shown in Fig. 3B, the points obtained by projecting the position of one camera 2 onto the image (imaging plane) of the other camera 2 are called epipolar points Ea and Eb, and the straight line passing through these points is called the epipolar line EL. The epipolar point Ea is the point obtained by projecting the viewpoint (optical center, lens focus) Pb of the second camera 2b onto the image (imaging plane) 60a of the first camera 2a, and the epipolar point Eb is the point obtained by projecting the viewpoint (optical center, lens focus) Pa of the first camera 2a onto the image (imaging plane) 60b of the second camera 2b.

[0028] When the position of the corresponding point Xa corresponding to the fixation point X on the image 60a captured by the first camera 2a is obtained, the fixation point X is on a straight line connecting the viewpoint Pa of the first camera 2a and the corresponding point Xa. The straight line projected onto the image 60b captured by the second camera 2b is an epipolar line EL, and the corresponding point Xb corresponding to the fixation point X on the image 60b exists on the epipolar line EL. This constraint condition is called the epipolar constraint condition, and is the basis of stereo vision.

[0029] FIG. 4 is an explanatory diagram of three-dimensional reconstruction of the overhead wire 36. In FIG. 4, the image 60a captured by the first camera 2a is the reference image A, and the image 60b captured by the second camera 2b is the paired image B. In the following description, the first camera 2a that captured the reference image A is referred to as the reference camera Ca, and the second camera 2b that captured the paired image B is referred to as the paired camera Cb. The object recognition device 100 sets an arbitrary point on the overhead wire 36 in the reference image A as a corresponding point (an element point to be described later) Xa corresponding to the gaze point X. The object recognition device 100 obtains a first line of sight Va by connecting the viewpoint Pa of the reference camera Ca and the corresponding point Xa on the reference image A. The first line of sight Va is a half line in three-dimensional space that extends from the viewpoint Pa of the reference camera Ca and passes through the corresponding point Xa. The object recognition device 100 projects the first line of sight Va onto the paired image B to obtain an epipolar line EL. The object recognition device 100 searches for a corresponding point Xb that corresponds to the gaze point X on the paired image B on the epipolar line EL.

[0030] The object recognition device 100 obtains a second line of sight Vb by connecting the viewpoint Pb of the paired camera Cb and the corresponding point Xb on the paired image B. The second line of sight Vb is a half line in a three-dimensional space that extends from the viewpoint Pb of the paired camera Cb and passes through the corresponding point Xb. The object recognition device 100 obtains the three-dimensional coordinates of a fixation point X that is the intersection point of the first line of sight Va and the second line of sight Vb in the three-dimensional space.

[0031] In the same manner as described above, the object recognition device 100 obtains corresponding points Xb corresponding to the multiple element points Xa on the overhead line 36 in the reference image A, and obtains the three-dimensional coordinates of the multiple gaze points X (i.e., the three-dimensional coordinates of the intersection points X of the first line of sight Va and the second line of sight Vb). The object recognition device 100 estimates the shape of the overhead line 36 based on the three-dimensional coordinates of the multiple intersection points X.

[0032] Here, the images 60a and 60b may contain many linear parts SL in addition to the predetermined overhead line 36 whose shape is to be estimated (see FIG. 6). For this reason, it is difficult to uniquely determine an appropriate corresponding point Xb in the paired image B. Therefore, in this embodiment, the object recognition device 100 calculates all of the intersections between the epipolar line EL and the multiple linear parts SL extracted in the paired image B as candidates for the corresponding point Xb (hereinafter, referred to as corresponding point candidates). The object recognition device 100 obtains the above-mentioned three-dimensional coordinates of the intersections X for all of the corresponding point candidates. Furthermore, the object recognition device 100 repeatedly executes the same process on another reference image A and the paired image B, and stores the accumulated three-dimensional coordinates of the intersections X as a three-dimensional point cloud. The object recognition device 100 executes noise processing based on the point cloud density on the three-dimensional point cloud. The object recognition device 100 estimates the shape of the linear part SL based on the three-dimensional point cloud after noise processing. The functions of the object recognition device 100 according to this embodiment will be described in detail below.

[0033] 5 is a block diagram showing a schematic functional configuration of the object recognition device 100 according to the first embodiment. The object recognition device 100 includes an image receiving unit 111, a reference image selecting unit 112, a paired image selecting unit 113, an element point arranging unit 121, a first gaze calculating unit 122, an epipolar line calculating unit 123, a corresponding point candidate calculating unit 124, a second gaze calculating unit 125, a three-dimensional intersection calculating unit 126, an object shape estimating unit 131, and an object shape output unit 132. Each of these functional units is realized in the form of software by the processing device 101 (see FIG. 1) executing a program stored in the non-volatile memory 102 (see FIG. 1).

[0034] The camera parameters 106 include internal parameters indicating the resolution (image size), horizontal angle of view, vertical angle of view, focal length, etc. of the camera 2, and external parameters indicating the position and attitude of the camera 2. The camera parameters 106 can be obtained by SfM or the like, but it is difficult to estimate them only from the image in front of the vehicle. When SfM is applied, it is better to configure the camera 2 so as to capture the surroundings of the vehicle including the sides of the vehicle (not shown). In FIG. 5, the camera parameters 106 estimated by another controller are provided to the object recognition device 100. The images 60a and 60b are configured to receive images that have been corrected for distortion by the first camera 2a and the second camera 2b. The distortion correction may be performed inside the object recognition device 100 by adding a distortion coefficient to the internal parameters of the camera parameters 106. The input may be an image captured by the camera 2, and the camera parameters may be estimated inside the object recognition device 100 in the same manner as SfM.

[0035] The image receiving unit 111 receives a plurality of images of an object captured from different viewpoints. For example, the image receiving unit 111 receives time history data including a plurality of images 60a captured by the first camera 2a and a plurality of images 60b captured by the second camera 2b at the same timing as the first camera 2a.

[0036] The reference image selection unit 112 selects a reference image A from a plurality of images. The reference image selection unit 112 selects one of the image 60a captured by the first camera 2a and the image 60b captured by the second camera 2b as the reference image A. In this embodiment, the reference image selection unit 112 selects the image 60a captured by the first camera 2a as the reference image A.

[0037] The reference image selection unit 112 selects the reference images A from the multiple images 60a so that they are not concentrated in the same place but are dispersed. For example, the reference image selection unit 112 refers to the time history data and the camera parameters 106, and based on the position of the first camera 2a, does not select the images 60a captured within a certain distance range from the position where the already selected reference image A was captured.

[0038] The reference image selection unit 112 may select the reference image A at a predetermined time interval with reference to the time history data so that the shooting times are dispersed. The in-vehicle controller 23 may store the images shot at each predetermined moving distance as time history data.

[0039] The paired image selection unit 113 selects, from a plurality of images, a paired image B corresponding to the reference image A. The paired image selection unit 113 selects, as the paired image B, an image 60b captured at the same time (or the closest time) to the image 60a selected as the reference image A.

[0040] The element point arrangement unit 121 arranges an element point Xa(i) at an arbitrary position on a linear portion of an object in the reference image A (60a). An example of a method for arranging the element point Xa(i) will be described with reference to FIG. 6. The element point arrangement unit 121 performs a known contour line extraction process on the reference image A (60a). As a result, linear portions SL such as the contour line (frame line) of a building and the overhead wire 36 are extracted in the reference image A. The element point arrangement unit 121 sets a reference point 37 in front of the road 33. The element point arrangement unit 121 sets n auxiliary lines AL(i) (i=1, 2, ..., n) radially upward from the reference point 37. The auxiliary lines AL(i) have parameters of an angle (inclination angle from a reference plane such as a horizontal plane) and the number (n). The element point arrangement unit 121 determines the intersections of the extracted line (linear portion) SL and a plurality of auxiliary lines AL(i) as element points Xa(i) for the reference image A.

[0041] The method of arranging the element points is not limited to the method of generating a plurality of auxiliary lines AL(i) and arranging the intersections of the plurality of auxiliary lines AL(i) and the linear portion SL extracted by image processing as element points Xa(i). For example, the element point Xa(i) may be arbitrarily set by the administrator (user). In this case, the element point arranging unit 121 displays the reference image A on the display screen of the display device 4. While looking at the reference image A displayed on the display device 4, the administrator performs an operation of specifying an arbitrary point on the overhead line 36 using the input device 5. The element point arranging unit 121 arranges the element points Xa(i) based on an operation signal input from the input device 5 in response to the administrator's operation on the input device 5.

[0042] Also, for example, the element point arrangement section 121 may arrange a plurality of element points Xa(i) at regular intervals or randomly on the linear portion SL of the reference image A (60a).

[0043] 5 calculates a first line of sight Va(i) corresponding to an element point Xa(i) arranged by the element point arrangement unit 121, using the position, posture, horizontal angle of view, vertical angle of view, and resolution (image size) of the reference camera Ca included in the camera parameters 106. The epipolar line calculation unit 123 calculates an epipolar line EL(i) on the paired image B (60b) corresponding to the first line of sight Va(i) calculated by the first line of sight calculation unit 122, using the positions, postures, horizontal angle of view, vertical angle of view, and resolution (image size) of the reference camera Ca and paired camera Cb included in the camera parameters 106.

[0044] The correspondence point candidate calculation unit 124 calculates a correspondence point candidate corresponding to the element point Xa(i) on the epipolar line EL(i) calculated by the epipolar line calculation unit 123. As described above, when a plurality of linear portions SL are arranged side by side, it is difficult to uniquely find one correspondence point corresponding to one element point Xa(i) on one epipolar line EL from the paired images B. For this reason, the correspondence point candidate calculation unit 124 calculates multiple correspondence point candidates corresponding to one element point Xa(i) on one epipolar line EL(i) calculated by the epipolar line calculation unit 123. A specific calculation method is as follows.

[0045] The corresponding point candidate calculation unit 124 first performs a known contour extraction process on the paired image B (60b). As a result, linear parts SL such as overhead lines 36 are extracted from the paired image B. The corresponding point candidate calculation unit 124 extracts all intersections between one epipolar line EL(i) and each of the multiple linear parts SL intersecting the epipolar line EL(i) as corresponding point candidates. As a result, multiple corresponding point candidates Xb(i,p) corresponding to one element point Xa(i) on one epipolar line EL(i) are calculated (p=1,...,R(i)). Note that R(i) corresponds to the number of linear parts SL intersecting the epipolar line EL(i) in the paired image B in the calculation process of the corresponding point candidates corresponding to the element point Xa(i).

[0046] The second gaze calculation unit 125 calculates multiple second gazes Vb(i,p) corresponding to each of the multiple correspondence point candidates Xb(i,p) calculated by the correspondence point candidate calculation unit 124, using the position, orientation, horizontal angle of view, vertical angle of view, and resolution (image size) of the pair camera Cb included in the camera parameters 106.

[0047] The three-dimensional intersection calculation unit 126 calculates an intersection X(i,p) between the first line of sight Va(i) and each of the multiple second lines of sight Vb(i,p). Here, the two straight lines (the first line of sight Va and the second line of sight Vb) do not necessarily exist on the same plane because they include calculation errors and the like. Therefore, the three-dimensional intersection calculation unit 126 approximately determines the nearest point between the first line of sight Va and the second line of sight Vb as the intersection X(i,p). For example, the three-dimensional intersection calculation unit 126 expresses the points on the first line of sight Va and the points on the second line of sight Vb with parameters, and minimizes the distance between them as a function of the parameters. The three-dimensional intersection calculation unit 126 determines the center point of the line segment connecting the points on the first line of sight Va and the points on the second line of sight Vb, which has the smallest distance, as the intersection X(i,p).

[0048] The object shape estimation unit 131 estimates the shape of an object (such as the overhead line 36) using the object shape model 133 based on the accumulated data of the multiple intersections X(i,p) calculated by the three-dimensional intersection calculation unit 126, i.e., the three-dimensional point cloud of the intersections X. The object shape estimation unit 131 first performs a noise removal process based on the density of the three-dimensional point cloud. The object shape estimation unit 131 estimates the shape of the object based on the three-dimensional point cloud after the noise removal process. The object shape estimation unit 131 estimates the shape of the overhead line 36 based on the three-dimensional point cloud of the intersections X after the noise removal process and Equation (1) representing the object shape model (overhead line model) 133, which is a function model representing a catenary curve. The object shape model 133 is stored in the non-volatile memory 102 of the object recognition device 100.

[0049]

number

[0050] Here, x, y, and z are the x-, y-, and z-coordinates of a point on the overhead wire 36 in a three-axis coordinate system. The z-axis is parallel to the vertical direction, and the x- and y-axes are parallel to the horizontal direction. tx, ty, and tz are the x-, y-, and z-coordinates of the lowest point of the overhead wire 36. s is the inclination of the overhead wire 36 in the xy plane (horizontal plane). The overhead wire 36 is a straight line when viewed from directly above (when viewed from the z-axis direction). Therefore, the inclination s of the overhead wire 36 in the xy plane (horizontal plane) is expressed as the ratio of the increment Δy of the y-coordinate of the overhead wire 36 to the increment Δx of the x-coordinate (s=Δy / Δx). a is a parameter indicating the slack of the overhead wire 36.

[0051] The object shape estimation unit 131 generates parameters of the object shape model (overhead line model) 133 as an estimation result of the shape of the overhead line 36. Note that the object shape estimation unit 131 may estimate the shape of the object by combining a functional model such as a plane or a cylinder, instead of the object shape model (overhead line model) 133. Furthermore, the object shape estimation unit 131 may generate three-dimensional point cloud data from which noise has been removed or CAD model data as an estimation result of the shape of the object, without using a functional model.

[0052] The object shape output unit 132 outputs the shape of the object (such as the overhead line 36) estimated by the object shape estimation unit 131. There are various output modes. For example, the object shape output unit 132 outputs the estimation result of the shape of the overhead line 36 to the display device 4. The display device 4 displays the overhead line 36 on the display screen based on the input estimation result of the shape of the overhead line 36. In this case, the display device 4 can be said to be an output device that outputs the estimation result of the shape of the overhead line 36 by a display image. Furthermore, the object shape output unit 132 may output the estimation result of the shape of the overhead line 36 to the communication device 6. In other words, the object shape output unit 132 may output the estimation result of the shape of the overhead line 36 to another computer via the communication device 6. In this case, the communication device 6 can be said to be an output device that outputs the estimation result of the shape of the overhead line 36 by communication data. Furthermore, the object shape output unit 132 may output the estimation result of the shape of the overhead line 36 to a printing device (not shown). The printing device prints the overhead wire 36 on a printing medium based on the inputted estimation result of the shape of the overhead wire 36. In this case, the printing device (not shown) can be said to be an output device that outputs the estimation result of the shape of the overhead wire 36 as a print image.

[0053] Fig. 7 is a diagram showing an example of a flow of processing executed by the object recognition device 100 according to the first embodiment. As shown in Fig. 7, in step S105, a first loop process is started. In the first loop process, a process of calculating an intersection point (candidate) between the lines of sight Va and Vb used to estimate the shape of an object is repeated for each time of the time history data based on two images captured by the first camera 2a and the second camera 2b (steps S105 to S158). As a result, a three-dimensional point group consisting of a large number of intersection points (candidates) in a three-dimensional space is obtained.

[0054] In step S110, the image receiving unit 111 receives two images of an object. In the next step S115, the reference image selecting unit 112 selects one of the two images 60a, 60b received in step S110 as the reference image A. In this embodiment, the image 60a captured by the first camera 2a is selected as the reference image A.

[0055] In the next step S120, the paired image selection unit 113 selects the other of the two images 60a, 60b accepted in step S110 as paired image B. In this embodiment, the image 60b captured by the second camera 2b is selected as paired image B corresponding to the reference image A.

[0056] In the next step S125, the element point arrangement unit 121 executes a contour extraction process for the reference image A. The element point arrangement unit 121 arranges n element points Xa(i) (i=1, . . . , n) on the extracted linear portion SL.

[0057] In the next step S130, the object recognition device 100 judges whether or not the second loop process, which repeats the processes from steps S135 to S155 from i=1 to i=n, ​​has been completed. Here, an example of a process flow in which each process (S135 to S155) is executed in order from i=1 (initial value) to n, will be described. If it is judged in step S130 that the second loop process has not been completed, the processes from steps S135 to S155 are executed, and then 1 is added to i (i is incremented), and the process returns to step S130. Then, if it is judged in step S130 that the second loop process has been completed, that is, if it is judged that each process (S135 to S155) for i=n has been completed, the process proceeds to step S158.

[0058] In step S135, the first gaze calculation unit 122 calculates the first gaze Va(i) corresponding to the element point Xa(i). In the next step S140, the epipolar line calculation unit 123 calculates an epipolar line EL(i) on the paired image B corresponding to the first gaze Va(i) calculated in step S135.

[0059] In the next step S145, the correspondence point candidate calculation unit 124 calculates a plurality of correspondence point candidates Xb(i,p) (p=1,...,R(i)) corresponding to the element point Xa(i) on the epipolar line EL(i) calculated in step S140. In the next step S150, the second gaze calculation unit 125 calculates a plurality of second gazes Vb(i,p) corresponding to each of the plurality of correspondence point candidates Xb(i,p) calculated in step S145.

[0060] In the next step S155, the three-dimensional intersection calculation unit 126 calculates the coordinates of the intersections between the first line of sight Va(i) calculated in step S135 and each of the multiple second lines of sight Vb(i,p) calculated in step S150. Upon completing the calculation of the coordinates of the intersections, the three-dimensional intersection calculation unit 126 increments i and proceeds to step S130.

[0061] Each process from step S125 to step S155 is performed for n element points Xa(i), thereby obtaining coordinates of a plurality of intersections (candidates) representing an object on which the element points are arranged. In other words, the processes from step S125 to step S155 can be said to be a 3D reconstruction process (S124) based on the reference image A and the paired image B. Therefore, the element point arrangement unit 121, the first gaze calculation unit 122, the epipolar line calculation unit 123, the corresponding point candidate calculation unit 124, the second gaze calculation unit 125, and the 3D intersection calculation unit 126 shown in FIG. 5 constitute a 3D reconstruction unit 120 that performs a 3D reconstruction process based on the reference image A and the paired image B.

[0062] When the 3D reconstruction process shown in FIG. 7 is completed, the process proceeds to step S158. In step S158, the object recognition device 100 determines whether the first loop process is completed. When the processes from steps S110 to S155 are completed for all target images in the time history data, the object recognition device 100 determines that the first loop process is completed. When the processes from steps S110 to S155 are not completed for all target images in the time history data, the object recognition device 100 determines that the first loop process is not completed. When the first loop process is not completed, the processes from steps S110 to S155 are executed for data for a time a predetermined time after the time the processes are completed.

[0063] If it is determined in step S158 that the first loop process is completed, the process proceeds to step S160. In step S160, the object shape estimation unit 131 estimates the shape of the object (the overhead line 36, etc.) based on the three-dimensional point group of the intersections. A method for estimating the shape of the overhead line 36 will be described with reference to FIG. 8. FIG. 8 shows display images 191, 192, and 193 when a road (not shown) is viewed from diagonally above, with a predetermined height from the ground as the viewpoint. Although not shown, the road extends from the lower left to the upper right in FIG. 8, and a plurality of overhead lines 36 are arranged so as to cross the road.

[0064] 8 shows a display image 191 of the 3D point cloud of the intersection before noise removal processing, a display image 192 of the 3D point cloud after noise removal processing, and a display image 193 showing the estimation result of the shape of the overhead line 36 using the object shape model (overhead line model) 133. As shown in the display image 191 of Fig. 8, the 3D point cloud of the intersection includes points that are not appropriate for estimating the overhead line 36. This is because the multiple corresponding point candidates Xb(i,p) include noise such as cloud contours, shadows, and overhead lines other than the target overhead line 36.

[0065] These noises are difficult to separate in the 3D reconstruction process. The noises included in the 3D point cloud are points that are spread out in the distance, so they can be separated by point cloud processing. For example, the object shape estimation unit 131 extracts only intersections with high point cloud density. For example, when the density of the 3D point cloud within a predetermined range in the 3D space is less than a predetermined density, the object shape estimation unit 131 removes the 3D point cloud within the predetermined range as noise. In addition, as a noise removal processing method, noise may be removed as follows. Specifically, the object shape estimation unit 131 calculates the spatial density of the intersection X in a predetermined space centered on the intersection X. When the spatial density is less than a density threshold, the object shape estimation unit 131 removes the intersection X as noise. When the spatial density is equal to or greater than the density threshold, the object shape estimation unit 131 extracts the intersection X as an estimation intersection used to estimate the shape of the object. As a result, a 3D point cloud after noise removal is obtained as shown in the display image 192 of FIG. 8.

[0066] The object shape estimation unit 131 estimates the shape of the overhead line 36 from the noise-removed 3D point cloud using the object shape model (overhead line model) 133. As shown in Fig. 7, when the object shape estimation process (step S160) is completed, the process proceeds to step S165.

[0067] In step S165, the object shape output unit 132 outputs data representing the shape of the object estimated in step S160 to the display device 4, and ends the process shown in the flowchart of Fig. 7. As shown in Fig. 8, the display device 4 displays a display image 193 of the shape of the overhead wire 36 on the display screen. This allows the manager to identify areas where the overhead wire 36 has a large slack and requires caution when a large vehicle passes, based on the display image 193.

[0068] According to the first embodiment, the following advantageous effects are achieved.

[0069] The object recognition device 100 includes an image receiving unit 111 that receives a plurality of images of a linear object such as an overhead wire 36 taken from different viewpoints, a reference image selecting unit 112 that selects a reference image A from the plurality of images, a paired image selecting unit 113 that selects a paired image B corresponding to the reference image A from the plurality of images, an element point arranging unit 121 that arranges an element point Xa on a linear portion (linear portion) SL of the object in the reference image A, a first line of sight calculation unit 122 that calculates a first line of sight Va corresponding to the element point Xa arranged by the element point arranging unit 121, and an epipolar line calculation unit 123 that calculates an epipolar line EL on the paired image B corresponding to the first line of sight Va calculated by the first line of sight calculation unit 122. The object shape estimation unit 131 includes a correspondence point candidate calculation unit 124 that calculates a plurality of correspondence point candidates Xb corresponding to one element point Xa on one epipolar line EL calculated by the epipolar line calculation unit 123, a second line of sight calculation unit 125 that calculates a plurality of second lines of sight Vb corresponding to each of the plurality of correspondence point candidates Xb calculated by the correspondence point candidate calculation unit 124, a three-dimensional intersection calculation unit 126 that calculates an intersection X between the first line of sight Va and each of the plurality of second lines of sight Vb, an object shape estimation unit 131 that estimates a shape of an object based on a three-dimensional point group of the intersection X calculated by the three-dimensional intersection calculation unit 126, and an object shape output unit 132 that outputs the shape of the object estimated by the object shape estimation unit 131. The correspondence point candidate calculation unit 124 calculates an intersection between a line portion SL extracted in the paired images B and the epipolar line EL as a correspondence point candidate Xb. Object shape estimation section 131 performs noise removal processing based on the density of the three-dimensional point cloud, and estimates the shape of the object based on the three-dimensional point cloud after noise removal processing.

[0070] As described above, when multiple overhead lines 36 are arranged side by side, multiple linear parts SL may intersect with one epipolar line EL, and multiple intersections may exist. However, at the stage where these intersections are calculated, it is difficult to determine which intersection corresponds to the appropriate gaze point. For this reason, the intersections are set as corresponding point candidates Xb, and the intersection X between the lines of sight is obtained, and noise is removed in the object shape estimation process, which is a later process, so that the shape of the object can be estimated with high accuracy. According to this embodiment, the shape of an object such as the overhead line 36 can be estimated with high accuracy. The administrator can appropriately perform road management, etc., based on the estimation result of the shape of the object output by the object recognition device 100.

[0071] In the present embodiment, the estimation of the shape of the overhead wire 36 has been described as an example, but the estimation of the shape of the linear portion SL of an object such as a pedestrian bridge girder, an overpass girder, the lower end of a traffic light, or the lower end of a sign may be performed. In this case, the estimation result of the shape of the linear portion SL can be used to measure the height of a vehicle that can travel, which can be useful for road management, etc. In addition, the estimation of the shape of the linear portion SL of an object such as the edge of the roof of a house, etc. may be performed. In this case, the estimation result of the shape of the linear portion SL can be used to measure the distance from the road to the house, which can be useful for laying a drop line, etc.

[0072] <Second embodiment> An object recognition device 100 according to a second embodiment of the present invention will be described with reference to Figs. 9 to 11. Configurations that are the same as or equivalent to those described in the first embodiment are given the same reference symbols, and differences will be mainly described. Fig. 9 is a diagram similar to Fig. 7, and is a flowchart showing an example of the flow of processing executed by the object recognition device 100 according to the second embodiment.

[0073] In the first embodiment, an example was described in which, at the same time, one of the images 60a and 60b captured by the first camera 2a and the second camera 2b is selected as the reference image A, and the other is selected as the pair image B. In contrast, in the second embodiment, at the same time, a combination of the reference image A(k) and the pair image B(k,j) is selected from among N images (N is the number of cameras 2) captured by each of three or more cameras 2. Here, it will be described with k < j.

[0074] As shown in FIG. 9, in step S210, the image reception unit 111 receives N images captured of an object from different viewpoints. In the next step S215, the reference image selection unit 112 selects the reference image A(k) from among the N images received in step S210. In this embodiment, the N images are sequentially selected one by one as the reference image A(k) (k = 1,..., N - 1). k is a number for identifying the camera 2 mounted on the vehicle 31.

[0075] In the next step S217, the object recognition device 100 determines whether or not a second loop process in which the processes from step S220 to S224 are repeated from k = 1 to k = N - 1 has been completed. In step S217, if it is determined that the second loop process has not been completed, the process proceeds to step S220, and if it is determined that the second loop process has been completed, the process proceeds to step S158.

[0076] In step S220, the pair image selection unit 113 selects the pair image B(k,j) corresponding to the reference image A(k) from among the N images received in step S210 (j = {M(k)}). {M(k)} corresponds to a list of pair images B selected corresponding to the reference image A(k) in a pair image selection process (see FIG. 10) described later among the images where j = k + 1,..., N. A specific example of the selection process of the pair image B(k,j) will be described later.

[0077] In the next step S222, the object recognition device 100 determines whether or not the third loop process, which repeats the process of step S224 for j={M(k)}, has been completed. If it is determined in step S222 that the third loop process has not been completed, the process proceeds to step S224, and if it is determined that the third loop process has been completed, k is incremented and the process returns to step S217.

[0078] In step S224, the 3D reconstruction unit 120 executes 3D reconstruction processing based on the reference image A(k) and the paired image B(k,j). The 3D reconstruction processing (S224) is similar to the 3D reconstruction processing (S124) of the first embodiment. In addition, the completion determination processing (S158) of the first loop processing S105, the object shape estimation processing (S160), and the object shape output processing (S170) are similar to the first embodiment, so their explanations are omitted.

[0079] Fig. 10 is a flowchart showing details of the pair image selection process (S220) by the pair image selection unit 113 according to the second embodiment. As shown in Fig. 10, in the pair image selection process (step S220 in Fig. 9) of the second embodiment, first, in step S2221, the pair image selection unit 113 judges whether or not the fourth loop process, which repeats the processes from steps S2222 to S2227 from j=k+1 to j=N, has been completed. If it is judged in step S2221 that the fourth loop process has not been completed, the process proceeds to step S2222, and if it is judged that the fourth loop process has been completed, the process shown in the flowchart in Fig. 10 ends and proceeds to step S222 in Fig. 9.

[0080] 10, in step S2222, paired image selection unit 113 selects image B(j). In the next step S2223, paired image selection unit 113 calculates a distance L between camera Ca that captured reference image A(k) and camera Cb that captured image B(j) (hereinafter also referred to as inter-camera distance) based on the position of camera 2 included in camera parameters 106.

[0081] When the distance calculation process (step S2223) is completed, the process proceeds to step S2224. When multiple cameras 2 are provided, if the distance between the positions of the reference camera Ca and the paired camera Cb is too short or too long, the error in the estimation accuracy of the corresponding point Xb increases. For this reason, if the distance between the reference camera Ca and the paired camera (candidate) Cb is equal to or less than the first threshold value Lmin or equal to or more than the second threshold value Lmax, the image B(j) is not selected as a paired image. The first threshold value Lmin and the second threshold value Lmax are values ​​that provide appropriate estimation accuracy and are stored in the non-volatile memory 102 (Lmin <Lmax)。

[0082] In step S2224, the pair image selection unit 113 determines whether the inter-camera distance L calculated in step S2223 is longer than the first threshold Lmin and shorter than the second threshold Lmax. If it is determined that the inter-camera distance L is longer than the first threshold Lmin and shorter than the second threshold Lmax, the process proceeds to step S2225. If it is determined that the inter-camera distance L is equal to or smaller than the first threshold Lmin or equal to or larger than the second threshold Lmax, j is incremented without selecting a pair image, and the process returns to step S2221.

[0083] In step S2225, the paired image selection unit 113 projects the optical axis Oa of the camera Ca shown in FIG. 11 and the four sides Sa of a pyramid that defines the angle of view range of the camera Ca onto the image B(j). The angle of view range is defined as, for example, a range in a quadrangular region that extends infinitely with the viewpoint Pa of the reference camera Ca as its apex, in which the distance or depth from the reference camera Ca is equal to or less than a predetermined value. The angle of view range may be defined by a quadrangular pyramid frustum. The paired image selection unit 113 finds the four sides Sa of the pyramid shape (quadrature pyramid or quadrangular pyramid frustum) that defines the angle of view range using the position, posture, horizontal angle of view, and vertical angle of view of the camera 2 included in the camera parameters 106.

[0084] 10, when the projection process (step S2225) is completed, the process proceeds to step S2226. The paired images B(k,j) must include a linear portion SL where an element point is arranged in the reference image A(k). The process of step S2226 is a process for determining whether or not the angle of view of the camera 2 that captured the image B(j) overlaps with the angle of view of the reference image A(k).

[0085] The process of step S2226 may be a process of geometrically determining whether or not the cone shape representing the angle of view range of reference image A(k) and the cone shape representing the angle of view range of image B(j) overlap, but in this case, there is a risk that a lot of time will be required for calculation. Therefore, in step S2226 according to this embodiment, the pair image selection unit 113 determines whether or not the optical axis Oa and at least a part of the four sides Sa are projected into image B(j).

[0086] If the optical axis Oa and at least a part of the four sides Sa are projected in the image B(j), the paired image selection unit 113 determines that the angle of view range of the camera 2 that captured the image B(j) overlaps with the angle of view range of the reference image A(k), and proceeds to step S2227. If neither the optical axis Oa nor the four sides Sa are projected in the image B(j), the paired image selection unit 113 determines that the angle of view range of the camera 2 that captured the image B(j) does not overlap with the angle of view range of the reference image A(k). In this case, j is incremented without selecting a paired image, and the process returns to step S2221.

[0087] The process of step S2226 may be a process of determining whether or not at least a part of the four sides Sa is projected in the image B(j). In this case, however, if the field angle range of one of the reference image A(k) and the image B(j) includes the whole field angle range of the other, it will be determined that there is no overlap. For this reason, as described above, it is preferable to also determine whether or not at least a part of the optical axis Oa is projected in the image B(j).

[0088] In step S2227, the paired image selection unit 113 selects image B(j) as paired image B(k,j), and adds j at this time to the list {M(k)} of paired images B. When paired image B(k,j) is selected, j is incremented and the process returns to step S2221.

[0089] According to the second embodiment, in addition to the same effects as those of the first embodiment, the following effects can be obtained.

[0090] The paired image selection unit 113 selects an image B(j) whose angle of view range overlaps with the reference image A(k) as a paired image B(k,j) (S2226, S2227). In other words, the paired image selection unit 113 selects the paired image (candidate) B(j) as the paired image B(k,j) when the angle of view range of the camera 2 that captured the reference image A(k) overlaps with the angle of view range of the camera 2 that captured the paired image (candidate) B(j). With this configuration, the number of paired images B(k,j) can be reduced, and therefore the time required for processing by the object recognition device 100 can be reduced.

[0091] The paired image selection unit 113 selects an image that includes at least a part of the line (side edge Sa) that defines the angle of view range of the reference image A(k) as the paired image B(k,j) (S2226, S2227). This configuration can reduce the time required for processing by the object recognition device 100 compared to a case where a process is performed to geometrically determine whether or not the pyramid shape representing the angle of view range of the reference image A(k) and the pyramid shape representing the angle of view range of the image B(j) overlap.

[0092] The paired image selection unit 113 selects an image B(j) that includes at least a part of the optical axis Oa of the camera (imaging device) Ca that captured the reference image A(k) as a paired image B(k,j) (S2226, S2227). With this configuration, even if the angle of view range of one of the reference image A(k) and image B(j) includes the entire angle of view range of the other, it is possible to appropriately select the paired image B(k,j).

[0093] The paired image selection unit 113 selects, as a paired image B(k, j), an image B(j) captured by a camera (second imaging device) 2 whose distance L from the camera (first imaging device) 2 that captured the reference image A(k) is within a predetermined range (Lmin < L < Lmax) (S2224, S2227). According to this configuration, the error in the estimation accuracy of the corresponding point Xb can be reduced, and as a result, the estimation accuracy of the shape of the object can be improved.

[0094] <Third Embodiment> With reference to FIGS. 12 and 13, an object recognition device 100 according to a third embodiment of the present invention will be described. Note that the same or corresponding components as those described in the first and second embodiments are denoted by the same reference numerals, and the differences will be mainly described. FIG. 12 is a diagram similar to FIG. 9 and is a flowchart showing an example of the flow of processing executed by the object recognition device 100 according to the third embodiment. FIG. 13 is a diagram similar to FIG. 10 and is a flowchart showing details of paired image selection processing (S320) by the paired image selection unit 113 according to the third embodiment.

[0095] In the first embodiment, an example in which, at the same time, one of the images 60a and 60b captured by the first camera 2a and the second camera 2b is used as the reference image A and the other is used as the paired image B was described. In contrast, in the third embodiment, the reference image A and the paired image B are selected from among a plurality of images captured by a single camera 2.

[0096] The in-vehicle controller 23 stores, at each predetermined sampling time, data in which a vehicle front image captured by a single camera 2, the time when the image was captured, and the position and orientation of the vehicle 31 when the image was captured are associated with each other. The in-vehicle controller 23 transmits the stored time history data to the object recognition device 100 at a predetermined timing. The time history data includes N images.

[0097] In the flowchart of Fig. 12, the processes of steps S105 and S158 in the flowchart of Fig. 9 are omitted, and instead of the processes of steps S210, S215, S217, and S220, the processes of steps S310, S315, S317, and S320 are executed. As shown in Fig. 12, in step S310, the image receiving unit 111 receives N images of the front of the vehicle captured by the single camera 2 while the vehicle is traveling.

[0098] In the next step S315, the reference image selection unit 112 selects a reference image A(k) from the N images accepted in step S310. In this embodiment, the N images are selected one by one in chronological order as the reference image A(k) (k=2, ..., N).

[0099] In the next step S317, object recognition device 100 determines whether or not a second loop process has been completed, in which the processes in steps S320 to S224 are repeated from k=2 to k=N. If it is determined in step S317 that the second loop process has not been completed, the process proceeds to step S320, and if it is determined that the second loop process has been completed, the process proceeds to step S160.

[0100] In the flowchart of Fig. 13, the process of step S3221 is executed instead of the process of step S2221 in the flowchart of Fig. 10. As shown in Fig. 13, in step S3221, the paired image selection unit 113 judges whether or not the fourth loop process, which repeats the processes of steps S2222 to S2227 from j = 1 to j = k-1, has been completed. If it is judged in step S2221 that the fourth loop process has not been completed, the process proceeds to step S2222, and if it is judged that the fourth loop process has been completed, the process of creating the list {M(k)} of paired images B shown in the flowchart of Fig. 13 ends, and the process proceeds to step S222 in Fig. 12.

[0101] In addition, cameras Ca and Cb in the processing of step S2223 are the same camera 2, and the distance between the position of camera 2 when reference image A was captured and the position of camera 2 when paired image B was captured is the inter-camera distance L.

[0102] For k, 1 is a value for identifying the reference image A captured at the earliest shooting time, and N is a value for identifying the reference image A captured at the latest shooting time. The value of k increases by 1 every time a sampling time elapses. In other words, in this embodiment, the later image in the chronological order is the reference image A, and the earlier image is the paired image B.

[0103] According to the third embodiment, it is possible to obtain the same effects as those of the first and second embodiments.

[0104] The following modified examples are also within the scope of the present invention, and it is possible to combine the configurations shown in the modified examples with the configurations described in the above-mentioned embodiments, to combine the configurations described in the different embodiments above, or to combine the configurations described in the different modified examples below.

[0105] <Variation 1> In the above-described embodiment, an example has been described in which the correspondence point candidate calculation unit 124 determines all intersections between one epipolar line EL and multiple linear portions SL extracted in the paired image B that intersect with the epipolar line EL as correspondence point candidates Xb. However, the present invention is not limited to this. The correspondence point candidate calculation unit 124 may calculate, in the paired image B, a point that is similar to the element point Xa of the reference image A as the correspondence point candidate Xb.

[0106] The correspondence point candidate calculation unit 124 narrows down the correspondence point candidates similar to the element point Xa by using local features (e.g., the slope of a straight line) near the element point Xa in the linear portion SL of the reference image A and local features (e.g., the slope of a straight line) near the intersection with the epipolar line EL in the linear portion SL of the paired image B.

[0107] For example, the correspondence point candidate calculation unit 124 calculates, as a correspondence point candidate Xb, an intersection point between the epipolar line EL and a linear portion SL extracted in the reference image 60a, on which the element point Xa is located, and the linear portion SL having an inclination angle within a predetermined angle range. In other words, even if a linear portion SL of the paired image B, on which the element point Xa is located, has an inclination angle outside a predetermined angle range with respect to the linear portion SL of the reference image A, the intersection point is not calculated as a correspondence point candidate Xb.

[0108] The correspondence point candidate calculation unit 124 calculates an angle between a local portion (straight line portion) near the element point Xa in the linear portion SL of the reference image A and a local portion (straight line portion) near an intersection point of the linear portion SLb intersecting with the epipolar line EL of the paired image B with the epipolar line EL. If the angle is within a predetermined angle range, the correspondence point candidate calculation unit 124 calculates the intersection point between the linear portion SLb and the epipolar line EL as a correspondence point candidate Xb. If the angle is outside the predetermined angle range, the correspondence point candidate calculation unit 124 does not calculate the intersection point between the linear portion SLb and the epipolar line EL as a correspondence point candidate Xb. The predetermined angle range can be set arbitrarily. For example, the predetermined angle range is a range of about ±20°.

[0109] According to this configuration, the number of corresponding point candidates Xb can be reduced, and therefore the processing load of the object recognition device 100 can be reduced.

[0110] <Variation 2> In the above embodiment, an example of removing noise based on the density of a three-dimensional point cloud has been described, but the present invention is not limited to this. For example, a three-dimensional point cloud calculated by SfM and a three-dimensional point cloud of intersection X calculated by the three-dimensional intersection calculation unit 126 may be compared, and points other than the overlapping portion of both may be removed as noise.

[0111] <Variation 3> In the second and third embodiments, steps S2225 and S2226 in Fig. 10 and Fig. 13 may be omitted. In particular, in the third embodiment, since the angle of view range of the reference image A and the angle of view range of the paired image B are considered to overlap in the chronologically close images of the front of the vehicle taken by the camera 2, the effect on the calculation accuracy is small even if the processing of steps S2225 and S2226 is omitted. Also, in the second and third embodiments, steps S2223 and S2224 in Fig. 10 and Fig. 13 may be omitted. Furthermore, in the second and third embodiments, steps S2223 to S2226 may be omitted. When steps S2223 to S2226 are omitted, the paired image B is selected in a brute force manner.

[0112] <Modification 4> In the second and third embodiments, the process of determining whether there is an overlap between the angle of view ranges of the camera Ca that captured the reference image A and the camera Cb that captured the paired image B (candidate) (step S2226 in Figures 10 and 13) is not limited to the examples described above.

[0113] <Modification 4-1> For example, the paired image selection unit 113 may execute a process of projecting an arbitrary line within the angle of view range of the reference image A(k) onto the image B(j) instead of step S2225 in FIG. 10 and FIG. 13, and may execute a process of determining whether or not at least a part of the arbitrary line is projected into the image B(j) instead of step S2226. That is, the paired image selection unit 113 according to this modification selects the image B(j) including at least a part of an arbitrary line within the angle of view range of the reference image A(k) as the paired image B(k,j). According to this configuration, as in the second and third embodiments, the time required for the processing of the object recognition device 100 can be reduced compared to the case of performing a process of geometrically determining whether or not the pyramid shape representing the angle of view range of the reference image A(k) and the pyramid shape representing the angle of view range of the image B(j) overlap.

[0114] <Modification 4-2> For example, the pair image selection unit 113 may execute a process of projecting the first line of sight Va onto the image B(j) instead of step S2225 in Fig. 10 and Fig. 13, and may execute a process of determining whether or not at least a part of the first line of sight Va is projected into the image B(j) instead of step S2226. That is, the pair image selection unit 113 according to this modification selects the image B(j) including at least a part of the first line of sight Va as the pair image B(k,j). This configuration can obtain the same effects as those of the second and third embodiments. Also, the corresponding point candidate Xb can be calculated more appropriately.

[0115] <Variation 5> In the above embodiment, an example has been described in which the object recognition method is realized by the object recognition device (server) 100. However, the present invention is not limited to this. A part of the processing executed by the object recognition device 100 described above may be realized by a manager who manages a road operating the input device 5 to input predetermined information to the object recognition device 100 while referring to information displayed on the display device 4. For example, the manager may operate the input device 5 to select the reference image A and the paired image B.

[0116] The object recognition method includes at least an image receiving step (S110 in FIG. 7) of receiving a plurality of images of an object (e.g., overhead wire 36) taken from different viewpoints, a reference image selecting step (S115 in FIG. 7) of selecting a reference image A from the plurality of images, a paired image selecting step (S120 in FIG. 7) of selecting a paired image B corresponding to the reference image A from the plurality of images, an element point arranging step (S125 in FIG. 7) of arranging an element point Xa on a linear portion SL of the object in the reference image A, a first gaze calculation step (S135 in FIG. 7) of calculating a first gaze Va corresponding to the element point Xa arranged by the element point arranging step, and an epipolar line calculation step (S140 in FIG. 7) of calculating an epipolar line EL on the paired image B corresponding to the first gaze Va calculated by the first gaze calculation step. The method may include a corresponding point candidate calculation step (S145 in FIG. 7) of calculating a plurality of corresponding point candidates Xb corresponding to one element point Xa on one epipolar line EL calculated by the epipolar line calculation step, a second line of sight calculation step (S150 in FIG. 7) of calculating a plurality of second lines of sight Vb corresponding to each of the plurality of corresponding point candidates Xb calculated by the corresponding point candidate calculation step, a three-dimensional intersection calculation step (S155 in FIG. 7) of calculating an intersection X between the first line of sight Va and each of the plurality of second lines of sight Vb, an object shape estimation step (S160 in FIG. 7) of estimating the shape of an object based on the three-dimensional point group of the intersection X calculated by the three-dimensional intersection calculation step, and an object shape output step (S165 in FIG. 7) of outputting the shape of the object estimated by the object shape estimation step. This allows the shape of a linear portion SL such as an overhead line 36 or a building frame line to be estimated with high accuracy.

[0117] Although the embodiments of the present invention have been described above, the above-mentioned embodiments merely show some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above-mentioned embodiments. [Explanation of symbols]

[0118] 1...vehicle, 2...camera (imaging device), 2a...first camera, 2b...second camera, 4...display device (output device), 5...input device, 6...communication device (output device), 10...object recognition system, 23...vehicle-mounted controller, 24...communication device, 31...vehicle, 32...vehicle-mounted camera device, 33...road, 36...overhead line (linear portion), 60a, 60b...image, 100...object recognition device (server), 101...processing device, 102...non-volatile memory (storage device), 103...volatile memory (storage device), 111...image receiving unit, 112...reference image selection unit, 113...pair image selection unit, 121...element point arrangement unit, 122...first view Line calculation unit, 123... epipolar line calculation unit, 124... corresponding point candidate calculation unit, 125... second gaze calculation unit, 131... object shape estimation unit, 132... object shape output unit, 133... object shape model (overhead line model), 191, 192, 193... display image, A... reference image, AL... auxiliary line, B... paired image, Ca... reference camera, Cb... paired camera, Ea, Eb... epipolar point, EL... epipolar line, ES... epipolar surface, L... distance between cameras, Oa... optical axis, Pa, Pb... viewpoint (optical center, lens focus), Va... first gaze, Vb... second gaze, X... gaze point, intersection of gazes, Xa... element point, Xb... corresponding point candidate

Claims

1. an image receiving unit that receives a plurality of images of an object taken from different viewpoints; a reference image selection unit for selecting a reference image from the plurality of images; a pair image selection unit that selects a pair image corresponding to the reference image from the plurality of images; an element point arrangement unit that arranges element points on a linear portion of the object in the reference image; a first line of sight calculation unit that calculates a first line of sight corresponding to the element points arranged by the element point arrangement unit; an epipolar line calculation unit that calculates an epipolar line on the pair of images corresponding to the first gaze calculated by the first gaze calculation unit; a corresponding point candidate calculation unit that calculates a plurality of corresponding point candidates corresponding to one of the element points on one of the epipolar lines calculated by the epipolar line calculation unit; a second gaze calculation unit that calculates a plurality of second gazes corresponding to the plurality of corresponding point candidates calculated by the corresponding point candidate calculation unit; a three-dimensional intersection calculation unit that calculates an intersection between the first line of sight and each of the plurality of second lines of sight; an object shape estimation unit that estimates a shape of the object based on the three-dimensional point cloud of intersections calculated by the three-dimensional intersection calculation unit; an object shape output unit that outputs the shape of the object estimated by the object shape estimation unit; An object recognition device comprising:

2. The object recognition device according to claim 1 , the corresponding point candidate calculation unit calculates an intersection between the line portion extracted in the pair of images and the epipolar line as the corresponding point candidate; Object recognition device.

3. The object recognition device according to claim 1 , The object shape estimation unit performing noise removal processing based on the density of the three-dimensional point cloud; Estimating a shape of the object based on the three-dimensional point cloud after the noise removal process. Object recognition device.

4. The object recognition device according to claim 2, the corresponding point candidate calculation unit calculates, as the corresponding point candidate, an intersection between the epipolar line and one of the linear portions extracted in the paired images, the linear portion having an inclination angle within a predetermined angle range with respect to the linear portion extracted in the reference image and on which the element point is arranged, Object recognition device.

5. The object recognition device according to claim 1 , the pair image selection unit selects an image having an overlap with the reference image in an angle of view range as the pair image; Object recognition device.

6. The object recognition device according to claim 1 , the pair image selection unit selects an image including at least a part of any line within an angle of view range of the reference image as the pair image; Object recognition device.

7. The object recognition device according to claim 1 , the pair image selection unit selects an image including at least a part of a line defining an angle of view range of the reference image as the pair image; Object recognition device.

8. The object recognition device according to claim 1 , the pair image selection unit selects, as the pair image, an image including at least a part of an optical axis of an imaging device that captured the reference image; Object recognition device.

9. The object recognition device according to claim 1 , The pair image selection unit selects an image including at least a part of the first line of sight as the pair image. Object recognition device.

10. The object recognition device according to claim 1 , the pair image selection unit selects, as the pair image, an image captured by a second imaging device whose distance from a first imaging device that captured the reference image is within a predetermined range; Object recognition device.

11. an image receiving step of receiving a plurality of images of an object taken from different viewpoints; a reference image selection step of selecting a reference image from the plurality of images; a pair image selection step of selecting a pair image corresponding to the reference image from the plurality of images; an element point arranging step of arranging element points on a linear portion of the object in the reference image; a first line of sight calculation step of calculating a first line of sight corresponding to the element points arranged by the element point arrangement step; an epipolar line calculation step of calculating an epipolar line on the pair of images corresponding to the first gaze calculated by the first gaze calculation step; a corresponding point candidate calculation step of calculating a plurality of corresponding point candidates corresponding to one of the element points on one of the epipolar lines calculated in the epipolar line calculation step; a second line of sight calculation step of calculating a plurality of second lines of sight corresponding to the plurality of corresponding point candidates calculated by the corresponding point candidate calculation step; a three-dimensional intersection calculation step of calculating an intersection between the first line of sight and each of the plurality of second lines of sight; an object shape estimating step of estimating a shape of the object based on the three-dimensional point group of intersections calculated in the three-dimensional intersection calculating step; an object shape output step of outputting the shape of the object estimated by the object shape estimation step, Object recognition method.

Images