Information processing device, information processing system, and information processing method

Abstract

To provide an information processing system and an information processing method that can estimate the position of a track regardless of the distance from a vehicle.SOLUTION: An information processing device according to an embodiment comprises: a first image acquisition unit that acquires a first image generated by an imaging device that captures a front environment of a vehicle; a track detection unit that detects a track on which the vehicle is running from the first image; a second image acquisition unit that acquires a second image including a three-dimensional position of the front environment on the basis of distance information acquired by a distance measuring device that measures a distance with respect to the front environment; and a track estimation unit that identifies a target range of the track included in the first image in which no part corresponding to the track in the second image is present, and estimates the three-dimensional position of the track part included in the target range from the track in the first image on the basis of information about the track on which the vehicle is running.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] 1. Field of the Invention The present invention relates to an information processing device, an information processing system, and an information processing method. [Background technology]

[0002] A detection technology is known that combines a camera mounted on a train with a distance measuring device so that a driver of a moving train or an automatic driving system can detect obstacles that may impede the train's running, such as a human or a fallen object within the construction limit of the track. The distance measuring device is, for example, a LiDAR (Light Detection and Ranging) device. In this technology, the distance measuring device is used to compensate for the fact that the positional relationship in the depth direction of objects shown in an image is unclear only from an image acquired by a camera. Such a technology can be particularly useful in cases where the driver does not visually determine the presence or absence of an obstacle, such as when an automatic driving system is adopted.

[0003] However, if the angle of incidence of the laser light emitted from the LiDAR device with respect to a given object becomes excessively large, most of the laser light reflected by the object does not return to the LiDAR device, making it impossible to measure the distance.

[0004] On the other hand, trains have a longer braking distance than automobiles, and in order to avoid collisions with obstacles, they need to detect obstacles several hundred meters ahead. However, the farther the track is from the train, the larger the angle of incidence of the laser light to the track, making it impossible for the distance measuring device to measure the distance from the train to the track. As a result, the position of tracks far from the train cannot be identified, and it becomes impossible to detect obstacles within the track's construction limit, resulting in a decrease in detection accuracy. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] A Camera and LiDAR Data Fusion Method for Railway Object Detection, IEEE SENSORS JOURNAL, VOL. 21, NO. 12, JUNE 15, 2021 [Non-Patent Document 2] Train Front Obstacle Detection Method with Camera-LiDAR Fusion, QR of RTRI, Vol. 63, No.3, Aug,2022. Summary of the Invention [Problem to be solved by the invention]

[0006] An embodiment of the present invention provides an information processing device, an information processing system, and an information processing method that are capable of estimating the position of a track regardless of the distance from a vehicle. [Means for solving the problem]

[0007] The information processing device of this embodiment includes a first image acquisition unit that acquires a first image generated by an imaging device that images the environment in front of the vehicle, a track detection unit that detects the track along which the vehicle runs from the first image, a second image acquisition unit that acquires a second image including a three-dimensional position of the environment in front based on distance information acquired by a ranging device that measures the distance to the environment in front, and a track estimation unit that identifies a target range of the track included in the first image where no portion corresponding to the track exists in the second image, and estimates the three-dimensional position of the track portion in the first image that is included in the target range based on information about the track along which the vehicle runs. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an obstacle detection system according to an embodiment. [Diagram 2] FIG. 1 shows an obstacle detection system installed on a train. [Diagram 3]FIG. 2 is a diagram showing an example of a front image of a train. [Figure 4] FIG. 4 is a diagram showing an example of a track image generated based on a forward image. [Diagram 5] 1A and 1B are diagrams for explaining the misalignment of the fields of view between a camera and a distance measuring device. [Figure 6] FIG. 13 is a diagram showing an example of a projected image expressed using a uv coordinate system. [Figure 7] FIG. 13 is a diagram showing an example of an extracted 3D line. [Figure 8] FIG. 1 shows the position of the track identified in the XYZ coordinate system. [Figure 9] FIG. 13 is a diagram showing points on a railway line in an area that was not observed. [Figure 10] A figure in which straight lines passing through the points shown in Figure 9 have been added to Figure 8. [Figure 11] A map showing the straight lines shown in Figure 10. [Figure 12] FIG. 10 shows the points shown in FIG. 9 estimated in the XYZ coordinate system. [Figure 13] A diagram explaining the estimated track and the calculated construction gauge. [Figure 14] 6 is a flowchart illustrating an example of a process performed by the obstacle detection system. [Figure 15] FIG. 2 is a block diagram showing an example of a hardware configuration according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] 1 is a block diagram showing an example of an obstacle detection system 1 according to an embodiment. The obstacle detection system 1 includes a camera 2, a distance measuring device 3, and an obstacle detection device 4.

[0011] FIG. 2 is a diagram showing an obstacle detection system 1 mounted on a vehicle, more specifically, on a leading vehicle of a train 10. The obstacle detection system 1 is a system for determining whether or not an object that may hinder the running of the train 10, i.e., an obstacle, exists within the construction limit of the track T on which the train 10 runs. An obstacle is a human being or a fallen object that exists within the construction limit of the track T, and does not include the track T itself or the ground on which the track T is placed. Note that the "leading vehicle" means the vehicle at the front in the direction of travel of the train 10. The train 10 runs on the track T. An object Obj1 is on the left side of the track T as seen from the train 10, and an object Obj2 is on the right side of the track T.

[0012] 2, the obstacle detection system 1 is disposed outside the train 10, but the obstacle detection system 1 may be disposed inside the train 10. In that case, the camera 2 and the distance measuring device 3 may capture images and measure distances through the windshield of the train 10. Also, only the camera 2 and the distance measuring device 3 may be mounted on the train 10, and the obstacle detection device 4 may not be mounted on the train 10.

[0013] The camera 2 is an imaging device that captures an image of the environment ahead of the train 10 to generate a forward image, which is a first image. The camera 2 is provided in a three-dimensional first coordinate system (first world coordinate system). The camera 2 is, for example, an optical camera such as a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera, an infrared camera (thermo camera), or a combination thereof. When the camera 2 includes an infrared camera, it becomes easier to detect obstacles even at night, etc. The forward image is, for example, a luminance image, and each pixel value of the luminance image is a luminance value.

[0014] 3 is a diagram showing an example of a forward image captured by the camera 2. In the example of FIG. 3, the track T curves to the right as seen from the train 10. The symbol H shown in FIG. 3 indicates the horizon. The forward image is assumed to be represented in the uv coordinate system as shown in FIG. 3. The uv coordinate system is the image coordinate system of the camera 2.

[0015] The distance measuring device 3 is a device or sensor that measures the distance to the forward environment including objects Obj1, Obj2, etc. The distance measuring device 3 generates a 3D image, which is a second image including a three-dimensional position of the forward environment based on the measured distance. The distance measuring device 3 is provided in a three-dimensional second coordinate system (second world coordinate system). Each pixel of the 3D image includes a three-dimensional position. Each 3D pixel may include a distance value. The object also includes the railroad T and the ground on which the railroad T is located. The distance between the object and the train 10 is, strictly speaking, the distance between the object and the distance measuring device 3. The distance measuring device 3 includes, for example, a LiDAR. When the distance measuring device 3 includes a LiDAR, the wavelength of the laser used may be, for example, an eye-safe band (for example, 1.4 μm to 2.6 μm). In the following description, the distance measuring device 3 is assumed to be a LiDAR. The distance measuring device 3 may generate a distance image including distance values ​​for the forward environment, and a 3D data acquisition unit 43 (described later) may generate a 3D image from the distance image and position information of the distance measuring device 3, etc.

[0016] The distance measuring device 3 scans a predetermined range of the forward environment by irradiating a laser pulse, for example. The distance measuring device 3 receives the pulse reflected and returned from the forward environment to obtain distance information for the forward environment (distance information to various objects included in the forward environment) and generates a 3D image.

[0017] The obstacle detection device 4 is an information processing device for determining whether or not there is an object that may be an obstacle on the track T ahead of the train 10, based on the forward image captured by the camera 2 and the 3D image acquired by the distance measuring device 3. In particular, the obstacle detection device 4 makes it possible to determine whether or not there is an obstacle within the construction gauge of a track location that is captured in the forward image but could not be measured by the distance measuring device 3. As shown in FIG. 1, the obstacle detection device 4 includes an image acquisition unit 41, a track detection unit 42, a 3D data acquisition unit 43, a projection image generation unit 44, a track estimation unit 46, and an obstacle detection unit 47.

[0018] The image acquisition unit 41 is a first image acquisition unit that acquires a forward image of the train 10 captured by the camera 2 (see FIG. 3).

[0019] The track detection unit 42 identifies the track T from the forward image acquired by the image acquisition unit 41, and acquires (generates) a track image showing the track T in the forward image. The track detection unit 42 generates the track image by extracting the track T by, for example, semantic segmentation or edge detection. FIG. 4 is an example of a track image detected by the track detection unit 42. The track image is also expressed in the uv coordinate system (the image coordinate system of the camera 2) like the forward image.

[0020] The 3D data acquisition unit 43 is a second image acquisition unit that acquires a 3D image from the distance measuring device 3. The 3D image is expressed in the image coordinate system of the distance measuring device 3. The 3D image acquired by the 3D data acquisition unit 43 is a 3D image acquired approximately simultaneously (approximately the same frame) as the forward image acquired by the image acquisition unit 41.

[0021] The projection image generating unit 44 projects the 3D image onto the track image. Projecting the 3D image onto the track image corresponds to converting the 3D image expressed in the image coordinate system of the distance measuring device 3 into the uv coordinate system, which is the image coordinate system of the camera 2 (expressing it in the uv coordinate system).

[0022] 5 is a diagram for explaining the misalignment between the field of view of camera 2 and the field of view of distance measuring device 3. As shown in Fig. 5, camera 2 and distance measuring device 3 have different physical arrangements, i.e., different viewpoints and orientations. Therefore, origin O2 and line of sight LOS2, which are the viewpoint of field of view FOV2 of camera 2, are different from origin O3 and line of sight LOS3, which are the viewpoint of field of view FOV3 of distance measuring device 3.

[0023] 5, the three-dimensional coordinates in the field of view FOV2 are referred to as the xyz coordinate system (the world coordinate system of camera 2 or the three-dimensional coordinate system of camera 2), and the three-dimensional coordinates in the field of view FOV3 are referred to as the XYZ coordinate system (the world coordinate system of distance measuring device 3 or the three-dimensional coordinate system of distance measuring device 3). Note that a 3D image expressed in the XYZ coordinate system only includes values ​​of the range (location) observed by distance measuring device 3, and does not include values ​​of the range (location) that is included in the field of view FOV3 but not observed by distance measuring device 3.

[0024] Consider an arbitrary point P (X, Y, Z) in the XYZ coordinate system shown in FIG. 5. The distance value of point P is measured, and the three-dimensional position of point P is included in the 3D image. Equation (1) is an equation for matching the origin O3 and line of sight LOS3 of the field of view FOV3 of the distance measuring device 3 with the origin O2 and line of sight LOS2 of the field of view FOV2 of the camera 2. In other words, it is an equation for converting point P (X, Y, Z) in the XYZ coordinate system to point p (x, y, z) in the xyz coordinate system. Matrix R is a rotation matrix that rotates the field of view FOV3 by an angle θ. Matrix tvec is a translation matrix that translates the field of view FOV3.

[0025] By performing the conversion of formula (1), the origin O2 and line of sight LOS2 of the field of view FOV2 coincide with the origin O3 and line of sight LOS3 of the field of view FOV3. In other words, point P included in the 3D image expressed in the XYZ coordinate system can be expressed as point p in the xyz coordinate system.

[0026]

number

[0027] Next, the projection image generating unit 44 converts a point p(x, y, z) in the 3D image expressed in the xyz coordinate system into a point p'(u, v) in the uv coordinate system using equations (2) to (4). The projection image generating unit 44 first normalizes the point p by the distance z in the depth direction using equation (2), and converts the point p included in the 3D image from three-dimensional coordinates (xyz coordinate system) to two-dimensional coordinates (x'y' coordinate system).

[0028]

number

[0029] Next, a distortion function f(x, y) specific to camera 2 is applied, as shown in equation (3).

[0030]

number

[0031] The coordinate system after application of the distortion function of the following formula (3) or the above-mentioned x'y' coordinate system corresponds to the two-dimensional coordinate system of the camera 2 or the camera coordinate system.

[0032] Finally, the coordinate system of (3) is transformed into the uv coordinate system (image coordinate system of camera 2) using the camera matrix K specific to camera 2. The camera matrix K is a 2 × 3 matrix.

[0033]

number

[0034] This results in a 3D image projected onto the track image (hereinafter referred to as a projected image), i.e., an image in which the 3D image is transformed into the uv coordinate system of the camera 2.

[0035] FIG. 6 is a diagram showing an example of a 3D image projected onto a uv coordinate system. A pixel (dot) DT corresponds to a position (observation point) where one (one) reflected pulse is received. A 3D image is expressed as a collection (point cloud) of multiple pixels DT. Each pixel of the projected image may correspond to the three-dimensional position and distance information of the corresponding pixel in the 3D image before conversion. In the example of FIG. 6, the distance from the train 10 is represented by the shade of the pixel DT, and the farther the position, the lighter the color of the pixel DT.

[0036] Here, a pixel without color means that the reflected pulse is not received, i.e., the pixel is not observed. As the train moves away from the train 10 to a certain distance, the incidence angle of the pulsed light emitted from the distance measuring device 3 with respect to the ground or the track T increases, so the number of cases where the object is not observed increases. Note that, from Figures 3 and 6, it can be seen that the object Obj3 is observed even though it is relatively far from the train 10, because the incidence angle of the pulsed light emitted from the distance measuring device 3 is small.

[0037] Based on the track image (see FIG. 4) and the projection image (see FIG. 6), the track extraction unit 45 extracts an image of a track portion corresponding to the track T in the track image in the projection image. The extracted image of the track portion is called a 3D track. The track extraction unit 45 superimposes the track image and the projection image, and extracts the portion of the projection image that overlaps the position of the track T as a 3D track. The track portion in the track image that overlaps with the 3D track corresponds to the track portion of the track T in the track image that has a corresponding portion in the projection image. The pixel DT group in FIG. 7 shows the extracted 3D track. Here, the area A shown in the figure is an area where a track portion exists that exists in the track image but does not exist in the projection image. In other words, the area A is an area where the track T is expected to exist from the track image, but the track portion is not observed by the distance measuring device 3 (unobserved area). Note that the area A does not mean that all of the area A is not observed, but is an area where at least the track portion is not observed, and is shown for reference purposes for ease of understanding.

[0038] For the range measured by the distance measuring device 3, the position of the track T in the XYZ coordinate system can be identified by tracing the above formulas in reverse, as shown in FIG. 8. FIG. 8 shows the portion in the XYZ coordinate system where the position of the track T is identified. As a result, for the portion where the position of the track T is identified, the construction gauge of the track T can be calculated, and therefore, it can be determined whether or not an obstacle (object) in the XYZ coordinate system interferes with the construction gauge. On the other hand, in the area A shown in FIG. 8, the existing track portion cannot be measured by the distance measuring device 3, so no track portion is displayed in the XYZ coordinate system. Although the existence of the track T is predicted from the track image in the area A, it is not observed by the distance measuring device 3, so the position of the track in the XYZ coordinate system (or the position in the depth direction in the traveling direction of the train 10) cannot be identified as it is. Therefore, it is necessary to estimate the position of the track T in the XYZ coordinate system in the area A.

[0039] The track estimation unit 46 estimates a 3D track in the area A. The track estimation unit 46 holds parameters related to the camera 2 and parameters related to the distance measuring device 3, and may use these parameter information for the estimation. The process performed by the track estimation unit 46 will be described in detail below.

[0040] A case where a 3D track is estimated in the XYZ coordinate system will be shown. As described above, the 3D image acquired by the distance measuring device 3 is converted into the uv coordinate system by the above-mentioned formulas (1) to (4). First, in order to explain a method for estimating a 3D track, a method for converting a point p'(u, v) expressed in the uv coordinate system into the XYZ coordinate system will be shown.

[0041] First, the inverse transformation of equation (4) is performed using the following equation (5), where K[1:2,1:2] means the 2×2 matrix on the left side of K, and K[1:2,3] means the rightmost column vector of K.

[0042]

number

[0043] Next, the inverse function f of the distortion function f shown in the following equation (6) is -1 Then, the inverse transformation of equation (3) is performed using

[0044]

number

[0045] Next, the inverse transformation of equation (2) is performed using equation (7). This makes it possible to transform point p' in the uv coordinate system to point p in the xyz coordinate system, which is the three-dimensional coordinate system of camera 2.

[0046]

number

[0047] Here, z is expressed as in the following equation (8). If the values ​​of u and v of point p' are known, the values ​​of x' and y' can be specified by equations (5) and (6), but the value of z of point p is unknown. Z0 shown in equation (8) represents the known Z-axis coordinate value of point P in the XYZ coordinate system. Here, instead of the Z-axis coordinate value, the X-axis coordinate value or the Y-axis coordinate value of point P may be known. In any case, in order to know the z value of point p, additional information is required in addition to the u and v values ​​of point p'.

[0048]

number

[0049] Vector S shown in equation (8) -1 is expressed as the following equation (9). R[3,1:3] is the third row vector from the top of the rotation matrix R.

[0050]

number

[0051] When the value of the X-axis coordinate is a known value, X0, Z0 in equation (8) is replaced with X0, and equation (9) becomes S -1 =R[1,1:3] -1 R[1,1:3] is the first row vector from the top of the rotation matrix R.

[0052] When the value of the Y-axis coordinate is a known value Y0, Z0 in equation (8) is replaced with Y0, and equation (9) becomes S -1 =R[2,1:3] -1 R[2,1:3] is the second row vector from the top of the rotation matrix R.

[0053] Finally, the inverse transformation of equation (1) is performed using the following equation (10). This makes it possible to transform point p in the xyz coordinate system of camera 2 into point P in the XYZ coordinate system of distance measuring device 3.

[0054]

number

[0055] Alternatively, the distance from the distance measuring device 3 to the point P may be known. In the XYZ coordinate system, if the distance from the coordinate O3 (0,0,0) of the distance measuring device 3 to the point P (X,Y,Z) is D, then X 2 +Y 2 +Z 2 =D 2 By substituting this into equation (10), we obtain the following equation (11). By solving equation (11), we can obtain the value of z at point p.

[0056]

number

[0057] Alternatively, the distance from the camera 2 to the point p may be known. In the xyz coordinate system, if the distance from the coordinate O2 (0,0,0) of the camera 2 to the point p (x,y,z) is d, then x 2 +y 2 +z 2 =d 2By substituting this into equation (7), we obtain the following equation (12). By solving equation (12), we can obtain the value of z at point p.

[0058]

number

[0059] In either case, the z value of point p must be known in order to convert from the uv coordinate system (the image coordinate system of the camera 2) to the XYZ coordinate system (the three-dimensional coordinate system of the distance measuring device 3). To find the z value of point p, additional information is required, such as the coordinate of one of the axes in the XYZ coordinate system of point P.

[0060] FIG. 9 is a diagram showing a track image for explaining a method for estimating the position of the track in area A. Point q' corresponds to a point included in a range of the track in the track image where no part corresponding to the projected image exists, i.e., a point on the track T in area A on the track image. This range (the range of the track portion included in area A) is set as the target range on which estimation is based. Point q is the point in the three-dimensional coordinate system of camera 2 that corresponds to point q'. Point q' is information that exists on the track image but does not exist on the 3D image. Consider estimating a point in the XYZ coordinate system (point Q (X,Y,Z)) that corresponds to point q'.

[0061] Although the position of point Q in the XYZ coordinate system is unknown, its coordinates in the uv coordinate system are known. Therefore, by solving equations (5) to (10), it is found that point Q is on a straight line (epipolar line L) passing through origin O3 in the XYZ coordinate system, which corresponds to a straight line passing through origin O2 and point q' in the xyz coordinate system (epipolar constraint). The epipolar line L can be calculated based on the parameters of the camera 2 and the parameters of the distance measuring device 3. It is found that point Q is located somewhere on the epipolar line L shown in FIG. 10. FIG. 10 is a diagram in which the epipolar line L passing through point Q has been added to FIG. 8.

[0062] Since it is known that point Q is on the epipolar line L, if any of the XYZ coordinate values ​​of point Q or the distance D from the origin O3 is known, the XYZ coordinate values ​​of point Q can be estimated using equation (8) or equation (11). Alternatively, if the distance d from point q to the origin O2 in the xyz coordinate system is known, the XYZ coordinate values ​​of point Q can be estimated using equation (12).

[0063] (First estimation method) In the first estimation method, the track estimation unit 46 has, for example, a train position estimation function and a map database. The train position estimation function acquires the current position of the train 10 from a positioning system such as a global positioning system (GPS). The track estimation unit 46 calculates, for example, the altitude of the train 10 from the current position of the train 10 and the map database. Then, the track estimation unit 46 acquires the position and altitude of the track T from the map database. The altitude is, for example, altitude above sea level or the like.

[0064] Now, consider a track T on a map database. FIG. 11 shows a map acquired from a map database, on which the current position of the train 10, that is, the origin O3 in the XYZ coordinate system, is displayed. The map shown in FIG. 11 shows contour lines indicating altitudes of 100 m, 105 m, and 110 m. The train 10 is illustrated for ease of understanding. When the epipolar line L is displayed on the map shown in FIG. 11, it can be seen that there is a point where the epipolar line L intersects with the track T. This intersection point is estimated to be point Q. Note that in FIG. 11, it appears that the inclination of the epipolar line L in the height direction is not taken into consideration, but in reality, the inclination of the epipolar line L in the height direction is also taken into consideration.

[0065] Furthermore, from the map shown in Fig. 11, it can be seen that the current altitude of train 10 is 100 m, and the altitude of point Q is 110 m. Therefore, as shown in Fig. 12, it can be seen that the coordinates of point Q in the XYZ coordinate system are (X1, 10 m, Z1). X1 and Z1 are the X and Z coordinate values ​​of point Q that become clearer as a result of the Y coordinate value of point Q being revealed.

[0066] In this way, in the first estimation method, the Y coordinate value of point Q is estimated from the map database, thereby estimating the XYZ coordinate values ​​of point Q. By repeating the above process, the position of track T in area A can be estimated in the XYZ coordinate system.

[0067] (Second estimation method) In the second estimation method, the track estimation unit 46 estimates any value of XYZ coordinates or distance from the pixel DT surrounding the pixel DT of the track T. For example, the point Q itself is not measured by the distance measuring device 3, but as shown in FIG. 6, when an object Obj3 near the point Q is observed, any value of XYZ coordinates or distance of the point Q can be estimated from the observed part. Specifically, a track part (a track part corresponding to the 3D track) that is not included in the 3D image among the tracks included in the track image is identified, and an object that exists close to the identified track part and includes a part corresponding to the 3D image is detected from the track image. Based on the three-dimensional position of the object (the three-dimensional position in the three-dimensional coordinate system of the camera 2 or the three-dimensional position in the three-dimensional coordinate system of the distance measuring device), the three-dimensional position in the XYZ coordinate system of the track part corresponding to the 3D track is estimated. Alternatively, the three-dimensional position in the XYZ coordinate system of the track portion corresponding to the 3D track is estimated based on the distance from the viewpoint of the camera 2 or the viewpoint of the distance measuring device 3 to the object. As a specific example of estimation, the same Z coordinate as that of the object may be used as the estimated value. In this way, in the second estimation method, the XYZ coordinate values ​​of point Q are estimated by estimating the distance from the origin O3 to point Q from observed points near point Q, etc.

[0068] (Third estimation method) In the third estimation method, the track estimation unit 46 estimates the distance from the camera 2 to the point q using the number of pixels corresponding to the width of the track T in the forward image (track image). The width of the track T is assumed to be known and constant. Under this precondition, the distance from the camera 2 to the point q can be calculated based on the number of pixels of the width of the track T in the forward image, the angle of view of the camera 2, and the number of pixels of the forward image. If the distance from the camera 2 to the point q is known, the position of the point q can be determined using equation (12), and the coordinates of the point q can be converted to the coordinates of the point Q in the XYZ coordinate system. This makes it possible to estimate the position of the track T in the area A. In this way, in the third estimation method, the distance d from the point q to the origin O2 in the xyz coordinate system is estimated using the number of pixels of the width of the track T in the forward image, and the value of the XYZ coordinates of the point Q is estimated.

[0069] (Fourth estimation method) In the fourth estimation method, the camera 2 is a stereo camera, and the track estimation unit 46 estimates the distance from the camera 2 to the point q. In the fourth estimation method, there are a first forward image and a second forward image as two first images simultaneously taken from two different places as forward images, and the distance from the camera 2 to the point q is estimated from the parallax between the first forward image and the second forward image taken of the point q. If the distance from the camera 2 to the point q is known, the coordinates of the point Q in the XYZ coordinate system are specified using equation (12), and the track T in the area A can be estimated in the XYZ coordinate system. In this way, in the fourth estimation method, the distance d from the point q to the origin O2 in the xyz coordinate system is estimated using a stereo camera, thereby estimating the value of the XYZ coordinates of the point Q.

[0070] By performing any one or a combination of the above processes, the three-dimensional position of the track portion not measured by the distance measuring device 3 can be estimated.

[0071] In the above process, the position of the track T is identified and estimated in the XYZ coordinate system, but the position of the track T may be estimated in the xyz coordinate system.

[0072] The obstacle detection unit 47 calculates the construction gauge from the track estimated by the track estimation unit 46 and the track extracted by the track extraction unit 45, and determines whether or not there is an object (obstacle) within the construction gauge that may hinder the running of the train 10. Fig. 13 is a diagram showing the construction gauge calculated based on the estimated and extracted tracks. The construction gauge specifies the range within which buildings must not be placed relative to the track T.

[0073] Then, the obstacle detection unit 47 outputs the result of determining whether or not there is an object within the building limit that may hinder travel. For example, if a large animal object is detected within the building limit, it may be determined that there is an object that may hinder travel. The determination result may be output to a display visible to the driver of the train 10, or may be output to the automatic driving system of the train 10. The obstacle detection unit 47 may output the determination result only when it is determined that there is an object that may hinder travel within the building limit, or may output the determination result regardless of the determination result.

[0074] The obstacle detection unit 47 may set a range wider than the construction limit and determine whether or not there is an obstacle within that range. This allows the driver to recognize a person who is not within the construction limit but is near the construction limit, and to take measures such as sounding the horn depending on the situation.

[0075] <Flowchart of processing performed by obstacle detection system 1> Fig. 14 is a flowchart of an example of processing performed by the obstacle detection system 1. Hereinafter, the processing performed by the obstacle detection system 1 will be described with reference to Fig. 14 and each figure.

[0076] First, the camera 2 captures an image of the environment ahead of the train 10 (forward image: see FIG. 3) (step S1). The captured forward image is captured by the image acquisition unit 41 into the obstacle detection device 4.

[0077] Next, the track detection unit 42 detects the position of the track T from the forward image, and obtains a track image (see FIG. 4) that extracts and shows the track T in the forward image (step S2).

[0078] At substantially the same time as step S1, the distance measuring device 3 acquires a 3D image of the environment ahead of the train 10 (step S3). The acquired 3D image is input to the obstacle detection device 4 by the 3D data acquisition unit 43.

[0079] Next, the projection image generating unit 44 projects the 3D image onto the track image to obtain a projection image (see FIG. 6) (step S4).

[0080] Next, based on the track image (see FIG. 4) and the projection image (see FIG. 6), the track extraction unit 45 extracts from the projection image a portion corresponding to the track T included in the track image (3D track: see FIG. 7) (step S5).

[0081] Next, the track estimation unit 46 estimates, based on the extracted 3D track and the track image, track portions that are included in the track image but not measured by the distance measuring device 3 in the XYZ coordinate system (step S6).

[0082] Next, the obstacle detection unit 47 calculates the construction gauge from the track portion estimated in the XYZ coordinate system and the 3D track (step S7).

[0083] Next, the obstacle detection unit 47 determines whether an obstacle is present within the calculated construction gauge (step S8). For example, if an object of a predetermined size or larger is present, it may be determined that an obstacle is present. Alternatively, the type of an object detected within the construction gauge may be determined using a track image and semantic segmentation, and if the type of object corresponds to a predetermined type of obstacle, it may be determined that an obstacle is present.

[0084] Next, the obstacle detection unit 47 outputs the determination result (step S9). The obstacle detection unit 47 outputs the determination result to, for example, an automatic driving system of the train 10.

[0085] When step S8 is completed, the process returns to steps S1 and S3, and each step is repeated in sequence. Each step may be performed in parallel with other steps. For example, even while the image acquisition unit 41 is acquiring the forward image in step S2, the camera 2 may continue to capture the forward image.

[0086] As described above, according to this embodiment, the distance measuring device 3 and the camera 2 are combined to estimate the position of the track in a range where distance measurement cannot be performed by the distance measuring device 3. This makes it possible to detect an obstacle even at a long distance where measurement cannot be performed by the distance measuring device 3. This makes it possible to improve the distance at which an obstacle can be detected and the detection accuracy of the obstacle.

[0087] (Another example 1) In this embodiment, the camera 2 and the distance measuring device 3 are separate devices, but they may be integrated into one device. In this case, if the viewpoint and field of view of both are the same and they can be considered to exist in the same coordinate system, the projection image generating unit 44 may be omitted and the 3D image of the distance measuring device 3 may be used as it is instead of the projection image.

[0088] (Another example 2) The obstacle detection device 4 may be linked to a brake control system of the train. The brake control system may control the brakes of the vehicle according to the determination result output by the obstacle detection device 4. For example, when an obstacle object is detected, the brake control system may control the train to stop at a position ahead of the position of the detected object.

[0089] (Hardware configuration) 16 shows the hardware configuration of an information processing device according to each embodiment. The information processing device is configured by a computer device 600. The computer device 600 includes a CPU 601, an input interface 602, a display device 603, a communication device 604, a main storage device 605, and an external storage device 606, which are interconnected by a bus 607.

[0090] The CPU (Central Processing Unit) 601 executes an information processing program, which is a computer program, on the main memory device 605. The information processing program is a program that realizes each of the above-mentioned functional configurations of the information processing device. The information processing program may be realized not as a single program, but as a combination of multiple programs and scripts. Each functional configuration is realized by the CPU 601 executing the information processing program.

[0091] The input interface 602 is a circuit for inputting operation signals from input devices such as a keyboard, a mouse, a touch panel, etc., to the information processing device. The input interface 602 corresponds to the input unit of the information processing device according to each embodiment.

[0092] The display device 603 displays data output from the information processing device. The display device 603 is, for example, but not limited to, an LCD (liquid crystal display), an organic electroluminescence display, a CRT (cathode ray tube), or a PDP (plasma display). Data output from the computer device 600 can be displayed on the display device 603. The display device 603 corresponds to the output unit of the information processing device according to each embodiment.

[0093] The communication device 604 is a circuit for the information processing device to communicate with an external device wirelessly or via a wire. Data can be input from the external device via the communication device 604. The data input from the external device can be stored in the main memory device 605 or the external memory device 606.

[0094] The main memory device 605 stores an information processing program, data required for executing the information processing program, data generated by executing the information processing program, and the like. The information processing program is deployed and executed on the main memory device 605. The main memory device 605 is, for example, a RAM, a DRAM, or an SRAM, but is not limited to these. Each storage unit or database of the information processing device according to each embodiment may be constructed on the main memory device 605.

[0095] The external storage device 606 stores an information processing program, data required for executing the information processing program, data generated by executing the information processing program, etc. These information processing programs and data are read into the main storage device 605 when the information processing program is executed. The external storage device 606 is, for example, but is not limited to, a hard disk, an optical disk, a flash memory, and a magnetic tape. Each storage unit or database of the information processing device may be constructed on the external storage device 606.

[0096] The information processing program may be pre-installed in the computer device 600, or may be stored in a storage medium such as a CD-ROM. The information processing program may also be uploaded onto the Internet.

[0097] Furthermore, the information processing device may be configured as a single computer device 600, or may be configured as a system made up of a plurality of computer devices 600 connected to each other.

[0098] The present invention is not limited to the above-described embodiments as they are, and the components can be modified and embodied in the implementation stage without departing from the gist of the invention. Various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, a configuration in which some components are deleted from all the components shown in each embodiment can also be considered. Furthermore, the components described in different embodiments can be appropriately combined.

[0099] This embodiment can also be configured as follows. [Item 1] a first image acquisition unit that acquires a first image generated by an imaging device that captures an image of an environment ahead of the vehicle; a track detection unit that detects a track on which the vehicle runs from the first image; a second image acquisition unit that acquires a second image including a three-dimensional position of the forward environment based on distance information acquired by a distance measuring device that measures a distance to the forward environment; a track estimation unit that identifies a target range of the track included in the first image where no portion corresponding to the track exists in the second image, and estimates a three-dimensional position of a track portion of the track in the first image that is included in the target range based on information about the track on which the vehicle runs; An information processing device comprising: [Item 2] The track estimation unit estimates a three-dimensional position of the track portion included in the target range based on map data including a three-dimensional position of the track on which the vehicle runs. Item 1. An information processing device according to item 1. [Item 3] the imaging device is provided in a three-dimensional first coordinate system, and the distance measuring device is provided in a three-dimensional second coordinate system; a projection image generating unit that generates a projection image by projecting the second image onto the first image, the target range is a range of the railroad tracks included in the first image in which a portion corresponding to the railroad tracks is not present in the projection image, The track estimation unit estimates a three-dimensional position in the second coordinate system of a track portion included in the target range. 3. The information processing device according to item 1 or 2. [Item 4] the track estimation unit calculates an epipolar line from a viewpoint of the distance measuring device, the epipolar line corresponding to a straight line passing through the track portion included in the target range from a viewpoint of the imaging device; Based on map data including three-dimensional position information of the track on which the vehicle runs, a position where the epipolar line intersects with the track in the map data is calculated, and the calculated position is regarded as an estimate of the three-dimensional position in the second coordinate system of the track portion included in the target range. Item 3. An information processing device according to item 3. [Item 5] the track estimation unit estimates a three-dimensional position in the first coordinate system of the track portion included in the target range under the assumption that the width of the track is constant, and converts the three-dimensional position of the track portion in the first coordinate system into a three-dimensional position in the second coordinate system; 5. The information processing device according to item 3 or 4. [Item 6] The track estimation unit detects an object that is close to a track portion included in the target range in the first image and includes a portion corresponding to the second image, and estimates a three-dimensional position of the track portion included in the target range in the second coordinate system based on the three-dimensional position of the object in the first coordinate system or the second coordinate system, or the distance from a viewpoint of the imaging device or a viewpoint of the ranging device to the object. 6. The information processing device according to any one of items 3 to 5. [Item 7] the imaging device is a stereo camera, the stereo camera generates the first images from a plurality of viewpoints, the track estimation unit estimates a three-dimensional position in the first coordinate system of the track portion included in the target range based on the first images from the multiple viewpoints, and converts the three-dimensional position of the track portion in the first coordinate system into a three-dimensional position in the second coordinate system. 7. The information processing device according to any one of items 3 to 6. [Item 8] an obstacle detection unit that calculates a construction gauge of the track portion estimated by the track estimation unit based on a three-dimensional position of the track portion estimated by the track estimation unit, and detects an object that is an obstacle to the travel of the vehicle within the construction gauge by using the first image and the second image. 8. The information processing device according to any one of items 1 to 7, comprising: [Item 9] The vehicle is a train Item 1. An information processing device according to item 1. [Item 10] acquiring a first image generated by an imaging device imaging an environment in front of the vehicle; Detecting a track on which the vehicle runs from the first image; acquiring a second image including a three-dimensional position of the forward environment based on distance information acquired by a distance measuring device that measures a distance to the forward environment; A target range of the track included in the first image where no portion corresponding to the track exists in the second image is identified, and a three-dimensional position of a track portion of the track included in the target range in the first image is estimated based on information about the track on which the vehicle runs. Information processing methods. [Item 11] an imaging device that captures an image of an environment ahead of the vehicle to generate a first image; A distance measuring device for measuring a distance to the forward environment; a track detection unit that detects a track on which the vehicle runs from the first image; a track estimation unit that acquires a second image including a three-dimensional position of the environment ahead based on distance information acquired by the distance measuring device, identifies a target range of the track included in the first image in which no portion corresponding to the track exists in the second image, and estimates a three-dimensional position of a track portion of the track included in the target range in the first image based on information about the track on which the vehicle runs; An information processing system comprising: [Explanation of symbols]

[0100] 1 Obstacle Detection System 2 Camera 3 Ranging device 4 Obstacle detection device 41 Image acquisition unit 42 Railroad track detector 43 3D Data Acquisition Section 44 Projection image generation unit 45 Line extraction part 46 Track Estimation Unit 47 Obstacle detection unit 10. Train 600 Computer equipment 601 CPU 602 Input Interface 603 Display device 604 Communication equipment 605 Main storage 606 External storage device 607 Bus Area A DT Pixels H horizon FOV2 field of view FOV3 field of view LOS2 Line of Sight LOS3 line of sight O2 Origin O3 Origin Obj1 and Obj2 objects T track L epipolar line

Claims

1. a first image acquisition unit that acquires a first image generated by an imaging device that captures an image of an environment ahead of the vehicle; a track detection unit that detects a track on which the vehicle runs from the first image; a second image acquisition unit that acquires a second image including a three-dimensional position of the forward environment based on distance information acquired by a distance measuring device that measures a distance to the forward environment; a track estimation unit that identifies a target range of the track included in the first image where no portion corresponding to the track exists in the second image, and estimates a three-dimensional position of a track portion of the track in the first image that is included in the target range based on information about the track on which the vehicle runs; An information processing device comprising:

2. The track estimation unit estimates a three-dimensional position of the track portion included in the target range based on map data including a three-dimensional position of the track on which the vehicle runs. The information processing device according to claim 1 .

3. the imaging device is provided in a three-dimensional first coordinate system, and the distance measuring device is provided in a three-dimensional second coordinate system; a projection image generating unit that generates a projection image by projecting the second image onto the first image, the target range is a range in which, among the tracks included in the first image, a portion corresponding to the tracks is not present in the projection image, The track estimation unit estimates a three-dimensional position in the second coordinate system of a track portion included in the target range. The information processing device according to claim 1 .

4. the track estimation unit calculates an epipolar line from a viewpoint of the distance measuring device, the epipolar line corresponding to a straight line passing through the track portion included in the target range from a viewpoint of the imaging device; Based on map data including three-dimensional position information of the track on which the vehicle runs, a position where the epipolar line intersects with the track in the map data is calculated, and the calculated position is regarded as an estimate of the three-dimensional position in the second coordinate system of the track portion included in the target range. The information processing device according to claim 3 .

5. the track estimation unit estimates a three-dimensional position in the first coordinate system of the track portion included in the target range under the assumption that the width of the track is constant, and converts the three-dimensional position of the track portion in the first coordinate system into a three-dimensional position in the second coordinate system; The information processing device according to claim 3 .

6. The track estimation unit detects an object that is close to a track portion included in the target range in the first image and includes a portion corresponding to the second image, and estimates a three-dimensional position of the track portion included in the target range in the second coordinate system based on the three-dimensional position of the object in the first coordinate system or the second coordinate system, or the distance from a viewpoint of the imaging device or a viewpoint of the ranging device to the object. The information processing device according to claim 3 .

7. the imaging device is a stereo camera, the stereo camera generates the first images from a plurality of viewpoints, the track estimation unit estimates a three-dimensional position in the first coordinate system of the track portion included in the target range based on the first images from the multiple viewpoints, and converts the three-dimensional position of the track portion in the first coordinate system into a three-dimensional position in the second coordinate system. The information processing device according to claim 3 .

8. an obstacle detection unit that calculates a construction gauge of the track portion estimated by the track estimation unit based on a three-dimensional position of the track portion estimated by the track estimation unit, and detects an object that is an obstacle to the travel of the vehicle within the construction gauge by using the first image and the second image. The information processing device according to claim 1 .

9. The vehicle is a train The information processing device according to claim 1 .

10. acquiring a first image generated by an imaging device imaging an environment in front of the vehicle; Detecting a track on which the vehicle runs from the first image; acquiring a second image including a three-dimensional position of the forward environment based on distance information acquired by a distance measuring device that measures a distance to the forward environment; A target range of the track included in the first image where no portion corresponding to the track exists in the second image is identified, and a three-dimensional position of a track portion of the track in the first image that is included in the target range is estimated based on information about the track on which the vehicle runs. Information processing methods.

11. an imaging device that captures an environment in front of the vehicle to generate a first image; a distance measuring device for measuring a distance to the forward environment; a track detection unit that detects a track on which the vehicle runs from the first image; a track estimation unit that acquires a second image including a three-dimensional position of the environment ahead based on distance information acquired by the distance measuring device, identifies a target range of the track included in the first image in which a portion corresponding to the track does not exist in the second image, and estimates a three-dimensional position of a track portion of the track included in the target range in the first image based on information about the track on which the vehicle travels; An information processing system comprising:

Images