Information processing device, information processing system, and information processing method

Abstract

To provide an information processing system and an information processing method that can estimate the position of a track regardless of the distance from a vehicle.SOLUTION: An information processing device according to an embodiment comprises: a first image acquisition unit that acquires a first image generated by an imaging device that captures a front environment of a vehicle; a track detection unit that detects a track on which the vehicle is running from the first image; a second image acquisition unit that acquires a second image including a three-dimensional position of the front environment on the basis of distance information acquired by a distance measuring device that measures a distance with respect to the front environment; and a track estimation unit that identifies a target range of the track included in the first image in which no part corresponding to the track in the second image is present, and estimates the three-dimensional position of the track part included in the target range from the track in the first image on the basis of information about the track on which the vehicle is running.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Embodiments of the present invention relate to an information processing device, an information processing system, and an information processing method. [Background technology]

[0002] A detection technology is known that combines a camera mounted on a train with a rangefinder to detect obstacles that could hinder train operation, such as people or fallen objects, that are within the track's clearance limits. The rangefinder is, for example, a LiDAR (Light Detection and Ranging) device. This technology compensates for the fact that images acquired solely by cameras lack clarity regarding the depth-direction positional relationships of objects in the image by using a rangefinder. Such technology can be particularly useful in situations where the driver cannot visually determine the presence or absence of obstacles, such as when using an automated driving system.

[0003] Incidentally, if the angle of incidence of the laser beam emitted from the LiDAR device to any given object becomes excessively large, a large portion of the laser beam reflected by the object may not return to the LiDAR device, making distance measurement impossible.

[0004] On the other hand, trains have longer braking distances than cars and other vehicles, and in order to avoid collisions with obstacles, it is necessary to detect obstacles hundreds of meters away. However, the further away the track is from the train, the larger the angle of incidence of the laser beam relative to the track becomes, making it impossible for the rangefinder to measure the distance from the train to the track. As a result, the location of tracks far from the train cannot be identified, and it becomes impossible to detect obstacles within the track's clearance limits, leading to a decrease in detection accuracy. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] A Camera and LiDAR Data Fusion Method for Railway Object Detection, IEEE SENSORS JOURNAL, VOL. 21, NO. 12, JUNE 15, 2021 [Non-Patent Document 2] Train Front Obstacle Detection Method with Camera-LiDAR Fusion, QR of RTRI, Vol. 63, No.3, Aug,2022. [Overview of the project] [Problems that the invention aims to solve]

[0006] Embodiments of the present invention provide an information processing device, an information processing system, and an information processing method that enable the estimation of the position of a railway track regardless of the distance from a vehicle. [Means for solving the problem]

[0007] The information processing apparatus according to this embodiment includes: a first image acquisition unit that acquires a first image generated by an imaging device that images the environment in front of a vehicle; a track detection unit that detects the track on which the vehicle travels from the first image; a second image acquisition unit that acquires a second image including the three-dimensional position of the environment in front of the vehicle based on distance information acquired by a distance measuring device that measures the distance to the environment in front of the vehicle; and a track estimation unit that identifies a target range in the first image where no portion of the track corresponding to the track exists in the second image, and estimates the three-dimensional position of the portion of the track in the first image that is included in the target range, based on information about the track on which the vehicle travels. [Brief explanation of the drawing]

[0008] [Figure 1] A block diagram showing an obstacle detection system according to one embodiment. [Figure 2] A diagram showing an obstacle detection system installed on a train. [Figure 3]A diagram showing an example of a front image of a train. [Figure 4] A diagram showing an example of a track image generated based on the front image. [Figure 5] A diagram for explaining the deviation of the visual fields of the camera and the distance measuring device. [Figure 6] A diagram showing an example of a projection image expressed in the uv coordinate system. [Figure 7] A diagram showing an example of the extracted 3D track. [Figure 8] A diagram showing the position of the track specified in the XYZ coordinate system. [Figure 9] A diagram showing points on the track within the observation omission range. [Figure 10] A diagram obtained by adding a straight line passing through the point shown in FIG. 9 to FIG. 8. [Figure 11] A diagram showing the straight line shown in FIG. 10 on a map. [Figure 12] A diagram showing the point shown in FIG. 9 estimated in the XYZ coordinate system. [Figure 13] A diagram for explaining the estimated track and the calculated building clearance. [Figure 14] A flowchart for explaining an example of the process performed by the obstacle detection system. [Figure 15] A block diagram showing an example of the hardware configuration in an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0010] [[ID=4,8]]FIG. 1 is a block diagram showing an example of an obstacle detection system 1 according to an embodiment. The obstacle detection system 1 includes a camera 2, a distance measuring device 3, and an obstacle detection device 4.

[0011] Figure 2 shows the obstacle detection system 1 installed on the vehicle, more specifically on the lead car of train 10. The obstacle detection system 1 is a system for determining whether or not there are objects that could obstruct the movement of train 10, i.e., obstacles, within the construction clearance of the track T on which train 10 is traveling. Obstacles include people and fallen objects that are within the construction clearance of the track T, and do not include the track T itself or the ground on which the track T is located. Note that "lead car" means the car at the very front of train 10 in the direction of travel. Train 10 is traveling on track T. Object Obj1 is on the left side of track T as seen from train 10, and object Obj2 is on the right side of track T.

[0012] In Figure 2, the obstacle detection system 1 is located outside the train 10, but it may also be located inside the train 10. In that case, the camera 2 and the rangefinder 3 may perform imaging and distance measurement through the windshield of the train 10, respectively. Alternatively, only the camera 2 and the rangefinder 3 may be mounted on the train 10, and the obstacle detection system 4 may not be mounted on the train 10.

[0013] Camera 2 is an imaging device that captures the environment in front of train 10 and generates a forward image, which is the first image. Camera 2 is located in a three-dimensional first coordinate system (first world coordinate system). Camera 2 is, for example, an optical camera such as a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera, or an infrared camera (thermal camera), or a combination thereof. If camera 2 includes an infrared camera, it becomes easier to detect obstacles even at night. The forward image is, for example, a luminance image, and each pixel value in the luminance image is a luminance value.

[0014] Figure 3 shows an example of a forward-facing image captured by camera 2. In the example in Figure 3, the railway track T curves to the right as seen from train 10. The symbol H shown in Figure 3 indicates the horizon. The forward-facing image is shown in the uv coordinate system, as shown in Figure 3. The uv coordinate system is the image coordinate system of camera 2.

[0015] The distance measuring device 3 is a device or sensor that measures the distance to the forward environment, including objects Obj1, Obj2, etc. Based on the measured distance, the distance measuring device 3 generates a 3D image, which is a second image containing the three-dimensional position of the forward environment. The distance measuring device 3 is located in a three-dimensional second coordinate system (second world coordinate system). Each pixel of the 3D image contains a three-dimensional position. Each pixel of the 3D image may also contain a distance value. The objects include the railway track T and the ground on which the railway track T is located. The distance between the object and the train 10 is, strictly speaking, the distance between the object and the distance measuring device 3. The distance measuring device 3 includes, for example, LiDAR. If the distance measuring device 3 includes LiDAR, the wavelength of the laser used may be, for example, in the eye-safe band (e.g., 1.4 μm to 2.6 μm). Hereafter, the distance measuring device 3 will be described assuming it is LiDAR. The distance measuring device 3 may generate a distance image that includes distance values ​​to the environment ahead, and the 3D data acquisition unit 43, described later, may generate a 3D image from the distance image and the position information of the distance measuring device 3.

[0016] The distance measuring device 3 scans a predetermined range of the forward environment by, for example, irradiating it with laser pulses. The distance measuring device 3 then receives the pulses reflected back from the forward environment to acquire distance information to the forward environment (distance information to various objects contained within the forward environment) and generates a 3D image.

[0017] The obstacle detection device 4 is an information processing device that determines whether or not there is an object that constitutes an obstacle on the track T in front of the train 10, based on the forward image captured by the camera 2 and the 3D image acquired by the distance measuring device 3. In particular, the obstacle detection device 4 makes it possible to determine whether or not there is an obstacle within the building clearance at a section of the track that is visible in the forward image but could not be measured by the distance measuring device 3. As shown in Figure 1, the obstacle detection device 4 comprises an image acquisition unit 41, a track detection unit 42, a 3D data acquisition unit 43, a projected image generation unit 44, a track estimation unit 46, and an obstacle detection unit 47.

[0018] The image acquisition unit 41 is a first image acquisition unit that acquires the forward image of the train 10 (see Figure 3) captured by the camera 2.

[0019] The track detection unit 42 identifies the track T from the forward image acquired by the image acquisition unit 41 and acquires (generates) a track image showing the track T in the forward image. The track detection unit 42 generates the track image by extracting the track T, for example, using semantic segmentation or edge detection. Figure 4 is an example of a track image detected by the track detection unit 42. The track image, like the forward image, is represented in the uv coordinate system (the image coordinate system of camera 2).

[0020] The 3D data acquisition unit 43 is a second image acquisition unit that acquires 3D images from the distance measuring device 3. The 3D images are represented in the image coordinate system of the distance measuring device 3. The 3D images acquired by the 3D data acquisition unit 43 are 3D images acquired approximately simultaneously (in approximately the same frame) as the forward image acquired by the image acquisition unit 41.

[0021] The projection image generation unit 44 projects the 3D image onto the track image. Projecting the 3D image onto the track image is equivalent to converting the 3D image, which is represented in the image coordinate system of the distance measuring device 3, into the uv coordinate system, which is the image coordinate system of the camera 2 (representing it in the uv coordinate system).

[0022] Figure 5 illustrates the discrepancy between the field of view of camera 2 and the field of view of the distance measuring device 3. As shown in Figure 5, camera 2 and distance measuring device 3 have different physical positions, i.e., different viewpoints and orientations. Therefore, the origin O2 and line of sight LOS2, which are the viewpoints of camera 2's field of view FOV2, and the origin O3 and line of sight LOS3, which are the viewpoints of distance measuring device 3's field of view FOV3, are different from each other.

[0023] As shown in Figure 5 below, the three-dimensional coordinates within the field of view (FOV) 2 are referred to as the xyz coordinate system (the world coordinate system of camera 2 or the three-dimensional coordinate system of camera 2), and the three-dimensional coordinates within the field of view (FOV) 3 are referred to as the XYZ coordinate system (the world coordinate system of the rangefinder 3 or the three-dimensional coordinate system of the rangefinder 3). Note that the 3D image represented in the XYZ coordinate system includes only the values ​​of the range (locations) observed by the rangefinder 3, and does not include the values ​​of the range (locations) that are included in the field of view (FOV) 3 but not observed by the rangefinder 3.

[0024] Consider an arbitrary point P(X,Y,Z) in the XYZ coordinate system, as shown in Figure 5. The distance value of point P is measured, and the 3D position of point P is included in the 3D image. Equation (1) is used to make the origin O3 and line of sight LOS3 of the field of view 3 of the distance measuring device 3 coincide with the origin O2 and line of sight LOS2 of the field of view 2 of the camera 2. In other words, it is an equation that transforms a point P(X,Y,Z) in the XYZ coordinate system to a point p(x,y,z) in the xyz coordinate system. Matrix R is a rotation matrix that rotates the field of view 3 by an angle θ. Matrix tvec is a translation matrix that translates the field of view 3.

[0025] By performing the transformation in equation (1), the origin O2 and line of sight LOS2 of the field of view FOV2 coincide with the origin O3 and line of sight LOS3 of the field of view FOV3. In other words, a point P included in a 3D image that was represented in the XYZ coordinate system can be represented as a point p in the xyz coordinate system.

[0026]

number

[0027] Next, the projection image generation unit 44 converts a point p(x,y,z) in the 3D image represented in the xyz coordinate system to a point p'(u,v) in the uv coordinate system using equations (2) to (4). First, the projection image generation unit 44 normalizes the point p in the 3D image by the distance z in the depth direction using equation (2), and converts the point p contained in the 3D image from a 3D coordinate system (xyz coordinate system) to a 2D coordinate system (x'y' coordinate system).

[0028]

number

[0029] Next, we apply the distortion function f(x,y), which is specific to camera 2, as shown in equation (3).

[0030]

number

[0031] The coordinate system after applying the distortion function in equation (3) below, or the aforementioned x'y' coordinate system, corresponds to the 2D coordinate system or camera coordinate system of camera 2.

[0032] Finally, using the camera matrix K specific to camera 2, we transform the coordinate system in (3) to the uv coordinate system (the image coordinate system of camera 2). The camera matrix K is a 2x3 matrix.

[0033]

number

[0034] This results in a 3D image projected onto the track image (hereinafter referred to as the projected image), that is, an image in which the 3D image has been transformed into the UV coordinate system of camera 2.

[0035] Figure 6 shows an example of a 3D image projected onto the uv coordinate system. Each pixel (dot) DT corresponds to the position (observation point) where a single reflected pulse was received. The 3D image is represented as a collection of multiple pixels DT (point cloud). Each pixel in the projected image may also be associated with the 3D position and distance information of the corresponding pixel in the original 3D image. In the example in Figure 6, the distance from train 10 is represented by the intensity of the pixels DT, with lighter colors indicating greater distance.

[0036] Here, pixels without color represent pixels where the reflected pulse was not received, i.e., pixels that were not observed. As you move further away from train 10, the angle of incidence of the pulsed light emitted from the distance measuring device 3 relative to the ground or track T increases, so the number of unobserved pixels increases. However, from Figures 3 and 6, it can be seen that object Obj3 is observed even though it is relatively far from train 10 because the angle of incidence of the pulsed light emitted from the distance measuring device 3 is small.

[0037] The track extraction unit 45 extracts images of the track portion corresponding to track T in the track image from the projected image, based on the track image (see Figure 4) and the projected image (see Figure 6). The extracted images of the track portion are called 3D tracks. The track extraction unit 45 overlays the track image and the projected image and extracts the portion of the projected image that overlaps with the position of track T as a 3D track. The track portion in the track image that overlaps with the 3D track corresponds to the portion of track T in the track image that has a corresponding portion in the projected image. The pixel group DT in Figure 7 shows the extracted 3D track. Here, region A shown in the figure is a region in which there is a track portion that exists in the track image but not in the projected image. In other words, region A is a region in which the existence of track T is expected from the track image, but the track portion has not been observed by the distance measuring device 3 (observation omission region). Note that region A does not mean that everything within region A is unobserved, but rather that at least the track portion has not been observed, and it is a region shown for reference for ease of understanding.

[0038] For the area measured by the distance measuring device 3, the position of track T in the XYZ coordinate system can be determined by following the above equations in reverse, as shown in Figure 8. Figure 8 shows the portion where the position of track T has been determined in the XYZ coordinate system. As a result, the building clearance of track T can be calculated for the portion where the position of track T has been determined, and thus it can be determined whether or not an obstruction (object) in the XYZ coordinate system is interfering with the building clearance. On the other hand, in region A shown in Figure 8, the track portion that exists has not been measured by the distance measuring device 3, so no track portion is displayed in the XYZ coordinate system. In region A, the presence of track T is expected from the track image, but it has not been observed by the distance measuring device 3, so the position of the track in the XYZ coordinate system (or the position in the depth direction in the direction of travel of train 10) cannot be determined as is. Therefore, it is necessary to estimate the position of track T in the XYZ coordinate system in region A.

[0039] The track estimation unit 46 estimates the 3D track in area A. The track estimation unit 46 also holds parameters related to the camera 2 and the distance measuring device 3, and may use this parameter information for estimation. The processing performed by the track estimation unit 46 will be described in detail below.

[0040] This section describes how to estimate a 3D transmission line in the XYZ coordinate system. As mentioned earlier, the 3D image acquired by the distance measuring device 3 is converted to the uv coordinate system using equations (1) to (4) described above. First, to explain how to estimate the 3D transmission line, we will show how to convert a point p'(u,v) expressed in the uv coordinate system to the XYZ coordinate system.

[0041] First, we perform the inverse transformation of equation (4) using equation (5). Here, K[1:2,1:2] represents the 2x2 matrix on the left side of K, and K[1:2,3] represents the rightmost column vector of K.

[0042]

number

[0043] Next, the inverse function of the strain function f shown in equation (6) is f -1 We use this to perform the inverse transform of equation (3).

[0044]

number

[0045] Next, we perform the inverse transformation of equation (2) using equation (7). This allows us to transform point p' in the uv coordinate system to point p in the xyz coordinate system, which is the 3D coordinate system of camera 2.

[0046]

number

[0047] Here, z is expressed as shown in equation (8). If the values ​​of u and v at point p' are known, then the values ​​of x' and y' can be determined by equations (5) and (6), but the value of z at point p is unknown. Z0 shown in equation (8) represents the known Z-axis coordinate value of point P in the XYZ coordinate system. Here, instead of the Z-axis coordinate value, the X-axis coordinate value or Y-axis coordinate value of point P may also be known. In any case, in order to know the z value of point p, additional information is needed in addition to the values ​​of u and v at point p'.

[0048]

number

[0049] The vector S shown in equation (8) -1 This can be expressed as shown in equation (9). R[3,1:3] is the third row vector from the top of the rotation matrix R.

[0050]

number

[0051] When the value of the X-axis coordinate is the known value X0, Z0 in Equation (8) is replaced by X0, and Equation (9) becomes S -1 = R[1, 1:3] -1 This is the case. R[1, 1:3] is the first row vector from the top of the rotation matrix R

[0052] When the value of the Y-axis coordinate is the known value Y0, Z0 in Equation (8) is replaced by Y0, and Equation (9) becomes S -1 = R[2, 1:3] -1 This is the case. R[2, 1:3] is the second row vector from the top of the rotation matrix R

[0053] Finally, the inverse transformation of Equation (1) is performed using the following Equation (10). In this way, the point p in the xyz coordinate system of Camera 2 can be transformed into the point P in the XYZ coordinate system of the distance measuring device 3

[0054]

Equation

[0055] Alternatively, the distance from the distance measuring device 3 to the point P may be known. In the XYZ coordinate system, if the distance from the coordinate O3(0, 0, 0) of the distance measuring device 3 to the point P(X, Y, Z) is D, then X 2 + Y 2 + Z 2 = D 2 This is the case. By substituting this into Equation (10), the following Equation (11) is obtained. By solving Equation (11), the value of z of the point p can be obtained

[0056]

Equation

[0057] Alternatively, the distance from Camera 2 to the point p may be known. In the xyz coordinate system, if the distance from the coordinate O2(0, 0, 0) of Camera 2 to the point p(x, y, z) is d, then x 2 + y 2 + z 2 = d 2Therefore, substituting this into equation (7), we obtain equation (12). By solving equation (12), we can obtain the value of z at point p.

[0058]

number

[0059] In any case, in order to convert from the uv coordinate system (the image coordinate system of camera 2) to the XYZ coordinate system (the 3D coordinate system of the distance measuring device 3), the z value of point p must be known. To find the z value of point p, additional information is needed, such as the coordinate of any axis of point P in the XYZ coordinate system.

[0060] Figure 9 is a diagram illustrating a method for estimating the position of a railway line in region A. Point q' is a point in the railway line image that is included in the region where the portion corresponding to the projected image does not exist, i.e., it corresponds to a point on railway line T in region A of the railway line image. This region (the portion of the railway line included in region A) will be the target range to be used as the basis for estimation. Point q is the point in the 3D coordinate system of camera 2 that corresponds to point q'. Point q' is information that exists in the railway line image but not in the 3D image. We consider estimating the point in the XYZ coordinate system (point Q(X,Y,Z)) that corresponds to point q'.

[0061] Although the position of point Q in the XYZ coordinate system is unknown, its coordinates in the uv coordinate system are known. Solving equations (5) to (10), we find that point Q lies on the line passing through the origin O3 in the XYZ coordinate system (epipolar line L), which corresponds to the line passing through the origin O2 and point q' in the xyz coordinate system (epipolar constraint). The epipolar line L can be calculated based on the parameters of camera 2 and the parameters of distance measuring device 3. Point Q is found to be located somewhere on the epipolar line L shown in Figure 10. Figure 10 is a diagram in which the epipolar line L passing through point Q is added to Figure 8.

[0062] Since we know that point Q lies on the epipolar line L, if we know the XYZ coordinates of point Q, or the distance D from the origin O3, we can estimate the XYZ coordinates of point Q using equation (8) or equation (11). Alternatively, if we know the distance d from point q to the origin O2 in the xyz coordinate system, we can estimate the XYZ coordinates of point Q using equation (12).

[0063] (First estimation method) In the first estimation method, the track estimation unit 46 includes, for example, a train position estimation function and a map database. The train position estimation function obtains the current position of the train 10 from a positioning system such as GPS (Global Positioning System). The track estimation unit 46 calculates the altitude of the train 10 from the current position of the train 10 and the map database. The track estimation unit 46 then obtains the position and altitude of the track T from the map database. Altitude can be, for example, elevation or sea level.

[0064] Now, let's consider the railway line T on the map database. Figure 11 shows the current position of train 10, i.e., the origin O3 in the XYZ coordinate system, on a map obtained from the map database. The map in Figure 11 shows contour lines indicating elevations of 100m, 105m, and 110m. Train 10 is shown for ease of understanding. When the epipolar line L is displayed on the map shown in Figure 11, it can be seen that there is a point where the epipolar line L and the railway line T intersect. This intersection point is presumed to be point Q. Note that although the slope of the epipolar line L in the vertical direction does not appear to be considered in Figure 11, the slope of the epipolar line L in the vertical direction is actually taken into account.

[0065] Furthermore, from the map shown in Figure 11, we can see that the current elevation of train 10 is 100m, and the elevation of point Q is 110m. Therefore, as shown in Figure 12, the coordinates of point Q in the XYZ coordinate system are (X1, 10m, Z1). X1 and Z1 are the X and Z coordinate values ​​of point Q, which were revealed in a chain reaction after the Y coordinate value of point Q was determined.

[0066] Thus, in the first estimation method, the XYZ coordinates of point Q are estimated by estimating the Y coordinate of point Q from a map database. By repeating the above process, the position of the railway line T within region A can be estimated in the XYZ coordinate system.

[0067] (Second estimation method) In the second estimation method, the track estimation unit 46 estimates either an XYZ coordinate value or distance from surrounding pixels DT of the track T. For example, even if point Q itself is not measured by the distance measuring device 3, if an object Obj3 near point Q is observed, as shown in Figure 6, the XYZ coordinate value or distance of point Q can be estimated from the observed portion. Specifically, the unit identifies the portion of the track included in the track image that is not included in the 3D image (the portion of the track corresponding to the 3D track), and detects objects from the track image that are close to the identified portion of the track and include a portion corresponding to the 3D image. Based on the three-dimensional position of the object (the three-dimensional position in the camera 2's three-dimensional coordinate system or the three-dimensional position in the distance measuring device's three-dimensional coordinate system), the unit estimates the three-dimensional position in the XYZ coordinate system of the portion of the track corresponding to the 3D track. Alternatively, the three-dimensional position of the track portion corresponding to the 3D track in the XYZ coordinate system is estimated based on the distance from the viewpoint of camera 2 or the viewpoint of distance measuring device 3 to the object. As a specific example of estimation, the same Z coordinate as the object may be used as the estimated value. In this way, the second estimation method estimates the XYZ coordinate values ​​of point Q by estimating the distance from the origin O3 to point Q from observed points near point Q.

[0068] (Third estimation method) In the third estimation method, the track estimation unit 46 estimates the distance from camera 2 to point q using the number of pixels corresponding to the width of the track T in the forward image (track image). The width of the track T is assumed to be known and constant. Under these preconditions, the distance from camera 2 to point q can be calculated based on the number of pixels of the width of the track T in the forward image, the field of view of camera 2, and the number of pixels in the forward image. Once the distance from camera 2 to point q is known, the position of point q can be determined using equation (12), and the coordinates of point q can be converted to the coordinates of point Q in the XYZ coordinate system. This allows the position of the track T within region A to be estimated. Thus, in the third estimation method, the distance d from point q to the origin O2 in the xyz coordinate system is estimated using the number of pixels of the width of the track T in the forward image to estimate the XYZ coordinates of point Q.

[0069] (Fourth estimation method) In the fourth estimation method, camera 2 is a stereo camera, and the track estimation unit 46 estimates the distance from camera 2 to point q. In the fourth estimation method, there are two first images, a first forward image and a second forward image, which are two forward images taken simultaneously from two different locations. The distance from camera 2 to point q is estimated from the parallax between the first forward image and the second forward image that captured point q. Once the distance from camera 2 to point q is known, the coordinates of point Q in the XYZ coordinate system can be determined using equation (12), and the track T within region A can be estimated in the XYZ coordinate system. Thus, in the fourth estimation method, the XYZ coordinate values ​​of point Q are estimated by estimating the distance d from point q to the origin O2 in the xyz coordinate system using a stereo camera.

[0070] By performing any or a combination of the above processes, it is possible to estimate the three-dimensional position of the track section that has not been measured by the distance measuring device 3.

[0071] In the above process, the position of the railway line T is identified and estimated in the XYZ coordinate system, but it is also acceptable to estimate the position of the railway line T in the xyz coordinate system.

[0072] The obstacle detection unit 47 calculates the building clearance from the tracks estimated by the track estimation unit 46 and the tracks extracted by the track extraction unit 45, and determines whether or not there are any objects (obstacles) within the building clearance that could obstruct the movement of the train 10. Figure 13 shows the building clearance calculated based on the estimated and extracted tracks. The building clearance defines the area in relation to the track T where no buildings should be constructed.

[0073] The obstacle detection unit 47 then outputs the result of determining whether or not there is an object within the building clearance that could obstruct train movement. For example, if a large animal is detected within the building clearance, it may be determined that there is an object that could obstruct train movement. The determination result may be output to a display visible to the train driver 10, or to the train's automatic driving system. The obstacle detection unit 47 may output the determination result only if it determines that there is an obstructing object within the building clearance, or it may output the determination result regardless of the determination result.

[0074] Furthermore, the obstacle detection unit 47 may set a range wider than the building clearance and determine whether or not there is an obstructing object within that range. This allows, for example, the driver to recognize a person who is near the building clearance but not inside it, and to take appropriate action, such as sounding a horn, depending on the situation.

[0075] <Flowchart of the processes performed by the obstacle detection system 1> Figure 14 is a flowchart of an example of the process performed by the obstacle detection system 1. The process performed by the obstacle detection system 1 will be explained below with reference to Figure 14 and the other figures.

[0076] First, camera 2 captures an image of the environment in front of train 10 (forward image: see Figure 3) (step S1). The captured forward image is taken up by the obstacle detection device 4 by the image acquisition unit 41.

[0077] Next, the track detection unit 42 detects the position of track T from the forward image and obtains a track image (see Figure 4) that shows track T extracted from the forward image (step S2).

[0078] Almost simultaneously with step S1, the distance measuring device 3 acquires a 3D image of the environment in front of the train 10 (step S3). The acquired 3D image is taken up by the obstacle detection device 4 by the 3D data acquisition unit 43.

[0079] Next, the projection image generation unit 44 projects the 3D image onto the track image and obtains the projected image (see Figure 6) (step S4).

[0080] Next, the track extraction unit 45 extracts from the projection image the portion corresponding to the track T included in the track image (3D track: see Figure 7) based on the track image (see Figure 4) and the projection image (see Figure 6) (step S5).

[0081] Next, the track estimation unit 46 estimates the portion of the track included in the track image that was not measured by the distance measuring device 3, based on the extracted 3D track and track image, in the XYZ coordinate system (step S6).

[0082] Next, the obstacle detection unit 47 calculates the building clearance from the estimated track section in the XYZ coordinate system and the 3D track (step S7).

[0083] Next, the obstacle detection unit 47 determines whether there is an obstacle within the calculated building clearance (step S8). For example, if there is an object larger than a predetermined size, it may be determined that there is an obstacle. Alternatively, the type of object detected within the building clearance may be determined using the track image and semantic segmentation, and if the type of object matches a predetermined type of obstacle, it may be determined that there is an obstacle.

[0084] Next, the obstacle detection unit 47 outputs the determination result (step S9). The obstacle detection unit 47 outputs the determination result to, for example, the automatic driving system of the train 10.

[0085] Once step S8 is completed, the process returns to steps S1 and S3, and each step is repeated sequentially. Note that each step may be performed in parallel with other steps. For example, while the image acquisition unit 41 is acquiring a forward image in step S2, the camera 2 may continue to capture a forward image.

[0086] As described above, according to this embodiment, the distance measuring device 3 and the camera 2 are combined to estimate the position of the railway track in an area where the distance measuring device 3 cannot measure. This makes it possible to detect obstacles even at long distances where the distance measuring device 3 cannot measure. This makes it possible to improve the distance at which obstacles can be detected and the accuracy of obstacle detection.

[0087] (Another example of operation 1) In this embodiment, the camera 2 and the distance measuring device 3 were separate devices, but they may be an integrated device. In this case, if the viewpoints and fields of view of both are the same and can be considered to exist in the same coordinate system, the projection image generation unit 44 may be omitted, and the 3D image from the distance measuring device 3 may be used directly instead of the projection image.

[0088] (Another example of operation 2) The obstacle detection device 4 may be linked to the train's brake control system. The brake control system may control the train's brakes according to the determination result output by the obstacle detection device 4. For example, if an obstructing object is detected, the system may control the train to stop at a position ahead of the detected object.

[0089] (Hardware configuration) Figure 16 shows the hardware configuration of the information processing device according to each embodiment. The information processing device consists of a computer device 600. The computer device 600 includes a CPU 601, an input interface 602, a display device 603, a communication device 604, a main memory 605, and an external memory device 606, which are interconnected by a bus 607.

[0090] The CPU (Central Processing Unit) 601 executes an information processing program, which is a computer program, on the main memory 605. An information processing program is a program that implements the aforementioned functional configurations of the information processing device. An information processing program may not be a single program, but rather a combination of multiple programs or scripts. The CPU 601 implements each functional configuration by executing the information processing program.

[0091] The input interface 602 is a circuit for inputting operation signals from input devices such as keyboards, mice, and touch panels to the information processing device. The input interface 602 corresponds to the input section of the information processing device according to each embodiment.

[0092] The display device 603 displays data output from the information processing device. The display device 603 is, for example, an LCD (liquid crystal display), an organic electroluminescent display, a CRT (cathode ray tube), or a PDP (plasma display), but is not limited to these. Data output from the computer device 600 can be displayed on this display device 603. The display device 603 corresponds to the output unit of the information processing device according to each embodiment.

[0093] The communication device 604 is a circuit for the information processing device to communicate with an external device wirelessly or via a wired connection. Data can be input from an external device via the communication device 604. The data input from the external device can be stored in the main memory 605 or the external memory 606.

[0094] The main memory 605 stores information processing programs, data necessary for executing the information processing programs, and data generated by the execution of the information processing programs. The information processing programs are deployed and executed on the main memory 605. The main memory 605 is, for example, RAM, DRAM, or SRAM, but is not limited to these. Each storage unit or database of the information processing apparatus according to each embodiment may be built on the main memory 605.

[0095] The external storage device 606 stores information processing programs, data necessary for executing the information processing programs, and data generated by the execution of the information processing programs. These information processing programs and data are read into the main memory 605 when the information processing programs are executed. The external storage device 606 is, for example, a hard disk, optical disk, flash memory, and magnetic tape, but is not limited to these. Each storage unit or database of the information processing device may be built on the external storage device 606.

[0096] The information processing program may be pre-installed on the computer device 600, or it may be stored on a storage medium such as a CD-ROM. Furthermore, the information processing program may be uploaded to the internet.

[0097] Furthermore, the information processing device may consist of a single computer device 600, or it may be configured as a system consisting of multiple interconnected computer devices 600.

[0098] It should be noted that the present invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments described above. For example, a configuration in which some components are removed from all the components shown in each embodiment is also conceivable. Moreover, components described in different embodiments may be appropriately combined.

[0099] This embodiment can also be configured as follows. [Item 1] A first image acquisition unit acquires a first image generated by an imaging device that captures the environment in front of the vehicle, A track detection unit that detects the track on which the vehicle is traveling from the first image, A second image acquisition unit acquires a second image including the three-dimensional position of the forward environment based on distance information acquired by a distance measuring device that measures the distance to the forward environment, A track estimation unit identifies a target area within the track included in the first image where no portion corresponding to the track exists in the second image, and estimates the three-dimensional position of the portion of the track included in the target area within the track in the first image based on information about the track on which the vehicle travels. Equipped with an information processing device. [Item 2] The track estimation unit estimates the three-dimensional position of the track portion included in the target area based on map data including the three-dimensional position of the track on which the vehicle travels. The information processing device described in item 1. [Item 3] The imaging device is provided in a three-dimensional first coordinate system, and the distance measuring device is provided in a three-dimensional second coordinate system. The system includes a projection image generation unit that projects the second image onto the first image to generate a projected image, The aforementioned target range is the range in which, among the lines included in the first image, there is no portion in the projected image that corresponds to the lines. The track estimation unit estimates the three-dimensional position of the track portion included in the target range in the second coordinate system. An information processing device as described in item 1 or 2. [Item 4] The track estimation unit calculates an epipolar line from the viewpoint of the distance measuring device that corresponds to a straight line passing through the track portion included in the target range from the viewpoint of the imaging device, Based on map data including the three-dimensional position information of the railway line on which the vehicle travels, the position where the epipolar line intersects with the railway line in the map data is calculated, and the calculated position is used as the estimated three-dimensional position of the railway section included in the target area in the second coordinate system. The information processing device described in item 3. [Item 5] The track estimation unit estimates the three-dimensional position of the track portion included in the target range in the first coordinate system, based on the premise that the width of the track is constant, and converts the three-dimensional position of the track portion in the first coordinate system to a three-dimensional position in the second coordinate system. An information processing device as described in item 3 or 4. [Item 6] The track estimation unit detects an object in the first image that is close to the track portion included in the target range and includes a portion corresponding to the second image, and estimates the three-dimensional position of the track portion included in the target range in the second coordinate system based on the three-dimensional position of the object in the first coordinate system or the second coordinate system, or the distance from the viewpoint of the imaging device or the viewpoint of the distance measuring device to the object. An information processing device as described in any one of items 3 to 5. [Item 7] The imaging device is a stereo camera, and the stereo camera generates the first image from multiple viewpoints. The track estimation unit estimates the three-dimensional position of the track portion included in the target range in the first coordinate system based on the first images of the multiple viewpoints, and converts the three-dimensional position of the track portion in the first coordinate system to a three-dimensional position in the second coordinate system. An information processing device as described in any one of items 3 to 6. [Item 8] Based on the three-dimensional position of the track section estimated by the track estimation unit, the building clearance of the track section estimated by the track estimation unit is calculated, and an obstacle detection unit detects objects that would obstruct the movement of the vehicle within the building clearance using the first image and the second image. An information processing device described in any one of items 1 to 7, which is equipped with the features described in item 1 to 7. [Item 9] The aforementioned vehicle is a train. The information processing device described in item 1. [Item 10] A first image is acquired by an imaging device that captures the environment in front of the vehicle. From the first image, the railway track on which the vehicle is traveling is detected. A second image is obtained, including the three-dimensional position of the forward environment, based on distance information acquired by a distance measuring device that measures the distance to the forward environment. Among the tracks included in the first image, a target area is identified where no portion corresponding to the tracks exists in the second image, and the three-dimensional position of the portion of the tracks included in the target area in the first image is estimated based on information about the tracks on which the vehicle travels. Information processing methods. [Item 11] An imaging device that captures the environment in front of the vehicle and generates a first image, A distance measuring device for measuring the distance to the forward environment, A track detection unit that detects the track on which the vehicle is traveling from the first image, A track estimation unit acquires a second image including the three-dimensional position of the forward environment based on distance information acquired by the distance measuring device, identifies a target range in the first image where no portion of the track corresponding to the track exists in the second image, and estimates the three-dimensional position of the portion of the track in the first image that is included in the target range, based on information about the track on which the vehicle is traveling. An information processing system equipped with [the following features]. [Explanation of Symbols]

[0100] 1. Obstacle detection system 2 cameras 3 Ranging device 4. Obstacle detection device 41 Image acquisition unit 42 Track detection unit 43 3D data acquisition unit 44 Projection Image Generation Unit 45 Line extraction part 46. ​​Track Estimation Section 47 Obstacle detection unit 10 trains 600 Computer devices 601 CPU 602 Input Interface 603 Display device 604 Communication equipment 605 Main storage 606 External storage device 607 Bus Area A DT pixels H horizon FOV2 field of view FOV3 field of view LOS2 Gaze LOS3 line of sight O2 Origin O3 Origin Obj1, Obj2 objects T track L Epipolar Ray

Claims

1. A first image acquisition unit is installed based on a three-dimensional first coordinate system and acquires a first image generated by an imaging device that images the environment in front of the vehicle, A track detection unit that detects the track on which the vehicle is traveling from the first image, A second image acquisition unit that acquires a second image including the three-dimensional position of the forward environment based on distance information acquired by a distance measuring device that measures the distance to the forward environment, which is set up with reference to a three-dimensional second coordinate system, A projection image generation unit generates a projected image by projecting the second image onto the image coordinate system of the first image, Among the railway lines included in the first image, a target range is identified in which no portion corresponding to the railway line exists in the projected image. The epipolar line from the viewpoint of the distance measuring device is calculated, corresponding to a straight line passing through the track portion included in the target range from the viewpoint of the imaging device. Based on map data including three-dimensional positional information of the railway line on which the vehicle travels, the position where the epipolar line intersects with the railway line in the map data is calculated. A track estimation unit that estimates the calculated position as the three-dimensional position of the track portion included in the target range in the second coordinate system, Equipped with an information processing device.

2. A first image acquisition unit is installed based on a three-dimensional first coordinate system and acquires a first image generated by an imaging device that images the environment in front of the vehicle, A track detection unit that detects the track on which the vehicle is traveling from the first image, A second image acquisition unit that acquires a second image including the three-dimensional position of the forward environment based on distance information acquired by a distance measuring device that measures the distance to the forward environment, which is set up with reference to a three-dimensional second coordinate system, A projection image generation unit generates a projected image by projecting the second image onto the image coordinate system of the first image, Among the railway lines included in the first image, a target range is identified in which no portion corresponding to the railway line exists in the projected image. Based on the information that the width of the railway track is constant, the distance from the imaging device to the railway track is calculated based on the number of pixels of the width of the railway track portion included in the target area in the first image and the field of view of the imaging device, and the three-dimensional position in the first coordinate system is determined. A track estimation unit that estimates the three-dimensional position of the track portion included in the target range by converting the identified three-dimensional position in the first coordinate system to the three-dimensional position in the second coordinate system, Equipped with an information processing device.

3. A first image acquisition unit is installed based on a three-dimensional first coordinate system and acquires a first image generated by an imaging device that images the environment in front of the vehicle, A track detection unit that detects the track on which the vehicle is traveling from the first image, A second image acquisition unit that acquires a second image including the three-dimensional position of the forward environment based on distance information acquired by a distance measuring device that measures the distance to the forward environment, which is set up with reference to a three-dimensional second coordinate system, A projection image generation unit generates a projected image by projecting the second image onto the image coordinate system of the first image, Among the railway lines included in the first image, a target range is identified in which no portion corresponding to the railway line exists in the projected image. In the first image, an object is detected that is located in close proximity to the track portion included in the target area and has a corresponding portion in the projection image. A track estimation unit estimates the three-dimensional position of the detected object in the first or second coordinate system, or the distance from the viewpoint of the imaging device or the viewpoint of the distance measuring device to the object, as the three-dimensional position or distance of the track portion included in the target range. Equipped with an information processing device.

4. A first image acquisition unit acquires a first image generated by a stereo camera, which is an imaging device that is set up based on a three-dimensional first coordinate system and images the environment in front of the vehicle. A track detection unit that detects the track on which the vehicle is traveling from the first image, A second image acquisition unit that acquires a second image including the three-dimensional position of the forward environment based on distance information acquired by a distance measuring device that measures the distance to the forward environment, which is set up with reference to a three-dimensional second coordinate system, A projection image generation unit generates a projected image by projecting the second image onto the image coordinate system of the first image, Among the railway lines included in the first image, a target range is identified in which no portion corresponding to the railway line exists in the projected image. A track estimation unit calculates the distance from the imaging device to the track portion included in the target range based on the parallax in the first image of multiple viewpoints generated by the stereo camera, identifies the three-dimensional position in the first coordinate system, and estimates the three-dimensional position of the track portion included in the target range by converting the identified three-dimensional position in the first coordinate system to a three-dimensional position in the second coordinate system. Equipped with an information processing device.

5. Based on the three-dimensional position of the track section estimated by the track estimation unit, the building clearance of the track section estimated by the track estimation unit is calculated, and an obstacle detection unit detects objects that obstruct the movement of the vehicle within the building clearance using the first image and the second image. An information processing apparatus according to any one of claims 1 to 4, comprising:

6. The aforementioned vehicle is a train. The information processing apparatus according to any one of claims 1 to 4.

7. An imaging device that captures the environment in front of the vehicle and generates a first image, A distance measuring device for measuring the distance to the forward environment, An information processing device according to any one of claims 1 to 4, An information processing system equipped with [the following features].

8. A first image is acquired by an imaging device that is set up with reference to a three-dimensional first coordinate system and images the environment in front of the vehicle. From the first image, the railway track on which the vehicle is traveling is detected. A second image is acquired, including the three-dimensional position of the forward environment, based on distance information obtained by a distance measuring device that is set up with reference to a three-dimensional second coordinate system and measures the distance to the forward environment. The second image is projected onto the image coordinate system of the first image to generate a projected image. Among the railway tracks included in the first image, a target area is identified in which no portion corresponding to the railway tracks exists in the projected image. An epipolar line is calculated from the viewpoint of the distance measuring device that corresponds to a straight line passing through the railway track portion included in the target area from the viewpoint of the imaging device. Based on map data including three-dimensional position information of the railway track on which the vehicle travels, the position where the epipolar line intersects with the railway track in the map data is calculated, and the calculated position is estimated as the three-dimensional position of the railway track portion included in the target area in the second coordinate system. The computer performs this. Information processing methods.

9. A first image is acquired by an imaging device that is set up with reference to a three-dimensional first coordinate system and images the environment in front of the vehicle. From the first image, the railway track on which the vehicle is traveling is detected. A second image is acquired, including the three-dimensional position of the forward environment, based on distance information obtained by a distance measuring device that is set up with reference to a three-dimensional second coordinate system and measures the distance to the forward environment. The second image is projected onto the image coordinate system of the first image to generate a projected image. Among the railway lines included in the first image, a target range is identified in which no portion corresponding to the railway line exists in the projected image. Based on the information that the width of the railway track is constant, the distance from the imaging device to the railway track is calculated based on the number of pixels of the width of the railway track portion included in the target area in the first image and the field of view of the imaging device to determine the three-dimensional position in the first coordinate system, and the three-dimensional position of the railway track portion included in the target area is estimated by converting the determined three-dimensional position in the first coordinate system to the three-dimensional position in the second coordinate system. A method of information processing performed by a computer.

10. A first image is acquired by an imaging device that is set up with reference to a three-dimensional first coordinate system and images the environment in front of the vehicle. From the first image, the railway track on which the vehicle is traveling is detected. A second image is acquired, including the three-dimensional position of the forward environment, based on distance information obtained by a distance measuring device that is set up with reference to a three-dimensional second coordinate system and measures the distance to the forward environment. The second image is projected onto the image coordinate system of the first image to generate a projected image. Among the railway lines included in the first image, a target range is identified in which no portion corresponding to the railway line exists in the projected image. In the first image, an object is detected that is located in close proximity to the track portion included in the target area and has a corresponding portion in the projection image. The three-dimensional position of the detected object in the first or second coordinate system, or the distance from the viewpoint of the imaging device or the viewpoint of the distance measuring device to the object, is estimated as the three-dimensional position or distance of the track portion included in the target range. A method of information processing performed by a computer.

11. The system acquires a first image generated by an imaging device, which is a stereo camera that is set up based on a three-dimensional first coordinate system and captures the environment in front of the vehicle. From the first image, the railway track on which the vehicle is traveling is detected. A second image is acquired, including the three-dimensional position of the forward environment, based on distance information obtained by a distance measuring device that is set up with reference to a three-dimensional second coordinate system and measures the distance to the forward environment. The second image is projected onto the image coordinate system of the first image to generate a projected image. Among the railway lines included in the first image, a target range is identified in which no portion corresponding to the railway line exists in the projected image. Based on the parallax in the first image of multiple viewpoints generated by the stereo camera, the distance from the imaging device to the track portion included in the target area is calculated to determine the three-dimensional position in the first coordinate system, and the three-dimensional position of the track portion included in the target area is estimated by converting the determined three-dimensional position in the first coordinate system to a three-dimensional position in the second coordinate system. A method of information processing performed by a computer.

Images