Positional information acquisition device

Abstract

To solve the problem that it is difficult to improve the accuracy in acquiring information such as XYZ coordinates on the basis of images.SOLUTION: The positional information acquisition device includes: an extraction unit configured to extract a rail to extract from each of a plurality of image data acquired by using different imaging devices for the same scene including a rail on which a railway vehicle runs; and an acquisition unit configured to acquire positional information corresponding to each point in data representing the shape of an object surface as a three-dimensional point cloud, by performing stereo matching processing with reference to the extraction result by the extraction unit.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to a location information acquisition device, a location information acquisition method, and a program. [Background technology]

[0002] When performing maintenance and management in the field of railway tracks, information such as XYZ coordinates that indicate the position of rails and other items may be obtained from images.

[0003] For example, Patent Document 1 describes a track inspection device that inspects track displacement of a first track including a first rail extending in a first direction in a planar view. According to Patent Document 1, the track inspection device includes a first acquisition unit, a first extraction unit, and a first selection unit. For example, the first acquisition unit acquires first point cloud data, using the results of a stereo matching process on image data acquired by a stereo camera or the like, for each of a plurality of first points on the surface of the first track and the surface of an object around the first track. The first point cloud data includes a first coordinate in the first direction, a second coordinate in a second direction orthogonal to the first direction in a planar view, a third coordinate in a third direction orthogonal to both the first and second directions, and a first value, which is a color value or a brightness value. Furthermore, the first extraction unit extracts, from the plurality of first points, first points whose second coordinates are within a first range, whose third coordinates are within a second range, and whose first values ​​are within a third range as second points representing the top surface of the first rail. As a result, the first extraction unit acquires second point cloud data including first coordinates, second coordinates, and third coordinates for each of the multiple second points representing the top surface of the first rail. Thereafter, the first selection unit selects, from the multiple second points, second points whose first coordinates are within a fourth range centered on the second value in the first direction as third points. Then, the first selection unit executes a first calculation process based on the first coordinates, second coordinates, and third coordinates for each of the selected multiple third points to calculate a fourth point whose first coordinate is the second value and is located on the center line of the top surface of the first rail. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-141444 Summary of the Invention [Problem to be solved by the invention]

[0005] When obtaining information such as XYZ coordinates corresponding to the position of rails from images, the information can be obtained by performing stereo matching using the principles of triangulation on stereo images or multiple images of the target object. In this process, matching incorrect corresponding points between images can result in errors. However, because rails have few features, there is a risk of matching incorrect corresponding points, such as matching rails with other parts. As a result, it has become difficult to improve the accuracy of obtaining information such as XYZ coordinates from images.

[0006] Therefore, one object of the present disclosure is to provide a location information acquisition device, a location information acquisition method, and a program that can solve the above-mentioned problems. [Means for solving the problem]

[0007] In order to achieve this purpose, the location information acquisition device in the present disclosure includes: an extraction unit that extracts the rails as extraction targets from each of a plurality of image data sets that include the rails on which the railway vehicle runs, each image data set being captured using a different imaging device of the same scene; an acquisition unit that acquires position information corresponding to each point in data representing the shape of the object surface as a three-dimensional point cloud by performing a stereo matching process that matches points corresponding to the rails with each other while referring to the extraction result by the extraction unit; have The structure is as follows.

[0008] Further, the location information acquisition method according to the present disclosure includes: The information processing device extracting the rails as an extraction target from each of a plurality of image data of the same scene including the rails on which the railway vehicle runs, each of which is acquired using a different imaging device; By performing stereo matching processing to match the points corresponding to the rails with reference to the extraction results, position information corresponding to each point in the data representing the shape of the object surface as a 3D point cloud is obtained. The structure is as follows.

[0009] In addition, the program in this disclosure In the information processing device, extracting the rails as an extraction target from each of a plurality of image data of the same scene including the rails on which the railway vehicle runs, each of which is acquired using a different imaging device; By performing stereo matching processing to match the points corresponding to the rails with reference to the extraction results, position information corresponding to each point in the data representing the shape of the object surface as a 3D point cloud is obtained. It is a program for realizing the processing. [Effects of the Invention]

[0010] According to the above-described configurations, it is possible to improve the accuracy when acquiring information such as XYZ coordinates based on an image. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an overview of an acquisition system according to the present disclosure. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of an acquisition device. [Figure 3] FIG. 10 is a diagram illustrating an example of a processing region. [Figure 4] FIG. 10 is a diagram illustrating an example of an extraction process. [Figure 5] FIG. 10 is a diagram illustrating an example of an extraction process. [Figure 6] FIG. 10 is a diagram illustrating an example of an extraction process. [Figure 7] FIG. 10 is a diagram for explaining an example of obtaining three-dimensional information. [Figure 8] FIG. 10 is a diagram illustrating an example of noise removal processing. [Figure 9] FIG. 10 is a diagram illustrating an example of noise removal processing. [Figure 10] FIG. 10 is a diagram illustrating an example of noise removal processing. [Figure 11] FIG. 10 is a diagram illustrating an example of fitting processing. [Figure 12] 10 is a flowchart illustrating an example of the operation of the acquisition device. [Figure 13] FIG. 2 is a diagram illustrating an example of the hardware configuration of a second acquisition device according to the present disclosure. [Figure 14] FIG. 2 is a block diagram illustrating an example of the configuration of an acquisition device. [Figure 15] 10 is a flowchart illustrating an example of the operation of the acquisition device. DETAILED DESCRIPTION OF THE INVENTION

[0012] [First embodiment] An example configuration of an acquisition system 100 in the present disclosure will be described with reference to FIGS. 1 to 12. FIG. 1 is a diagram illustrating an overview of the acquisition system 100. FIG. 2 is a block diagram illustrating an example configuration of an acquisition device 200. FIG. 3 is a diagram illustrating an example of a processing region. FIGS. 4 to 6 are diagrams illustrating an example of extraction processing. FIG. 7 is a diagram illustrating an example of acquisition of information such as XYZ coordinates. FIGS. 8 to 10 are diagrams illustrating an example of noise removal processing. FIG. 11 is a diagram illustrating an example of fitting processing. FIG. 12 is a flowchart illustrating an example operation of the acquisition device 200. Note that in the present disclosure, the drawings may be associated with one or more embodiments.

[0013] In a first embodiment of the present disclosure, an acquisition system 100 is described that acquires information such as XYZ coordinates indicating the position of the rail R on which the rail vehicle runs from image data acquired by an imaging device such as a camera 310 or a camera 320 installed at the front of the rail vehicle. As will be described later, when acquiring information such as XYZ coordinates, which are position information, the acquisition system 100 extracts a predetermined extraction target, such as the rail R, from the image data. For example, the acquisition system 100 extracts the extraction target using one or a combination of a method focusing on the RGB (Red, Green, Blue) values ​​of the rail R, a method using machine learning such as semantic segmentation, and a method using edge extraction. The acquisition system 100 then acquires information such as the XYZ coordinates by performing stereo matching processing with reference to the extracted results. In this case, the acquisition system 100 may acquire only point cloud information corresponding to the rail R that can be identified using the extracted results, or may acquire point cloud information corresponding to the rail R that can be identified using the extracted results and also acquire point cloud information corresponding to points other than the rail R. For example, as described above, the acquisition system 100 performs preprocessing to extract the extraction target before acquiring information such as XYZ coordinates, and can then recognize the rail R, which is the extraction target, and perform stereo matching processing or the like to acquire information.

[0014] In the present disclosure, information such as XYZ coordinates acquired by the acquisition system 100 includes values ​​indicating the X, Y, and Z coordinates of each point in the data representing the shape of the object surface as a three-dimensional point cloud. For example, the X coordinate may indicate the coordinate in the X-axis direction, which is the width direction of the rail R. The Y coordinate may indicate the coordinate in the Y-axis direction, which is the longitudinal direction of the rail R and perpendicular to the X-axis. The Z coordinate may indicate the coordinate in the Z-axis direction, which is the height direction of the rail R and perpendicular to the X-axis and Y-axis. In other words, the acquisition system 100 can acquire information such as XYZ coordinates of each point constituting the three-dimensional point cloud data, such that the XY plane is the horizontal plane and the Z-axis direction is directed vertically upward. The granularity of each point constituting the three-dimensional point cloud data may be set arbitrarily. The information acquired by the acquisition system 100 may also include information indicating the three-dimensional position of each point other than those exemplified above. The information acquired by the acquisition system 100 may also include information other than XYZ coordinates, such as color information.

[0015] Fig. 1 shows an example of the configuration of an acquisition system 100. Referring to Fig. 1, the acquisition system 100 has cameras 310 and 320 which are imaging devices, and an acquisition device 200 (position information acquisition device) which is an information processing device. As shown in Fig. 1, the cameras 310 and 320 and the acquisition device 200 are connected to each other via wire or wirelessly so as to be able to communicate with each other.

[0016] Camera 310 and camera 320 are imaging devices each installed at a predetermined position on the railway vehicle. For example, camera 310 and camera 320 may be stereo cameras installed parallel to each other at the same height at a predetermined position in front of the railway vehicle. Camera 310 and camera 320 acquire image data at the same time as the railway vehicle travels on rail R, thereby enabling them to acquire image data of the same scene including rail R on which the railway vehicle travels from different positions. Camera 310 and camera 320 may each acquire time-series image data at predetermined intervals.

[0017] Acquisition device 200 is an information processing device that acquires information such as XYZ coordinates, which are positional information indicating the position of rail R, from image data acquired by cameras 310 and 320. Fig. 2 shows an example configuration of acquisition device 200. Referring to Fig. 2, acquisition device 200 has, as main components, for example, an operation input unit 210, a screen display unit 220, a communication I / F unit 230, a storage unit 240, and an arithmetic processing unit 250.

[0018] 2 illustrates an example in which the functions of the acquisition device 200 are realized using one information processing device. However, at least some of the functions of the acquisition device 200 may be realized using multiple information processing devices, for example, on the cloud. Furthermore, the acquisition device 200 may not include some of the components exemplified above, such as not having the operation input unit 210 or the screen display unit 220, or may have a configuration other than those exemplified above.

[0019] The operation input unit 210 is composed of operation input devices such as a keyboard, a mouse, etc. The operation input unit 210 detects operations by an operator operating the acquisition device 200 and outputs the operations to the calculation processing unit 250.

[0020] The screen display unit 220 is composed of a screen display device such as a liquid crystal display, an organic EL (electro-luminescence) display, etc. The screen display unit 220 can display various information stored in the storage unit 240 on the screen in response to instructions from the arithmetic processing unit 250.

[0021] The communication I / F unit 230 is composed of a data communication circuit, etc. The communication I / F unit 230 performs data communication with the camera 310, the camera 320, and other external devices connected via a communication line.

[0022] The storage unit 240 is a storage device such as a hard disk or memory. The storage unit 240 stores processing information and a program 244 required for various processes in the arithmetic processing unit 250. The program 244 is read into the arithmetic processing unit 250 and executed to realize various processing units. The program 244 is read in advance from an external device or recording medium via a data input / output function such as the communication I / F unit 230, and is stored in the storage unit 240. Main information stored in the storage unit 240 includes, for example, processing region information 241, image information 242, and three-dimensional information 243.

[0023] Processing area information 241 is information indicating an area where processing to extract an extraction target is performed from image data acquired by camera 310 or camera 320. For example, processing area information 241 is acquired in advance by a method such as accepting input using operation input unit 210 or accepting input from an external device via communication I / F unit 230, and is stored in storage unit 240.

[0024] For example, if the cameras 310 and 320 are fixed at fixed positions on the railway vehicle, it is possible to roughly limit the positions in the image where the rails R appear. Therefore, as shown in Fig. 3, the processing area information 241 indicates, as an area to be processed, an area in the image data acquired by the cameras 310 and 320 where the rails R, which are the extraction target, may be present. This allows the target extraction unit 253, which will be described later, to perform processing to extract the extraction target from the area identified in accordance with the processing area information 241 from the entire image data.

[0025] The area where processing is performed, indicated by the processing area information 241, may be specified in advance using any method. For example, the area where processing is performed may be specified by specifying an area where rail R is likely to be captured using image data acquired during past travel with the camera installed at a fixed position, etc. Furthermore, the processing area information 241 may include only one piece of information indicating the area where processing is performed, or may include multiple pieces of information. For example, the processing area information 241 may include information indicating the area where processing is performed in a more detailed state, such as by route or by route characteristics, or by specific time or location.

[0026] Image information 242 includes image data acquired by camera 310 and camera 320. Image information 242 may include time-series image data acquired by camera 310 and time-series image data acquired by camera 320. For example, image information 242 is updated in response to image acquisition unit 251 (described later) acquiring image data from camera 310 and camera 320.

[0027] The three-dimensional information 243 includes values ​​indicating the XYZ coordinates (X coordinate, Y coordinate, Z coordinate) of each point in data that represents the shape of the object surface as a three-dimensional point cloud. The three-dimensional information 243 may include the XYZ coordinates of each point corresponding to the rail R in the entire image data, or may include the XYZ coordinates of a point cloud other than those exemplified above. The three-dimensional information 243 is updated according to the results of processing by the three-dimensional information acquisition unit 254, which will be described later. Furthermore, the three-dimensional information 243 can be updated according to the results of processing by the noise removal unit 255 and the fitting unit 256, which will be described later.

[0028] The arithmetic processing unit 250 has an arithmetic device such as a CPU (Central Processing Unit) and its peripheral circuits. The arithmetic processing unit 250 reads and executes a program 244 from the storage unit 240, thereby realizing various processing functions by causing the above hardware and the program 244 to work together. Major processing units realized by the arithmetic processing unit 250 include, for example, an image acquisition unit 251, a processing region identification unit 252, a target extraction unit 253, a three-dimensional information acquisition unit 254, a noise removal unit 255, a fitting unit 256, and an output unit 257.

[0029] In addition, the arithmetic processing unit 250 may have a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof, instead of the above-mentioned CPU.

[0030] The image acquisition unit 251 acquires image data from the camera 310, the camera 320, etc. The image acquisition unit 251 also stores the acquired image data as image information 242 in the storage unit 240.

[0031] The processing area specifying unit 252 specifies an area in the image data acquired by the camera 310 or the camera 320 where the target extraction unit 253 performs processing to extract an extraction target. For example, the processing area specifying unit 252 can specify an area where the target extraction unit 253 performs processing to extract an extraction target by referring to the processing area information 241.

[0032] The target extraction unit 253 performs processing to extract the rail R, which is the extraction target, from the region of the image data identified by the processing region identification unit 252. The target extraction unit 253 can perform the extraction processing on each piece of image data included in the image information 242. In other words, the target extraction unit 253 can perform extraction processing on image data acquired by the camera 310, and can also perform extraction processing on image data acquired by the camera 320. For example, the target extraction unit 253 extracts the extraction target by using any one or a combination of a method focusing on the RGB values ​​of the rail R, a method using machine learning such as semantic segmentation, and a method using edge extraction.

[0033] For example, the target extraction unit 253 extracts, from among the points included in the processing area, points whose RGB values ​​satisfy a predetermined condition, such as RGB values ​​within a predetermined range, as a point cloud corresponding to rail R. For example, the target extraction unit 253 can extract points whose R values ​​fall within a predetermined range, whose G values ​​fall within a predetermined range, and whose B values ​​fall within a predetermined range, as a point cloud corresponding to rail R. Note that the ranges of each RGB value may be adjusted arbitrarily based on past results, etc., and the ranges of R values, G values, and B values ​​may be different from each other. For example, RGB values ​​may vary depending on the lighting conditions and camera settings at the time of shooting. For this reason, it is desirable to set the range of RGB values ​​to a certain extent.

[0034] Furthermore, if the range of RGB values ​​extracted as the point cloud corresponding to rail R is set too wide, there are many cases where point clouds other than rail R are erroneously extracted. Therefore, as a method for setting the range of RGB values, ranges may be set for each of the R, G, and B values, or alternatively, ranges may be set that take advantage of the strong correlation between the RGB values ​​of the point cloud corresponding to rail R. For example, as shown in FIG. 4, it has been confirmed that there is a strong correlation between the R and G values ​​at points corresponding to rail R. Furthermore, as shown in FIG. 5, it has been confirmed that there is also a strong correlation between the B and G values ​​at points corresponding to rail R. Therefore, the target extraction unit 253 can extract, from among the points included in the processing area, points whose R and G values ​​lie within a range satisfying a predetermined relationship as exemplified in FIG. 4 and whose B and G values ​​lie within a range satisfying a predetermined relationship as exemplified in FIG. 5 as the point cloud corresponding to rail R.

[0035] Specifically, for example, the target extraction unit 253 can determine that the R value and the G value satisfy a predetermined relationship when the R value is equal to or greater than the value of A×G value+C and equal to or less than the value of D×G value+E. Furthermore, the target extraction unit 253 can determine that the B value and the G value satisfy a predetermined relationship when the B value is equal to or greater than the value of F×G value+H and equal to or less than the value of I×G value+J. The values ​​of A, C, D, E, F, H, I, and J may be set arbitrarily based on past results, such as those illustrated in FIGS. 4 and 5. Furthermore, the value of D×G value+E is assumed to be greater than the value of A×G value+C, and the value of I×G value+J is assumed to be greater than the value of F×G value+H.

[0036] For example, as described above, the target extraction unit 253 can extract points where the relationship between the R value and the G value satisfies a condition and the relationship between the B value and the G value satisfies a predetermined condition as a point cloud corresponding to the rail R. As described above, the target extraction unit 253 may also extract points where each of the RGB values ​​is within a predetermined range, where the relationship between the R value and the G value satisfies a condition, and where the relationship between the B value and the G value satisfies a predetermined condition as a point cloud corresponding to the rail R. Note that the ranges of the RGB values, the relationship between the R value and the G value, the relationship between the B value and the G value, etc. may be configured to vary depending on the external environment, such as the time of day and sunlight conditions.

[0037] Furthermore, the object extraction unit 253 may extract a point cloud corresponding to a rail R from among the points included in the processing region using a machine learning method such as semantic segmentation. For example, the object extraction unit 253 may obtain a result in which each pixel of the image data is labeled as a rail R, as shown in FIG. 6, by inputting the image data into a pre-trained model. For example, the object extraction unit 253 may extract points labeled as a rail R as a point cloud corresponding to the rail R by referring to the result. Note that the present disclosure does not particularly limit the model training method. The object extraction unit 253 may perform the above processing using a model trained using any method that enables labeling such as a rail R.

[0038] The target extraction unit 253 may also extract a point cloud corresponding to the rail R from among the points included in the processing area by, for example, performing edge extraction of the rail R. For example, the target extraction unit 253 can extract a point cloud corresponding to the rail R by performing edge extraction processing in response to changes in brightness values ​​of the image data. In this case, the rail R has the characteristic of continuously existing in a straight or curved line. Therefore, the target extraction unit 253 may extract only points that satisfy a continuity condition from among the points extracted by edge extraction as a point cloud corresponding to the rail R. The target extraction unit 253 may determine whether continuity is satisfied in accordance with any condition. For example, the target extraction unit 253 may check whether continuity is satisfied in accordance with the distance between each point extracted by edge extraction, or may check whether continuity is satisfied by checking whether points exist within a frame set to include at least a predetermined percentage of the extracted points. The target extraction unit 253 may also check whether continuity is satisfied using a method other than the above examples.

[0039] For example, the target extraction unit 253 may extract the rails R, which are the extraction target, by using any one of the methods exemplified above, or a combination thereof. Furthermore, for example, inside a tunnel, conditions change, so it may be difficult to extract the rails R by focusing on RGB values. Therefore, the target extraction unit 253 may be configured to acquire information indicating whether or not the vehicle is inside a tunnel, and the external environment, such as the weather and time of day, and change the method for extracting the rails R depending on the acquired information.

[0040] The three-dimensional information acquisition unit 254 references the extraction results by the object extraction unit 253 and performs stereo matching processing using the principles of triangulation to acquire information such as XYZ coordinates corresponding to each point in data representing the shape of the object surface as a three-dimensional point cloud. In other words, the three-dimensional information acquisition unit 254 references the extraction results by the object extraction unit 253 and performs stereo matching processing to match points having the same feature amounts in paired image data, thereby acquiring information such as XYZ coordinates, which are positional information. In this case, the three-dimensional information acquisition unit 254 may acquire only information on a point cloud corresponding to a rail R that can be identified using the extraction results, or may acquire information on a point cloud corresponding to a rail R that can be identified using the extraction results and information on a point cloud corresponding to areas other than the rail R. The three-dimensional information acquisition unit 254 also stores the acquired information in the storage unit 240 as three-dimensional information 243.

[0041] For example, as described above, cameras 310 and 320 are installed parallel to each other at the same height at a predetermined position in front of the railway vehicle. Therefore, as shown in FIG. 7, when image data is acquired simultaneously using cameras 310 and 320, the center of the image data acquired by camera 310 corresponds to the center of the image data acquired by camera 320. Under such circumstances, the three-dimensional information acquisition unit 254 refers to the extraction result by the target extraction unit 253 and performs stereo matching processing to match points corresponding to the rail R or points corresponding to areas other than the rail R. This allows the three-dimensional information acquisition unit 254 to acquire information on a point cloud corresponding to the rail R and information on other point clouds, without risk of erroneously matching the rail R with other areas.

[0042] The noise removal unit 255 removes position information determined to be noise from the position information corresponding to each point acquired by the three-dimensional information acquisition unit 254. For example, the noise removal unit 255 can remove noise by focusing on the continuity of the rail R. Furthermore, the noise removal unit 255 can update the three-dimensional information 243 according to the result of the noise removal process, for example, by deleting information such as the position information removed as noise from the three-dimensional information 243.

[0043] For example, the noise removal unit 255 can remove noise according to the positional relationship on the XY plane in the point cloud acquired by the three-dimensional information acquisition unit 254. As an example, as shown in FIG. 8, the noise removal unit 255 estimates the center position of the rail R from the density on the XY plane of the point cloud corresponding to the rail R acquired by the three-dimensional information acquisition unit 254. Then, the noise removal unit 255 removes, as noise, information corresponding to points determined to correspond to the rail R whose X coordinates or Y coordinates are farther away from the estimated center position of the rail R than a predetermined threshold value. Note that the threshold value for determining noise may be set arbitrarily. Furthermore, the noise removal unit 255 may remove noise using a method other than the above-mentioned example, such as the method of setting a frame.

[0044] 9, the noise removal unit 255 may remove noise by smoothing the image by performing a projection process within a range that maintains the continuity of the rail R on the XY plane. For example, the noise removal unit 255 estimates the shape of the rail R, such as its outer shape, on the XY plane from the information on the point cloud corresponding to the rail R. Then, the noise removal unit 255 can remove noise by moving the XY coordinates of points that are away from the estimated outer shape or tangent of the rail R by a predetermined threshold or more onto the estimated outer shape.

[0045] Furthermore, the noise removal unit 255 may remove noise according to the positional relationship on the Z axis of the point cloud acquired by the three-dimensional information acquisition unit 254. For example, the XYZ coordinates of the point cloud acquired by the three-dimensional information acquisition unit 254 generally correspond to the shape of the object surface. Furthermore, the rails R are laid generally horizontally. Therefore, it is assumed that the point cloud corresponding to the rails R is concentrated at a specific position in the Z axis direction. Therefore, as shown in FIG. 10, the noise removal unit 255 can determine that the point cloud at which positions on the Z axis are concentrated is the point cloud corresponding to the rails R, and can remove, as noise, information corresponding to points that are away from the concentrated position on the Z axis by a predetermined threshold or more.

[0046] For example, the noise removal unit 255 can remove noise by using any one of the methods exemplified above or a combination of the methods.

[0047] The fitting unit 256 acquires additional information that is insufficient to be acquired by the three-dimensional information acquisition unit 254, according to the positional relationship on the XYZ coordinates in the point cloud corresponding to the rail R. For example, the fitting unit 256 acquires coordinates (points) corresponding to a predetermined position on the GC-side (inside gauge) side surface of the rail R by fitting the position on the XYZ coordinates in the point cloud corresponding to the rail R with a pre-stored head shape of the rail R. Furthermore, the fitting unit 256 can store the additionally acquired information in the storage unit 240 as three-dimensional information 243.

[0048] For example, when acquiring a 3D shape using image analysis techniques such as stereo matching, only information such as the XYZ coordinates of parts captured in multiple image data can be acquired, and information on parts that are shaded cannot be acquired. On the other hand, when inspecting track irregularities, the XYZ coordinates of predetermined parts of the rail R are required, and it is desirable to have information on a predetermined position on the GC-side side of the rail R. Therefore, the fitting unit 256 acquires information on the above points by focusing on the continuity and head shape of the rail R. For example, as shown in FIG. 11 , the fitting unit 256 fits the XYZ coordinate positions in the point cloud corresponding to the rail R with the pre-stored head shape of the rail R to find the state that best matches. Then, based on the fitting results, the fitting unit 256 acquires the XYZ coordinates of the necessary points in the pre-stored head shape of the rail R that correspond to the predetermined position on the GC side of the rail R.

[0049] The output unit 257 outputs the three-dimensional information 243 and the like. The output unit 257 may display the information included in the three-dimensional information 243 on the screen display unit 220, or may transmit the information to an external device via the communication I / F unit 230 or the like. The output unit 257 may output information other than the above-mentioned examples, such as the image information 242.

[0050] The above is an example of the configuration of the acquisition device 200. Next, an example of the operation of the acquisition device 200 will be described with reference to FIG.

[0051] Fig. 12 is a flowchart showing an example of the operation of acquisition device 200. Referring to Fig. 12, processing area specification unit 252 refers to processing area information 241 to specify an area in the image data acquired by camera 310 or camera 320 where target extraction unit 253 will perform processing to extract an extraction target (step S101).

[0052] The target extraction unit 253 performs processing to extract the rail R, which is the extraction target, from the area of ​​the image data identified by the processing area identification unit 252 (step S102). The target extraction unit 253 can extract the extraction target by using any one of a method focusing on the RGB values ​​of the rail R, a method using machine learning such as semantic segmentation, and a method using edge extraction, or a combination thereof.

[0053] The three-dimensional information acquisition unit 254 refers to the extraction result by the object extraction unit 253 and performs stereo matching processing using the principle of triangulation to acquire information such as XYZ coordinates corresponding to each point in the data representing the shape of the object surface as a three-dimensional point cloud (step S103). At this time, the three-dimensional information acquisition unit 254 may acquire only information on the point cloud corresponding to the rail R that can be identified using the extraction result, or may acquire information on the point cloud corresponding to the rail R that can be identified using the extraction result and also acquire information on the point cloud corresponding to points other than the rail R.

[0054] The noise removal unit 255 removes noise included in each point acquired by the three-dimensional information acquisition unit 254 (step S104). For example, the noise removal unit 255 can remove noise by focusing on the continuity of the rail R.

[0055] The fitting unit 256 acquires additional information that is insufficient to be acquired by the three-dimensional information acquisition unit 254, according to the positional relationship on the XYZ coordinates in the point cloud corresponding to the rail R (step S105). For example, the fitting unit 256 acquires coordinates corresponding to a predetermined position on the GC-side side surface of the rail R by fitting the position on the XYZ coordinates in the point cloud corresponding to the rail R with the pre-stored head shape of the rail R.

[0056] The above is an example of the operation of the acquisition device 200.

[0057] As described above, the acquisition device 200 has the object extraction unit 253 and the three-dimensional information acquisition unit 254. With this configuration, the three-dimensional information acquisition unit 254 can acquire information such as XYZ coordinates corresponding to each point in the data representing the shape of the object surface as a three-dimensional point cloud, by referring to the extraction result by the object extraction unit 253. This allows the three-dimensional information acquisition unit 254 to acquire information on the point cloud corresponding to the rail R and information on other point clouds, without the risk of erroneously matching the rail R with other locations. As a result, the accuracy of information acquisition can be improved.

[0058] The configuration of the acquisition device 200 is not limited to the example shown in Fig. 2. For example, the acquisition device 200 may not have the processing region specifying unit 252. Furthermore, the acquisition device 200 may not have at least some of the noise removal unit 255, the fitting unit 256, etc. For example, as described above, the acquisition device 200 may be configured with some of the configurations shown in Fig. 2.

[0059] [Second embodiment] Next, a second acquisition device 400 in the present disclosure will be described with reference to Fig. 13 to Fig. 15. Fig. 13 is a diagram illustrating an example of the hardware configuration of the acquisition device 400. Fig. 14 is a block diagram illustrating an example of the configuration of the acquisition device 400. Fig. 15 is a flowchart illustrating an example of the operation of the acquisition device 400.

[0060] In the second embodiment of the present disclosure, an acquisition device 400 (position information acquisition device) will be described, which is an information processing device that acquires information such as XYZ coordinates indicating the position of rails from image data. Fig. 13 shows an example of the hardware configuration of the acquisition device 400. Referring to Fig. 13, the acquisition device 400 has, as an example, the following hardware configuration. ·CPU(Central Processing Unit)401(Arithmetic unit) ROM (Read Only Memory) 402 (storage device) RAM (Random Access Memory) 403 (storage device) Programs 404 loaded into RAM 403 A storage device 405 for storing the program group 404 A drive device 406 that reads and writes data from a recording medium 410 outside the information processing device A communication interface 407 for connecting to a communication network 411 outside the information processing device Input / output interface 408 for inputting and outputting data Bus 409 connecting each component

[0061] 14 by the CPU 401 acquiring the program group 404 and executing it. The program group 404 is stored in advance in the storage device 405 or the ROM 402, for example, and is loaded into the RAM 403 or the like by the CPU 401 for execution as needed. The program group 404 may be supplied to the CPU 401 via the communication network 411, or may be stored in advance in the recording medium 410, and the drive device 406 may read out the programs and supply them to the CPU 401.

[0062] 13 shows an example of the hardware configuration of the acquisition device 400. The hardware configuration of the acquisition device 400 is not limited to the above-described case. For example, the acquisition device 400 may be configured with only a part of the above-described configuration, such as not including the drive device 406. Furthermore, the CPU 401 may be a GPU or the like exemplified in the first embodiment.

[0063] The extraction unit 421 extracts rails, which are extraction targets, from each of a plurality of image data sets that capture the same scene, including rails on which railroad vehicles run, using different imaging devices. For example, the extraction unit 421 may extract the extraction targets by using one or a combination of a method that focuses on the RGB values ​​of the rails, a method that uses machine learning such as semantic segmentation, and a method that uses edge extraction.

[0064] The acquisition unit 422 acquires position information corresponding to each point in the data representing the shape of the object surface as a three-dimensional point cloud by performing stereo matching processing with reference to the extraction result by the extraction unit 421. For example, the acquisition unit 422 may acquire XYZ coordinates according to the position of the rail as the position information.

[0065] The above is an example of the configuration of the acquisition device 400. Next, an example of the operation of the acquisition device 400 will be described with reference to FIG.

[0066] Fig. 15 is a flowchart showing an example of the operation of the acquisition device 400. Referring to Fig. 15, the extraction unit 421 extracts rails, which are the extraction target, from each of a plurality of image data sets that capture the same scene, including the rails on which a railway vehicle runs, using different imaging devices (step S201).

[0067] The acquisition unit 422 performs stereo matching processing by referring to the extraction result by the extraction unit 421, thereby acquiring position information corresponding to each point in the data representing the shape of the object surface as a three-dimensional point cloud (step S202).

[0068] As described above, the acquisition device 400 includes the extraction unit 421 and the acquisition unit 422. With this configuration, the acquisition unit 422 performs stereo matching processing by referring to the extraction result by the extraction unit 421, thereby acquiring position information corresponding to each point in the data representing the shape of the object surface as a 3D point cloud. This allows the acquisition unit 422 to acquire position information of the point cloud corresponding to the rails, etc., without the risk of erroneously matching the rails with other locations. As a result, the accuracy of acquiring position information can be improved.

[0069] The acquisition device 400 described above can be realized by incorporating a predetermined program into an information processing device such as the acquisition device 400. Specifically, a program according to another embodiment of the present disclosure is a program for implementing processing in an information processing device such as the acquisition device 400 to extract rails, which are the extraction target, from each of a plurality of image data sets that are acquired using different imaging devices of the same scene, including rails on which a railway vehicle runs, and perform stereo matching processing that matches points corresponding to the rails with reference to the extraction results, thereby acquiring position information corresponding to each point in data that represents the shape of the surface of an object as a three-dimensional point cloud.

[0070] Furthermore, the acquisition method executed by an information processing device such as the acquisition device 400 described above is a method in which the information processing device such as the acquisition device 400 extracts the rails to be extracted from each of a plurality of image data captured by different imaging devices of the same scene, including the rails on which a railway vehicle runs, and performs a stereo matching process that matches points corresponding to the rails with reference to the extraction results, thereby acquiring positional information corresponding to each point in data that represents the shape of the object surface as a three-dimensional point cloud.

[0071] A program having the above-described configuration, a computer-readable recording medium having the program recorded thereon, an acquisition method, etc. can achieve the same functions and effects as the above-described acquisition device 400, and therefore can achieve the above-described objective of the present disclosure.

[0072] <Additional Notes> A part or all of the above-described embodiments can be described as follows: The following provides an overview of the location information acquisition device and the like according to the present disclosure. However, the present disclosure is not limited to the following configuration.

[0073] (Appendix 1) an extraction unit that extracts the rails as extraction targets from each of a plurality of image data sets that include the rails on which the railway vehicle runs, each image data set being captured using a different imaging device of the same scene; an acquisition unit that acquires position information corresponding to each point in data representing the shape of the object surface as a three-dimensional point cloud by performing a stereo matching process that matches points corresponding to the rails with each other while referring to the extraction result by the extraction unit; have Location information acquisition device. (Appendix 2) 10. The location information acquisition device according to claim 1, The extraction unit extracts points, the RGB (Red, Green, Blue) values ​​of which satisfy predetermined conditions, as the rails. Location information acquisition device. (Appendix 3) 3. The location information acquisition device according to claim 2, The extraction unit extracts, as the rail, a point where the R value and the G value of the RGB values ​​are within a range satisfying a predetermined relationship and the B value and the G value are within a range satisfying a predetermined relationship. Location information acquisition device. (Appendix 4) 10. The location information acquisition device according to claim 2, wherein: The extraction unit extracts, as the rail, a point where each of the RGB values ​​is within a predetermined range, where the R value and the G value are within a range that satisfies a predetermined relationship, and where the B value and the G value are within a range that satisfies a predetermined relationship. Location information acquisition device. (Appendix 5) 10. The location information acquisition device according to claim 1, wherein: a noise removal unit that removes position information determined to be noise from the position information corresponding to the rail acquired by the acquisition unit; Location information acquisition device. (Appendix 6) 6. The location information acquisition device according to claim 5, the acquiring unit acquires XYZ coordinates according to the position of the rail as position information, The noise removal unit removes noise from the position information corresponding to the rail acquired by the acquisition unit according to a positional relationship on an XY plane. Location information acquisition device. (Appendix 7) 10. The location information acquisition device according to claim 5 or 6, the acquiring unit acquires XYZ coordinates according to the position of the rail as position information, The noise removal unit removes noise from the position information corresponding to the rail acquired by the acquisition unit according to a positional relationship on the Z axis. Location information acquisition device. (Appendix 8) 10. A location information acquisition device according to claim 1, wherein: and a fitting unit that acquires position information corresponding to a predetermined position on the rail by fitting the position information corresponding to the rail acquired by the acquisition unit to a pre-stored rail head shape. Location information acquisition device. (Appendix 9) The information processing device extracting the rails as an extraction target from each of a plurality of image data of the same scene including the rails on which the railway vehicle runs, each of which is acquired using a different imaging device; By performing stereo matching processing to match the points corresponding to the rails with reference to the extraction results, position information corresponding to each point in the data representing the shape of the object surface as a 3D point cloud is obtained. How to obtain location information. (Appendix 10) In the information processing device, extracting the rails as an extraction target from each of a plurality of image data of the same scene including the rails on which the railway vehicle runs, each of which is acquired using a different imaging device; By performing stereo matching processing to match the points corresponding to the rails with reference to the extraction results, position information corresponding to each point in the data representing the shape of the object surface as a 3D point cloud is obtained. A program to realize the processing.

[0074] Note that some or all of the configurations described in Supplementary Notes 2 to 8 that are dependent on the location information acquisition device described in Supplementary Note 1 may also be dependent in a similar dependent relationship on the location information acquisition method described in Supplementary Note 9 and the program described in Supplementary Note 10. Furthermore, not limited to Supplementary Notes 9 and 10, some or all of the configurations described as Supplements may also be dependent on various hardware, software, various recording means for recording software, or systems within the scope of the above-mentioned embodiments.

[0075] The programs described in the above embodiments and appendices may be stored in a storage device or a computer-readable recording medium, such as a portable medium such as a flexible disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0076] Although the present disclosure has been described above with reference to the above-described embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate. [Explanation of symbols]

[0077] 100 Acquisition System 200 Acquisition device 210 Operation input section 220 Screen display section 230 Communication I / F section 240 Storage section 241 Processing area information 242 Image Information 243 3D Information 244 Programs 250 Processing Unit 251 Image Acquisition Unit 252 Processing area specification unit 253 Target Extraction Unit 254 3D information acquisition unit 255 Noise Reduction Section 256 Fitting section 257 output section 321 Camera 322 Camera 400 Acquisition device 401 CPU 402 ROM 403 RAM 404 Programs 405 Storage device 406 Drive Unit 407 Communication Interface 408 Input / Output Interface 409 Bus 410 Recording Media 411 Communication Network 421 Extraction part 422 Acquisition Department

Claims

1. an extraction unit that extracts the rails as extraction targets from each of a plurality of image data sets that include the rails on which the railway vehicle runs, each image data set being captured using a different imaging device of the same scene; an acquisition unit that acquires position information corresponding to each point in data representing the shape of the object surface as a three-dimensional point cloud by performing a stereo matching process that matches points corresponding to the rails with each other while referring to the extraction result by the extraction unit; have Location information acquisition device.

2. 2. The location information acquisition device according to claim 1, The extraction unit extracts points, the RGB (Red, Green, Blue) values ​​of which satisfy predetermined conditions, as the rails. Location information acquisition device.

3. 3. The location information acquisition device according to claim 2, The extraction unit extracts, as the rail, a point where the R value and the G value of the RGB values ​​are within a range satisfying a predetermined relationship and where the B value and the G value are within a range satisfying a predetermined relationship. Location information acquisition device.

4. 3. The location information acquisition device according to claim 2, The extraction unit extracts, as the rail, a point where each of the RGB values ​​is within a predetermined range, where the R value and the G value are within a range that satisfies a predetermined relationship, and where the B value and the G value are within a range that satisfies a predetermined relationship. Location information acquisition device.

5. 2. The location information acquisition device according to claim 1, a noise removal unit that removes position information determined to be noise from the position information corresponding to the rail acquired by the acquisition unit; Location information acquisition device.

6. 6. The location information acquisition device according to claim 5, the acquiring unit acquires XYZ coordinates according to the position of the rail as position information, The noise removal unit removes noise from the position information corresponding to the rail acquired by the acquisition unit in accordance with a positional relationship on an XY plane. Location information acquisition device.

7. 6. The location information acquisition device according to claim 5, the acquiring unit acquires XYZ coordinates according to the position of the rail as position information, The noise removal unit removes noise from the position information corresponding to the rail acquired by the acquisition unit in accordance with a positional relationship on the Z axis. Location information acquisition device.

8. 2. The location information acquisition device according to claim 1, and a fitting unit that acquires position information corresponding to a predetermined position on the rail by fitting the position information corresponding to the rail acquired by the acquisition unit to a pre-stored rail head shape. Location information acquisition device.

9. The information processing device extracting the rails as an extraction target from each of a plurality of image data of the same scene including the rails on which the railway vehicle runs, each of which is acquired using a different imaging device; By performing stereo matching processing to match points corresponding to the rails with reference to the extraction results, position information corresponding to each point in the data representing the shape of the object surface as a 3D point cloud is obtained. How to obtain location information.

10. In the information processing device, extracting the rails as an extraction target from each of a plurality of image data of the same scene including the rails on which the railway vehicle runs, each of which is acquired using a different imaging device; By performing stereo matching processing to match points corresponding to the rails with reference to the extraction results, position information corresponding to each point in the data representing the shape of the object surface as a 3D point cloud is obtained. A program to realize the processing.

Images