Information processing device and program

Abstract

To provide an information processing unit and a program which effectively recognize a rail track from images.SOLUTION: According to an embodiment, an information processing unit comprises an image interface, a communication interface and a processor. The image interface acquires taken images including rail tracks and facilities. The communication interface acquires rail track information indicating geodesic coordinates of the rail track and facility information indicating geodesic coordinates of the facilities. The processor specifies facility domain where the facilities appear from the taken image, generates a transformation matrix which makes the geodesic coordinates correspond to the coordinates of the taken image based on coordinates of the facility domain in the taken images and the geodesic coordinates of the facilities, transforms the geodesic coordinates of the rail track into the coordinates of the taken image based on the transformation matrix and specifies coordinates of the rail track in the taken image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing device and a program. [Background technology]

[0002] Some vehicles traveling on railroad tracks use cameras to capture images of the area ahead and detect obstacles. The vehicle recognizes the railroad tracks it will travel on from the images captured by the camera. Based on the recognized railroad tracks, the vehicle identifies the area in the image through which the vehicle will travel. The vehicle then detects obstacles based on images within the identified area.

[0003] There is a demand for technology that can effectively recognize railroad tracks from images, for example, to detect obstacles. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-164013 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to solve the above problem, an information processing device and a program are provided that can effectively recognize railroad tracks from an image. [Means for solving the problem]

[0006] According to an embodiment, an information processing device includes an image interface, a communication interface, and a processor. The image interface acquires a photographed image including a track and equipment. The communication interface acquires track information indicating the geodetic coordinates of the track and equipment information indicating the geodetic coordinates of the equipment. The processor identifies an equipment area in which the equipment is captured from the photographed image, generates a transformation matrix that associates the geodetic coordinates with coordinates in the photographed image based on the coordinates of the equipment area in the photographed image and the geodetic coordinates of the equipment, and transforms the geodetic coordinates of the track into coordinates in the photographed image based on the transformation matrix, thereby identifying the coordinates of the track in the photographed image. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram of a vehicle system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing a control system of the vehicle according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of facility information according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the operation of the vehicle according to the first embodiment. [Figure 5] FIG. 5 is a diagram showing tracks projected by a vehicle according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing feature amounts of a captured image according to the first embodiment. [Figure 7] FIG. 7 is a flowchart showing an example of the operation of the vehicle according to the first embodiment. [Figure 8] FIG. 8 is a schematic diagram of a railroad track on which a vehicle according to the second embodiment travels. [Figure 9] FIG. 9 is a top view of a railroad track on which a vehicle according to the second embodiment travels. [Figure 10] FIG. 10 is a diagram illustrating an example of facility information according to the second embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of the operation of the vehicle according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments will be described with reference to the drawings. (First embodiment) The vehicle provided in the vehicle system according to the embodiment constitutes a train traveling on a railroad track. The vehicle photographs the area ahead so that the railroad track the vehicle is traveling on is included in the photographed area. The vehicle recognizes the railroad track from the photographed image. The vehicle sets a travel area in the photographed image in which the vehicle will travel based on the recognized position of the railroad track, etc. The vehicle detects obstacles that may impede the vehicle's travel based on the set travel area.

[0009] Fig. 1 is a schematic diagram of a vehicle system 100 according to an embodiment. As shown in Fig. 1, the vehicle system 100 includes a vehicle 10 and a host device 20. The vehicle 10 and the host device 20 are connected to each other so that they can communicate with each other.

[0010] The vehicle 10 constitutes a train that runs on a railway line R. The vehicle 10 runs on the railway line R either by operation of a driver or automatically.

[0011] The vehicle 10 includes a housing 1, a camera 2, an antenna 3, and the like.

[0012] The housing 1 forms the outer shape of the vehicle 10. For example, the housing 1 is formed so that it can accommodate people, luggage, etc. inside.

[0013] A camera 2 is installed in front of the housing 1. The camera 2 captures images in the traveling direction of the vehicle 10. The camera 2 also captures images of the track R on which the vehicle 10 is traveling and the surrounding area of the track R.

[0014] For example, the camera 2 is a CCD (Charge Coupled Device) camera. The camera 2 may also be equipped with a light that illuminates the area ahead of the vehicle 10.

[0015] The antenna 3 is an antenna for receiving signals for positioning the vehicle 10. For example, the antenna 3 is an antenna for receiving signals of the Global Navigation Satellite System (GNSS).

[0016] The host device 20 provides various information to the vehicle 10. For example, the host device 20 is a server managed by a railway company.

[0017] The higher-level device 20 transmits to the vehicle 10 facility information relating to facilities (such as utility poles, speed signs, or stop signs) that may be photographed by the camera 2 of the vehicle 10.

[0018] For example, the host device 20 receives a request for facility information from the vehicle 10. The request includes the current coordinates (e.g., latitude and longitude) of the vehicle 10. Upon receiving the request, the host device 20 identifies a facility that the camera 2 may photograph based on the current coordinates included in the request. After identifying the facility, the host device 20 transmits facility information about the identified facility to the vehicle 10 as a response to the request. The facility information will be described in detail later.

[0019] Furthermore, the higher-level device 20 transmits to the vehicle 10 track information indicating the geodetic coordinates (coordinates measured by GNSS or the like) of the track R along which the vehicle 10 is scheduled to travel.

[0020] For example, the host device 20 receives a request for facility information from the vehicle 10. Upon receiving the request, the host device 20 identifies the track R on which the vehicle 10 is traveling. After identifying the track R, the host device 20 transmits a response to the vehicle 10 that includes track information indicating the geodetic coordinates of the identified track R. For example, the track information indicates a sequence of geodetic coordinates (plurality of geodetic coordinates) indicating each point of the track R.

[0021] In addition, the higher-level device 20 transmits limit information indicating the rolling stock gauge and construction gauge of the track R to the vehicle 10.

[0022] Vehicle clearance is the limit range for the size of the vehicle cross section. The construction gauge is the clearance from the tracks within which no structures may be erected. For example, the limit information indicates the rolling stock gauge and construction gauge at each geodetic coordinate of the track R.

[0023] The host device 20 receives a request for limit information from the vehicle 10. Upon receiving the request, the host device 20 transmits a response including the limit information to the vehicle 10.

[0024] Next, the vehicle 10 will be described. Fig. 2 shows an example of the configuration of a vehicle 10 according to an embodiment. Fig. 2 is a block diagram showing the example of the configuration of the vehicle 10. As shown in Fig. 2, the vehicle 10 includes a camera 2, an antenna 3, an information processing device 4, and the like.

[0025] The information processing device 4 identifies the coordinate sequence of the track R from the image captured by the camera 2. The information processing device 4 also detects obstacles that obstruct the travel of the vehicle.

[0026] The information processing device 4 includes a processor 11, a ROM 12, a RAM 13, an NVM 14, a communication unit 15, an operation unit 16, a display unit 17, a camera interface 18, an antenna interface 19, and the like.

[0027] The processor 11, ROM 12, RAM 13, NVM 14, communication unit 15, operation unit 16, display unit 17, camera interface 18, and antenna interface 19 are connected to one another via a data bus or the like. The camera interface 18 is connected to the camera 2. The antenna interface 19 is connected to the antenna 3.

[0028] The vehicle 10 and the information processing device 4 may be provided with other components as needed in addition to the components shown in FIG. 2, or certain components may be excluded from the vehicle 10.

[0029] The processor 11 has the function of controlling the overall operation of the vehicle 10. The processor 11 may include an internal cache and various interfaces. The processor 11 performs various processes by executing programs stored in advance in the internal memory, the ROM 12, or the NVM 14.

[0030] Some of the various functions realized by the processor 11 executing the programs may be realized by hardware circuits. In this case, the processor 11 controls the functions executed by the hardware circuits.

[0031] The ROM 12 is a non-volatile memory that stores in advance control programs, control data, etc. The control programs and control data stored in the ROM 12 are installed in advance in accordance with the specifications of the vehicle 10.

[0032] RAM 13 is a volatile memory. RAM 13 temporarily stores data being processed by processor 11. RAM 13 stores various application programs based on instructions from processor 11. RAM 13 may also store data necessary for executing application programs and execution results of application programs.

[0033] The NVM 14 is a nonvolatile memory to which data can be written and rewritten. The NVM 14 is configured, for example, with a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The NVM 14 stores control programs, applications, and various data according to the operational use of the vehicle 10.

[0034] The communication unit 15 (communication interface) is an interface for transmitting and receiving data to and from the higher-level device 20, etc. For example, the communication unit 15 connects to the higher-level device 20, etc. via a network. For example, the communication unit 15 is an interface that supports wired or wireless LAN (Local Area Network) connection.

[0035] The operation unit 16 receives input of various operations from an operator (for example, a driver, etc.). The operation unit 16 transmits a signal indicating the input operation to the processor 11. The operation unit 16 may be configured as a touch panel.

[0036] The display unit 17 displays image data from the processor 11. For example, the display unit 17 is configured with a liquid crystal monitor. When the operation unit 16 is configured with a touch panel, the display unit 17 may be formed integrally with the operation unit 16.

[0037] The camera interface 18 (image interface) is an interface that connects to the camera 2. The camera interface 18 transmits a signal from the processor 11 to the camera 2. The camera interface 18 also acquires a signal (such as a captured image) from the camera 2 and transmits it to the processor 11.

[0038] The antenna interface 19 is an interface that connects to the antenna 3. The antenna interface 19 measures the current position (geodetic coordinates) of the vehicle 10 based on a signal from the antenna 3 or the like.

[0039] For example, the antenna interface 19 measures the position based on a signal from the antenna 3. After measuring the position, the antenna interface 19 fits the measured position onto a nearby railroad track R based on map information including the position of the railroad track R. The antenna interface 19 obtains the fitted position as the current position of the vehicle 10. The antenna interface 19 transmits the obtained current position to the processor 11.

[0040] The function of the antenna interface 19 may be realized by the processor 11. Furthermore, when the vehicle 10 travels by automatic driving, the information processing device 4 does not need to include the operation unit 16 and the display unit 17.

[0041] Next, a description will be given of functions realized by the vehicle 10. The functions realized by the vehicle 10 are realized by the processor 11 executing a program stored in the internal memory, the ROM 12, the NVM 14, or the like.

[0042] First, the processor 11 has a function of using the camera 2 to capture an image of the area ahead of the vehicle 10.

[0043] The processor 11 starts capturing images with the camera 2 through the camera interface 18. When capturing images starts, the processor 11 acquires captured images (captured images) from the camera 2. The processor 11 acquires the captured images from the camera 2 in real time.

[0044] The processor 11 also has a function of acquiring facility information. For example, the processor 11 acquires the current location of the vehicle 10 using the antenna 3 and the antenna interface 19. After acquiring the current location, the processor 11 generates a request for facility information based on the current location. The request includes the current location. After generating the request, the processor 11 transmits the generated request to the higher-level device 20 via the communication unit 15.

[0045] When the request is transmitted to the host device 20, the processor 11 receives a response including facility information from the host device 20 via the communication unit 15.

[0046] Next, the facility information will be described. The facility information indicates the geodetic coordinates of the facility that may appear in the image captured by the camera 2.

[0047] An example of the configuration of facility information is shown in Fig. 3. As shown in Fig. 3, the facility information stores "ID," "type," "geodetic coordinates," and "positional relationship from the track" in association with each other.

[0048] "ID" indicates an identifier for identifying the facility. Here, "ID" is a numerical value.

[0049] "Type" indicates the type of facility. In this example, "Type" is a utility pole, a stop sign, or a speed sign.

[0050] "Geodetic coordinates" refers to the geodetic coordinates of the facility. Here, "geodetic coordinates" refers to the latitude and longitude of the facility.

[0051] The "positional relationship from the tracks" indicates on which side of the tracks R the facility is located in the captured image. In other words, the "positional relationship from the tracks" indicates that the facility is located on the right or left side of the tracks R.

[0052] The facility information may include other configurations as needed in addition to the configuration shown in Fig. 3, and specific configurations may be excluded from the facility information. The configuration of the facility information is not limited to a specific configuration.

[0053] The processor 11 also has a function of acquiring track information. The processor 11 transmits a request for track information to the higher-level device 20 via the communication unit 15. The processor 11 receives a response including the track information from the higher-level device 20 via the communication unit 15.

[0054] The processor 11 also has a function of recognizing equipment that appears in the captured image.

[0055] The processor 11 recognizes the equipment from the captured image according to a predetermined image recognition algorithm.

[0056] For example, the NVM 14 stores dictionary information for recognizing equipment in advance. For example, the dictionary information may be information about the feature amounts of images showing the equipment, or a network obtained by deep learning.

[0057] The processor 11 recognizes the equipment based on the dictionary information. That is, the processor 11 identifies the coordinates (for example, one vertex coordinate or center coordinate) of the area in which the equipment is captured (equipment area).

[0058] 4 shows an example of an operation of the processor 11 to recognize equipment. 4 shows an example of a captured image. Here, it is assumed that the processor 11 has recognized equipment areas 21 to 25.

[0059] The facility area 21 is an area in which utility poles are photographed as facilities. The facility area 22 is an area in which utility poles are photographed as facilities. The facility area 23 is an area in which speed signs are captured as facilities. The facility area 24 is an area in which a stop sign is captured as a facility. The facility area 25 is an area in which utility poles are photographed as facilities.

[0060] The processor 11 may also recognize the facility based on the facility information. For example, processor 11 may limit the search area for each facility based on the "positional relationship from the tracks" in the facility information. For example, processor 11 may set the search area for a facility whose "positional relationship from the tracks" indicates "right" to the right side of the captured image. Processor 11 may also identify the position of the tracks by image recognition and set the search area to the right or left of the position of the tracks.

[0061] The processor 11 may also limit the search area for each facility based on the “geodetic coordinates” of the facility information. For example, the processor 11 may set the search area around an area where the facility is expected to appear in the captured image based on the “geodetic coordinates” and the current position of the vehicle 10.

[0062] The processor 11 also has a function of generating a transformation matrix that associates coordinates in the captured image with geodetic coordinates, based on the coordinates of the facility area and the track information.

[0063] Here, the transformation matrix is a matrix that transforms geodetic coordinates into coordinates in a captured image, i.e., a matrix for calculating the coordinates at which an object existing at predetermined geodetic coordinates appears in a captured image.

[0064] The processor 11 selects a predetermined number (for example, four) of equipment areas from the equipment areas in the captured image. For example, the processor 11 selects the equipment areas so that the area of a polygon connecting the coordinates of the equipment areas is maximized.

[0065] When an equipment area is selected, processor 11 acquires the geodetic coordinates of the equipment shown in the selected equipment area from the equipment information. For example, processor 11 acquires geodetic information corresponding to the ID of the equipment shown in each selected equipment area from the equipment information.

[0066] When the geodetic coordinates of the equipment are acquired, the processor 11 generates a transformation matrix that associates the coordinates of each equipment area in the captured image with the geodetic coordinates of the equipment depicted in each equipment area, according to a predetermined algorithm.

[0067] The processor 11 also has a function of generating a coordinate sequence of the track R in the captured image based on the transformation matrix.

[0068] After generating the transformation matrix, the processor 11 applies the transformation matrix to the geodetic coordinate sequence of the track R indicated by the track information to calculate the coordinate sequence of the track R in the photographed image. That is, the processor 11 transforms the geodetic coordinate sequence indicating each point of the track R into a coordinate sequence in the photographed image.

[0069] Fig. 5 shows the coordinates of the tracks in the captured image. In the example shown in Fig. 5, processor 11 calculates a coordinate sequence 31 indicating track R. Here, processor 11 calculates, as coordinate sequence 31, a coordinate sequence of the right rail and a coordinate sequence of the left rail of track R.

[0070] After calculating the coordinate sequence 31, the processor 11 calculates an offset, based on the feature amount of the captured image, to be added to the coordinate sequence 31 to calculate the coordinates of the track.

[0071] For example, the processor 11 extracts the coordinates of the edges in the captured image by edge detection as the feature amount of the captured image. Fig. 6 shows feature amounts detected by edge detection. As shown in Fig. 6, processor 11 detects edges at the coordinates of the tracks (here, the left and right rails) in the captured image.

[0072] When the coordinates of the edge in the captured image are detected, the processor 11 matches the coordinate sequence 31 with the feature amount (edge coordinates) and calculates an evaluation value. Here, the evaluation value increases as the similarity between the coordinate sequence 31 and the feature amount increases.

[0073] Processor 11 calculates an evaluation value by adding an offset (x coordinate and y coordinate) to coordinate sequence 31. Processor 11 moves the offset in the x-axis direction and the y-axis direction to identify the offset that maximizes the evaluation value. For example, processor 11 may move the offset randomly, or may move the offset in a direction that improves the evaluation value.

[0074] When the offset with the maximum evaluation value is identified, the processor 11 acquires the coordinate sequence obtained by adding the offset to the coordinate sequence 31 as the coordinate sequence indicating the track R in the captured image.

[0075] The processor 11 also has a function of acquiring limit information indicating vehicle gauges and construction gauges.

[0076] The processor 11 transmits a request for limit information to the host device 20 via the communication unit 15. The processor 11 receives a response including the limit information from the host device 20 via the communication unit 15.

[0077] The processor 11 also has a function of setting a travel area in which the vehicle 10 travels in the photographed image based on the coordinate sequence of the track R in the photographed image and the limit information.

[0078] For example, the processor 11 calculates the size and position of the vehicle gauge and construction gauge at each coordinate point of the track R in the photographed image based on the coordinate sequence of the track R in the photographed image and the geodetic coordinates of the track R. After calculating the size and position of the vehicle gauge and construction gauge, the processor 11 sets the vehicle gauge and construction gauge as a running area according to the size and position.

[0079] The processor 11 also has a function of detecting an obstacle that may hinder the travel of the vehicle 10 based on the travel area.

[0080] Once the travel area is set, the processor 11 detects an obstacle in the travel area that may obstruct the travel of the vehicle 10. That is, the processor 11 detects an obstacle interfering with the travel area.

[0081] When an obstacle is detected, the processor 11 displays the fact that an obstacle has been detected on the display unit 17. For example, the processor 11 displays a warning or the like on the display unit 17.

[0082] The processor 11 may display differently when an obstacle is detected at the vehicle gauge and when an obstacle is detected at the structure gauge. For example, when the processor 11 detects an obstacle at the structure gauge, the processor 11 displays a caution on the display unit 17. When the processor 11 detects an obstacle at the vehicle gauge, the processor 11 displays a warning on the display unit 17.

[0083] The processor 11 may also output a warning sound or the like through a speaker or the like.

[0084] Next, an example of the operation of the vehicle 10 will be described. 7 is a flowchart for explaining an example of the operation of the vehicle 10. Here, it is assumed that the vehicle 10 is traveling on a railroad track R.

[0085] First, the processor 11 of the vehicle 10 acquires a captured image from the camera 2 (S11). After acquiring the captured image, the processor 11 acquires facility information from the higher-level device 20 via the communication unit 15 (S12).

[0086] After acquiring the facility information, the processor 11 acquires track information from the higher-level device 20 via the communication unit (S13). After acquiring the track information, the processor 11 identifies a facility area from the captured image (S14).

[0087] When the equipment area is identified, processor 11 generates a transformation matrix that converts the geodetic coordinates into coordinates in the captured image based on the coordinates of the identified equipment area (S15). When the transformation matrix is generated, processor 11 uses the transformation matrix to convert the geodetic coordinate sequence of track R indicated by the track information into a coordinate sequence in the captured image (S16).

[0088] After converting the coordinate sequence, processor 11 matches the converted coordinate sequence with the feature amount of the photographed image to calculate the coordinate sequence of the track R in the photographed image (S17). After calculating the coordinate sequence of the track R in the photographed image, processor 11 acquires limit information from upper device 20 via communication unit 15 (S18).

[0089] When the limit information is acquired, the processor 11 sets a traveling area in the photographed image based on the coordinate sequence of the track R in the photographed image and the limit information (S19). After setting the traveling area, the processor 11 detects obstacles based on the traveling area (S20).

[0090] If it is determined that an obstacle is present (YES in S20), the processor 11 displays a warning or the like on the display unit 17 (S21).

[0091] When it is determined that there is no obstacle (S20, NO), or when a warning or the like is displayed on the display unit 17 (S21), the processor 11 ends the operation.

[0092] The processor 11 may repeatedly execute steps S11 to S21 at predetermined intervals. Furthermore, the equipment that the processor 11 recognizes from the captured image may be a marker or the like.

[0093] The processor 11 may also be configured to superimpose the coordinate sequence of the track R in the photographed image on the display unit 17. The processor 11 may also be configured to superimpose the running area on the photographed image on the display unit 17.

[0094] Furthermore, when an obstacle is detected, the processor 11 may automatically stop the vehicle 10. Furthermore, the processor 11 may notify the higher-level device 20 via the communication unit 15 that an obstacle has been detected.

[0095] A vehicle configured as described above recognizes equipment from a captured image. The vehicle generates a transformation matrix from the geodetic coordinates of the equipment and the coordinates of the equipment in the captured image. The vehicle converts the geodetic coordinates of the tracks into coordinates in the captured image based on the transformation matrix. As a result, the vehicle can effectively acquire the coordinates of the tracks in the captured image even when the tracks are difficult to recognize through image processing, such as when the tracks are far away and small in the captured image.

[0096] (Second embodiment) Next, a second embodiment will be described. The vehicle according to the second embodiment differs from that according to the first embodiment in that it recognizes the track from the captured image based on the gradient of the location where the facility is located. Therefore, the other features are given the same reference numerals and detailed explanations are omitted.

[0097] Fig. 8 is a side view showing the track R on which the vehicle 10 according to the second embodiment runs and facilities around the track R. Fig. 9 is a top view showing the track R and facilities.

[0098] As shown in Figures 8 and 9, the gradient of track R changes at the gradient change point. Here, track R before the gradient change point is installed in an area where the gradient value is Z1. Track R further back than the gradient change point is installed in an area where the gradient value is Z2. For example, the gradient value may be expressed in per mille or degrees.

[0099] The vehicle 10 is on a track R that is installed in an area with a gradient value of Z1. The vehicle 10 travels from its current location to a track R that is installed in an area with a gradient value of Z2.

[0100] Furthermore, facilities (here, utility poles) are installed in the area where the gradient value is Z1, and facilities (here, utility poles) are also installed in the area where the gradient value is Z2.

[0101] Furthermore, the image captured by camera 2 shows the track R and equipment installed in an area with a gradient value of Z1. The image captured by camera 2 also shows the track R and equipment installed in an area with a gradient value of Z2.

[0102] The track information also includes the gradient value of the track R. That is, the track information includes the gradient value of the track R at each geodetic coordinate.

[0103] The facility information further includes a gradient value. Fig. 10 shows an example of the configuration of facility information according to the second embodiment. As shown in Fig. 10, the facility information stores "ID," "type," "geodetic coordinates," and "gradient value" in association with each other.

[0104] "ID", "type", and "geodetic coordinates" are as described above.

[0105] The "gradient value" indicates the gradient value of the area in which the equipment is installed. Here, the "gradient value" indicates Z1 or Z2. As shown in FIG. 10, equipment with "ID" of 1 to 4 is installed in an area with a gradient value of Z1. Furthermore, equipment with "ID" of 5 to 8 is installed in an area with a gradient value of Z2.

[0106] The facility information may further include "positional relationship from the railroad tracks." The facility information may include other configurations as needed in addition to the configuration shown in Fig. 10, or specific configurations may be excluded from the facility information. The configuration of the facility information is not limited to a specific configuration.

[0107] Next, a description will be given of functions realized by the vehicle 10. The functions realized by the vehicle 10 are realized by the processor 11 executing a program stored in the internal memory, the ROM 12, the NVM 14, or the like. The vehicle 10 according to the second embodiment realizes the following functions in addition to the functions according to the first embodiment.

[0108] The processor 11 has a function of generating a transformation matrix for each gradient value.

[0109] Here, it is assumed that the processor 11 has identified the facility area from the captured image.

[0110] The processor 11 acquires a gradient value corresponding to the recognized facility from the facility information. For example, the processor 11 acquires a gradient value corresponding to the ID of the recognized facility from the geodetic information.

[0111] When the gradient value corresponding to the facility is obtained, the processor 11 sets a predetermined gradient value (for example, Z1).

[0112] When the gradient value is set, processor 11 identifies the equipment area of the equipment corresponding to the set gradient value. When the equipment area is identified, processor 11 selects a predetermined number (for example, four) of equipment areas from the identified equipment areas. For example, processor 11 selects the equipment areas so that the area of a polygon connecting the coordinates of the equipment areas is maximized.

[0113] When an equipment area is selected, processor 11 acquires the geodetic coordinates of the equipment shown in the selected equipment area from the equipment information. For example, processor 11 acquires geodetic information corresponding to the ID of the equipment shown in each selected equipment area from the equipment information.

[0114] When the geodetic coordinates of the equipment are acquired, the processor 11 generates a transformation matrix that associates the coordinates of each equipment area in the captured image with the geodetic coordinates of the equipment depicted in each equipment area. The processor 11 acquires the generated transformation matrix as a transformation matrix for a set gradient value (for example, Z1). That is, the transformation matrix is a matrix for converting the geodetic coordinates of an object installed in an area having the set gradient value into coordinates in the captured image.

[0115] The processor 11 sets another gradient value (for example, Z2). When the other gradient value is set, the processor 11 generates a transformation matrix for the gradient value (for example, Z2) that has been set in the same manner.

[0116] The processor 11 has a function of generating a coordinate sequence of the track R in the captured image based on a transformation matrix for each gradient.

[0117] The processor 11 sets a predetermined gradient value (for example, Z1). After setting the gradient value, the processor 11 acquires a geodetic coordinate sequence corresponding to the set gradient value from the track information.

[0118] When the geodetic coordinate sequence is acquired, the processor 11 applies a transformation matrix of the gradient values set to the geodetic coordinate sequence to calculate a coordinate sequence in the captured image. That is, the processor 11 transforms the geodetic coordinate sequence into a coordinate sequence in the captured image.

[0119] After calculating the coordinate sequence in the captured image, processor 11 sets another gradient value (for example, Z2). After setting the other gradient value, processor 11 similarly uses a transformation matrix for the gradient value to convert the geodetic coordinate sequence corresponding to the set gradient value into a coordinate sequence in the captured image.

[0120] The processor 11 similarly converts the geodetic coordinate sequence into a coordinate sequence in the captured image for each gradient value.

[0121] The processor 11 generates one coordinate sequence by integrating the coordinate sequences transformed by the transformed coordinates of each gradient value. After generating one coordinate sequence, the processor 11 calculates an offset to be added to the coordinate sequence to calculate the coordinates of the track, based on the feature amount of the captured image, as in the first embodiment.

[0122] The processor 11 acquires a coordinate sequence obtained by adding an offset to the integrated single coordinate sequence as a coordinate sequence indicating the track R in the photographed image.

[0123] The processor 11 also has a function of setting a travel area in which the vehicle 10 travels in the captured image based on the coordinate sequence of the track R, the gradient value of the track R, and the limit information.

[0124] For example, the processor 11 calculates the size and position of the vehicle gauge and construction gauge at each coordinate point of the track R in the photographed image based on the coordinate sequence of the track R in the photographed image, the gradient value of the track R, and the geodetic coordinates of the track R. After calculating the size and position of the vehicle gauge and construction gauge, the processor 11 sets the vehicle gauge and construction gauge as a running area according to the size and position.

[0125] Next, an example of the operation of the vehicle 10 will be described. 11 is a flowchart for explaining an example of the operation of the vehicle 10. Here, it is assumed that the vehicle 10 is traveling on a railroad track R.

[0126] First, the processor 11 of the vehicle 10 acquires a captured image from the camera 2 (S31). After acquiring the captured image, the processor 11 acquires facility information from the higher-level device 20 via the communication unit 15 (S32).

[0127] After acquiring the facility information, the processor 11 acquires track information from the higher-level device 20 via the communication unit (S33). After acquiring the track information, the processor 11 identifies a facility area from the captured image (S34).

[0128] When the equipment area is identified, processor 11 generates a transformation matrix for each gradient value based on the coordinates of the identified equipment area (S35). After generating the transformation matrix for each gradient value, processor 11 converts the geodetic coordinate sequence of track R into a coordinate sequence in the captured image based on the transformation matrix for each gradient value and the track information (S36).

[0129] After converting the coordinate sequence, processor 11 matches the converted coordinate sequence with the feature amount of the photographed image to calculate the coordinate sequence of the track R in the photographed image (S37). After calculating the coordinate sequence of the track R in the photographed image, processor 11 acquires limit information from upper device 20 via communication unit 15 (S38).

[0130] After generating the limit information, processor 11 sets a traveling area in the captured image based on the coordinate sequence of track R in the captured image, the gradient value of track R, and the limit information (S39). After setting the traveling area, processor 11 detects obstacles based on the traveling area (S40).

[0131] If it is determined that an obstacle is present (YES in S40), the processor 11 displays a warning or the like on the display unit 17 (S41).

[0132] When it is determined that there is no obstacle (NO in S40) or when a warning or the like is displayed on the display unit 17 (S41), the processor 11 ends the operation.

[0133] The processor 11 may repeatedly execute steps S31 to S41 at predetermined intervals. The processor 11 may also acquire the gradient value at each point of the rail R separately from the rail information. The processor 11 may also acquire the gradient value of the facility separately from the facility information.

[0134] The vehicle configured as described above generates a transformation matrix for each gradient value of the area where the equipment is installed. The vehicle uses the transformation matrix for each gradient value to transform the geodetic coordinate sequence of the track into a coordinate sequence in the captured image. As a result, the vehicle can effectively acquire the coordinate sequence of the track in the captured image even if the gradient of the area where the equipment and track are installed changes.

[0135] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0136] 1...housing, 2...camera, 3...antenna, 4...information processing device, 10...vehicle, 11...processor, 12...ROM, 13...RAM, 14...NVM, 15...communication unit, 16...operation unit, 17...display unit, 18...camera interface, 19...antenna interface, 20...higher-level device, 21...equipment area, 22...equipment area, 23...equipment area, 24...equipment area, 25...equipment area, 31...coordinate sequence, 100...vehicle system

Claims

1. an image interface for acquiring photographed images including tracks and facilities; a communication interface that acquires track information indicating the geodetic coordinates of the track and equipment information indicating the geodetic coordinates of the equipment; Identifying an equipment area in which the equipment is captured from the captured image; generating a transformation matrix that associates geodetic coordinates with coordinates in the photographed image based on coordinates of the equipment area in the photographed image and geodetic coordinates of the equipment; transforming the geodetic coordinates of the track into coordinates in the photographed image based on the transformation matrix to identify the coordinates of the track in the photographed image; a processor; An information processing device comprising:

2. The processor: extracting a feature amount of the photographed image; adding an offset to the coordinates converted from the geodetic coordinates of the track, and matching the coordinates to which the offset has been added with the feature amount to calculate an evaluation value; Identifying an offset to be added to the coordinates converted from the geodetic coordinates of the track based on the evaluation value to calculate the coordinates of the track; adding the specified offset to the coordinates converted from the geodetic coordinates of the track to calculate the coordinates of the track in the captured image; The information processing device according to claim 1 .

3. The feature amount is extracted by edge detection. The information processing device according to claim 2 .

4. The communication interface acquires limit information indicating a rolling stock gauge or a construction gauge of the track, the processor sets a travel area of a vehicle traveling on the track in the photographed image based on the limit information and the coordinates of the track in the photographed image. The information processing device according to claim 1 .

5. The processor detects an obstacle that may hinder the travel of the vehicle based on the travel area. The information processing device according to claim 4 .

6. The processor: selecting a predetermined number of the facility areas from the facility areas; generating the transformation matrix based on the coordinates of the selected equipment area; The information processing device according to claim 1 .

7. the processor selects the facility area so that the area of a polygon connecting the coordinates of the facility area is maximized. The information processing device according to claim 6 .

8. The equipment information includes a gradient value of an area in which the equipment is installed, the processor generates the transformation matrix for each of the gradient values. The information processing device according to claim 1 .

9. the facility includes one of a utility pole, a speed sign, or a stop sign; The information processing device according to any one of claims 1 to 8.

10. The image interface acquires the captured image from a camera capturing an image of the area ahead of the vehicle.

6. The information processing device according to claim 4 or 5.

11. an antenna interface connected to an antenna for receiving a signal for positioning the vehicle; The facility information indicates the geodetic coordinates of the facility photographed from the geodetic coordinates of the vehicle.

6. The information processing device according to claim 4 or 5.

12. A program executed by a processor, the processor, The function to capture images including tracks and facilities a function of acquiring track information indicating the geodetic coordinates of the track and equipment information indicating the geodetic coordinates of the equipment; a function of identifying an equipment area in which the equipment is captured from the captured image; a function of generating a transformation matrix that associates geodetic coordinates with coordinates in the photographed image based on coordinates of the equipment area in the photographed image and geodetic coordinates of the equipment; a function of transforming the geodetic coordinates of the track into coordinates in the captured image based on the transformation matrix, and identifying the coordinates of the track in the captured image; A program to make this happen.

Images