Visibility confirmation device and visibility confirmation method

Abstract

To generate vision information without using a dedicated sensor such as a laser scanner.SOLUTION: A vision checking device 12 includes: a video acquisition unit 21 that acquires video of a railway track in a vehicle traveling direction, imaged by an imaging device 11; a target detection unit 23 that detects, from the acquired video, at least one of a vision checking target and a reference target having a predetermined positional relationship with the vision checking target; a frame number calculation unit 24 that based on a result of detection performed by the target detection unit 23, detects a reference image frame corresponding to the position of the vision checking target; a frame extraction unit 25 that extracts, from the video, an image frame group on the near side in the vehicle traveling direction by a predetermined vision checking required distance from the reference image frame; a three-dimensional image generation unit 26 that generates a three-dimensional image based on the image frame group extracted by the frame extraction unit 25; and a position detection unit 27 that detects a position at which the vision checking target is visible based on the generated three-dimensional image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a visibility confirmation device and a visibility confirmation method. [Background technology]

[0002] Special signal lights are known as wayside equipment installed along railway tracks, and they emit a light signal to indicate the occurrence of an incident that may disrupt train operations.The special signal light is equipped with an emergency stop button, and when the emergency stop button is pressed, the special signal light emitter outputs a light signal.When the train crew sees the light signal from the special signal light, they are required to stop the train, so it is necessary for the train crew to be able to see the light signal from the special signal light when the train is located at a point more than a certain distance in front of the special signal light.

[0003] Therefore, a method has been proposed (for example, Patent Document 1) for generating visibility information for target facilities without much manual intervention, based on three-dimensional point cloud data around the tracks. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7209913 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology of Patent Document 1 mentioned above assumes the use of a three-dimensional point cloud measurement device such as a laser scanner to generate three-dimensional point cloud data, and its introduction requires a dedicated sensor, which is costly.

[0006] Therefore, the present invention aims to solve the above-mentioned problem, that is, to provide a visibility confirmation device and a visibility confirmation method that can generate a three-dimensional spatial image from video obtained using a general imaging device such as a consumer camcorder, and use the three-dimensional spatial image to generate visibility information for the target equipment. [Means for solving the problem]

[0007] a position detection unit that detects a position at which the visibility confirmation object can be seen based on the three-dimensional image generated by the three-dimensional image generation unit; and a position detection unit that detects a position at which the visibility confirmation object can be seen based on the three-dimensional image generated by the three-dimensional image generation unit.The position detection unit enlarges, to a predetermined magnification, a predetermined area image in which the visibility confirmation object is located in an image corresponding to a virtual field of view directed toward the visibility confirmation object at a predetermined position on the three-dimensional image through which the vehicle is moving, and detects a position at which the visibility confirmation object can be seen based on the result of a determination of whether the visibility confirmation object can be seen in the area image enlarged at the magnification.

[0008] Another aspect of the visibility confirmation device of the present invention further includes a recognition unit that recognizes a visibility confirmation object from a three-dimensional image, and the position detection unit enlarges an area image to a predetermined confirmation magnification in an image corresponding to a virtual field of view directed toward the visibility confirmation object at a predetermined position on the three-dimensional image where the vehicle is moving, and detects a position at which the visibility confirmation object can be seen based on the determination result of whether the recognition unit can recognize the visibility confirmation object in the area image enlarged at the confirmation magnification.

[0009] Another aspect of the visibility confirmation device of the present invention is characterized in that the video acquisition unit assigns relative distance information to each image frame of the acquired video based on the number of pixels corresponding to the size of a predetermined object whose size is known and the amount of movement of predetermined pixels corresponding to the same location in adjacent frames, and the image extraction unit extracts a group of image frames from the video based on the distance information assigned to each image frame.

[0010] Another aspect of the visibility confirmation device of the present invention is characterized in that the position detection unit calculates the line of sight direction from a predetermined position and the position of the visibility confirmation object, calculates the straight-line distance between the predetermined position and the line of sight direction of the visibility confirmation object, and calculates the magnification rate according to the straight-line distance.

[0011] a position detection step for detecting a position at which the visibility confirmation object can be seen based on the three-dimensional image generated by the three-dimensional image generation step; and a position detection step for detecting a position at which the visibility confirmation object can be seen based on the three-dimensional image generated by the three-dimensional image generation step. The position detection step is characterized in that, in an image corresponding to a virtual field of view directed toward the visibility confirmation object at a predetermined position on the three-dimensional image through which the vehicle is moving, a predetermined area image in which the visibility confirmation object is located is enlarged to a predetermined magnification, and the position at which the visibility confirmation object can be seen is detected based on the result of a determination of whether the visibility confirmation object can be seen in the area image enlarged at the magnification. [Effects of the Invention]

[0012] According to the present invention, line-of-sight information for a target facility can be generated from an image obtained using a general imaging device such as a consumer camcorder. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram for explaining a visibility confirmation system 1. As shown in FIG. [Figure 2] FIG. 2 is a functional block diagram of the visibility confirmation device 12. [Figure 3] FIG. 3 is a diagram for explaining the assignment of relative distance information. [Figure 4] FIG. 4 is a diagram for explaining detection of a visibility confirmation object and a reference object. [Figure 5] FIG. 5 is a diagram for explaining the extracted image frame group. [Figure 6] FIG. 6 is a diagram for explaining the generated three-dimensional image. [Figure 7] FIG. 7 is a diagram for explaining an example of the process of enlarging the area image and the determination of whether or not the visibility confirmation object is visible. [Figure 8] FIG. 8 is a diagram for explaining an example of the process of enlarging the area image and the determination of whether or not the visibility confirmation object is visible. [Figure 9] FIG. 9 is a diagram for explaining an example of the process of enlarging the area image and the determination of whether or not the visibility confirmation object is visible. [Figure 10] FIG. 10 is a flowchart for explaining the visibility confirmation process. [Figure 11] FIG. 11 is a flowchart for explaining the three-dimensional image generation process. [Figure 12] FIG. 12 is a flowchart for explaining the visibility confirmation object detection process. [Figure 13] FIG. 13 is a functional block diagram of the visibility confirmation device 111. [Figure 14]FIG. 14 is a flowchart for explaining the visibility confirmation process of the visibility confirmation device 111. [Figure 15] FIG. 15 is a diagram for explaining detection of an object for confirming visibility on a three-dimensional image. [Figure 16] FIG. 16 is a diagram for explaining an image in which an enlarged image Z1 of an enlarged visibility confirmation target is superimposed on a self-viewpoint position image P1. [Figure 17] FIG. 17 is a diagram for explaining an image in which an enlarged image Z2 of an enlarged visibility confirmation target is superimposed on a self-viewpoint position image P2. [Figure 18] FIG. 18 is a diagram for explaining an image in which an enlarged image Z3 of an enlarged visibility confirmation target is superimposed on a self-viewpoint position image P3. [Figure 19] FIG. 19 is a diagram for explaining another example of a method for generating an enlarged image. DETAILED DESCRIPTION OF THE INVENTION

[0014] The visibility confirmation device and visibility confirmation method will be described below with reference to the drawings.

[0015] (Visibility confirmation system) The visibility confirmation system 1 will be described with reference to Figure 1. The visibility confirmation system 1 is configured to include an imaging device 11, a visibility confirmation device 12, an input device 13, and an output device 14. These do not need to be separate devices, and may be configured as devices with multiple functions.

[0016] The imaging device 11 is configured to have the functions of a general imaging device 11 such as a consumer camcorder. The imaging device 11 captures an image of the area ahead of the front of a train traveling on a railway line where visibility is to be confirmed, and supplies the obtained image to the visibility confirmation device 12. If absolute position information obtained by, for example, a GPS (Global Positioning System) or the like is added to each frame of the image obtained by the imaging device 11, it is possible to use that information in subsequent processing, but it is also possible to execute subsequent processing even if that information is not added.

[0017] The visibility confirmation device 12 executes processing for visibility confirmation based on information input by the input device 13 and the image captured by the imaging device 11, and outputs the results to the output device 14.

[0018] The input device 13 is configured to include input devices such as a keyboard, a mouse, a touch panel, a microphone, and a reader for an external storage device or an image input device, and receives user operation inputs and input of information required for processing by the visibility confirmation device 12, and supplies the information to the visibility confirmation device 12. Specifically, information such as the visibility confirmation object to be the subject of visibility confirmation, a reference object that is in a predetermined positional relationship with the visibility confirmation object, positional relationship information between the reference object and the visibility confirmation object, and information on the distance at which visibility to the visibility confirmation object must be ensured (hereinafter referred to as the required visibility confirmation distance) is input to the input device 13 and supplied to the visibility confirmation device 12. Note that it is preferable that the reference object be one that is unlikely to be difficult to detect from any position due to trees or weeds along the railway line, for example.

[0019] Specific examples of visibility confirmation objects include special signal lights, red rotating lights, various traffic lights, repeaters, etc. installed within loop line crossings. Reference objects include, for example, loop line crossings.

[0020] The output device 14 includes output devices such as a display, a printer, and a speaker, and outputs the results of processing by the visibility confirmation device 12.

[0021] (Visibility confirmation device) 2 is a functional block diagram of the visibility confirmation device 12. The visibility confirmation device 12 is configured to have the functions of an image acquisition unit 21, an input information acquisition unit 22, an object detection unit 23, a frame number calculation unit 24, a frame extraction unit 25, a three-dimensional image generation unit 26, a position detection unit 27, a recognition unit 28, and an output processing unit 29.

[0022] The image acquisition unit 21 acquires images of the tracks in the direction of travel of the train, captured by the imaging device 11 installed on the train. The image acquisition unit 21 then acquires or calculates relative or absolute position information between each of the acquired frames. The image acquisition unit 21 then supplies the acquired images and the relative or absolute position information between each of the frames to the object detection unit 23, the frame number calculation unit 24, and the frame extraction unit 25.

[0023] If absolute position information has not been assigned to each acquired frame, the video acquisition unit 21 calculates relative position information between each acquired frame. Any method for adding relative distance information may be used, but for example, relative distance information can be assigned to each image frame of the acquired video based on the size of a known predetermined object, the corresponding number of pixels, and the amount of movement of the predetermined object in adjacent frames (which does not have to be the same object as the object for which the correspondence between size and number of pixels is used).

[0024] To assign relative distance information, it is possible to use, for example, the technology described in "Maeda Riho, Hazuki Kota, Tsuru Naoto, Nagamine Nozomi. Equipment Management Support System Using Train Front Images. Proceedings of the 60th Railway Cybernetics Symposium, 803, 2023-11." Specifically, as shown in FIG. 3, in an image, at a location where the size of a sleeper or other object is known, for example, if the actual size indicated by arrow 41 in the figure is 2100 mm and the number of pixels at the corresponding location is 1050, it can be calculated that one pixel corresponds to 2 mm in actuality. Then, in adjacent frames, relative distance information can be assigned to each image frame of the video based on the number of pixels of movement at the same location indicated by point 42 in the figure.

[0025] In this way, the image acquisition unit 21 can add relative distance information to each image frame of the image without using various sensors to measure displacement or adding position information to the image using a separate sensor.

[0026] Furthermore, by calculating the relative positions between each frame in this way and adding relative distance information to each image frame of the video, it becomes possible to obtain the relative positional relationship between the reference object and the visibility confirmation object based on the video, even if the vehicle speed is not constant when the imaging device 11 obtains the video.

[0027] Returning to FIG. 2, the input information acquisition unit 22 supplies each piece of information input from the input device 13 to each part of the visibility confirmation device 12.

[0028] The object detection unit 23 detects at least one of a line-of-sight confirmation object and a reference object from the video supplied from the video acquisition unit 21 based on various information supplied from the input information acquisition unit 22, and supplies the detection result to the frame number calculation unit 24 and the frame extraction unit 25. The reference object is an object that is not a line-of-sight confirmation object but is in a predetermined positional relationship with the line-of-sight confirmation object.

[0029] Specifically, as shown in FIG. 4, the object detection unit 23 detects, in one frame of video, a red rotating light 43 installed within the loop line crossing as a visibility confirmation object and a loop line crossing 44 as a reference object that is in a predetermined positional relationship with the visibility confirmation object. The object detection unit 23 may use any method to detect the visibility confirmation object and the reference object, such as pattern matching or the use of AI. Specifically, for example, the technology described in "Maeda Riho, Hazuki Kota, Tsuru Naoto, Nagamine Nozomi. Facility Management Support System Using Train Front Images. Proceedings of the 60th Railway Cybernetics Symposium, 803, 2023-11" may be used.

[0030] Returning to FIG. 2, the frame number calculation unit 24 detects a reference image frame corresponding to the position of the visibility confirmation object based on the detection result of the object detection unit 23.

[0031] The frame number calculation unit 24 can detect the reference image frame in the following three ways, for example.

[0032] (1) In consecutive frames in the vehicle traveling direction, the frame just before the last frame in which an object to be confirmed in visibility can be detected is detected as the reference image frame.

[0033] However, to prepare for false detections where an object to be confirmed by line of sight cannot be temporarily detected due to some kind of obstacle in consecutive frames when it should actually be possible to detect it continuously, it is preferable to determine that it is being detected continuously even if the detection interval is a specified distance that is sufficiently shorter than the spacing between the objects to be confirmed by line of sight, or a specified number of frames that roughly corresponds to that distance, or to determine that an object to be confirmed by line of sight has been detected even if, for example, only part of the object to be confirmed by line of sight is detected.

[0034] (2) The frame just before the last frame in which the reference object detected by the object detection unit 23 was detected is set as the position of the reference object, and the position of the object to be confirmed as visible, i.e., the reference image frame, is detected based on the distance information from the reference object.

[0035] (3) Detecting a reference image frame using absolute position information of a line-of-sight object or a reference object.

[0036] Then, based on the distance information added to each frame by the video acquisition unit 21 and the information on the distance required for visibility supplied from the input information acquisition unit 22, the frame number calculation unit 24 calculates the number of frames required for generating a three-dimensional image for the distance required for visibility by the three-dimensional image generation unit 26 described below, and supplies this to the frame extraction unit 25. For example, the distance required for visibility of a special traffic light emitting device is 800 m, so when the object to be visible is a special traffic light emitting device, the frame number calculation unit 24 calculates the number of frames corresponding to 800 m and supplies this to the frame extraction unit 25.

[0037] Based on the information supplied from the frame number calculation unit 24, the frame extraction unit 25 extracts from the video a group of image frames, such as those shown in Figure 5, which are located a predetermined distance in front of the reference image frame in the direction of vehicle travel required to confirm visibility, and supplies these to the three-dimensional image generation unit 26.

[0038] Returning to FIG. 2, the three-dimensional image generator 26 generates a three-dimensional image, for example, as shown in FIG. 6, based on the extracted image frame group.

[0039] The three-dimensional image generation unit 26 may use any method for generating a three-dimensional image. Specifically, it is possible to use the technology described in, for example, "Yoshino, Sumiki, Takahashi, Hiroyuki, Nagamine, Nozomi. "Basic study on the construction of three-dimensional track space using images of the train ahead." Institute of Electrical Engineers of Japan Study Group Materials. TER= The papers of Technical Meeting on "Transportation and Electric Railway," IEE Japan / Transportation and Electric Railway Study Group [edited]. Institute of Electrical Engineers of Japan, May 2023, pp. 1-6." and "Hongo, Takataka, Goda, Wataru, Nakasone, Ryuta, Nagamine, Nozomi. "Improving the accuracy of three-dimensional track space construction methods using mask images." Institute of Electrical Engineers of Japan Study Group Materials. TER= The papers of Technical Meeting on "Transportation and Electric Railway," IEE Japan / Transportation and Electric Railway Study Group [edited]. Institute of Electrical Engineers of Japan, September 2023, pp. 25-30."

[0040] Returning to FIG. 2, the position detection unit 27 generates an image corresponding to a virtual field of view directed toward the visibility confirmation object at a predetermined position where a vehicle traveling on the track passes in the three-dimensional image generated by the three-dimensional image generation unit 26, and also generates an image obtained by enlarging a predetermined area image in which the visibility confirmation object exists at a predetermined confirmation magnification. The position detection unit 27 then detects a position at which the visibility confirmation object is visible based on the determination result of whether the visibility confirmation object is visible in the area image enlarged at the confirmation magnification. The position detection unit 27 then determines whether the visibility confirmation object is visible from each position on the track in the three-dimensional image, i.e., from a position a predetermined required visibility distance away from the visibility confirmation object to just before the visibility confirmation object, and supplies the determination result to the output processing unit 29.

[0041] The determination result of whether the visibility confirmation object is visible at a predetermined position may be obtained by outputting an area image enlarged by the position detection unit 27 to the output device 14 via the output processing unit 29, and the determination result entered by the input device 13 by the user who has confirmed it may be supplied from the input information acquisition unit 22, or the enlarged area image may be supplied to the recognition unit 28 described below, and the recognition result by the recognition unit 28 may be used. Hereinafter, the position detection unit 27 will be described as using the recognition result by the recognition unit 28 to detect a position where the visibility confirmation object is visible.

[0042] The recognition unit 28 recognizes the visibility confirmation object from the three-dimensional image supplied from the position detection unit 27 and supplies the recognition result to the position detection unit 27. Any method for recognizing the visibility confirmation object by the recognition unit 28 may be used, for example, recognition processing such as pattern matching or recognition processing using AI. In addition, here, it is preferable that the visibility confirmation object can be visually recognized from the corresponding position when the entire object is recognized more precisely than the detection of the visibility confirmation object by the object detection unit 23 described above.

[0043] 7 to 9, an example of the process of enlarging the area image by the position detection unit 27 and the determination of whether or not the visibility confirmation object is visible will be described.

[0044] As shown in FIG. 7, at a position somewhat close to the visibility confirmation object, a red rotating light 51-1, which is the visibility confirmation object, is visible both in the unenlarged image and in the enlarged area image 52-1.

[0045] As shown in Figure 8, at a position some distance away from the visibility confirmation object, it becomes difficult to determine whether the visibility confirmation object, red rotating light 51-2, is visible in the unenlarged image, but it can be easily determined that it is visible in the enlarged area image 52-2.

[0046] As shown in Figure 9, at a position further away from the visibility confirmation object, it becomes difficult to determine whether the visibility confirmation object, red rotating light 51-3, is visible in the unenlarged image, but in the enlarged area image 52-3, it can be easily determined that it is blocked by the fence and cannot be seen.

[0047] Here, the position detection unit 27 determines the direction from the self-viewpoint position, which is assumed to be used for checking the visibility of the object, to the visibility of the object as the gaze direction. By determining the gaze direction based on a three-dimensional image, the correct gaze direction can be obtained even when the visibility of the object is not visible in the captured image due to some kind of obstacle or the like.

[0048] Then, the position detection unit 27 calculates the linear distance in the line of sight using the three-dimensional image, and calculates the magnification rate according to the linear distance so that the size of the visibility confirmation object in the enlarged area images 52-1 to 52-3 becomes a certain size that can be easily determined. By performing a calculation process according to the linear distance in the line of sight, the area image can be enlarged based on the distance according to the line of sight, rather than the moving distance on the route.

[0049] Typically, when an image captured at a predetermined resolution is enlarged, the higher the enlargement ratio, the coarser the pixels become. That is, as explained with reference to FIGS. 8 and 9, even if an image captured at a predetermined resolution from a position farther away from the line-of-sight object is enlarged, the pixels of the corresponding area image become coarse, making it difficult to recognize. However, because the image used as the source of the area image enlargement process by the position detection unit 27 is a three-dimensional image, even if the area image is enlarged in a virtual field of view from a position farther away from the line-of-sight object, the pixels do not become coarse as with normal image enlargement.

[0050] To recognize a specific object from an image using pattern matching, AI, or the like, image quality must be high enough to recognize the object. However, the enlarged area image generated by the position detection unit 27 is enlarged without blurring while maintaining a certain image quality. In other words, the recognition unit 28 does not encounter any recognition difficulties due to the distance from the line-of-sight confirmation object. This is particularly true when the recognition unit 28 uses AI recognition processing. For example, as described with reference to FIG. 9, it can correctly determine that an obstacle blocking the line of sight exists between the viewpoint and the line-of-sight confirmation object, making the line-of-sight confirmation object invisible.

[0051] Returning to Figure 2, the output processing unit 29 receives the detection results of the positions at which the visibility confirmation object detected by the position detection unit 27 can be seen, i.e., the determination results of whether the visibility confirmation object can be seen at all positions from the position a predetermined distance away from the visibility confirmation object to just before the visibility confirmation object, and executes output processing to the output device 14.

[0052] In this way, the visibility confirmation device 12 generates a 3D image using the image frames obtained as a result of imaging, and enlarges the image of a predetermined area where the visibility confirmation object exists at a predetermined confirmation magnification, so the area image can be enlarged while maintaining a constant image quality without blurring.As a result, it is possible to accurately determine from the image whether there is an obstacle that blocks the line of sight between the viewpoint position and the visibility confirmation object, making it impossible to see the visibility confirmation object.In other words, accurate visibility information can be generated without using a dedicated sensor such as a laser scanner.

[0053] (Outlook confirmation process) Next, the visibility confirmation process will be described with reference to the flowchart of FIG.

[0054] In step S1, the input information acquisition unit 22 supplies the visibility confirmation object and reference object, and their respective required visibility confirmation distances, input from the input device 13, to the object detection unit 23, the frame number calculation unit 24, and the frame extraction unit 25. The object detection unit 23, the frame number calculation unit 24, and the frame extraction unit 25 set this information.

[0055] In step S2, the image acquisition unit 21 acquires frame image data constituting an image of the track in the direction of travel of the train, captured by the imaging device 11 provided on the train, and supplies the data to the object detection unit 23, the frame number calculation unit 24, and the frame extraction unit 25. At this time, if absolute position information has not been assigned to each acquired frame, the image acquisition unit 21 assigns relative distance information to each image frame, for example, as described with reference to FIG.

[0056] In step S3, the object detection unit 23 determines whether or not at least one of the line-of-sight confirmation object and a reference object that is in a predetermined positional relationship with the line-of-sight confirmation object has been detected from the image acquired by the image acquisition unit 21, based on the various information acquired from the input information acquisition unit 22. If it is determined in step S3 that neither the line-of-sight confirmation object nor the reference object has been detected, the processing of step S3 is repeated until it is determined that at least one of the line-of-sight confirmation object and the reference object has been detected.

[0057] If it is determined in step S3 that at least one of the visibility confirmation object and the reference object has been detected, then in step S4, a three-dimensional image generation process, which will be described later with reference to FIG. 11, is executed.

[0058] In step S5, a visibility confirmation object detection process, which will be described later with reference to FIG. 12, is executed.

[0059] In step S6, the object detection unit 23 determines whether or not processing has been completed for all of the acquired frame images. If it is determined in step S6 that processing has not been completed for all of the frame images, the process returns to step S3, and the subsequent processes are repeated. If it is determined in step S6 that processing has been completed for all of the frame images, the process ends.

[0060] (3D image generation processing) Next, the three-dimensional image generation process executed in step S4 of FIG. 10 will be described with reference to the flowchart of FIG.

[0061] In step S21, the frame number calculation unit 24 detects a reference image frame corresponding to the position of the visibility confirmation object based on the detection result of the object detection unit 23.

[0062] In step S22, the image acquisition unit 21 calculates the size in pixel units from the number of pixels corresponding to a predetermined object whose size is known, as described with reference to FIG.

[0063] In step S23, the video acquisition unit 21 assigns relative distance information to each image frame of the acquired video based on the amount of movement of a predetermined object (which does not have to be the same as the object for which the correspondence between size and pixel count is used) in adjacent frames. Then, the frame number calculation unit 24 calculates the number of frames required for generating a three-dimensional image for the required visibility distance using the three-dimensional image generation unit 26 (described later) based on the distance information assigned to each frame by the video acquisition unit 21 and the information on the required visibility distance supplied from the input information acquisition unit 22, and supplies the calculated number of frames to the frame extraction unit 25.

[0064] In step S24, based on the information supplied from the frame number calculation unit 24, the frame extraction unit 25 extracts from the video a group of image frames, such as those shown in Figure 5, which are located a predetermined distance in front of the reference image frame in the direction of vehicle travel required to confirm visibility, and supplies these to the three-dimensional image generation unit 26.

[0065] In step S25, the three-dimensional image generating unit 26 generates a three-dimensional image corresponding to a three-dimensional space, for example, as shown in FIG. 6, based on the extracted image frame group.

[0066] In step S26, the three-dimensional image generating unit 26 calculates the camera geometry in three-dimensional space, calculates the trajectory of the rail center, and supplies it to the position detecting unit 27, and the process proceeds to step S5 in FIG.

[0067] (Confirmation object detection process) Next, the visibility confirmation object detection process executed in step S5 of FIG. 10 will be described with reference to the flowchart of FIG.

[0068] First, the position detection unit 27 sets the initial position of the own viewpoint to immediately before the object to be confirmed in visibility. In step S41, the position detection unit 27 moves the own viewpoint position 1 m away from the object to be confirmed in the direction of the tracks.

[0069] In step S42, the position detection unit 27 calculates the line of sight direction based on the position of the user's viewpoint and the position of the visibility confirmation object in three-dimensional space.

[0070] In step S43, the position detection unit 27 calculates the magnification ratio according to the linear distance between the position of the own viewpoint and the object to be confirmed as visible, as described with reference to FIGS.

[0071] 7 to 9, position detection unit 27 generates an image from its own viewpoint position, enlarged to a predetermined confirmation magnification, of a predetermined area image in which an object for visibility confirmation exists, in accordance with the line of sight direction and magnification rate, and supplies this image to recognition unit 28. Recognition unit 28 recognizes the object for visibility confirmation from the three-dimensional image supplied from position detection unit 27, and supplies the recognition result to position detection unit 27.

[0072] In step S45, the position detection unit 27 determines whether the own viewpoint position is at least the distance required for visibility confirmation in the direction of the tracks from the object for visibility confirmation. If it is determined in step S45 that the own viewpoint position is not at least the distance required for visibility confirmation, the process returns to step S41, and the subsequent processes are repeated.

[0073] If it is determined in step S45 that the object is located at a distance greater than the required distance for visibility confirmation, then in step S46 the position detection unit 27 determines whether the entire object for visibility confirmation could be confirmed at all positions from the position that is the predetermined required distance from the object for visibility confirmation to just before the object for visibility confirmation, and supplies the determination result to the output processing unit 29.

[0074] If it is determined in step S46 that the entire visibility confirmation object can be confirmed at all positions, in step S47 the output processing unit 29 outputs to the output device 14 the result that there is no problem with visibility of the visibility confirmation object, and the processing is terminated.

[0075] If it is determined in step S46 that the entire object to be viewed could not be seen at any position, then in step S48 the output processing unit 29 outputs information indicating the location where the view was determined to be incomplete to the output device 14 as a result, and the processing is terminated.

[0076] Here, we have explained that the judgment is made by moving away from the previous position by 1 m at a time, but this order does not have to be the case as long as the judgment can be made at all positions within the required distance for visibility confirmation, and it goes without saying that the judgment interval can be any interval other than 1 m as long as it allows for accurate judgment.

[0077] 10 to 12, a 3D image is generated using the image frames obtained as a result of imaging, and a predetermined area image where the visibility confirmation object exists is enlarged to a predetermined confirmation magnification, so that the area image is enlarged while maintaining a constant image quality without blurring. As a result, it is possible to accurately determine from the image whether there is an obstacle that blocks the line of sight between the viewpoint position and the visibility confirmation object, making it impossible to see the visibility confirmation object. In other words, accurate visibility information can be generated without using a dedicated sensor such as a laser scanner.

[0078] (Example of output data) Table 1 shows a specific example of information output in step S48 indicating areas where visibility is determined to be incomplete. In the example of Table 1, "probability," "obscuration rate," and "visibility determination" are shown for each predetermined distance from the visibility confirmation object. "Probability" is the recognition probability when the recognition unit 28 recognizes the visibility confirmation object using AI. "Oscuration rate" is the proportion of the area recognized as the visibility confirmation object that is obscured by something, as calculated by AI. "Visibility determination" is the visibility result of the recognition unit 28 for the visibility confirmation object at that distance, based on the "probability" and "obscuration rate," as determined by AI. A "○" mark indicates that the confirmation object can be recognized, and an "×" mark indicates that the confirmation object cannot be recognized. [Table 1]

[0079] (Other visibility confirmation devices) In the visibility confirmation device 12 of Figure 2, the recognition unit 28 performs recognition processing such as pattern matching or recognition processing using AI on the three-dimensional image supplied from the position detection unit 27 to confirm the visibility of the visibility confirmation object, but it is also possible for the user to determine whether the visibility confirmation object is visible at a specified position by displaying an enlarged area image on the output device 14 and checking it.

[0080] 13 is a functional block diagram of the visibility confirmation device 111 when the user determines whether or not the visibility confirmation object is visible by checking an enlarged area image displayed on the output device 14. The same components as those in the visibility confirmation device 12 are assigned the same reference numerals.

[0081] 14 is a flowchart illustrating visibility confirmation processing by the visibility confirmation device 111. The components of the visibility confirmation device 111 will be described together with the description of the visibility confirmation processing flowchart.

[0082] In step S101, the three-dimensional image generating unit 121 generates a three-dimensional image from the video supplied from the video acquiring unit 21, and causes the output device 14 to display the image via the output processing unit 29.

[0083] In step S102, the object detection unit 122 detects a line-of-sight confirmation object on the three-dimensional image designated by the user. The user can designate a line-of-sight confirmation object on the three-dimensional image displayed on the output device 14 in step S101. For example, as shown in FIG. 15, if the user frames an area on the displayed three-dimensional image that includes the line-of-sight confirmation object, the object detection unit 23 detects a special signal light from the image of the framed area. Note that the speech bubbles in FIG. 15 are provided for illustrative purposes and are not actually displayed.

[0084] In step S103, the position detection unit 123 acquires the user's own viewpoint position specified by the user via the input information acquisition unit 22. The user can specify the user's own viewpoint position on the three-dimensional image displayed on the output device 14 in step S101. The position detection unit 123 corresponds to the enlarged image generation unit in the claims.

[0085] In steps S104 and S105, the position detection unit 123 calculates the line of sight direction based on the position of the user's viewpoint and the position of the object to be confirmed as being visible in three-dimensional space, similar to the processing in steps S42 and S43 of Figure 12, and calculates a magnification factor as a predetermined magnification factor according to the straight-line distance between the user's viewpoint and the object to be confirmed as being visible.

[0086] In step S106, the position detection unit 123 generates an image (hereinafter referred to as the "self-viewpoint position image") directed from the self-viewpoint position toward the visibility confirmation object according to the line of sight direction, and a predetermined area (hereinafter referred to as the "enlarged image") including the visibility confirmation object according to the line of sight direction and magnification, and displays them on the output device 14 via the output processing unit 29. For example, as shown in FIG. 16, an image is displayed in which an enlarged image Z1 of a special traffic light, which is an enlarged visibility confirmation object, is superimposed on a self-viewpoint position image P1 directed from the self-viewpoint position toward the visibility confirmation object according to the line of sight direction. In the enlarged image Z1 of FIG. 16, only the traffic light, which is an visibility confirmation object, is shown for the sake of simplicity, but the displayed image also displays scenery other than the visibility confirmation object. The same applies to FIGS. 17 and 18, which will be described later.

[0087] In step S107, the position detection unit 123 accepts the user's judgment result, and in step S108, stores the judgment result. The user judges the visibility of the visibility confirmation object by referring to the self-viewpoint position image and the enlarged image displayed on the output device 14, and inputs the judgment result to the input device 13. The user's judgment result input to the input device 13 is supplied to the position detection unit 123 via the input information acquisition unit 22. In the example of Fig. 16, the special signal light is visible, so the judgment result is that it is visible.

[0088] In step S109, the position detection unit 123 waits until the next user's viewpoint position is designated. If so, the process is repeated. FIG. 17 shows a display example in which the user's viewpoint position is set to a position farther from the special signal light emitting device than in the case of FIG. 16. In this case, an enlarged image Z2 of the special signal light emitting device is displayed, which is approximately the same size as the special signal light emitting device in the enlarged image Z1 of FIG. 16, allowing the user to confirm the view from the designated user's viewpoint position. In this case, the special signal light emitting device is visible, and the determination result indicates that the special signal light emitting device is visible. FIG. 18 shows a display example in which the user's viewpoint position is set to a position on the right side of the drawing, as compared to the case of FIG. 17. In this case, an enlarged image Z3 of the special signal light emitting device is displayed, which is approximately the same size as the special signal light emitting device in the enlarged image Z1 of FIG. 16, allowing the user to confirm the view from the designated user's viewpoint position. In this case, the user's viewpoint position is blocked by the traffic light pole, and the determination result indicates that the special signal light emitting device is not visible.

[0089] As described above, an enlarged image including the visibility confirmation target is displayed, so that the user can check the visibility using the image.

[0090] The user's own viewpoint position is not necessarily designated by the user, and the position detection unit 123 can also display an image from any own viewpoint position.

[0091] (Another example of generating an enlarged image) In the above description, the position detection units 27 and 123 generate the enlarged image by calculating the enlargement ratio, but they may also generate the enlarged image by other methods. Next, another example of a method for generating an enlarged image will be described with reference to FIG.

[0092] First, the position detection unit 27, 123 acquires the distance L between the user's viewpoint position and the position of the visibility confirmation object in three-dimensional space. The position detection unit 27, 123 acquires the height H of the visibility confirmation object. Then, the position detection unit 27, 123 calculates the FOV (Field of View) using equation (1). In this example, the FOV is the viewing angle from the user's viewpoint position O to the visibility confirmation object.

number

[0093] 19A, the position detection units 27, 123 use the calculated FOV to generate an enlarged image Q. This makes it possible to generate an enlarged image in which, for example, the height of the object in the enlarged image Q is adjusted to match the vertical length of the enlarged image.

[0094] 19B, when the distance between the own viewpoint position and the position of the visibility confirmation object changes to L', the position detection unit 27, 123 can recalculate the FOV using equation (1) and regenerate the enlarged image Q'. The size of the visibility confirmation object becomes constant in the enlarged images Q and Q'.

[0095] [Supplementary explanation of the embodiment] The above-described embodiments each show a preferred specific example of the present invention. The numerical values, components, arrangement and connection order of the components, processing order in the flowcharts, etc. shown in the embodiments are merely examples and are not intended to limit the present invention. Furthermore, the drawings are not necessarily strict illustrations.

[0096] The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the programs constituting the software are installed from a program recording medium into a computer incorporated in dedicated hardware, or into an information processing device such as a general-purpose personal computer that can execute various functions by installing various programs.

[0097] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0098] Furthermore, the embodiments of the present invention are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present invention.

[0099] [Note] (1) The visibility confirmation device 12 described above is an image acquisition unit 21 that acquires an image of the track in the direction of travel of the train, taken by an imaging device 11 installed on the train; an object detection unit (23) that detects, from the image acquired by the image acquisition unit (21), at least one of a visibility confirmation object (e.g., a red rotating light (43)) and a reference object (e.g., a loop line crossing (44)) that is in a predetermined positional relationship with the visibility confirmation object; a frame number calculation unit 24 and a frame extraction unit 25 which detect a reference image frame corresponding to the position of the visibility confirmation object based on the detection result of the object detection unit 23, and extract from the video a group of image frames on the front side in the traveling direction of the vehicle by a predetermined distance required for visibility confirmation from the reference image frame; a three-dimensional image generating unit (26) that generates a three-dimensional image based on the image frame group extracted by the frame extracting unit (25); a position detection unit (27) that detects a position where an object to be confirmed in visibility can be seen based on the three-dimensional image generated by the three-dimensional image generation unit (26); Equipped with The position detection unit 27 In an image corresponding to a virtual field of view directed toward the visibility confirmation object at a predetermined position where the vehicle is moving on the three-dimensional image, an image of a predetermined area where the visibility confirmation object exists is enlarged to a predetermined confirmation magnification; Based on the result of determining whether the visibility confirmation object is visible in the area image enlarged at the confirmation magnification, the position where the visibility confirmation object is visible is detected.

[0100] This allows the area image including the visibility confirmation object to be enlarged without blurring while maintaining a certain image quality, and as a result, it is possible to accurately determine from the image whether there is an obstacle blocking the line of sight between the viewpoint and the visibility confirmation object, making it impossible to see the visibility confirmation object. In other words, visibility information can be generated without using a dedicated sensor such as a laser scanner.

[0101] (2) The visibility confirmation device 12 also: Further, a recognition unit 28 is provided to recognize an object to be confirmed from the three-dimensional image, The position detection unit 27 In an image corresponding to a virtual field of view directed toward a visibility confirmation object at a predetermined position where the vehicle is moving on the three-dimensional image, an area image is enlarged to a predetermined confirmation magnification; Based on the determination result of whether or not the recognition unit 28 can recognize the line-of-sight confirmation object in the area image enlarged at the confirmation magnification, the position where the line-of-sight confirmation object can be visually recognized is detected.

[0102] For example, in order for AI or other devices to recognize a specific object from an image, the image quality must be high enough to recognize the object. As described above, the area image can be enlarged without blurring while maintaining a certain level of image quality, so the AI ​​can detect the object. This detection then makes it possible to accurately determine whether there is an obstacle blocking the line of sight between the viewpoint and the object, making it impossible to see the object.

[0103] (3) The visibility confirmation device 12 also The video acquisition unit 21 assigns relative distance information to each image frame of the acquired video based on the number of pixels corresponding to the size of a predetermined object (for example, a sleeper) whose size is known and the amount of movement of predetermined pixels corresponding to the same location in adjacent frames, The frame extraction unit 25 extracts a group of image frames from the video based on the distance information assigned to each image frame.

[0104] In this way, distance information corresponding to the position where the image was captured is added to each image frame of the video, based on the pixel-size correspondence according to the size of the captured object, so that the image frame group required to generate a 3D image can be extracted based on the distance information added to each image frame, meaning that a sensor for measuring position is not required.

[0105] (4) The visibility confirmation device 12 also The position detection unit 27 Calculate the line of sight direction from the specified position and the position of the object to be checked for visibility, Calculate the straight-line distance between a specified position and the line of sight of the object to be checked, The magnification ratio is calculated according to the straight-line distance.

[0106] In this way, the line of sight direction is calculated from the predetermined position and the position of the object for line of sight confirmation, the straight-line distance between the predetermined position and the object for line of sight confirmation in that line of sight direction is calculated, and the magnification rate is calculated according to that straight-line distance, so that the area image can be enlarged based on the distance according to the line of sight direction.

[0107] (5) The recognition unit 28 recognizes the visibility confirmation object at each predetermined distance from the visibility confirmation object, The position detection unit 27 outputs the recognition result obtained by the recognition unit 28 for each of the predetermined distances.

[0108] Since the recognition results are output for each specified distance, the user can see at a glance at which positions the visibility confirmation object can or cannot be seen, making it easy to grasp the overall visibility situation.

[0109] (6) The visibility confirmation device 111 also an image acquisition unit 21 that acquires an image of the track in the direction of travel of the train, taken by an imaging device 11 installed on the train; an object detection unit 122 that detects an object to be confirmed as visible (for example, a special signal light) from the image acquired by the image acquisition unit 21; a three-dimensional image generating unit 121 that generates a three-dimensional image based on the image acquired by the image acquiring unit 21; a position detection unit 123 that enlarges a predetermined area image in which the visibility confirmation object exists to a predetermined confirmation magnification in an image corresponding to a virtual field of view directed toward the visibility confirmation object at a predetermined position on the three-dimensional image where the vehicle is moving, and generates an enlarged image; an output processing unit (29) that outputs an image corresponding to a virtual field of view directed toward the visibility confirmation object at a predetermined position on the three-dimensional image and the enlarged image created by the position detection unit (123); Equipped with.

[0110] With this configuration, the user can view both an image corresponding to the virtual field of view directed toward the visibility confirmation object at a specified position on the three-dimensional image, and an enlarged image, thereby enabling the user to directly grasp the visibility situation of the visibility confirmation object.

[0111] (7) The visibility confirmation device 111 also further comprising an input information acquisition unit 22 that acquires the input at the predetermined position, The position detection unit 123 enlarges an image of a specified area in which the visibility confirmation object exists to a specified confirmation magnification in an image corresponding to a virtual field of view directed toward the visibility confirmation object at the specified position acquired by the input information acquisition unit 22.

[0112] This configuration allows the user to specify the position where they want to check visibility and check the image corresponding to the field of view from that position, allowing the user to check visibility from any position and angle. [Explanation of symbols]

[0113] 1... Visibility confirmation system, 11... Imaging device, 12,111... Visibility confirmation device, 13... Input device, 14... Output device, 21... Image acquisition unit, 22... Input information acquisition unit, 23,122... Object detection unit, 24... Frame number calculation unit, 25... Frame extraction unit, 26,121... Three-dimensional image generation unit, 27,123... Position detection unit, 28... Recognition unit, 29... Output processing unit

Claims

1. an image acquisition unit that acquires an image of the track in the direction of travel of the vehicle, the image being captured by a camera installed on the vehicle; an object detection unit that detects at least one of a visibility confirmation object and a reference object that is in a predetermined positional relationship with the visibility confirmation object from the image acquired by the image acquisition unit; an image extraction unit that detects a reference image frame corresponding to the position of the visibility confirmation object based on the detection result of the object detection unit, and extracts from the video a group of image frames located a predetermined distance in front of the reference image frame in the direction of travel of the vehicle; a three-dimensional image generating unit that generates a three-dimensional image based on the image frame group extracted by the image extracting unit; a position detection unit that detects a position at which the visibility confirmation object can be viewed based on the three-dimensional image generated by the three-dimensional image generation unit; Equipped with The position detection unit enlarging a predetermined area image in which the visibility confirmation object exists to a predetermined confirmation magnification in an image corresponding to a virtual field of view directed toward the visibility confirmation object at a predetermined position on the three-dimensional image where the vehicle is moving; A position where the visibility confirmation object can be seen is detected based on a determination result of whether the visibility confirmation object can be seen in the area image enlarged at the confirmation magnification. A visibility confirmation device characterized by:

2. 2. The visibility confirmation device according to claim 1, a recognition unit that recognizes the visibility confirmation object from the three-dimensional image; The position detection unit In an image corresponding to a virtual field of view directed toward the visibility confirmation object at a predetermined position on the three-dimensional image where the vehicle is moving, the area image is enlarged to a predetermined confirmation magnification; A position where the visibility confirmation object can be visually recognized is detected based on a determination result of whether the recognition unit can recognize the visibility confirmation object in the area image enlarged at the confirmation magnification. A visibility confirmation device characterized by:

3. 2. The visibility confirmation device according to claim 1, the video acquisition unit assigns relative distance information to each image frame of the acquired video based on the number of pixels corresponding to the size of a predetermined object whose size is known and the amount of movement of predetermined pixels corresponding to the same location in adjacent frames; The image extraction unit extracts the image frame group from the video based on the distance information assigned to each of the image frames. A visibility confirmation device characterized by:

4. 2. The visibility confirmation device according to claim 1, The position detection unit calculating a line of sight direction from the predetermined position and the position of the visibility confirmation object; Calculating a straight-line distance between the predetermined position and the visibility confirmation object in the line of sight direction; Calculating the magnification rate according to the linear distance A visibility confirmation device characterized by:

5. A visibility confirmation method executed by an information processing device, an acquisition step of acquiring an image of the track in the direction of travel of the vehicle, the image being captured by a camera installed on the vehicle; an object detection step of detecting at least one of a visibility confirmation object and a reference object that is in a predetermined positional relationship with the visibility confirmation object from the image acquired by the processing of the acquisition step; an image extraction step of detecting a reference image frame corresponding to the position of the visibility confirmation object based on the detection result of the object detection step, and extracting from the video a group of image frames located a predetermined distance in front of the reference image frame in the direction of travel of the vehicle; a three-dimensional image generating step of generating a three-dimensional image based on the image frame group extracted by the processing of the image extracting step; a position detection step of detecting a position at which the visibility confirmation object can be viewed based on the three-dimensional image generated by the processing of the three-dimensional image generation step; Including, In the processing of the position detection step, enlarging a predetermined area image in which the visibility confirmation object exists to a predetermined confirmation magnification in an image corresponding to a virtual field of view directed toward the visibility confirmation object at a predetermined position on the three-dimensional image where the vehicle is moving; A position where the visibility confirmation object can be seen is detected based on a determination result of whether the visibility confirmation object can be seen in the area image enlarged at the confirmation magnification. A method for checking visibility.

6. 3. The visibility confirmation device according to claim 2, the recognition unit recognizes the visibility confirmation object at each predetermined distance from the visibility confirmation object, The position detection unit outputs a recognition result for each of the predetermined distances by the recognition unit. A visibility confirmation device characterized by:

7. an image acquisition unit that acquires an image of the track in the direction of travel of the vehicle, the image being captured by an imaging device installed on the vehicle; an object detection unit that detects an object for which visibility is to be confirmed from the image acquired by the image acquisition unit; a three-dimensional image generating unit that generates a three-dimensional image based on the video acquired by the video acquiring unit; an enlarged image generating unit that enlarges an image of a predetermined area where the visibility confirmation object exists in an image corresponding to a virtual field of view directed toward the visibility confirmation object at a predetermined position on the three-dimensional image where the vehicle is moving, to a predetermined confirmation magnification, and generates an enlarged image; an output processing unit that outputs an image corresponding to a virtual field of view directed toward the visibility confirmation object at a predetermined position on the three-dimensional image and the enlarged image created by the enlarged image generating unit; an input information acquisition unit that acquires the input at the predetermined position; Equipped with The enlarged image generation unit enlarges, to a predetermined confirmation magnification, an image of a predetermined area in which the visibility confirmation object exists in an image corresponding to a virtual field of view directed toward the visibility confirmation object at the predetermined position acquired by the input information acquisition unit. A visibility confirmation device characterized by:

Images