Camera automatic inspection system

Abstract

The present invention provides a camera inspection system that can automatically and easily determine whether calibration of a camera mounted on a railway vehicle is required. The camera inspection system of this embodiment includes a shooting unit that is mounted on a railway vehicle and shoots a sign set around the track on which the railway vehicle is traveling at a first viewpoint and a second viewpoint. The operation processing unit calculates the parallax between a first image of the sign at the first viewpoint and a second image of the sign at the second viewpoint. The operation processing unit calculates a first distance between the railway vehicle and the sign based on the parallax between the first image and the second image. The operation processing unit calculates a second distance between the railway vehicle and the sign based on known attribute information related to the size and / or position of the sign and the length of the attribute information on the first image or the second image. The operation processing unit determines whether the difference between the first distance and the second distance is greater than a threshold value. The storage unit stores the attribute information.

Description

Technical Field

[0001] This embodiment relates to an automatic camera inspection system. Background Art

[0002] A system is being developed that uses a stereo camera mounted on a railway vehicle to capture images of the area in front of the moving railway vehicle and uses the resulting images to detect obstacles near the track.

[0003] Railway vehicles must detect obstacles at a distance closer than the stopping distance, which is the sum of the free-running distance until the system detects the obstacle and begins decelerating the railway vehicle, and the braking distance from the start of deceleration until the railway vehicle stops. Furthermore, compared to automobiles, railway vehicles have longer braking distances and travel on paved tracks, making it more difficult to avoid obstacles by braking or steering. Therefore, railway vehicles are required to accurately detect obstacles on their route from a relatively long distance. To accurately detect the presence and distance of obstacles, the railway vehicle's stereo cameras must be calibrated with high precision.

[0004] However, as time passes, there arises a problem that the relative positions of the cameras shift, causing a decrease in detection performance.

[0005] Therefore, in order to prevent performance degradation due to aging, a system is required that can automatically and easily determine whether camera calibration is necessary.

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Registration No. 6122365

[0009] Patent Document 2: Japanese Patent Application Laid-Open No. 2019-84881

[0010] Patent Document 3: Japanese Patent Application Laid-Open No. 2019-188996 Summary of the Invention

[0011] Problems to be solved by the invention

[0012] Provided is a camera inspection system capable of automatically and easily determining whether a camera mounted on a railway vehicle requires calibration.

[0013] Means for solving problems

[0014] The camera inspection system of this embodiment includes a shooting unit that is mounted on a railway vehicle and shoots a sign set around the track on which the railway vehicle is traveling at a first viewpoint and a second viewpoint. The operation processing unit calculates the parallax between the first image of the sign at the first viewpoint and the second image of the sign at the second viewpoint. The operation processing unit calculates the first distance between the railway vehicle and the sign based on the parallax between the first image and the second image. The operation processing unit calculates the second distance between the railway vehicle and the sign based on known attribute information related to the size and / or position of the sign and the length of the attribute information on the first image or the second image. The operation processing unit determines whether the difference between the first distance and the second distance is greater than a threshold value. The storage unit stores the attribute information. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a diagram showing an example of a schematic configuration of a railway vehicle according to the present embodiment.

[0016] Figure 2 This is a block diagram showing an example of the functional configuration of the running obstacle detection device included in the railway vehicle according to the present embodiment.

[0017] Figure 3 It is a diagram showing a setting example of a monitoring area in a railway vehicle according to the present embodiment.

[0018] Figure 4 It is a diagram showing a setting example of a monitoring area in a railway vehicle according to the present embodiment.

[0019] Figure 5 This is a diagram showing an example of the flow of obstacle detection processing in the railway vehicle according to the present embodiment.

[0020] Figure 6 This is a schematic diagram for explaining an example of a method for setting a monitoring area in a railway vehicle according to the present embodiment.

[0021] Figure 7 This is a diagram for explaining an example of a reference plane setting process in the railway vehicle according to the present embodiment.

[0022] Figure 8 This is a block diagram showing an example of the configuration of the camera inspection system according to the first embodiment.

[0023] Figure 9 This is a conceptual diagram showing a method for calculating the second distance.

[0024] Figure 10 This is a flowchart showing an example of the operation of the camera inspection system according to the first embodiment.

[0025] Figure 11This is a conceptual diagram showing a method for calculating the second distance in Modification 1 of the first embodiment.

[0026] Figure 12 This is a conceptual diagram showing a method for calculating the second distance in Modification 2 of the first embodiment. DETAILED DESCRIPTION

[0027] The following describes embodiments of the present invention with reference to the accompanying drawings. These embodiments do not limit the present invention. The accompanying drawings are schematic or conceptual diagrams, and the proportions of the various parts are not necessarily the same as the actual proportions. In the specification and drawings, the same reference numerals are used for elements that are identical to those described in the accompanying drawings, and detailed descriptions are omitted as appropriate.

[0028] Figure 1 1 is a diagram showing an example of the schematic configuration of a railway vehicle according to this embodiment. Figure 1 As shown, the railway vehicle RV of this embodiment includes a sensor 10, a running obstacle detection device 20, a storage device 30, and a display device 40. The sensor 10 is an example of an imaging unit configured to capture images of the direction of travel of the railway vehicle RV. The running obstacle detection device 20 is an example of an obstacle detection device that detects obstacles that may hinder the travel of the railway vehicle RV. The storage device 30 stores the obstacle detection results of the running obstacle detection device 20. The display device 40 displays various information, including images captured by the sensor 10 in the direction of travel of the railway vehicle RV and the obstacle detection results of the running obstacle detection device 20.

[0029] Figure 2 1 is a block diagram showing an example of the functional configuration of the running obstacle detection device of the railway vehicle according to the present embodiment. Figure 2 As shown, the traveling obstacle detection device 20 includes an image acquisition unit 201, a track recognition unit 202, a monitoring area recognition unit 203, an obstacle candidate recognition unit 204, an obstacle determination unit 205, a result output unit 206, and a storage unit 207. In this embodiment, some or all of the image acquisition unit 201, the track recognition unit 202, the monitoring area recognition unit 203, the obstacle candidate recognition unit 204, the obstacle determination unit 205, and the result output unit 206 are implemented by a processor such as a CPU (Central Processing Unit) included in the railway vehicle RV executing software stored in the storage unit 207.

[0030] Furthermore, some or all of the image acquisition unit 201, track recognition unit 202, monitoring area recognition unit 203, candidate obstacle recognition unit 204, obstacle determination unit 205, and result output unit 206 may be implemented using a circuit board, i.e., hardware, such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). Furthermore, the image acquisition unit 201, track recognition unit 202, monitoring area recognition unit 203, candidate obstacle recognition unit 204, obstacle determination unit 205, and result output unit 206 may be implemented through a combination of software executed by a processor and hardware.

[0031] The storage unit 207 includes nonvolatile storage media such as ROM (Read Only Memory), flash memory, HDD (Hard Disk Drive), and SD card, and volatile storage media such as RAM (Random Access Memory) and registers. Furthermore, the storage unit 207 stores various information such as programs executed by the processor of the railway vehicle RV.

[0032] The image acquisition unit 201 is an example of an acquisition unit that acquires a visible light image (hereinafter referred to as a "color image") obtained by the sensor 10 capturing images of the direction of travel of the railway vehicle RV, and a range image that enables determination of the distance to objects within the imaging range of the sensor 10. In this embodiment, the image acquisition unit 201 acquires, as a color image, any one of a plurality of images captured by the plurality of imaging units included in the sensor 10 in the direction of travel of the railway vehicle RV from a plurality of different viewpoints.

[0033] In addition, the image acquisition unit 201 acquires the multiple images as distance images. Specifically, the image acquisition unit 201 converts the multiple images obtained by shooting with the multiple shooting units included in the sensor 10 into grayscale images (hereinafter referred to as "grayscale images"). Next, the image acquisition unit 201 geometrically converts the grayscale image of one camera into an image obtained when the optical axes of the multiple shooting units are hypothetically parallel (hereinafter referred to as "parallel equipotential images"). Next, the image acquisition unit 201 uses the parallel equipotential images and other grayscale images in accordance with the principle of parallel stereo vision to acquire a distance image that can determine the distance to an object within the shooting range of the shooting unit included in the sensor 10.

[0034] The track recognition unit 202 functions as an example of a track detection unit that detects the track on which the railway vehicle RV is traveling within a color image. In this embodiment, the track recognition unit 202 converts the grayscale image obtained by capturing the color image by the sensor 10 into an image with emphasized edges (hereinafter referred to as an "edge-emphasized image"). Next, the track recognition unit 202 generates a pixel group obtained by grouping pixels in the edge-emphasized image whose edge strength is greater than a predetermined edge strength and pixels with the same edge strength among the surrounding pixels. Next, the track recognition unit 202 extracts from the generated pixel group a group of pixels that are connected to form a line segment in the direction of travel of the railway vehicle RV. Furthermore, the track recognition unit 202 detects from the extracted pixel group a group of pixels that meets a predetermined evaluation condition as the track on which the railway vehicle RV is traveling. Here, the evaluation condition is a group of pixels that is determined to be a track, for example, a group of pixels that has an edge strength that is determined to be a track.

[0035] The monitoring area identification unit 203 functions as an example of a monitoring area setting unit that sets a three-dimensional monitoring area for the color image based on the track detected by the track identification unit 202. Here, the monitoring area is a three-dimensional area defined by the vehicle boundaries of the railway vehicle RV or the building boundaries within the color image. Furthermore, the monitoring area identification unit 203 determines range information, or a range, that allows the range of the distance from the sensor 10 to the monitoring area to be determined based on the color image.

[0036] Figure 3 as well as Figure 4 : is a diagram showing an example of setting a monitoring area in a railway vehicle according to this embodiment. Figure 3 As shown, the monitoring area identification unit 203 determines the vehicle boundary cross-section X and the distance to the vehicle boundary cross-section X at any set position P of the detected track, based on constraints such as the installation conditions of the sensor 10 and the constant width of the track detected by the track identification unit 202. Here, the vehicle boundary cross-section X is the cross-section of the vehicle boundary in a direction perpendicular to the direction of travel of the railway vehicle RV. The monitoring area identification unit 203 sets a tunnel-shaped area with the vehicle boundary cross-section X determined for different set positions P of the track in the direction of travel of the railway vehicle RV as the monitoring area. Furthermore, the monitoring area identification unit 203 determines, as range information, the range (set) of distances to the vehicle boundary cross-section X determined for each set position P.

[0037] Or, as Figure 4As shown, the monitoring area identification unit 203 determines the building boundary cross-section Y and the distance to the building boundary cross-section Y at any set position P of the detected track, based on constraints such as the installation conditions of the sensor 10 and the constant width of the track detected by the track identification unit 202. Here, the building boundary cross-section Y is the cross-section of the building boundary at the set position P, perpendicular to the direction of travel of the railway vehicle RV. The monitoring area identification unit 203 sets a tunnel-shaped area with the building boundary cross-section Y determined for different set positions P of the track in the direction of travel of the railway vehicle RV as the monitoring area. Furthermore, the monitoring area identification unit 203 determines, as range information, the range (set) of distances to the building boundary cross-section Y determined for each set position P.

[0038] return Figure 2 Based on the range image and the range information, the obstacle candidate identification unit 204 extracts objects within the monitoring area as obstacle candidates (hereinafter referred to as "obstacle candidates"). In this embodiment, the obstacle candidate identification unit 204 extracts three-dimensional objects as obstacle candidates. These three-dimensional objects are located within the monitoring area and are formed by grouping pixels within the range image that have a distance that matches the distance determined by the range information.

[0039] The obstacle determination unit 205 detects obstacles from the obstacle candidates based on at least one of their size and their motion vectors, as extracted by the obstacle candidate identification unit 204. This allows objects within the monitoring area that are more likely to be obstacles to be detected as obstacles, thereby improving obstacle detection accuracy. In this embodiment, the obstacle determination unit 205 detects as obstacles any obstacle candidate with a predetermined size or larger. Furthermore, the obstacle determination unit 205 detects as obstacles any obstacle candidate with a motion vector pointing toward the track. Alternatively, the obstacle determination unit 205 may detect as obstacles any obstacle candidate with a predetermined size or larger and a motion vector pointing toward the track.

[0040] Therefore, in this embodiment, the obstacle candidate identification unit 204 cooperates with the obstacle determination unit 205 to function as an example of an obstacle detection unit that detects obstacles that are present within the monitoring area and that hinder the movement of the railway vehicle RV based on the range image and range information. Thus, when using a color image captured in the direction of travel of the railway vehicle RV to detect obstacles that hinder the movement of the railway vehicle RV, the obstacle detection monitoring area can be dynamically and appropriately set within the color image, thereby enabling high-precision detection of obstacles that hinder the movement of the railway vehicle RV.

[0041] The result output unit 206 functions as an example of an output unit that outputs the obstacle detection results from the obstacle candidate identification unit 204 and the obstacle determination unit 205. In this embodiment, when the obstacle determination unit 205 detects an obstacle, the result output unit 206 displays information indicating the detection of the obstacle along with the color image on the display device 40. Furthermore, the result output unit 206 stores the obstacle detection results in the storage device 30.

[0042] Next, use Figure 5 An example of the flow of obstacle detection processing in the railway vehicle according to the present embodiment will be described. Figure 5 This is a diagram showing an example of the flow of obstacle detection processing in the railway vehicle according to the present embodiment.

[0043] The image acquisition unit 201 acquires a color image and a distance image from the sensor 10 (step S501). The sensor 10 is an optical sensor capable of simultaneously outputting a color image and a distance image, and employs a pattern projection method, a photogrammetry method, or a time-of-flight method. Furthermore, if the sensor 10 includes two imaging units (left and right imaging units) spaced apart in the width direction of the railway vehicle RV, the image acquisition unit 201 may also acquire two color images captured by the two imaging units as the distance image.

[0044] When two color images are acquired as range images, the image acquisition unit 201 converts each of the two color images into a grayscale image (step S502). This embodiment describes an example in which the image acquisition unit 201 converts the two color images into grayscale images to reduce the computational complexity required for obstacle detection and then uses the grayscale images to detect obstacles. However, obstacle detection can also be performed using color images directly.

[0045] Next, the image acquisition unit 201 corrects the distortion caused by the aberration of the lens of the sensor 10, etc. for each of the two grayscale images, and geometrically converts one of the two grayscale images into a parallel equidistant image (step S503). The lens of a general camera has distortion in the radial direction and distortion in the circumferential direction. Therefore, the image acquisition unit 201 stores the coefficients and tables required for the distortion correction of the image obtained by shooting with the sensor 10 as image conversion information in the storage unit 207. In addition, the image acquisition unit 201 uses the image conversion information stored in the storage unit 207 to correct the distortion of the grayscale image. In addition, the internal parameters of the sensor 10 and the external parameters based on the relative positional relationship can be obtained through calibration, and the external parameters can be stored in the storage unit 207 as image conversion information.

[0046] Next, the image acquisition unit 201 uses the parallel equidistant image as the reference image and the grayscale image as the reference image, and calculates the parallax between the reference image and the reference image as a distance image (step S504). Based on geometric constraints such as epipolar geometry, the pixel in the reference image corresponding to the pixel of interest in the reference image, i.e., the corresponding pixel, is found by searching only on epipolar lines that are at the same height as the pixel of interest in the reference image. This allows for high-precision searching of corresponding pixels with minimal computational effort. The search for corresponding pixels is performed by, for example, template matching that searches for corresponding pixels in a small area within the reference image, global matching that searches for the best corresponding pixel (solution) from the entire reference image, or semi-global matching that searches for the best corresponding pixel from pixels surrounding the pixel of interest in the reference image. However, when the search for corresponding pixels requires real-time performance, template matching is often used to search for corresponding pixels.

[0047] After obtaining the corresponding pixel, the image acquisition unit 201 obtains the parallax between the pixel of interest of the base image and the corresponding pixel of the reference image. In the present embodiment, the image acquisition unit 201 obtains the corresponding pixel of the reference image while offsetting the pixel of interest of the base image by 1 pixel at a time. Therefore, when the reference image is a digital image, the coordinates of the corresponding pixel are represented by integers. Therefore, when estimating the coordinates of the corresponding pixel with an accuracy below the decimal point, the sub-pixel estimation method is used. When the evaluation function used in the search for the corresponding pixel is a linear function such as SAD (Sum of Absolute Difference), the sub-pixel estimation method uses equiangular straight line fitting. On the other hand, when the evaluation function used in the search for the corresponding pixel is a quadratic function such as SSD (Sum of Squared Difference), the sub-pixel estimation method uses parabola fitting.

[0048] Furthermore, when it is not necessary to calculate the parallax with sub-decimal accuracy or when it is desired to shorten the processing time required to obtain the parallax, it is possible to calculate the parallax without using the sub-pixel estimation method. Furthermore, when the evaluation value of the coordinates of the corresponding pixel based on the evaluation function used in the sub-pixel estimation method is greater than a preset threshold, or when the difference in the feature quantity (e.g., grayscale change) between the pixel of interest and the corresponding pixel is small, the calculated parallax can be invalidated.

[0049] Through the above processing, the image acquisition unit 201 calculates the parallax between all pixels of the reference image or pixels within a portion of the effective area of ​​the reference image and the pixels of the reference image. Alternatively, the image acquisition unit 201 can use the following equation 1 to calculate the distance Z to the object represented by the pixels of the reference image as a distance image. In equation 1, f represents the focal length of the two cameras included in the sensor 10, B represents the distance between the two cameras included in the sensor 10, and D represents the parallax D between the pixels of the reference image and the pixels of the reference image.

[0050] Z = f × B / D (Equation 1)

[0051] Next, the track recognition unit 202 detects candidates for the track on which the railway vehicle RV is traveling, i.e., track candidates, from the reference image (step S505). In this embodiment, the track recognition unit 202 converts the reference image into an edge-emphasized image. Next, the track recognition unit 202 detects, in the edge-emphasized image, a group of pixels that are connected to form a line segment in the direction of travel of the railway vehicle RV, from a group of pixels obtained by grouping pixels having an edge strength greater than a predetermined edge strength and pixels having the same edge strength among their surrounding pixels, as track candidates. Then, the track recognition unit 202 detects, among the detected track candidates, those that meet the predetermined evaluation conditions as tracks (step S506).

[0052] The monitoring area recognition unit 203 sets a three-dimensional monitoring area for the reference image based on the track detected by the track recognition unit 202 (step S507 ).

[0053] Figure 6 Schematic diagram for explaining an example of a method for setting a monitoring area in a railway vehicle according to this embodiment. Figure 6As shown, the monitoring area recognition unit 203 establishes a horizontal line segment connecting the left and right rails at any set position P on the track R detected by the track recognition unit 202. Here, the line segment has a length of L pixels in the reference image. Furthermore, if the pitch, yaw, and roll angles of the camera included in the sensor 10 are all 0° and the track on which the railway vehicle RV travels is a narrow-gauge track, the resolution per horizontal pixel is (1067 / L) mm.

[0054] Alternatively, the monitoring area identification unit 203 uses the range image to obtain the parallax between the target pixel at the set position P and the corresponding pixel, or the distance to the set position P. Next, the monitoring area identification unit 203 pre-calculates the resolution per pixel corresponding to the parallax or distance, and determines the resolution per pixel corresponding to the obtained parallax or distance. Furthermore, the monitoring area identification unit 203 may set the determined resolution per pixel as the resolution per pixel in the horizontal direction at the set position P. Furthermore, when the aspect ratio of the imaging element of the sensor 10 is 1:1, the resolution per pixel in the direction perpendicular to the line segment (i.e., the vertical direction) is the same as the resolution per pixel in the horizontal direction.

[0055] Then, if Figure 6 As shown, the monitoring area recognition unit 203 sets a vehicle boundary cross section X for a set position P within the reference image based on the dimensions of the vehicle boundary in real space (e.g., width: 3800 mm, height: 4300 mm) and the resolution per pixel of the reference image. In this embodiment, the monitoring area recognition unit 203 sets an area that mimics the cross section of the vehicle boundary as the vehicle boundary cross section X. However, this is not limited to setting an area that can identify the vehicle boundary as the vehicle boundary cross section X. For example, the monitoring area recognition unit 203 may also set an area with a predetermined shape, such as a rectangle, that approximates the cross section of the vehicle boundary as the vehicle boundary cross section X. While this example describes setting the vehicle boundary cross section X for the reference image, the same applies when setting a building boundary cross section Y for the reference image. The monitoring area recognition unit 203 then repeatedly sets the vehicle boundary cross section X for the set position P within the reference image while moving the set position P along the direction of travel of the railway vehicle RV, thereby establishing a tunnel-shaped monitoring area within the reference image.

[0056] Furthermore, the monitoring area identification unit 203 calculates the distance to the set position P or the parallax between the target pixel and the corresponding pixel at the set position P for each set position P based on the resolution of each pixel of the reference image and the number of pixels from the bottom end of the reference image (i.e., the near-front end of the imaging range of the sensor 10). The monitoring area identification unit 203 sets the range of the distance or parallax calculated for each set position P as range information. Furthermore, the monitoring area identification unit 203 stores the range information as monitoring area information in the storage unit 207.

[0057] return Figure 5 The obstacle candidate identification unit 204 sets a reference plane in the reference image that is in contact with the upper surface of the track detected by the track identification unit 202 (step S508). This prevents objects located below the track and less likely to be obstacles from being detected as obstacle candidates, thereby reducing the processing load of obstacle detection.

[0058] Figure 7 1 is a diagram for explaining an example of a process for setting a reference plane in a railway vehicle according to this embodiment. Figure 7 As shown, the obstacle candidate identification unit 204 sets the plane or curved surface on the upper surface of the track R identified by the track identification unit 202 as the reference plane P1 or P2. The plane or curved surface approximates the surface contacting the area with a short distance or parallax from the sensor 10.

[0059] return Figure 5 Next, the obstacle candidate identification unit 204 extracts objects within the monitoring area of ​​the reference image as obstacle candidates based on the range image representing the disparity (or distance) between the reference image and the reference image and the range information (step S509). Specifically, the obstacle candidate identification unit 204 extracts, from among the objects within the reference image, objects whose disparity (or distance) between the reference image and the reference image falls within the range indicated by the range information, as obstacle candidates.

[0060] However, there is a high probability that an object within the monitoring area, where only a portion of it enters the monitoring area, will not be detected as an obstacle due to reasons such as the object's size not meeting a predetermined size. Therefore, in this embodiment, the obstacle candidate identification unit 204 extracts objects located above the reference plane as obstacle candidates. Specifically, the obstacle candidate identification unit 204 extracts objects within the monitoring area of ​​the reference image that are located in front of the reference plane, in other words, objects that are closer to the sensor 10 than the reference plane (or objects that have a larger parallax than the reference plane) as obstacle candidates.

[0061] The obstacle determination unit 205 detects obstacles from the obstacle candidates based on at least one of their size and their motion vectors (step S510). In this embodiment, the obstacle determination unit 205 discards obstacle candidates whose aspect ratios of the circumscribed rectangles within the reference image are extremely unbalanced as noise or external interference. Furthermore, the obstacle determination unit 205 calculates the motion vectors of obstacle candidates between two or more reference images and discards them as external interference, such as birds or smoke, if the motion vectors point upward. The obstacle determination unit 205 then detects as obstacles any remaining obstacle candidates whose centers of gravity enter the monitoring area.

[0062] Next, the result output unit 206 outputs the obstacle detection results of the obstacle determination unit 205 to the display device 40, causing the display device 40 to display the obstacle detection results (step S511). In this embodiment, the result output unit 206 displays the obstacle detection results, such as the obstacle detection status and the distance to the detected obstacle, on the display device 40. Furthermore, the result output unit 206 outputs a sound notifying the obstacle detection results or outputs the obstacle detection results to the vehicle control device controlling the movement of the railway vehicle RV. Furthermore, the result output unit 206 stores detection result information indicating the obstacle detection results in the storage unit 207. Furthermore, the result output unit 206 can also store the detection result information in an external storage unit. This allows external devices to analyze the obstacle detection results using the detection result information stored in the external storage unit, even in an offline state where communication with the railway vehicle RV's travel obstacle detection device 20 is impossible.

[0063] The image acquisition unit 201 then determines whether it has been instructed to terminate the obstacle detection process (step S512). If it has been instructed to terminate the obstacle detection process (step S512: Yes), the image acquisition unit 201 terminates the obstacle detection process without acquiring a new color image or range image. On the other hand, if it has not been instructed to terminate the obstacle detection process (step S512: No), the image acquisition unit 201 returns to step S501 and acquires a new color image and range image.

[0064] In this way, according to the railway vehicle RV of this embodiment, when using a color image obtained by photographing the direction of travel of the railway vehicle RV to detect an obstacle that becomes an obstacle to the movement of the railway vehicle RV, it is possible to dynamically and appropriately set a monitoring area for detecting the obstacle in the color image and detect the obstacle, thereby being able to detect the obstacle that becomes an obstacle to the movement of the railway vehicle RV with high precision.

[0065] (First embodiment)

[0066] In the first embodiment, a camera inspection system 21 is further configured based on the aforementioned traveling obstacle detection device 20. Instead of detecting obstacles, the camera inspection system 21 detects markers 50 located around the rails on which the railway vehicle RV travels. The camera inspection system 21 only needs to detect the markers 50 in the same manner as it detects obstacles. The markers 50 will be described later. The following describes the configuration of the camera inspection system 21 and the processing performed after detecting the markers 50.

[0067] Figure 8 This is a block diagram showing an example of the configuration of a camera inspection system according to the first embodiment. The camera inspection system 21 includes a sensor 10, a display device 40, a storage device 30, a storage unit 207, and a processing unit 208. The sensor 10, the storage device 30, the display device 40, and the storage unit 207 are basically as described in the embodiment of the present invention. Figure 2 As explained.

[0068] The processing unit 208 includes an image acquisition unit 201, a marker recognition unit 210, a first distance calculation unit 220, a second distance calculation unit 230, a distance comparison and determination unit 240, and an output unit 250. A portion or all of the processing unit 208 is implemented by a processor such as a CPU included in the railway vehicle RV executing software stored in the storage unit 207.

[0069] A sensor 10, serving as an imaging device, is mounted on a railway vehicle and images a sign 50 located around the track on which the railway vehicle travels, from a first viewpoint and a second viewpoint. The first viewpoint and the second viewpoint are separated by a predetermined distance and are located at different positions. The sensor 10 is a so-called stereo camera, which uses multiple cameras with these first and second viewpoints to capture the same sign 50 almost simultaneously.

[0070] The image acquisition unit 201 is Figure 2 The image acquisition unit 201 has basically the same configuration as that of FIG. The image acquisition unit 201 calculates the parallax between the first image of the marker 50 at the first viewpoint and the second image of the marker at the second viewpoint.

[0071] The identification unit 210 includes Figure 2Track recognition unit 202, monitoring area recognition unit 203, obstacle candidate recognition unit 204, and obstacle determination unit 205 recognize marker 50 from the first and second images. Images of marker 50 are pre-registered in storage unit 207 as a recognition dictionary. Marker recognition unit 210 uses any pattern image recognition technology to determine whether the image matches marker 50. The method for recognizing marker 50 may be the same as the obstacle recognition method described in the above embodiment.

[0072] The first distance calculator 220 receives the parallax between the first image of the marker 50 at the first viewpoint and the second image of the marker 50 at the second viewpoint, calculated by the image acquisition unit 201. Based on the parallax between the first and second images, the first distance calculator 220 calculates a first distance Z1 between the railway vehicle RV and the marker 50. Specifically, the first distance Z1 is calculated using the stereo camera function of the sensor 10.

[0073] For example, the first distance Z1 is the distance between the railway vehicle RV and the marker 50, calculated using the parallax between the first and second images. F is the focal length of the sensor 10. B is the distance (interval) between the first and second viewpoints of the sensor 10. In other words, B is the distance between the two cameras of the sensor 10. D is the parallax between the first and second images. In this case, the first distance Z1 is expressed as follows using the above-mentioned equation 1.

[0074] Z1=f×B / D (Equation 1)

[0075] Furthermore, the focal length f of the sensor 10 and the distance B between the first viewpoint and the second viewpoint are known in advance and stored in the storage unit 207 .

[0076] The first distance Z1 is calculated based on both the first and second images using the parallax between the two cameras of the sensor 10. Therefore, if the positions, angles, and other configurations of the two cameras of the sensor 10 change, the first distance Z1 deviates from the distance between the railway vehicle RV and the marker 50. In other words, the first distance Z1 is susceptible to the effects of aging of the sensor 10.

[0077] The second distance calculator 230 calculates the second distance Z2 between the railway vehicle and the marker 50 based on known attribute information related to the size and / or position of the marker 50 and the length of the attribute information on the first image or the second image.

[0078] For example, Figure 9This is a conceptual diagram showing a method for calculating the second distance. The marker 50 is fixed on the ground around the track R and has a shape of a predetermined size. When the actual lateral length (width) of the marker 50 is set to d1, the length d1 is stored in the storage unit 207 as pre-known (pre-set) attribute information. In addition, the marker 50 is designed to be easily recognized by the sensor 10 and is fixedly installed in the monitoring area around the track R. The marker 50 is preferably installed in an environment with a good field of view and is not easily affected by external interference. For example, the marker 50 is suitable for installation in a vehicle base, etc.

[0079] Meanwhile, a first image and a second image are acquired by capturing the marker 50 while the railway vehicle RV is traveling. The size of the marker 50 in the first and second images varies depending on the distance from the railway vehicle RV to the marker 50. When the marker 50 is farther from the railway vehicle RV, the size of the marker 50 in the image appears smaller. Conversely, when the marker 50 is closer to the railway vehicle RV, the marker 50 appears larger. Using this change in the size of the marker 50 in the image according to the distance between the railway vehicle RV and the marker 50, the actual distance Z2 between the railway vehicle RV and the marker 50 is determined.

[0080] For example, let x be the number of pixels representing the width of the marker 50 displayed on the first or second image of the display device 40, and let s be the distance between two adjacent pixels of the display device 40. In this case, the width of the marker 50 displayed on the display device 40 on the screen is x × s. Furthermore, when the focal length of the sensor 10 is f, the actual second distance Z2 between the marker 50 and the railway vehicle RV is expressed as shown in Equation 2.

[0081] Z2=f×d1 / (x×s) (Equation 2)

[0082] The second distance Z2 can be calculated using either the first or second image and is not affected by parallax between the first and second images. Therefore, even if the positions, angles, and other configurations of the two cameras of the sensor 10 change, the second distance Z2 can still accurately represent the distance between the railway vehicle RV and the sign 50. In other words, the second distance Z2 is less susceptible to aging of the sensor 10. Alternatively, the second distance calculator 230 may calculate the second distance Z2 for each of the first and second images and selectively use either one, or may use the average of the second distances Z2 for each of the first and second images.

[0083] The attribute information may be any known information related to the size and / or position of the marker 50. For example, the attribute information may be the horizontal length (width) d1 of the marker 50 or the vertical length of the marker 50. Alternatively, the attribute information may be the length d2 of the pattern depicted on the marker 50. Other examples of attribute information will be described later.

[0084] The distance comparison and determination unit 240 calculates the difference between the first distance Z1 and the second distance Z2 and compares the difference with a threshold value. If the difference between the first distance Z1 and the second distance Z2 is greater than the threshold value, the distance comparison and determination unit 240 determines that there is an abnormality in the arrangement of the two cameras of the sensor 10. On the other hand, if the difference between the first distance Z1 and the second distance Z2 is less than the threshold value, the distance comparison and determination unit 240 determines that the arrangement of the two cameras of the sensor 10 is normal. If it is determined that there is an abnormality in the sensor 10, it is necessary to calibrate the two cameras of the sensor 10. The threshold value may be pre-set and stored in the storage unit 207. Furthermore, the result information of the normal / abnormal determination performed by the distance comparison and determination unit 240 may be stored in the storage unit 207.

[0085] The output unit 250 outputs the result information of the normal / abnormal determination performed by the distance comparison and determination unit 240. The result information of the normal / abnormal determination output from the output unit 250 is stored in the storage device 30 and displayed on the display device 40.

[0086] When the difference between the first distance and the second distance is greater than a threshold value, the display device 40 displays an alarm indicating an abnormality in the sensor 10. The display device 40 may display both the normal determination and the abnormal determination of the sensor 10, or may be configured to display only the abnormal determination without displaying the normal determination.

[0087] When the display device 40 displays an abnormality determination, the arrangement of the two cameras of the sensor 10 and the like are corrected and maintained.

[0088] Next, the operation of the camera inspection system 21 according to this embodiment will be described.

[0089] Figure 10 This is a flowchart showing an example of the operation of the camera inspection system 21 according to the first embodiment.

[0090] First, the sensor 10 captures the image of the marker 50 at the first viewpoint and the second viewpoint ( S10 ). The sensor 10 captures the image of the same marker 50 substantially simultaneously using a plurality of cameras having the first viewpoint and the second viewpoint.

[0091] Next, the image acquisition unit 201 obtains a parallax between the first image of the marker 50 at the first viewpoint and the second image of the marker at the second viewpoint ( S20 ).

[0092] Next, the first distance calculation unit 220 receives the parallax between the first image of the identifier 50 at the first viewpoint obtained by the image acquisition unit 201 and the second image of the identifier 50 at the second viewpoint. Based on the parallax between the first image and the second image, the first distance calculation unit 220 calculates the first distance Z1 between the railway vehicle RV and the identifier 50 using Equation 1 (S30). That is, the first distance Z1 is calculated using the function of the stereo camera of the sensor 10.

[0093] Next, the second distance calculation unit 230 calculates the second distance Z2 between the railway vehicle RV and the identifier 50 using Equation 2 based on the size and / or position of the identifier 50 and the length of the size and / or position of the identifier 50 on the first image or the second image (S40).

[0094] Next, the distance comparison determination unit 240 calculates the difference |Z1 - Z2| between the first distance Z1 and the second distance Z2, and compares this difference with the threshold Th (S50). When the difference between the first distance Z1 and the second distance Z2 is greater than or equal to the threshold (|Z1 - Z2| ≧ Th), the distance comparison determination unit 240 determines that there is an abnormality in the configuration of the two cameras of the sensor 10 (S60). On the other hand, when the difference between the first distance Z1 and the second distance Z2 is less than the threshold (|Z1 - Z2| < Th), the distance comparison determination unit 240 determines that the configuration of the two cameras of the sensor 10 is normal (S70). In addition, the threshold only needs to be preset and stored in the storage unit 207. In addition, the result information of the normal / abnormal determination performed by the distance comparison determination unit 240 only needs to be stored in the storage unit 207.

[0095] The output unit 250 outputs the result information of the normal / abnormal determination performed by the distance comparison determination unit 240 (S80). The result information of the normal / abnormal determination output from the output unit 250 is stored in the storage device 30 and displayed on the display device 40 (S90). The display device 40 can display both the normal determination and the abnormal determination of the sensor 10, but it can also be set to only display the abnormal determination and not display the normal determination.

[0096] When the display device 40 displays an abnormal determination, the configuration of the two cameras of the sensor 10 and the like is corrected.

[0097] The process is periodically repeated during the running of the railway vehicle RV Figure 10 By doing so, the camera inspection system 21 can judge the abnormality of the sensor 10 in real time during the running of the railway vehicle RV and learn about the abnormality on the spot. Therefore, there is no need to wait until the regular inspection of the railway vehicle RV, and it is possible to automatically and easily determine whether the sensor 10 needs to be corrected.

[0098] The railway vehicle RV usually travels on a specific track R. Therefore, for example, if the camera inspection process of the present embodiment is executed every time the vehicle departs from the vehicle base, it is possible to detect the degradation of the obstacle detection performance of the sensor 10 at an early stage.

[0099] (Variation 1)

[0100] Figure 11 This is a conceptual diagram illustrating a method for calculating the second distance Z2 in Variation 1 of the first embodiment. Multiple markers 50_1 and 50_2 may be provided, and the attribute information may be the distance d3 between markers 50_1 and 50_2. Markers 50_1 and 50_2 are arranged in a direction approximately perpendicular to the track R and are positioned approximately the same distance from the railway vehicle RV. Therefore, distance d3 is the distance between markers 50_1 and 50_2 in a direction approximately perpendicular to the track R. In this manner, the attribute information may also be the distance between multiple markers 50_1 and 50_2. The actual distance d3 is stored in advance in the storage unit 207.

[0101] The patterns of the marks 50_1 and 50_2 may be the same, but are not particularly limited as long as they are registered in the storage unit 207 and can be recognized as marks, and may have different patterns.

[0102] (Variation 2)

[0103] Figure 12 This is a conceptual diagram illustrating a method for calculating the second distance Z2 in a second variation of the first embodiment. Multiple markers 50_3 and 50_4 are provided, and the attribute information is the distance d4 between markers 50_3 and 50_4 along the track R. Markers 50_3 and 50_4 are located at different positions on either side of the track R, in a direction generally parallel to the track R. Therefore, distance d4 is the distance between markers 50_3 and 50_4 in a direction generally parallel to the track R. In this manner, the attribute information may also be the distance between multiple markers 50_3 and 50_4. The actual distance d4 is pre-stored in the storage unit 207.

[0104] Furthermore, the marks 50_3 and 50_4 are distinguished from each other by making their patterns different from each other. By distinguishing the marks 50_3 and 50_4, it can be seen that the mark 50_3 is arranged at the front and the mark 50_4 is arranged at the back.

[0105] Even in such modified examples 1 and 2, the same effects as those of the above-mentioned first embodiment can be obtained.

[0106] While several embodiments of the present invention have been described, these embodiments are provided as examples and are not intended to limit the scope of the invention. These embodiments may be implemented in various other ways, and various omissions, substitutions, and modifications may be made without departing from the gist of the invention. These embodiments and their variations are intended to be included within the scope and gist of the invention, and are intended to be included within the invention set forth in the claims and their equivalents.

Claims

1. A camera inspection system comprising: an imaging unit mounted on the railway vehicle and configured to capture images of signs provided around a track on which the railway vehicle is traveling from a first viewpoint and a second viewpoint; a processing unit that obtains a parallax between a first image of the marker at the first viewpoint and a second image of the marker at the second viewpoint, calculates a first distance between the railway vehicle and the marker based on the parallax between the first image and the second image, calculates a second distance between the railway vehicle and the marker based on known attribute information related to the size and / or position of the marker and the attribute information on the first image or the second image, and determines whether the imaging unit is normal or abnormal based on whether the difference between the first distance and the second distance is greater than a threshold value; and A storage unit stores the above attribute information. A plurality of the above-mentioned markers are provided, and the above-mentioned attribute information is the distance between the above-mentioned plurality of markers.

2. The camera inspection system according to claim 1, wherein: The arithmetic processing unit calculates the first distance using Formula 1. Z1=f×B / D (Equation 1) Here, Z1 is the first distance between the railway vehicle and the sign, f is the focal length of the imaging unit, B is the distance between the first viewpoint and the second viewpoint of the imaging unit, and D is the parallax.

3. The camera inspection system according to claim 1, wherein: The arithmetic processing unit calculates the second distance using Formula 2. Z2=f×d / (x×s) (Equation 2) Here, Z2 is the second distance between the railway vehicle and the sign, f is the focal length of the shooting unit, d is the actual length of the attribute information of the sign, x is the number of pixels of the attribute information on the first image or the second image, and s is the distance between adjacent pixels of the display unit displaying the first image and / or the second image.

Images