A method for detecting foreign object intrusion on track based on light field depth image
Through the combination of light field camera and Lytro Desktop software, the edge and depth images of tracks are extracted, and foreign objects on railway tracks are automatically detected and alarms are issued, solving the problems of high detection costs and high labor consumption in the existing technology, achieving high accuracy and low accuracy Cost of railway track foreign matter detection.
Patent Information
- Application Number
- CN202111300346.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-04
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-11-04
AI Technical Summary
The prior art has failed to effectively detect foreign matter intrusions in railway track areas, resulting in railway safety threats. The detection methods are costly and labor-intensive, and lack of automated alarm functions.
The light field camera is used to capture the background image and depth image of the track, and the edge of the track is extracted through Lytro Desktop software, a hazardous area detection window is established, pre-processing and significant object detection are performed, foreign objects are automatically identified and alarm signals are issued.
It realizes high-accuracy detection of foreign objects on railway tracks, saves labor costs, realizes automated alarms, and reduces detection costs.
Smart Images

Figure CN114140742B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method for detecting track foreign object intrusion based on a light field camera. Background Art
[0002] Railway transportation has the advantages of strong transportation capacity, high efficiency and low operating cost. It has developed rapidly in my country in recent years. Foreign objects invading the railway safety clearance may cause traffic accidents and endanger the safety and stability of railway transportation, which has attracted great attention from the society. Once foreign objects invade the railway track, it will endanger the safe operation of the railway system and may even threaten the lives of passengers. Therefore, timely and accurate detection of foreign objects in the railway track area is crucial to railway safety.
[0003] As a new type of camera, light field camera has great development potential and broad application prospects. So far, there has been no research on the application of light field camera to track foreign body intrusion detection. The appearance of the light field camera is similar to that of ordinary traditional cameras, but the internal structure is very different. The working principle of traditional cameras is that the main lens captures light and focuses it on the film or photoreceptor behind the lens. The sum of all the light forms a small dot on the photo to display the image. The biggest structural feature of the light field camera is that there is a microscope array filled with 90,000 micro lenses between the main lens and the photoreceptor. After each micro lens array receives the light from the main mirror, although it only records the intensity information of the light, it records the direction information of the light due to its position relative to a certain micro lens, and converts the light data and records it digitally. The built-in software operation of the camera has "expanded the light field", tracking the landing point of each light on the image at different distances, and after digital refocusing, a perfect photo can be taken. In addition, the light field camera can maintain the increase in light intensity, reduction in shooting time and graining brought by the large aperture of the original camera without sacrificing depth of field and image clarity. More importantly, light field cameras are low-cost, efficient and quick to use, can output 2D images and depth images at the same time, and the subsequent image processing using Lytro Desktop software is also convenient and quick. Summary of the invention
[0004] The purpose of this application is to provide a railway foreign body intrusion monitoring method based on light field depth image, which has high detection accuracy, saves manpower, automatically alarms, and has low cost.
[0005] To achieve the above-mentioned purpose, the present application provides a railway foreign object intrusion monitoring method, which includes: using a light field camera to shoot and collect a background image of a straight track, and obtaining an original RGB image and a depth image of the background image through Lytro Desktop software; extracting the straight track from the RGB image of the background image and establishing a dangerous area foreign object detection window; using a light field camera to shoot and collect a track dangerous area detection image, and preprocessing the image; performing significant object detection on the preprocessed track dangerous area detection image to obtain a foreign object image, and marking the foreign object in the foreign object image; if a foreign object is detected, an alarm signal is issued, otherwise, the foreign object is continued to be monitored.
[0006] Among them, the method for extracting the background image track danger zone detection window includes the following sub-steps: extracting the track edge of the background image track through a Gaussian high-pass filter, and using Hough transform to extract straight lines; establishing a danger zone foreign object detection window on the outer side of the track according to the shape and position of the track; segmenting the image located outside the danger zone foreign object detection window, retaining the image located inside the danger zone foreign object detection window, and forming the background image track danger zone detection window.
[0007] As mentioned above, in the track extraction of the background image, the Gaussian low-pass filter is used to obtain obvious track edge features, and the calculation formula is as follows:
[0008]
[0009] Among them, the method for preprocessing the track danger area detection image includes the following sub-steps: performing fast guided filtering processing on the depth image of the track danger area detection image and performing depth difference calculation; performing salient object extraction processing on the RGB image of the track danger area detection image using the ITTI model to fuse the RGB saliency map and depth difference map of the track danger area detection image to obtain the final difference map; and ending the preprocessing.
[0010] The algorithm steps for fast guided filtering of the depth image of the track danger zone detection image are as follows:
[0011] The input image is a depth image P, the guide image is an RGB image R, and the output image is an optimized depth image Q. The input P and output Q of the filter function are a local linear model. The calculation formula is as follows:
[0012]
[0013] Coefficient a k and b k The linear regression least squares method can be used to obtain:
[0014]
[0015]
[0016] Where n is the window w k The number of pixels in Respectively represent the depth map P to be optimized and the guide image R in the window ω k The average value within, ε is the regularization parameter. The filter output can be calculated using the following formula:
[0017]
[0018] and is the window ω centered at pixel i k The mean of a and b within the box, the computational effort of the algorithm is the box filter.
[0019] The specific algorithm steps for calculating the depth difference of the depth image of the track danger area detection image are as follows:
[0020] Scanning is performed along eight directions of image pixels. In the depth map obtained by the LYTRO camera, the greater the depth, the greater the grayscale value; for each scan line, the pixel with the maximum depth value is used as the background pixel according to the relationship between the image depth information and the grayscale information, and the difference between the current pixel and the background pixel is calculated as the scan value; the depth difference value is the sum of the scan values in the eight directions. In the circular coordinate system, the eight directions are: 0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°, and the depth difference is mathematically described as:
[0021]
[0022]
[0023] represents the grayscale value of pixel p along the scan line L, where L is the length of the scan line and depends on the size of the image. max(d L ) represents the maximum grayscale value on the scan line L, d(p) is the depth value of pixel p, D df (p) is the sum of the difference values of pixel p in eight directions, that is, the depth difference value.
[0024] The formula for integrating the RGB saliency map and depth difference map of the track danger area detection image is expressed as:
[0025] S(i)=C(i)·D(i)
[0026] Where C(i) is the RGB saliency map, D(i) is the depth difference map, and S(i) represents the i-th pixel in the image. The fusion map S can further highlight the target, suppress the background, and achieve better detection results.
[0027] The present invention uses a light field camera based on light field imaging technology as experimental equipment. The light field camera only includes one sensor, and can simultaneously obtain the original RGB image and the depth image through a single exposure. For the first time, the depth characteristics of track foreign objects are introduced into the detection algorithm, and have a good feasibility effect, realizing the automatic detection of track foreign object intrusion, saving a certain amount of manpower and cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 This is a flow chart of a railway foreign object intrusion detection method according to an embodiment of the present application.
[0029] Figure 2 This is an example of an operation for calculating a difference value for a depth image. DETAILED DESCRIPTION
[0030] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.
[0031] like Figure 1 As shown, the present application provides a railway foreign body intrusion detection method, the method comprising the following steps:
[0032] Step S1: Establish a foreign body detection window in a dangerous area.
[0033] Step S110: Use the LYTRO light field camera to capture a linear track background image, and only its RGB image is required;
[0034] Step S120: Gaussian high-pass filtering. The frequency of the image is an indicator of the intensity of grayscale changes in the image, and is the gradient of grayscale in the plane space. Details such as edges or lines correspond to the high-frequency components of the image spectrum. Therefore, high-pass filtering is used to allow high-frequency components to pass smoothly, making the edge or line details of the image clear and realizing the edge extraction of the track. The track map after FFT transformation is passed through a high-pass filter, which blocks all low frequencies and only allows high frequencies to pass. The image that has passed the filter is subjected to an inverse FFT transform to obtain obvious track edge features. The filter formula is as follows:
[0035]
[0036] S130: Obtain a straight line equation through Hough transformation according to the extracted track edge, adjust the equation parameters, define the foreign object detection range in the dangerous area, and establish a foreign object detection window in the dangerous area.
[0037] Step S2: monitoring image preprocessing.
[0038] Step S210: using a LYTRO light field camera to shoot a scene simulating foreign body invasion, and obtaining an original RGB image and a depth image of the shot image through Lytro Desktop software;
[0039] Step S220: Using the RGB image with rich edge information as the edge information reference, the depth map is guided by the improved fast guided filtering algorithm to achieve edge preservation of the guided image and local smoothing characteristics of the guided image, so as to obtain a depth map with lower noise and more accurate edges. The time complexity of the algorithm is independent of the filter window size, greatly improves the processing speed, and can highlight the edges of the depth image and play a smoothing role.
[0040] The input image P is a depth image, the guide image R is the corresponding RGB image, the output image Q is the optimized depth image, and the input R and output Q of the filter function are a local linear model.
[0041]
[0042] Coefficient a k and b k The linear regression least squares method can be used to obtain:
[0043]
[0044] n is the window w k The number of pixels within They represent the depth map P to be optimized and the guide image R processed in the previous section in the window w. k The filter output can be calculated using the following formula:
[0045]
[0046] and is the window w centered at pixel i k The mean of a and b within .
[0047] The algorithm implementation process of fast guided filtering, the pseudo code is:
[0048] 1: R'=f subsample (R,s)
[0049] P'=f subsample (P,s)
[0050] r'=r / s
[0051] 2: mean R =f mean (R',r')
[0052] meanp =f mean (P',r')
[0053] corr R =f mean (R.*R)
[0054] corr RP =f mean (R.*P)
[0055] 3: var R =corr R -mean R .*mean R
[0056] cov RP =corr R -mean R .*mean R
[0057] 4: a=cov RP . / (var R +ε)
[0058] b=mean P -a.*mean R
[0059] 5: mean a =f mean (a,r')
[0060] mean b =f mean (b,r')
[0061] 6: mean a =f upsample (mean a ,s)
[0062] mean b =f upsample (mean b ,s)
[0063] Q=mean a .*R+mean b
[0064] and are two smooth graphs. The edge and structure of the output image Q are mainly obtained by adjusting the guided graph R, but the main computational cost of the guided filter is to smooth and Fast guided filtering is based on guided filtering. It first downsamples the image input R and P.
[0065] The coefficient is s, and then the low-resolution images R' and P' are used as input to calculate, and then upsampled to calculate. Using this acceleration method, the computational complexity can be reduced from O(N) to O(N / S 2 ), which greatly improves the calculation speed.
[0066] Step S3: extracting the salient features of the preprocessed detection image.
[0067] Step S310: Use the ITTI model to extract salient features from the RGB image of the detection image. Use the ITTI visual attention model to extract the initial color, brightness and direction feature maps from the original image, and perform multi-scale geometric decomposition, merging and normalization on these feature maps to obtain salient maps of the three channels of color, brightness and direction, and add them together to obtain the final salient map of the RGB image.
[0068] Step S320: Calculate the depth difference of the depth image of the track dangerous area detection image.
[0069] Scanning is performed along eight directions of image pixels. In the depth map obtained by the LYTRO camera, the greater the depth, the greater the grayscale value; for each scan line, the pixel with the maximum depth value is used as the background pixel according to the relationship between the image depth information and the grayscale information, and the difference between the current pixel and the background pixel is calculated as the scan value; the depth difference value is the sum of the scan values in the eight directions. In the circular coordinate system, the eight directions are: 0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°, and the depth difference is mathematically described as:
[0070]
[0071]
[0072] represents the grayscale value of pixel p along the scan line L, where L is the length of the scan line and depends on the size of the image. max(d L ) represents the maximum grayscale value on the scan line L, d(p) is the depth value of pixel p, D df (p) is the sum of the difference values of pixel p in eight directions, that is, the depth difference value.
[0073] Step S330: Fuse the RGB saliency map and the depth difference map. Directly multiply the saliency values of the corresponding pixels of the two saliency maps as the saliency value of the fused map. The formula is as follows:
[0074] S(i)=C(i)·D(i)
[0075] Where C(i) is the RGB saliency map, D(i) is the depth difference map, and S(i) represents the i-th pixel in the image. The fusion map S can further highlight the target, suppress the background, and achieve better detection results.
[0076] Step S4: Detect whether there is foreign matter in the image and display the ROI area.
[0077] Step S410: The fusion image is segmented by threshold using the Ostu method to extract the region of interest and display it in the RGB image of the track danger zone detection image. If a foreign object is detected, the system sends an alarm signal, otherwise, the foreign object detection continues.
[0078] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.
Claims
1. A method for detecting foreign body intrusion on a track based on light field depth images, characterized in that: The method comprises the following steps: 1) Use a light field camera to capture and collect the background image of the linear track, extract the linear track from the RGB image of the background image, and establish a foreign object detection window in the dangerous area; 2) Use a light field camera to capture and collect images of track danger areas and pre-process the images; 3) Extracting salient features of the preprocessed detection image; 4) Detect whether there are foreign objects in the image and extract the ROI area, and issue an alarm signal; In the step 2), the method of using a light field camera to capture and collect the track danger zone detection image and preprocessing the image includes: a) Performing fast guided filtering on the depth image of the track danger zone detection image; b) calculating the depth difference of the optimized depth image; c) fusing the RGB image saliency map and the optimized depth difference map of the detected image; The method for calculating the depth difference of the optimized depth image in step b) includes: Scanning is performed along the eight directions of the image pixels. For each scan line, according to the relationship between the image depth information and the grayscale information, the pixel with the maximum depth value is taken as the background pixel, and the difference between the current pixel and the background pixel is calculated as the scan value; the depth difference value is the sum of the scan values in the eight directions. In the circular coordinate system, the eight directions are: 0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°, and the depth difference is mathematically described as: represents the grayscale value of pixel p along the scan line L, where L is the length of the scan line and depends on the size of the image. max(d L ) represents the maximum grayscale value on the scan line L, d(p) is the depth value of pixel p, D df (p) is the sum of the difference values of pixel p in eight directions, that is, the depth difference value; The formula for fusing the RGB image saliency map and the optimized depth difference map of the detection image in step c) is as follows: S(i)=C(i)·D(i) Where C(i) is the RGB saliency map, D(i) is the depth difference map, S(i) represents the i-th pixel in the image, and the fusion map S can further highlight the target, suppress the background, and achieve better detection results.
2. According to claim 1, a method for detecting foreign body intrusion on a track based on light field depth image is characterized in that: The light field cameras in step 1) and step 2) are second generation Lytro Illum cameras produced by LYTRO.
3. The method for detecting track foreign body intrusion based on light field depth image according to claim 1, characterized in that: The method of extracting a straight track from the RGB image of the background image and establishing a foreign body detection window in the dangerous area in the step 1) includes: The track edges of the background image are extracted by using a Gaussian high-pass filter: After the track edge is extracted, the image is subjected to straight line extraction using Hough transform. According to the shape and position of the track, a foreign object detection window in a dangerous area is established on the outer side of the track. The image outside the foreign object detection window in the dangerous area is segmented, and the image inside the foreign object detection window in the dangerous area is retained to form a background image track dangerous area detection window.
4. The method for detecting foreign body intrusion on a track based on light field depth image according to claim 1, characterized in that: The method for performing fast guided filtering on the depth image of the track danger zone detection image in step a) comprises: The input image is a depth image P, the guide image is an RGB image R, and the output image is an optimized depth image Q. The input P and output Q of the filter function are a local linear model; the calculation formula is as follows: Coefficient a k and b k The linear regression least squares method can be used to obtain: Where n is the window ω k The number of pixels in Respectively represent the depth map P to be optimized and the upper guide image R in the window ω k The average value within, ε is the regularization parameter, and the filter output can be calculated using the following formula: and is the window ω centered at pixel i k The mean of a and b within the box, the computational effort of the algorithm is the box filter.
