Railway infrared and visible image registration method combining multiple features and pixel information

Abstract

The present application relates to the technical field of image registration, in particular to a railway infrared and visible light image registration method combining multiple features and pixel information, which takes a target function designed by heterogenous structure similarity measurement as a search target, and to some extent overcomes the modal difference between infrared and visible light images; the information of uniquely explicit matching rail blanking points is used to guide global search to obtain coarse registration parameters, effectively reducing the difficulty of subsequent rail straight line feature matching; based on the coarse registration result, the distance from the midpoint of a rail line segment in the infrared image to a line segment in the visible light image after registration is used as the main judgment basis to easily realize the rail straight line matching of the heterogenous images, providing more reliable constraint conditions for subsequent fine registration and accelerating the search process; then the rail straight line matching pair information is used to guide global search to obtain the final fine registration parameters, avoiding falling into local optimal solution, and improving the overall calculation efficiency.

Description

Technical Field

[0001] This invention relates to the field of image registration technology, and specifically to a method for registering railway infrared and visible light images by combining multiple features and pixel information. Background Technology

[0002] my country's railway lines have been equipped with integrated video surveillance systems, and with the rapid development of deep learning, these systems have acquired the initial capabilities for video analysis and automatic identification of intrusion behaviors and events.

[0003] Currently, there has been considerable research on perimeter intrusion detection algorithms based on video images to address the issues of high false alarm rates and low detection rates under adverse weather conditions, at night, and in poor lighting conditions, as well as challenges such as small target detection, multi-scene adaptability, and real-time performance. However, because video surveillance equipment relies on the light reflection of objects for imaging, image quality is poor in low-light conditions at night and in adverse weather. Even with the application of deep learning algorithms, video-based perimeter intrusion detection technology cannot completely overcome the influence of lighting and weather changes. Therefore, to achieve all-weather and effective railway perimeter security, perimeter protection methods need to shift from a single, personnel-based approach to a combined, technology-based approach.

[0004] In recent years, a number of research results have emerged, including railway perimeter intrusion monitoring methods based on the collaboration of radar and video images, railway safety video monitoring and intelligent analysis systems developed using high-precision linkage of laser cameras and infrared thermal imaging cameras, and composite perimeter safety monitoring systems that combine vibration fiber optic sensing and video. Combining video images with other means and leveraging the complementary advantages of different technologies to accurately identify perimeter intrusion targets has become the mainstream trend in railway perimeter protection research.

[0005] Among various combinations of railway perimeter protection methods, infrared thermal imaging, which images objects based on their own thermal radiation, is unaffected by changes in light or inclement weather. It has a natural advantage in all-weather monitoring of targets such as pedestrians and animals that generate their own thermal radiation. Video surveillance, on the other hand, can acquire clear textures and detailed information about targets. Therefore, the combination of infrared thermal imaging and visible light video surveillance can work effectively in all weather conditions while significantly improving the accuracy, detection rate, and reliability of target detection at night and under poor lighting conditions. Compared to the combined approach of radar and video imaging, both infrared and visible light video images are two-dimensional planar images with excellent visualization effects. Combining the two enhances the intuitiveness and understandability of railway monitoring scenarios, facilitating further intelligent analysis.

[0006] In target detection tasks involving the fusion of infrared and visible light images, image registration is a prerequisite for fusion detection. Currently commonly used methods include:

[0007] The calibration method for infrared and visible light cameras based on lookup tables depends on the fineness of the grid division for registration accuracy. Moreover, the grid division needs to be repeated after changing the scene, which requires a large investment of manpower in the early stage and has a low degree of automation.

[0008] While image registration using camera calibration offers high accuracy, it requires recalibration when the scene changes and cannot automatically adapt to different scenarios.

[0009] Pixel-based image registration methods are easy to implement, but they are sensitive to changes in image intensity, especially nonlinear changes in illumination. Moreover, if the background features are highly repetitive, mutual information and similar statistical similarity metrics are prone to getting stuck in local optima. Furthermore, as the complexity of the transformation model increases, the computational cost also increases significantly.

[0010] Feature-based image registration differs from pixel-based registration methods. Feature-based registration methods are computationally less expensive and faster, and the design of feature descriptors makes them highly flexible and insensitive to complex geometric deformations. However, the registration accuracy of feature-based methods heavily depends on the accuracy of feature extraction and feature matching. Any erroneous feature detection or mismatch will degrade the performance of image registration. This method is also affected by complex scenes, lighting interference, and modal differences, making it not entirely reliable.

[0011] While deep learning-based image registration methods are the most efficient, they require rich datasets for training and face generalization issues in unknown scenarios. Current end-to-end deep learning image registration methods handle relatively simple image transformation relationships, such as only considering translation between images, and have not yet conducted in-depth research on more complex transformation relationships such as scaling. Summary of the Invention

[0012] To achieve automatic registration of infrared and visible light images under large field-of-view differences and complex lighting conditions in railway scenarios, and to solve the problems of low registration accuracy and poor adaptability to different scenarios in infrared and visible light image registration in railway scenarios, this invention provides a railway infrared and visible light image registration method that combines multiple features and pixel information to provide a reliable data source for subsequent target fusion detection.

[0013] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0014] A method for registering railway infrared and visible light images by combining multiple features and pixel information includes the following steps:

[0015] S1. Using the visible light image focused on a local scene as the image to be registered, and the infrared image with a fixed large field of view as the reference image, the transformation parameters that need to be solved are determined based on the simplified transformation model between infrared and visible light images in the railway scene.

[0016] S2. The heterogeneous structure similarity index is used as the similarity measurement criterion, and the objective function designed for the heterogeneous structure similarity measurement is used as the search target.

[0017] S3. Perform the following operations respectively: extracting straight line features from infrared and visible light images, detecting edge images, filtering straight lines on rails, and calculating the coordinates of rail hidden point disappearance points.

[0018] S4. Use point feature matching and line feature matching information to guide and constrain the search for the global optimal transformation parameters based on pixel information, and perform the transformation of the image to be registered based on the optimal transformation parameters to obtain the registered visible light image.

[0019] S4.1. Use the information from the unique and clearly matched rail hidden point to guide the global search and obtain coarse registration parameters;

[0020] S4.2. Based on the coarse registration results, complete the straightness feature matching of the rails;

[0021] S4.3. Use the information from the straight-line matching of the rails to guide the global search and obtain the final fine registration parameters;

[0022] S4.4. Based on the obtained fine registration parameters, transform the image to be registered to obtain the registered visible light image.

[0023] Furthermore, in step S1, it is assumed that a certain pixel in the visible light image... , corresponding to points in infrared images The simplified transformation model between infrared and visible light images in a railway scene is defined as shown in equation (1):

[0024] (1)

[0025] in, and For scaling transformation parameters, and These are the translation transformation parameters.

[0026] Furthermore, in step S2, the objective function designed using the heterogeneous structural similarity metric is shown in equation (2):

[0027] (2)

[0028] in, Represents the visible light edge image to be registered. and fixed infrared edge image The transformation model between the two edge images, SSIM, is used to measure the similarity between them, as shown in equation (3):

[0029] (3)

[0030] in, and Representing edge images respectively , The mean, and Representing edge images respectively , variance Representing edge image and Covariance between and It is a constant that maintains stability. , , It is the dynamic range of pixel values. , The SSIM value reaches its maximum of 1 when two edge images are completely identical.

[0031] Furthermore, in step S3, the M-LSD (Mobile Line Segment Detector) algorithm is used for line feature extraction. M-LSD (Mobile Line Segment Detector) is an effective deep learning-based line feature extraction algorithm that does not require frequent parameter tuning. The RCF (Richer Convolutional Features for edge detection) algorithm is used for edge image detection. RCF (Richer Convolutional Features for edge detection) is a deep learning-based edge detection algorithm that can output ideal edge images without parameter tuning.

[0032] Furthermore, in step S3, adjacent straight lines are merged and those that are clearly not rails are deleted through a rail straight line filtering operation. Specifically, this includes the following steps:

[0033] The two endpoints are , straight line segments Represented as:

[0034] (4)

[0035] in, It is the intercept. It is the slope angle; assuming there are line segments in the image. and , , , , These are the endpoints of the corresponding line segments. and These are the midpoints of the line segments. It is the intersection of the extensions of two line segments. The angle between the extensions of the line segments. Let be the distance between the midpoints of a line segment. Then the distance between two lines can be defined as:

[0036] (5)

[0037] The above distance It can be derived from points The equation of the straight line expressed in equation (4) is obtained as follows:

[0038] (6)

[0039] Considering that each line segment has a different length and endpoint position, the merging of adjacent lines is based on the following constraints:

[0040] (7)

[0041] In addition, if If the two lines are connected end to end, then the new line is formed by connecting the two line segments. Otherwise, the new line is formed by connecting the center points of the two line segments.

[0042] After obtaining a set of different candidate lines, the slope angle is used. Select straight rail lines and use the following constraints to eliminate lines that are clearly not rail lines:

[0043] (8).

[0044] Furthermore, in step S3, when calculating the coordinates of the rail hidden point, all intersections of the rail straight lines are polar-projected onto the edge map, and the number of non-zero pixels in all projection directions is calculated. The intersection with the largest cumulative number of non-zero pixels is taken as the hidden point.

[0045] Furthermore, step S4.1 specifically includes the following steps:

[0046] Based on a simplified transformation model between infrared and visible light images in a railway scene, four parameters need to be solved, including the scale value. and Translation value and In addition, the following conditions must also be met:

[0047] Regarding scaling, the search range for scaling is set to [0.5, 1]; meanwhile, the scaling ratios for horizontal and vertical values ​​are limited as follows:

[0048] (9)

[0049] For translation transformation parameters, the range of translation amount... It cannot exceed the width and height of the image itself. Assuming the image size is... For the vertices of the image The vertex coordinates after image transformation will become The constraints on the translation amount are as follows:

[0050] (10)

[0051] The above formula can be further derived as follows:

[0052] (11)

[0053] The constraint on the pairs of hidden point disappearances is set to a threshold. Assuming the hidden point in the visible light image is In visible light images, The constraints for the hidden point pairs in infrared and visible light images are as follows:

[0054] (12)

[0055] Based on the above constraints, the search range, search step size, threshold size, and initial maximum similarity of the parameters to be solved are first set according to the above constraints. Then, the scaling parameters are used to expand the search. If the ratio between the scaling parameters satisfies equation (9), the translation parameter search is entered. The translation parameters and the scaling parameters at this time are used to transform the coordinates of the hidden point of the visible light image. Then, the distance between the transformed visible light hidden point coordinates and the fixed infrared hidden point coordinates is calculated. It is determined whether the distance satisfies equation (12). If it does, the visible light edge image is transformed based on the current translation and scaling parameters. The similarity between the transformed visible light edge image and the fixed infrared edge image is calculated according to equation (3). If the similarity is greater than the set maximum similarity, the similarity is set as the new maximum similarity and the corresponding parameter values ​​are retained. This process is repeated until the scaling parameters and translation parameters complete the search of the entire search range. The parameter values ​​corresponding to the final maximum similarity are used as the optimal estimates for the coarse registration stage.

[0056] Further, step S4.2 specifically includes the following steps: using straight lines in the infrared image midpoint Straight lines in visible light images distance Line matching is performed. In a single image, if a line segment is higher than the hidden point, it is directly eliminated. In a pair of images, if the candidate line pair is in a different quadrant of the coordinate system constructed with the corresponding hidden point as the origin, the line pair will be directly eliminated. Outliers in the shortest distance are eliminated using the Z-Score method based on the upper limit threshold constraint of the shortest distance.

[0057] Further, in step S4.3, based on the obtained coarse registration parameters, the transformation model parameters are searched twice using different intervals and step sizes. During the search, it is assumed that the shape of the infrared image is... The shape of the VIS image is ,coefficient The design is as follows:

[0058] (13)

[0059] And horizontal scaling value With vertical scaling value The relationship between them can be described as follows:

[0060] (14)

[0061] Considering the significant differences in size between infrared and VIS camera imaging sensors, The search scope is around And expand.

[0062] This invention achieves automatic registration of infrared and visible light images under large field-of-view differences and complex lighting conditions in railway scenes, improving the accuracy and robustness of image registration in railway scenes and providing a reliable data source for subsequent fusion detection. Compared with existing infrared and visible light image registration methods, this invention has the following advantages:

[0063] 1) Regarding the registration method, this invention simplifies the image transformation model for railway scenes, designs an objective function based on heterogeneous structural similarity measurement, and deeply mines the straight-line features and related information of railway scenes. Unlike existing combined registration methods based on features and pixel information, this invention uses a pixel-based registration approach throughout, with the mined feature information used to assist in constraining the hierarchical global search process. This overcomes the problems of long processing time and easy getting trapped in local optima in pixel-based registration methods, while avoiding the potential mismatch problem in feature-based registration methods. The algorithm implementation ensures both accuracy and efficiency while also making it highly adaptable to different railway scenes.

[0064] 2) In terms of solving the challenges, this invention not only overcomes the modal differences between infrared and visible light images, but also overcomes the large field-of-view differences between images. Most existing infrared and visible light image registration methods study image registration under similar fields of view, such as medical remote sensing. Because professional acquisition equipment is used instead of ordinary monitoring equipment, the difference in sharpness between images is also small. The difficulty of registering heterogeneous images in these fields is not as great as the registration difficulty in railway scenes. However, this invention overcomes these difficulties in railway scenes, achieves accurate registration of infrared and visible light images, and has good robustness. Attached Figure Description

[0065] Figure 1 This is a framework diagram of a railway infrared and visible light image registration method that combines multiple features and pixel information according to an embodiment of the present invention.

[0066] Figure 2 The results of infrared and visible light image line detection in this embodiment of the invention are shown.

[0067] Figure 3 These are infrared and visible light images and their corresponding edge images in embodiments of the present invention.

[0068] Figure 4 This refers to the linear merging calculation theory in the embodiments of the present invention.

[0069] Figure 5 The line is the result of merging the infrared and visible light images in this embodiment of the invention.

[0070] Figure 6 Candidate rail straight lines are retained from infrared and visible light images in embodiments of the present invention.

[0071] Figure 7 These are the hidden points of the rail in the infrared and visible light images in this embodiment of the invention and their corresponding longest polar coordinate lines.

[0072] Figure 8 This is the coarse / fine registration process in the embodiments of the present invention.

[0073] Figure 9 This is a flowchart of the coarse registration algorithm in an embodiment of the present invention.

[0074] Figure 10 The superimposed image obtained after coarse registration in this embodiment of the invention.

[0075] Figure 11 An example of the distance from the midpoint of a line segment to a straight line in an embodiment of the present invention.

[0076] Figure 12 The line matching result in the embodiment of the present invention. Implementation

[0077] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0078] This invention aims to combine feature and pixel information from infrared and visible light images. By fully exploiting the rich and stable straight-line features in railway scenes, it utilizes point and straight-line feature matching information to guide and constrain the process of searching for globally optimal transformation parameters based on pixel information. This achieves accurate registration under significant field-of-view differences while improving registration efficiency. Furthermore, for commonly used image transformation models, this invention simplifies the image transformation model in railway scenes by analyzing the positional relationship between infrared and visible light cameras, reducing the number of transformation parameters to be solved and further accelerating the solution process. Moreover, to overcome the modal differences between infrared and visible light images, this invention designs a similarity metric function based primarily on image structural similarity, providing a reliable search basis for the search for globally optimal transformation parameters. The final railway infrared and visible light image registration method can adapt to different scenes with varying lighting conditions and field-of-view differences.

[0079] The overall structure of the algorithm of this invention is as follows: Figure 1As shown, a visible light image focusing on a local scene is used as the image to be registered, while a fixed infrared image with a large field of view is used as the reference image. Before registration, the number of parameters to be solved is determined based on a simplified model. The heterogeneous structure similarity index is used as the similarity criterion, and the objective function designed by the heterogeneous structure similarity criterion is used as the search target. Then, the infrared and visible light image pairs are input for coarse-to-fine registration. The optimal parameters are solved step by step using a registration method based on pixel information throughout the process. Hidden point pairs and straight line feature matching pairs are used as search conditions to constrain the search space and improve search efficiency. The specific registration process is as follows: First, straight line feature extraction, edge image detection, and calculation of rail hidden point coordinates are performed on the infrared and visible light images respectively to complete all preprocessing operations. Then, the information of the uniquely matched rail hidden point pairs guides the global search to obtain coarse registration parameters. Then, rail straight line feature matching is completed based on the coarse registration results. Finally, the information of rail straight line matching pairs guides the global search to obtain the final fine registration parameters. Based on the fine registration parameters, the image to be registered is transformed to obtain the registered visible light image.

[0080] 1.1 Simplification of Image Transformation Model

[0081] Since the relative positions of the infrared and PTZ (Pan, Tilt, and Zoom) cameras in the railway scene are fixed, and the visible light camera controlled by the PTZ can only zoom and move in the vertical and horizontal directions, the complexity of the infrared to visible light image transformation can be simplified. The scaling motion of the visible light camera corresponds to the scaling transformation between images, the vertical motion corresponds to the vertical translation between images, and the horizontal motion corresponds to the horizontal translation between images. Therefore, the transformation model can be simplified by removing rotation and shearing transformations to accelerate the process of solving for transformation parameters.

[0082] Suppose a pixel in a visible light image , corresponding to points in infrared images The simplified transformation model between infrared and visible light images in a railway scene is defined as shown in equation (1).

[0083] (1)

[0084] in, and For scaling transformation parameters, and These are the translation transformation parameters.

[0085] Observing the infrared and visible light images captured of the railway scene, it is easy to see that the overhead contact line posts and pedestrians are always perpendicular, consistent with the theoretical analysis, meaning that there is no rotation or shearing transformation in the image. Although the railway scene image is non-planar, because the visible light camera captures a local scene with only a small change in depth of field, the parallax effect between the infrared and visible light images can be ignored.

[0086] 1.2 Design of Objective Function Based on Heterogeneous Structure Similarity Measurement

[0087] In pixel-based image registration methods, designing a suitable objective function to measure the similarity between images is crucial for achieving accurate registration. Considering the rich structural information in railway scenarios, including rails and overhead contact line posts, edge images can weaken smooth texture features and highlight abrupt changes, making them more suitable for similarity measurement than the images themselves.

[0088] SSIM (Structural Similarity Index) is a similarity index that is insensitive to pixel brightness and color, but sensitive to the location of edges and textures. The features of SSIM closely match the representation of edge mappings; therefore, our objective function is designed as shown in Equation (2):

[0089] (2)

[0090] in, Represents the visible light edge image to be registered. and fixed infrared edge image The transformation model between the two edge images, SSIM, is used to measure the similarity between them, as shown in equation (3):

[0091] (3)

[0092] in, and Representing edge images respectively , The mean, and Representing edge images respectively , variance Representing edge image and Covariance between and It is a constant that maintains stability. , , It is the dynamic range of pixel values. , The SSIM value reaches its maximum of 1 when two edge images are completely identical.

[0093] 1.3 Preprocessing The preprocessing process for infrared and visible light image pairs specifically includes four steps: straight line feature extraction, edge detection, rail straight line filtering, and calculation of the coordinates of the rail straight line hidden point.

[0094] Line Feature Extraction: Accurate line feature extraction is helpful for subsequent line matching. M-LSD (Mobile Line Segment Detector) is an effective deep learning-based line feature extraction algorithm that does not require frequent parameter tuning. The line detection results obtained using this method are as follows: Figure 2 As shown, it is clear that there are basically no errors in line detection, but there are cases where the same line is detected repeatedly.

[0095] Edge detection: RCF (Richer Convolutional Features for edge detection) is a deep learning-based edge detection algorithm that can output ideal edge images without parameter tuning. For example... Figure 3 As shown, infrared and visible light images are significantly more similar in edge images, and the modal differences between infrared and visible light images are almost overcome in edge images.

[0096] Rail line filtering: Because the same straight line may be repeatedly detected in both infrared and visible light images, it is necessary to merge adjacent straight lines and delete those that are clearly not rails to reduce the difficulty of subsequent rail line matching. Straight line segments You can use two endpoints directly. , Indicates, that is To further illustrate the slope angle and intercept of a straight line segment, the following can be considered:

[0097] in, This represents a point on a straight line segment, while It is the intercept. It is the slope angle. Combining the two endpoints of the line segment, a straight line can ultimately be represented as:

[0098] (4)

[0099] Suppose there are line segments in the image. and , , , , These are the endpoints of the corresponding line segments. and These are the midpoints of the line segments. It is the intersection of the extensions of two line segments. The angle between the extensions of the line segments. Let be the distance between the midpoints of a line segment. Then the distance between two lines can be defined as:

[0100] (5)

[0101] The above distance It can be derived from points The equation of the line is obtained from:

[0102] (6)

[0103] Considering that each line segment has a different length and endpoint position, the merging of adjacent lines is based on the following constraints:

[0104] (7)

[0105] In addition, if If the two lines are connected end to end, then the new line is formed by connecting the two line segments. Otherwise, the new line is formed by connecting the center points of the two line segments.

[0106] Once a set of different candidate lines is obtained, the slope angle can be used. Filter out straight lines on the rails. Because the cameras are mounted beside the rails and capture images along the rail direction, the rails in the images are neither perfectly horizontal nor perfectly vertical. Therefore, the following constraints are used to eliminate lines that are clearly not straight lines on the rails.

[0107] (8).

[0108] The merged straight line and candidate rail straight line results are as follows: Figure 4 and Figure 5 As shown, the only and accurate railway rail is exactly what is needed.

[0109] Calculation of Hidden Line Coordinates for Rail Straight Lines: For calculating hidden line coordinates, a set of parallel lines will not intersect in three-dimensional space, but will intersect on the two-dimensional image plane under the camera's perspective projection. The intersection point of these parallel lines in the image is called their hidden line. In a railway scene, all rail straight lines are parallel to each other, and the intersection point of these rail straight lines in the image is unique. Therefore, the hidden lines of these rail straight lines in infrared and visible light images must have a one-to-one correspondence.

[0110] However, in practical applications, the accuracy of straight rail lines can be affected by factors such as low infrared camera resolution, low infrared image clarity, poor quality of visible light images at night, or the extraction of straight-line features from straight-to-curved railway sections. Therefore, the hidden-line point calculated from these candidate straight rail lines is not unique. To address this issue, all intersections of the straight rail lines are polar-projected onto the edge map, and the number of non-zero pixels in all projection directions is calculated. The intersection with the largest cumulative number of non-zero pixels is selected as the hidden-line point. Figure 6 The diagram shows the calculated hidden point and the corresponding longest polar coordinate line with the maximum number of non-zero pixels in the edge maps of infrared and visible light images. Although the obtained hidden points are not very accurate, as long as the overall trend is correct, it will not affect the accuracy of subsequent registration.

[0111] 1.4 Coarse-to-fine registration

[0112] The coarse-to-fine registration stage includes three parts: coarse registration, straight line matching, and fine registration. Since coarse registration under the constraint of hidden point pairs can only achieve matching of straight rail regions, resulting in insufficient registration accuracy, the coarse registration results are used to achieve straight rail feature matching. Then, multiple pairs of straight rail matching pairs are combined to constrain information for fine registration, achieving the final accurate registration. Both the coarse and fine registration processes employ pixel-based registration methods, and their specific processes are as follows: Figure 7 As shown, the input consists of edge images from infrared and visible light images. A structural similarity metric is used to measure the similarity between the two images, and an objective function designed based on this structural similarity criterion is used as the optimization objective. The search space and search step size are set by simplifying the number of model parameters and the variation range of various parameters. Then, the search process is optimized using hidden point pair matching information or line matching information as constraints. The optimal model parameters are obtained through multiple searches that satisfy the constraints; these parameters represent the global optimal solution. Based on the image transformation model parameters obtained in the fine registration stage, image transformation and image fusion of the image to be registered can be achieved.

[0113] 1.41 Coarse registration based on hidden point pair constraints:

[0114] Coarse registration of infrared and visible light images helps reduce the difficulty of subsequent rail straightness matching. Since rails are a major component of images in railway scenes, the SSIM value based on edge mapping between the infrared image and the transformed visible light image can largely represent the degree of overlap of the rail region. According to the proposed transformation model, four parameters need to be solved, including the scale value. and Translation value and In addition, the following conditions must also be met.

[0115] Regarding scaling, since the visible light image is part of the infrared image in the railway scene, the scaling value does not exceed 1. Experience shows that when the scaling value is less than 0.5, matching the track area of ​​the entire image will be unreliable; therefore, the scaling search range is set to [0.5, 1]. Meanwhile, the scaling ratios for horizontal and vertical values ​​are limited as follows:

[0116] (9)

[0117] For translation transformation parameters, the range of translation amount... The overlap cannot exceed the width and height of the original image. As a rule of thumb, if the transformed visible light image appears within an infrared image and is less than half its width and height, the overlap area between the two images will be too small, and the matching result will be unreliable. Assuming the image size is... For the vertices of the image The vertex coordinates after image transformation will become The constraints on the translation amount are as follows:

[0118] (10)

[0119] The above formula can be further derived as follows:

[0120] (11)

[0121] Regarding the constraints on the hidden point pairs, since the railway line extraction and its hidden point calculation results are not very accurate, a certain degree of error distance is allowed between the hidden point pairs of the infrared image and the transformed visible light image, and this distance is set as a threshold. Assuming the hidden point in the visible light image is In visible light images, The constraints for the hidden point pairs in infrared and visible light images are as follows:

[0122] (12)

[0123] Based on the above constraints, the flowchart of the coarse registration algorithm is as follows: Figure 8As shown, firstly, the search range, search step size, threshold values, and initial maximum similarity of the parameters to be solved are set according to the above constraints. Then, the scaling parameters are used to expand the search. If the ratio between the scaling parameters satisfies equation (9), the translation parameter search is entered. The coordinates of the hidden point of the visible light image are transformed using the translation parameters and the scaling parameters at this time. Then, the distance between the transformed visible light hidden point coordinates and the fixed infrared hidden point coordinates is calculated. It is determined whether the distance satisfies equation (12). If it does, the visible light edge image is transformed based on the current translation and scaling parameters, and the similarity between the transformed visible light edge image and the fixed infrared edge image is calculated according to equation (3). If the similarity is greater than the set maximum similarity, the similarity is set as the new maximum similarity and the corresponding parameter values ​​are retained. This process is repeated until the scaling parameters and translation parameters complete the search of the entire search range, and the parameter values ​​corresponding to the final maximum similarity are used as the optimal estimates for the coarse registration stage.

[0124] The fused image obtained through coarse registration is as follows: Figure 9 As shown in the figure, the straight sections of the rails largely overlap, which greatly facilitates subsequent rail alignment.

[0125] 1.42 Line Matching: After coarse registration, straight lines in the infrared image are used for matching. midpoint Straight lines in visible light images distance Line matching is performed, as illustrated in the following diagram. Figure 10 As shown.

[0126] Two strategies for eliminating hidden point pairs were employed to further verify the accuracy of candidate rail straight lines and the accuracy of eliminating mismatched straight line pairs: First, in a single image, if a line segment is higher than the hidden point (i.e., the maximum value of the ordinate of the line segment's endpoint is less than the maximum value of the hidden point's coordinate), it is directly eliminated. Second, in an image pair, if a candidate straight line pair lies in a different quadrant of a coordinate system constructed with the corresponding hidden point as the origin, the straight line pair will be directly eliminated.

[0127] To ensure that the shortest distance between potential line pairs does not exceed an acceptable limit, a threshold constraint is applied to the upper limit of the shortest distance, and the Z-Score method is used to eliminate outliers in the shortest distance. Simultaneously, the difference between the shortest and second shortest distances, as well as the number of line pairs, are considered to obtain the final line result. Figure 11 As shown, after coarse registration, an accurate rail straight matching pair can be easily obtained.

[0128] 1.43 Fine Registration Based on Rail Straightness Matching Constraints: To achieve efficient and fine registration, two searches are performed on the transformation model parameters using different intervals and step sizes. This search process is similar to that of the coarse registration stage. The difference is that the first search introduces rail straightness matching constraints in addition to the hidden point pair constraints, while the second search is based on the results of the first search with a smaller step size and finer search intervals. Considering time costs, , The step size is small and is not set to 1.

[0129] Since the ratio of horizontal to vertical scaling is largely affected by the resolution difference between infrared and visible light images, a coefficient was designed to counteract this effect. Assuming the shape of the infrared image is... The shape of the VIS image is ,coefficient The design is as follows:

[0130] (13)

[0131] And horizontal scaling value With vertical scaling value The relationship between them can be described as follows:

[0132] (14)

[0133] Considering the significant differences in size between infrared and VIS camera imaging sensors, The search scope is around And expand. The fused image obtained through fine registration via two global searches is as follows: Figure 12 As shown, the first search can achieve a basically accurate registration result, while the second search can make the registration result more accurate.

[0134] In summary, this invention simplifies the infrared and visible light image transformation model in railway scenarios, involving only translation and scaling transformations. This simplification reduces the complexity of image transformations, decreases the number of optimal parameter solutions in the global search, and significantly improves search efficiency. A target function based on heterogeneous structural similarity measurement is designed, which to some extent overcomes the modal differences between infrared and visible light images. Optimizing this target function facilitates obtaining the globally optimal solution for the infrared and visible light image transformation model parameters. A stepwise registration method is proposed, using different feature information as search conditions to guide and constrain the global search. This fully utilizes the stable features of the railway scene to ensure the accuracy and efficiency of infrared and visible light image registration, while also overcoming the large field-of-view differences between images. The hidden point pairs of rails are the only clearly matched feature point pairs between images. Global search based on the constraints of these hidden point pairs completes coarse registration, effectively reducing the difficulty of subsequent rail straight line feature matching. Based on the coarse registration results, the distance from the midpoint of a rail segment in the infrared image to a corresponding line segment in the registered visible light image can be used as the primary criterion to easily achieve rail straight-line matching between heterogeneous images. This provides more reliable constraints for subsequent fine registration and accelerates the search process. Finally, fine registration obtains the final transformation model parameters by using the straight-line matching results as constraints for searching, avoiding getting trapped in local optima and improving overall computational efficiency.

[0135] It is worth noting that in the preprocessing stage, the line detection and edge detection algorithms in this embodiment can be replaced by other detection algorithms with the same functionality, such as the traditional Hough line detection algorithm, the Canny edge detection operator, and the deep learning algorithm HAWP (Holistically-Attacted Wireframe Parsing). In the entire coarse-to-fine registration process, the constraints on the search range and search step size can be adjusted according to the actual situation. The coarse registration part can utilize methods based on feature point extraction and matching, as well as deep learning methods, to simplify subsequent line matching without relying on hidden point pair constraints. Examples include traditional feature point detection and matching algorithms such as LGHD (Log-Gabor Histogram Descriptor) and RIFT (Radiation variation Insensitive Feature Transform), and deep learning-based feature point detection algorithms such as D2-Net (Detect-and Describe Network) Super Point. The fine registration part can involve one or multiple searches.

[0136] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for registering railway infrared and visible light images by combining multiple features and pixel information, characterized in that: Includes the following steps: S1. Using the visible light image focused on a local scene as the image to be registered, and the infrared image with a fixed large field of view as the reference image, the transformation parameters that need to be solved are determined based on the simplified transformation model between infrared and visible light images in the railway scene. S2. The heterogeneous structure similarity index is used as the similarity measurement criterion, and the objective function designed for the heterogeneous structure similarity measurement is used as the search target. S3. Perform the following operations respectively: extracting straight line features from infrared and visible light images, detecting edge images, filtering straight lines on rails, and calculating the coordinates of rail hidden point disappearance points. S4. Use point feature matching and line feature matching information to guide and constrain the search for the global optimal transformation parameters based on pixel information, and perform the transformation of the image to be registered based on the optimal transformation parameters to obtain the registered visible light image. S4.

1. Use the information from the unique and clearly matched rail hidden point to guide the global search and obtain coarse registration parameters; S4.

2. Based on the coarse registration results, complete the straightness feature matching of the rails; S4.

3. Use the information from the straight-line matching of the rails to guide the global search and obtain the final fine registration parameters; S4.

4. Based on the obtained fine registration parameters, transform the image to be registered to obtain the registered visible light image; S4.1 specifically includes the following steps: Based on the simplified transformation model between infrared and visible light images in a railway scene, four parameters need to be solved, including the scale value. and Translation value and In addition, the following conditions must also be met: Regarding scaling, the search range for scaling is set to [0.5, 1]; meanwhile, the scaling ratios for horizontal and vertical values ​​are limited as follows: (10) For translation transformation parameters, the range of translation amount... It cannot exceed the width and height of the image itself. Assuming the image size is... For the vertices of the image The vertex coordinates after image transformation will become The constraints on the translation amount are as follows: (11) The above formula can be further derived as follows: (12) The constraint on the pairs of hidden point disappearances is set to a threshold. Assuming the hidden point in the visible light image is In visible light images, The constraints for the hidden point pairs in infrared and visible light images are as follows: (13) Based on the above constraints, the search range, search step size, threshold size, and initial maximum similarity of the parameters to be solved are first set according to the above constraints. Then, the scaling parameters are used to expand the search. If the ratio between the scaling parameters satisfies equation (10), the translation parameter search is entered. The translation parameters and the scaling parameters at this time are used to transform the coordinates of the hidden point of the visible light image. Then, the distance between the transformed visible light hidden point coordinates and the fixed infrared hidden point coordinates is calculated. It is determined whether the distance satisfies equation (13). If it does, the visible light edge image is transformed based on the current translation and scaling parameters. The similarity between the transformed visible light edge image and the fixed infrared edge image is calculated according to equation (3). If the similarity is greater than the set maximum similarity, the similarity is set as the new maximum similarity and the corresponding parameter values ​​are retained. This process is repeated until the scaling parameters and translation parameters complete the search of the entire search range. The parameter values ​​corresponding to the final maximum similarity are used as the optimal estimates for the coarse registration stage.

2. The method for registering railway infrared and visible light images by combining multiple features and pixel information as described in claim 1, characterized in that: In step S1, it is assumed that a certain pixel in the visible light image... , corresponding to points in infrared images The simplified transformation model between infrared and visible light images in a railway scene is defined as shown in equation (1): (1) wherein and are scaling transformation parameters, and are translation transformation parameters.

3. The method for registering railway infrared and visible light images by combining multiple features and pixel information as described in claim 1, characterized in that: In step S2, the objective function designed using heterogeneous structural similarity measurement is shown in equation (2): (2) in, Represents the visible light edge image to be registered. and fixed infrared edge image The transformation model between the two edge images, SSIM, is used to measure the similarity between them, as shown in equation (3): (3) in, and Representing edge images respectively , The mean, and Representing edge images respectively , variance Representing edge image and Covariance between and It is a constant that maintains stability. , It is the dynamic range of pixel values. , The SSIM value reaches its maximum of 1 when two edge images are completely identical.

4. A method for registering railroad infrared and visible light images with multi-features and pixel information as claimed in claim 1, wherein: In step S3, the M-LSD extraction algorithm is used to extract straight line features, and the RCF detection algorithm is used to detect edge images.

5. A method for registering railroad infrared and visible light images with multi-features and pixel information as claimed in claim 1, wherein: In step S3, adjacent straight lines are merged and those that are clearly not rails are deleted through a rail straight line filtering operation. Specifically, this includes the following steps: The two endpoints are , straight line segments Represented as: (5) in, It is the intercept. It is the slope angle; Suppose there are line segments in the image. and , These are the endpoints of the corresponding line segments. and These are the midpoints of the line segments. It is the intersection of the extensions of two line segments. The angle between the extensions of the line segments. Let be the distance between the midpoints of a line segment. Then the distance between two lines can be defined as: (6) The above distance It can be derived from points The equation of the straight line expressed in equation (4) is obtained as follows: (7) Considering that each line segment has a different length and endpoint position, the merging of adjacent lines is based on the following constraints: (8) In addition, if If the two lines are connected end to end, then the new line is formed by connecting the two line segments. Otherwise, the new line is formed by connecting the center points of the two line segments. After obtaining a set of different candidate lines, the slope angle is used. Select straight rail lines and use the following constraints to eliminate lines that are clearly not rail lines: (9)。 6. A method for registering railroad infrared and visible light images with multi-features and pixel information as claimed in claim 1, wherein: In step S3, when calculating the coordinates of the rail hidden point, all intersections of the rail straight lines are polar-projected onto the edge map, and the number of non-zero pixels in all projection directions is calculated. The intersection with the largest cumulative number of non-zero pixels is taken as the hidden point.

7. A method for registering railroad infrared and visible light images with multi-features and pixel information as claimed in claim 1, wherein: Step S4.2 specifically includes the following steps: using straight lines in the infrared image midpoint Straight lines in visible light images distance Line matching is performed. In a single image, if a line segment is higher than the hidden point, it is directly eliminated. In a pair of images, if the candidate line pair is in a different quadrant of the coordinate system constructed with the corresponding hidden point as the origin, the line pair will be directly eliminated. Outliers in the shortest distance are eliminated using the Z-Score method based on the upper limit threshold constraint of the shortest distance.

8. The method for registering railway infrared and visible light images by combining multiple features and pixel information as described in claim 1, characterized in that: In step S4.3, based on the obtained coarse registration parameters, the transformation model parameters are searched twice using different intervals and step sizes. During the search, it is assumed that the shape of the infrared image is... The shape of the VIS image is ,coefficient The design is as follows: (14) And horizontal scaling value With vertical scaling value The relationship between them can be described as follows: (15) Considering the significant differences in size between infrared and VIS camera imaging sensors, The search scope is around And expand.

Images