A high-precision turnout panoramic image synthesis method, system, device and storage medium

Abstract

The present invention discloses a high-precision turnout panorama synthesis method, system, device and storage medium, including the steps of: extracting features and matching features of original images to construct a maximum splicable image set; splicing images according to the maximum splicable image set to obtain a spliced ​​image; correcting the spliced ​​image to obtain a corrected image; converting the corrected image into a top view to obtain a target image. The present invention realizes efficient, accurate and high-quality railway turnout image splicing by adopting a series of technical means such as SIFT feature extraction algorithm, nearest neighbor and next nearest neighbor matching, union-find query of the maximum splicable image set, homography matrix estimation, bundle adjustment correction, etc., and has strong versatility and practicality.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a high-precision turnout panoramic image synthesis method, system, device and storage medium. Background Art

[0002] A turnout is a line connection device that allows trains to transfer from one track to another. It is also one of the weak links of the track and is usually laid in large quantities at stations and marshaling yards. With a turnout, the line's capacity can be fully utilized. Even for a single-track railway, by laying a turnout and building a fork line that is longer than the length of the train, trains can be turned in opposite directions. Therefore, turnouts play an important role on railway lines. When using a turnout to change the track of a train, the lateral pressure on the track is large and some problems are prone to occur. Therefore, it is necessary to focus on the railway lines near the turnouts and monitor the turnouts.

[0003] When using image acquisition equipment to monitor turnouts, it is necessary to monitor minor problems on the line. Therefore, image acquisition equipment with a small shooting range is required. Although the obtained images are very high-definition, the viewing experience is not good due to the large number of images. Image stitching technology is required to stitch multiple images with a small shooting range into a larger panoramic image, so as to monitor the turnout intuitively.

[0004] However, when processing images, due to the high complexity of the image stitching algorithm itself and the fact that the stitched image data set often contains a large number of redundant images and interference images, these image information not only increases the amount of calculation, but may also introduce unnecessary matching and errors, resulting in low overall processing efficiency. At the same time, traditional image stitching technology often pays too much attention to the image stitching algorithm itself, while ignoring the specific needs in practical applications. As a result, in practical applications, the stitched images may not meet specific needs, such as railway line monitoring, terrain analysis, etc. Especially when processing railway turnout images, due to the importance and particularity of the image content, higher requirements are placed on the accuracy and reliability of the stitching results. However, traditional technologies often ignore this point, resulting in local distortion and information loss in the stitched images, affecting their reliability and practicality in practical applications. In addition, there may be obvious seam marks in the stitched images, affecting the overall visual effect. Summary of the invention

[0005] The main problem solved by the present invention is to provide a high-precision turnout panoramic image synthesis method to solve the problems of low processing efficiency of existing splicing technology, low quality of spliced ​​images and poor visual effects when using high-definition image acquisition equipment to monitor turnouts.

[0006] To solve the above technical problems, a technical solution adopted by the present invention is to provide a high-precision turnout panoramic image synthesis method, comprising the steps of: performing feature extraction and feature matching on the original image to construct a maximum splicable image set; performing image stitching according to the maximum splicable image set to obtain a stitched image; performing image correction on the stitched image to obtain a corrected image; performing a top view conversion on the corrected image to obtain a target image.

[0007] In some embodiments, the step of extracting features and matching features on the original image to construct a maximum set of stitchable images includes: using a SIFT feature extraction algorithm to extract feature points of the original image; using a FLANN feature matching algorithm to perform feature matching on the feature points of the original image to obtain feature matching results; and using a union-find data structure and the feature matching results to obtain a maximum set of stitchable images.

[0008] In some embodiments, while obtaining the maximum set of stitchable images by using the union-find data structure and the feature matching results, it is also necessary to determine camera parameters, including: calculating a homography matrix based on the feature matching results; estimating camera parameters based on the homography matrix; and correcting the camera parameters using a bundle adjustment method.

[0009] In some embodiments, the step of performing image stitching according to the maximum stitchable image set to obtain a stitched image includes: performing waveform correction on the original images in the maximum stitchable image set to obtain a distortion-free image set; performing distortion transformation on the distortion-free images in the distortion-free image set to obtain a distortion-transformed image set; performing exposure compensation on the distortion-transformed images in the distortion-transformed image set to obtain a registered image set; and performing multi-band fusion on the registered images in the registered image set to obtain the stitched image.

[0010] In some embodiments, the step of performing multi-band fusion on the registered images in the registered image set to obtain the stitched image includes: using a seam estimation method based on minimum graph cut to find an optimal seam; and performing multi-band fusion on the registered images according to the optimal seam to generate the stitched image.

[0011] In some embodiments, the step of performing image correction on the stitched image to obtain a corrected image includes: performing correction preprocessing on the stitched image to obtain a filtered stitched image; using a Canny edge detection algorithm to extract corrected edge features in the filtered stitched image; and using a least squares method to establish a correction mapping relationship based on the corrected edge features to obtain the corrected image.

[0012] In some embodiments, the step of converting the corrected image into a top view to obtain a target image includes: performing top view preprocessing on the corrected image to obtain a filtered corrected image; using a Canny edge detection algorithm to extract top view edge features in the filtered corrected image; and using a least squares method to establish a top view mapping relationship based on the top view edge features to obtain the target image.

[0013] The present invention also provides a high-precision turnout panoramic image synthesis system, comprising a feature processing module, a stitching module, a correction module and a bird's-eye view conversion module, wherein the feature processing module is used to perform feature extraction and feature matching on the original image to construct a maximum stitchable image set; the stitching module is used to perform image stitching according to the maximum stitchable image set to obtain a stitched image; the correction module is used to perform image correction on the stitched image to obtain a corrected image; and the bird's-eye view conversion module is used to perform bird's-eye view conversion on the corrected image to obtain a target image.

[0014] The present invention also provides a high-precision turnout panoramic image synthesis device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0016] The beneficial effects of the present invention are as follows: the present invention discloses a high-precision turnout panoramic image synthesis method, system, device and storage medium, including the steps of: extracting features and matching features of original images to construct a maximum set of stitchable images; stitching images according to the maximum set of stitchable images to obtain a stitched image; correcting the stitched image to obtain a corrected image; converting the corrected image into a top view to obtain a target image. The present invention realizes efficient, accurate and high-quality railway turnout image stitching by adopting a series of technical means such as SIFT feature extraction algorithm, nearest neighbor and next nearest neighbor matching, union-find query of the maximum set of stitchable images, homography matrix estimation, bundle adjustment correction, etc., and has strong versatility and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a high-precision turnout panorama synthesis method of the present invention;

[0018] Figure 2 It is a schematic diagram of an original image obtained by a high-precision turnout panoramic image synthesis method of the present invention;

[0019] Figure 3It is a specific flow chart of step S1 in a high-precision turnout panoramic image synthesis method of the present invention;

[0020] Figure 4 It is a specific flow chart of determining camera parameters in a high-precision turnout panoramic image synthesis method of the present invention;

[0021] Figure 5 It is a specific flow chart of step S2 in a high-precision turnout panoramic image synthesis method of the present invention;

[0022] Figure 6 It is a specific flow chart of step S24 in a high-precision turnout panoramic image synthesis method of the present invention;

[0023] Figure 7 It is a schematic diagram of a spliced ​​image obtained by a high-precision turnout panorama synthesis method of the present invention;

[0024] Figure 8 It is a specific flow chart of step S3 in a high-precision turnout panoramic image synthesis method of the present invention;

[0025] Fig. 9 It is a specific flow chart of step S4 in a high-precision turnout panoramic image synthesis method of the present invention;

[0026] Fig.10 It is a partial schematic diagram of a target image obtained by a high-precision turnout panoramic image synthesis method of the present invention;

[0027] Fig.11 It is a schematic diagram of the composition of a high-precision turnout panoramic image synthesis system of the present invention;

[0028] Fig.12 It is a schematic diagram of the specific composition of a feature processing module in a high-precision turnout panoramic image synthesis system of the present invention;

[0029] Fig.13 It is a schematic diagram of the specific composition of a splicing module in a high-precision turnout panoramic image synthesis system of the present invention;

[0030] Fig.14 It is a schematic diagram of the specific composition of a correction module in a high-precision turnout panoramic image synthesis system of the present invention;

[0031] Fig.15 It is a schematic diagram of the specific composition of a top view conversion module in a high-precision turnout panoramic image synthesis system of the present invention. DETAILED DESCRIPTION

[0032] In order to facilitate the understanding of the present invention, the present invention is described in more detail below in conjunction with the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are provided in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0033] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more related listed items.

[0034] like Figure 1 As shown, a high-precision turnout panorama synthesis method of the present invention is shown, comprising the steps of:

[0035] Step S1: Extract and match features of the original image to construct the largest set of stitchable images.

[0036] In this embodiment, a PTZ (Pan Tilt Zoom) camera is used. The PTZ camera can perform all-round (left / right / up / down) movement and lens magnification and zoom control to capture high-definition original images.

[0037] like Figure 2 As shown, multiple original images captured by the PTZ camera are displayed, and the field of view of the original image is very small, which is a very small part of the railway turnout, and may be even smaller in reality.

[0038] Specifically, if Figure 3 As shown, step S1 includes the following sub-steps:

[0039] Step S11: Using the SIFT feature extraction algorithm, extract feature points of the original image.

[0040] Among them, the SIFT (Scale Invariant Feature Transform) feature extraction algorithm is a feature extraction method widely used in computer vision, which can detect and describe local features in images. In addition, the SIFT feature extraction algorithm can extract stable feature points at different scales and rotation angles, and has strong robustness to interference such as illumination and noise. The SIFT feature extraction method includes constructing a DOG (Difference of Gaussian) scale space, key point search and positioning, direction assignment, and generation of key point descriptors. The extracted SIFT features have a large amount of information and are suitable for fast and accurate matching in massive databases. SIFT features are based on some local appearance interest points on objects, are independent of the size and rotation of the image, and have a high tolerance for changes in light, noise, and perspective.

[0041] Furthermore, feature points are composed of key points and descriptors. Key points are points found in an image, and their function is to find landmarks (representative points) in an image. For each key point, there are three pieces of information: position, scale, and direction. The function of the descriptor is to match the corresponding key points in the two images when the key points of the two images are found, usually vectors.

[0042] Descriptors, also known as feature descriptors, are vectors or feature value sets used to describe the features of the area around feature points. In the fields of image processing, computer vision, and point cloud processing, feature descriptors are used to characterize local feature information in images, point clouds, or other data, and are usually used for tasks such as matching, recognition, and tracking. The properties that feature descriptors need to have include robustness, uniqueness, compactness, and computational efficiency to ensure that they have certain invariance to multiple geometric transformations and illumination transformations. At the same time, the feature descriptor of an area can be well distinguished from other areas, and only a lower-dimensional vector can be used to robustly and uniquely describe the feature, and the feature descriptor can be quickly calculated and efficiently compared. Commonly used feature descriptors include, but are not limited to, SIFT and SURF, which play an important role in computer vision tasks such as target detection, image matching, and 3D reconstruction.

[0043] In this embodiment, the SIFT feature extraction algorithm is used to extract feature points from the original image. Among them, the local features of the original image are extracted. These feature points remain invariant to the rotation, scale scaling, brightness change, etc. of the original image. The scale invariance is to ensure that the key points can be detected from far to near, and the rotation invariance is to ensure that the key points can still be detected after the image is rotated. At the same time, a certain degree of stability is maintained for changes in viewing angles, affine transformations, and noise, so that effective feature extraction and feature matching can be performed in complex environments.

[0044] Furthermore, in the process of feature point extraction, for each original image, key points are first extracted, and then the descriptors corresponding to the key points are calculated. A descriptor is established for each key point so that it does not change with various changes, such as changes in lighting, changes in viewing angle, etc., and the descriptor has a high uniqueness, so as to increase the probability of correct matching of feature points.

[0045] Among them, feature extraction is the basis of feature matching. The feature points extracted by the SIFT feature extraction algorithm can be further used in subsequent feature matching and feature recognition.

[0046] Step S12: Using the FLANN feature matching algorithm, feature matching is performed on the feature points of the original image to obtain a feature matching result.

[0047] Among them, the FLANN (Fast Approximate Nearest Neighbor Search Library) feature matching algorithm can achieve fast and efficient matching. The FLANN feature matching algorithm significantly reduces the complexity of searching for nearest neighbors in high-dimensional space by constructing data structures such as KD trees and KMeans trees. Compared with traditional linear search methods, the FLANN feature matching algorithm can reduce the search complexity from O(N) to O(log N), greatly improving the search efficiency.

[0048] For the feature points in one original image, the FLANN feature matching algorithm is used to find the two feature points that are most similar to its descriptor in another original image, and the feature matching is achieved by calculating the distance between the corresponding descriptors in the two original images. At the same time, the reliability of the matching is judged by the ratio test, specifically: comparing the distance ratio between the nearest neighbor and the second nearest neighbor of the descriptor. If the distance ratio between the nearest neighbor and the second nearest neighbor is less than the ratio threshold, the two corresponding feature points are matched as a feature point pair, and the matching is considered reliable, which also indicates that there is a correlation between the corresponding two original images, and all reliable matches are retained; if the distance ratio between the nearest neighbor and the second nearest neighbor is greater than or equal to the ratio threshold, the matching is considered unreliable, and the unreliable matching is eliminated to obtain the feature matching result.

[0049] In this process, by comparing the distance ratio between the nearest neighbor and the next nearest neighbor, the correctness of the match can be judged, thereby improving the accuracy and efficiency of the match.

[0050] Step S13: using the union-find data structure and feature matching results to obtain the largest set of stitchable images.

[0051] Specifically, the original images to which the matched feature points belong are grouped to form an image set. In the image set, each original image is regarded as an element in an image set, and the matched feature point pairs represent the connection relationship between these elements. A union-find data structure is used to manage the image set, and all matching feature point pairs are traversed. If two feature points come from two different image sets and the matching confidence between them is higher than the feature matching confidence threshold, the two image sets are considered to be related and merged into a new image set. After completing the merging of all image sets, the image set containing the most elements, that is, the maximum splicable image set, is selected and returned. This maximum splicable image set represents the main image sequence or group because it contains the most interrelated original images, thereby obtaining the maximum splicable image subset.

[0052] At the same time, the camera parameters need to be determined. Figure 4 As shown, the specific steps include:

[0053] Step S141: Calculate the homography matrix according to the feature matching result.

[0054] Specifically, feature points and descriptors extracted from each original image obtained previously, as well as matching relationships between feature points of different original images, are obtained, and a set of homography matrices are calculated based on the feature points and matching relationships between feature points of different original images.

[0055] Among them, the homography matrix is ​​used to describe the relationship matrix between a point in three-dimensional space and different projection planes. Since the position of a point in three-dimensional space is fixed, but for different image acquisition devices, the two-dimensional plane of the original image acquired is different. The homography matrix can map points on one plane to another plane, that is, find the feature matching relationship between the feature points in one original image and the feature points in another original image, that is, there are corresponding feature point pairs in the two original images. After finding the feature point pairs, it is helpful to stitch the two different original images.

[0056] Step S142: Estimate camera parameters according to the homography matrix.

[0057] In this embodiment, the homography matrix is ​​a 3×3 matrix used to describe the mapping relationship between two planes. The calculated homography matrix is ​​used to further estimate the camera parameters and save the camera parameters, thereby ensuring that the original images can be correctly aligned during the stitching process, reducing misalignment and ghosting, and improving the accuracy of the registration. Among them, the camera parameters include the camera's intrinsic parameters (such as focal length, principal point, etc.) and extrinsic parameters (i.e., camera posture).

[0058] Step S143: Correct the camera parameters using the bundle adjustment method.

[0059] The camera parameters are corrected by using the bundle adjustment method. The specific operation is: setting the image matching confidence threshold to screen the matching original image pairs.

[0060] At the same time, in image processing, masking is a technique used to specify specific areas or pixels in an image. The mask is usually a matrix of the same size as the original image, in which the element values ​​can be used to identify whether the pixels at the corresponding position should participate in a specific processing operation. Therefore, the optimization mask is set to:

[0061] ,

[0062] Among them, mask represents the optimization mask, which is used to specify which parameters in the camera parameters need to be optimized. 1 indicates that optimization is required, and 0 indicates that optimization is not required.

[0063] In this process, the bundle adjustment method is used to adjust and save the camera parameters using the previously acquired feature points and the matching relationship between their descriptors and feature points. Specifically, the sum of the distances between the light rays passing through the camera center and the feature points is minimized to optimize the camera intrinsic parameters and camera extrinsic parameters. At the same time, the 3D positions of the feature points are optimized so that these points can more accurately match the feature points detected on the image after being projected by the camera parameters, so that the optimized camera parameters and feature point positions can ensure that the images can be more accurately aligned when stitched, reduce misalignment and ghosting, and make the stitched images more visually natural and coherent.

[0064] In this embodiment, the essence of feature extraction is to obtain the key points and descriptors corresponding to each original image. The feature points extracted by the SIFT feature extraction algorithm have scale invariance and rotation invariance. By searching in a multi-scale space, the stability and accuracy of feature matching are improved. Feature matching is performed based on the descriptors, and then the corresponding original images are constructed into a maximum set of splicable images based on the matched feature point pairs.

[0065] In step S1, the SIFT feature extraction algorithm is used to accurately extract image feature points, which reduces the processing of redundant information and improves the overall processing efficiency of image stitching. The strategy of querying the largest subset of stitchable images by using the union-find set effectively reduces the complexity of the algorithm and further improves the processing speed. The nearest neighbor and next nearest neighbor matching methods are used to match the feature points of any two original images, which can accurately find the corresponding relationship between the original images and avoid mismatching. The camera parameters are roughly estimated by the homography matrix, which provides a reliable basis for subsequent precise stitching. The camera parameters are corrected using the bundle adjustment method to further improve the accuracy of stitching.

[0066] Step S2: performing image stitching according to the maximum stitchable image set to obtain a stitched image.

[0067] Among them, by traversing all the original images in the maximum stitchable image set in turn, the original image after waveform correction is performed using spherical projection transformation according to the camera intrinsic parameters and rotation matrix, and the final transformed distorted transformed image and the corresponding mask are obtained. According to the final transformed distorted transformed image and the corresponding mask and feature point information, the distorted transformed image is exposed compensated to obtain the registered image, and then all the registered images are fused into a stitched image using multi-band fusion.

[0068] Specifically, if Figure 5 As shown, step S2 includes the following sub-steps:

[0069] Step S21: performing waveform correction on the original images in the maximum stitchable image set to obtain a set of distortion-free images.

[0070] Among them, the rotation matrix and transformation parameters of the previously saved camera are obtained, and a method based on image remapping is used to perform waveform correction on all original images in the maximum stitchable image set according to the rotation matrix of the acquired camera, so as to correct the wavy distortion caused by camera lens distortion, poor shooting conditions or image stitching algorithm.

[0071] First, a set of feature points in the original image with known real coordinates are used to estimate the camera's distortion parameters, which describe the degree of distortion of the original image. Based on the estimated distortion parameters, a correction algorithm is used to calculate an inverse transformation that can map points in the distorted original image to the correct position in the undistorted image.

[0072] Then, each pixel in the original image is remapped using the calculated inverse transform. Specifically, for each pixel in the original image, its corresponding undistorted position in the undistorted image is calculated, and the value of the pixel is copied to the undistorted position. All original images in the maximum stitchable image set are waveform corrected to obtain a distortion-free image set, where all distortion-free images in the distortion-free image set correspond one-to-one to the original images in the maximum stitchable image set.

[0073] Furthermore, in order to improve the accuracy of waveform correction, a nonlinear optimization algorithm can be used to fine-tune the distortion parameters. By iteratively minimizing the reprojection error, the optimal distortion parameters can be found to correct the ripple distortion in the original image, making the original image clearer and more realistic.

[0074] In this process, waveform correction is performed on the original images to help ensure that the overlapping areas between different original images can be matched more accurately during the image stitching process, reducing misalignment or distortion caused by distortion.

[0075] Step S22: performing a distortion transformation on the undistorted images in the undistorted image set to obtain a distortion transformed image set.

[0076] First, all the undistorted images in the obtained undistorted image set are mapped to the spherical coordinate system to obtain the distorted image set; at the same time, the corresponding masks are processed for subsequent image stitching. Among them, all the distorted images in the distorted image set correspond one-to-one to the undistorted images in the undistorted image set. The mapping formula is:

[0077] ,

[0078] ,

[0079] ,

[0080] ,

[0081] in, Represents the three-dimensional coordinates of the pixels on the original image, R represents the rotation matrix, K represents the intrinsic parameter matrix of the camera, and Respectively R and K The inverse matrix of represents the 3D coordinates of pixels on the undistorted image, represents the polar coordinates of the pixels on the distorted image, Represents the scaling matrix.

[0082] Specifically, the spherical projection method is used to map all the undistorted images in the undistorted image set to the spherical coordinate system for image stitching and processing the corresponding masks. The mapping process is to convert the pixel coordinates (two-dimensional) on the original image into three-dimensional space coordinates, and convert the three-dimensional coordinates of the pixels on the original image into Multiply by the camera's intrinsic matrix K The inverse matrix and the rotation matrix R The inverse matrix , thus obtaining the three-dimensional coordinates of the pixels on the undistorted image , and then the three-dimensional coordinates of the pixels on the undistorted image Convert to spherical polar coordinates (radians corresponding to elevation and opening angles), that is, the polar coordinates of the pixels on the distorted image .

[0083] Finally, all the distorted images in the obtained distorted image set are mapped to a rectangular plane to obtain a distorted transformation image set, wherein all the distorted transformation images in the distorted transformation image set correspond one-to-one to the distorted images in the distorted image set.

[0084] Specifically, based on the spherical coordinate system, the position of each point on the surface corresponding to the rectangular plane is determined by the interpolation algorithm, and the spherical coordinates are converted into three-dimensional space coordinates again. The three-dimensional space coordinates are multiplied by the intrinsic parameter matrix of the camera. K and the rotation matrix R , the final pixel coordinates and the corresponding warped transformed image set are obtained, thereby improving the stitching accuracy, enhancing image continuity, optimizing feature matching, and simplifying the stitching process.

[0085] Step S23: performing exposure compensation on the distorted transformed images in the distorted transformed image set to obtain a registered image set.

[0086] Wherein, exposure compensation is performed on the distorted transformed image based on gain compensation of the image block.

[0087] Specifically, the parameters required for exposure compensation are calculated based on the position of the feature points, the set of warped transformed images, and the masks corresponding to the warped transformed images, so as to adjust the brightness of the images in the subsequent image stitching process to ensure that the stitched images are more visually consistent. Specifically, first, the warped transformed images are divided into different image blocks of the same size. The sizes of these image blocks can be set according to specific needs, but it is generally recommended to select an appropriate image block size to balance the computational complexity and compensation accuracy.

[0088] In this embodiment, the empirical value of the image block size is 32×32 pixels. For each image block, its exposure gain coefficient is calculated, and this gain coefficient is used to adjust the brightness of the image block to make it more consistent with the brightness of other image blocks or reference images. The calculation of the gain coefficient is usually based on the brightness difference between image blocks or the brightness statistics of overlapping areas. By multiplying each pixel value in the image block by the corresponding gain coefficient, the brightness of the corresponding image block is adjusted. This process changes the exposure of the image block to make it more consistent with other image blocks.

[0089] After applying the gain coefficients of all image blocks, in order to eliminate the brightness discontinuity or "block" effect caused by the block division, the gain coefficients need to be smoothed and filtered. This can be achieved by applying some form of filtering algorithm to the gain coefficients of all image blocks of the same warped image, such as segmented linear filtering, where the kernel of the segmented filter can be set to [1 / 4, 1 / 2, 1 / 4] to perform weighted averaging of the gain coefficients of adjacent image blocks.

[0090] After completing the block compensation and smoothing filtering, the processed image blocks can be reassembled into a complete registered image, thereby obtaining a registered image set, wherein all the registered images in the registered image set correspond one-to-one to the warped transformed images in the warped transformed image set.

[0091] Since the exposure levels of different registered images are now more consistent, the stitched images will have more natural transitions and less brightness differences in the subsequent stitching, thereby reducing the problem of stitching seams or brightness inconsistencies caused by exposure differences between different distorted transformed images, making it easier to improve image quality, enhance stitching consistency, and reduce stitching errors.

[0092] Step S24: performing multi-band fusion on the registered images in the registered image set to obtain a stitched image.

[0093] like Figure 6 As shown, step S24 includes the following sub-steps:

[0094] Step S241: using a seam estimation method based on minimum graph cut to find the best seam.

[0095] Specifically, the optimal seam is calculated by using a set of images that have been transformed to be aligned, a set of feature points that match between images, and a corresponding set of mask images. The optimal seam will be used to guide the subsequent image stitching process to ensure that the most similar pixels are selected for stitching in the overlapping area, thereby performing optimal seam estimation to reduce blur and artifacts during stitching, improve stitching quality, and reduce visual discontinuities.

[0096] Step S242: performing multi-band fusion on the registered images according to the optimal seam to generate a stitched image.

[0097] Using the multi-band fusion method, all the registered images in the registered image set are fused to obtain a stitched image. Multi-band fusion is used to effectively merge image information from different frequency bands or different sources to generate a stitched image with higher quality and richer information.

[0098] First, the registered image is decomposed into multiple frequency bands or scales, each of which contains the image information of the registered image at different frequencies. This is usually achieved through techniques such as filtering or pyramid decomposition, such as Laplacian pyramid or Gaussian pyramid.

[0099] Then, at each frequency band or scale, the image information from different registered images is mixed together, which usually involves weight assignment, fusion algorithms (such as weighted averaging, wavelet transform, convolution, etc.) and possible optimization steps to ensure that the image information at different frequency bands or scales can be smoothly transitioned and fused.

[0100] Finally, the mixed image information of each frequency band or scale is reassembled to generate the final output image. This step usually involves techniques such as inverse filtering or pyramid reconstruction, which can produce a stitched image with higher quality, smoother transitions and richer information.

[0101] In step S2, the waveform correction technology effectively eliminates the image distortion caused by camera shake or scene changes, making the stitched image smoother and more natural. The image spherical projection transformation and exposure compensation technology ensure the consistency of visual attributes such as brightness and contrast of different images, improving the visual effect of the stitched image. The seam processing and multi-band fusion technology further eliminate the seam marks in the stitched image, making the image transition more natural.

[0102] like Figure 7 As shown, a stitched image obtained by using this method is shown. The image stitching is natural and smooth. There is an obvious curved part in the lower half of the image, which requires the next step of correction operation.

[0103] Step S3: performing image correction on the stitched image to obtain a corrected image.

[0104] Specifically, if Figure 8 As shown, step S3 includes the following sub-steps:

[0105] Step S31: performing correction preprocessing on the stitched image to obtain a filtered stitched image.

[0106] First, the stitched image is binarized. The stitched image is first converted into a grayscale image, and then the first pixel threshold is set to 160, the pixel values ​​below the first pixel threshold are set to 0, and the pixel values ​​above the first pixel threshold are set to 255, and the first feature map is obtained, thereby simplifying the image information, highlighting the key features, reducing the amount of data, and improving the efficiency and accuracy of image processing.

[0107] Secondly, the connected areas are analyzed and small connected areas are filtered. The first connected image is obtained by searching for connected blocks in the first feature map, setting the first connected threshold to 1000, and setting the pixel values ​​corresponding to the connected blocks whose area is smaller than the first connected threshold to 0, so that various noises are filtered out and the main features are highlighted.

[0108] Finally, the image opening operation is used to filter the connected images, and the first connected image is eroded and then expanded to obtain the filtered spliced ​​image. The convolution kernel size of the erosion and expansion is set to , thereby eliminating small spots and tiny protrusions, smoothing images, improving edges, separating adjacent objects, and removing small objects.

[0109] Step S32: using the Canny edge detection algorithm to extract the corrected edge features in the filtered stitched image.

[0110] Among them, the connected blocks in the filtered spliced ​​image are sorted from large to small according to the area size. According to the number of tracks in the image, the first few connected blocks with the same number are selected from the sorted connected blocks, and the pixel values ​​corresponding to other connected blocks are set to 0. The Canny edge detection algorithm is used to detect the corrected edge feature points of the processed image, so as to obtain the corrected edge features of each track.

[0111] Step S33: According to the corrected edge features, a least square method is used to establish a correction mapping relationship to obtain a corrected image.

[0112] Among them, for the correction edge features corresponding to each track, the least squares method is used to fit the track curve and straight line, so as to establish a correction mapping relationship based on this curve and straight line, and use this correction mapping relationship as the full image mapping relationship to eliminate the image distortion caused by stitching and generate a corrected image.

[0113] In this process, the image distortion caused by image stitching is eliminated by correcting the stitched image, further improving the reliability of the result.

[0114] Step S4: convert the corrected image into a top view to obtain a target image.

[0115] Specifically, if Fig. 9 As shown, step S4 includes the following sub-steps:

[0116] Step S41: performing overhead preprocessing on the corrected image to obtain a filtered corrected image.

[0117] First, the corrected image is binarized. The corrected image is first converted into a grayscale image, and then the second pixel threshold is set to 160, the pixel values ​​below the second pixel threshold are set to 0, and the pixel values ​​above the second pixel threshold are set to 255, and the second feature map after binarization is obtained, thereby simplifying the image information, highlighting the key features, reducing the amount of data, and improving the efficiency and accuracy of image processing.

[0118] Secondly, the connected areas are analyzed and small connected areas are filtered. The second connected image is obtained by searching for connected blocks in the second feature map, setting the second connected threshold to 1000, and setting the pixel values ​​corresponding to the connected blocks whose area is smaller than the second connected threshold to 0, thereby filtering out various noises and highlighting the main features.

[0119] Finally, the image opening operation is used to filter the connected image, and the second connected image is subjected to morphological operations of first erosion and then dilation to obtain the filtered corrected image. The convolution kernel size of the erosion and dilation is set to , thereby eliminating small spots and tiny protrusions, smoothing images, improving edges, separating adjacent objects, and removing small objects.

[0120] Step S42: using the Canny edge detection algorithm to extract the top-view edge features in the filtered and corrected image.

[0121] Firstly, the connected blocks in the filtered corrected image are sorted from large to small according to their area size. According to the number of tracks in the filtered corrected image, the first few connected blocks with the same number are selected from the sorted connected blocks, and the pixel values ​​corresponding to other connected blocks are set to 0. The Canny edge detection algorithm is used to detect the top-view edge feature points of the processed image, so as to obtain the top-view edge features of each track.

[0122] Then, calculate the mean coordinates of the feature points of the overlooking track , the track overlooking edge feature points detected by the Canny edge detection algorithm, calculate the coordinate mean of each track overlooking edge feature point, and get .

[0123] Step S43: According to the top-view edge features, a least square method is used to establish a mapping relationship to obtain a target image.

[0124] For each track’s corresponding edge feature point, the track curve is fitted using the least squares method. , so according to this curve and straight line A mapping relationship is established, and this mapping relationship is used as the mapping relationship of the entire image, which is converted into a top view to obtain the target image.

[0125] In this process, the corrected image is converted into a bird's-eye view, providing an intuitive and comprehensive perspective for applications such as railway turnout monitoring and terrain analysis.

[0126] like Fig.10 As shown, a partial schematic diagram of a target image obtained by the present method is shown, which intuitively shows the specific condition of the turnout, and the characteristics of the track can be directly observed and analyzed, so as to more accurately identify and understand the shape, direction, distribution and other information of the turnout.

[0127] like Fig.11 As shown, based on the same inventive concept, the present invention also provides a high-precision turnout panoramic image synthesis system, including a feature processing module 2, a stitching module 3, a correction module 4 and a bird's-eye view conversion module 5, wherein the feature processing module 2 is used to perform feature extraction and feature matching on the original image to construct a maximum stitchable image set; the stitching module 3 is used to perform image stitching according to the maximum stitchable image set to obtain a stitched image; the correction module 4 is used to perform image correction on the stitched image to obtain a corrected image; the bird's-eye view conversion module 5 is used to perform a bird's-eye view conversion on the corrected image to obtain a target image.

[0128] Furthermore, the system also includes an acquisition module 1, which is used to capture high-quality original images from an external device and supports a variety of common image formats. Then, the acquisition module 1 loads these image data into the system's memory or cache for subsequent processing. During the loading process, the acquisition module 1 will also perform necessary preprocessing on the image, such as noise reduction and contrast enhancement, to improve image quality and enhance the visibility and detectability of feature points. In addition, the acquisition module 1 will also evaluate the clarity and overall quality of the original image (ensuring that adjacent images have a suitable degree of overlap) to ensure that only original images that meet the requirements will be passed to the subsequent feature processing module, thereby ensuring the accuracy and reliability of the entire image stitching process. Through this series of sophisticated operations, the acquisition module 1 provides a solid foundation for the entire image stitching process.

[0129] like Fig.12 As shown, the feature processing module 2 includes a feature extraction module 21, a feature matching module 22 and an image aggregation module 23. The feature extraction module 21 is used to extract feature points of the original image using the SIFT feature extraction algorithm; the feature matching module 22 is used to perform feature matching on feature points using the FLANN feature matching algorithm; the image aggregation module 23 is used to obtain the maximum set of stitchable images using a union-find data structure, and estimate the camera parameters based on the homography matrix, and then fine-tune these parameters through the bundle adjustment method.

[0130] like Fig.13 As shown, the stitching module 3 includes a waveform correction module 31, an image transformation module 32, an image registration module 33 and a fusion module 34. The waveform correction module 31 is used to perform waveform correction on the original images in the maximum stitchable image set to obtain a distortion-free image set; the image transformation module 32 is used to perform distortion transformation on the distortion-free images in the distortion-free image set to obtain a distortion transformation image set; the image registration module 33 is used to perform exposure compensation on the distortion transformation images in the distortion transformation image set to obtain a registered image set; the fusion module 34 is used to perform multi-band fusion on the registered images in the registered image set to obtain a stitched image.

[0131] In this embodiment, the stitching module 3 is the core module of the entire system, which is responsible for accurately stitching multiple original images that have undergone feature extraction and feature matching into a seamless stitching image. During the image stitching process, the waveform correction module 31 and the image transformation module 32 will also perform waveform correction and image distortion transformation on multiple original images to ensure the alignment accuracy between the distorted transformed images. Afterwards, the stitching effect is further improved by the image registration module 33, and the fusion module 34 uses multi-band fusion technology to smooth the transition between different registered images, eliminate stitching traces, and perform global correction on the stitched image to eliminate any distortion or deformation caused by stitching.

[0132] like Fig.14 As shown, the correction module 4 includes a correction preprocessing module 41, a correction edge feature extraction module 42 and a correction mapping module 43. The correction preprocessing module 41 is used to perform correction preprocessing on the stitched image to obtain a filtered stitched image; the correction edge feature extraction module 42 is used to use the Canny edge detection algorithm to extract the correction edge features in the filtered stitched image; the correction mapping module 43 is used to establish a correction mapping relationship using the least squares method to obtain a corrected image.

[0133] In this embodiment, the correction module 4 is specifically responsible for eliminating the distortion caused by the image stitching process to ensure the quality of the final generated image. The correction preprocessing module 41, the correction edge feature extraction module 42 and the correction mapping module 43 perform detailed correction and conversion steps, which can significantly improve the quality and visual effect of the stitched image.

[0134] like Fig.15 As shown, the overhead view conversion module 5 includes a overhead view preprocessing module 51, a overhead view edge feature extraction module 52 and a overhead view mapping module 53. The overhead view preprocessing module 51 is used to perform overhead view preprocessing on the corrected image to obtain a filtered corrected image; the overhead view edge feature extraction module 52 is used to use the Canny edge detection algorithm to extract the overhead edge features in the filtered corrected image; the overhead view mapping module 53 is used to establish a overhead mapping relationship using the least squares method to obtain a target image.

[0135] In this embodiment, the top view conversion module 5 is responsible for converting the corrected image into a top view so as to better analyze and identify specific features therein, such as tracks. The top view preprocessing module 51 first performs binarization processing on the input corrected image, simplifying the pixel values ​​in the corrected image into two levels, namely black and white, to facilitate subsequent feature extraction. Then, through the connected region analysis technology, the top view preprocessing module 51 can identify and mark each connected region in the image, while filtering out those connected regions with smaller areas to reduce noise and interference. Subsequently, the top view preprocessing module 51 applies an image opening operation to further smooth the image, eliminate small objects, disconnect narrow connections, and keep larger bright areas basically unchanged. This helps to reduce noise and details in the image and make the track features more prominent. Next, the top view edge feature extraction module 52 uses the Canny edge detection algorithm to extract the edge features of the track. The Canny algorithm can detect strong edges in the image and is insensitive to noise, so it is very suitable for extracting objects with obvious edge features such as tracks. After extracting the track edge features, the top view edge feature extraction module 52 calculates the coordinate mean of these feature points , to get the approximate position of the track and establish a straight line . Finally, the top-view mapping module 53 uses the least squares method to fit these feature points and establish a curve to more accurately describe the shape of the track. Based on the established straight lines and curves, as well as the spatial relationship in the filtered and corrected image, a mapping relationship is established to convert the track features in the filtered and corrected image into corresponding positions in the top view, that is, to obtain the target image. In this way, the user can observe and analyze the features of the turnout from a unified, top-down perspective, so as to more accurately identify and understand the shape, direction, distribution and other information of the turnout.

[0136] like Fig.11 As shown, the system also includes a data storage module 6 for storing the stitched image, the corrected image and the target image, and at the same time, storing the coordinates of the track feature points extracted during the use of the system, the fitted straight line and curve parameters and other data. Based on the stored data, subsequent analysis, monitoring tasks and user interaction are convenient.

[0137] Based on the same inventive concept, the present invention also provides a high-precision turnout panoramic image synthesis device, which can be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (PDA), a server, etc.

[0138] Specifically, the device may include a processor and a memory.

[0139] Further, the processor may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, which may implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0140] Further, the memory can be used to store non-volatile software programs, non-volatile computer executable programs and modules as a non-volatile computer readable storage medium. The memory may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0141] Based on the same inventive concept, the present invention further provides a computer-readable medium, in which program codes are stored. The program codes can be called by a processor to execute the method described in the above method embodiment.

[0142] The computer readable storage medium may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium includes a non-transitory computer-readable storage medium. The computer readable storage medium has storage space for program codes that execute any of the method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code can be compressed in an appropriate form.

[0143] It can be seen that the present invention discloses a high-precision turnout panorama synthesis method, system, device and storage medium, including the steps of: extracting features and matching features of original images to construct a maximum set of stitchable images; stitching images according to the maximum set of stitchable images to obtain a stitched image; correcting the stitched image to obtain a corrected image; converting the corrected image into a top view to obtain a target image. The present invention realizes efficient, accurate and high-quality railway turnout image stitching by adopting a series of technical means such as SIFT feature extraction algorithm, nearest neighbor and next nearest neighbor matching, union-find query of the maximum set of stitchable images, homography matrix estimation, bundle adjustment correction, etc., and has strong versatility and practicality.

[0144] The above are only embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A high-precision turnout panorama synthesis method, characterized in that: Includes steps: Extracting and matching features of the original image to construct a maximum set of stitchable images, wherein the original image is captured by a PTZ camera and the field of view of the original image is a very small part of the railway turnout; Performing image stitching according to the maximum stitchable image set to obtain a stitched image; Performing image correction on the stitched image to obtain a corrected image; Performing a top view conversion on the corrected image to obtain a target image; The step of obtaining the spliced ​​image includes: The original images in the maximum stitchable image set are subjected to waveform correction to obtain a set of distortion-free images; the distortion-free images in the set of distortion-free images are subjected to distortion transformation to obtain a set of distortion-transformed images, including: First, all the undistorted images in the obtained undistorted image set are mapped to a spherical coordinate system to obtain a distorted image set, wherein all the distorted images in the distorted image set correspond one-to-one to the undistorted images in the undistorted image set; Then, all the distorted images in the obtained distorted image set are mapped to a rectangular plane to obtain a distorted transformed image set, wherein all the distorted transformed images in the distorted transformed image set correspond one-to-one to the distorted images in the distorted image set; Exposure compensation is performed on the distorted transformed images in the distorted transformed image set to obtain a registered image set; and multi-band fusion is performed on the registered images in the registered image set to obtain the spliced ​​image.

2. The panoramic image synthesis method according to claim 1, characterized in that: The step of extracting and matching features of the original image to construct the largest set of stitchable images includes: Using SIFT feature extraction algorithm to extract feature points of the original image; Using the FLANN feature matching algorithm, feature matching is performed on the feature points of the original image to obtain a feature matching result; The maximum set of stitchable images is obtained by using the union-find data structure and the feature matching results.

3. The panoramic image synthesis method according to claim 2, characterized in that: While obtaining the maximum set of stitchable images by using the union-find data structure and feature matching results, it is also necessary to determine the camera parameters, including: Calculating a homography matrix according to the feature matching result; estimating camera parameters according to the homography matrix; The camera parameters are corrected by using a bundle adjustment method.

4. The panoramic image synthesis method according to claim 1, characterized in that: The step of performing multi-band fusion on the registered images in the registered image set to obtain the stitched image includes: Use the seam estimation method based on minimum graph cut to find the optimal seam; According to the optimal seam, the registered images are fused in multiple frequency bands to generate the stitched image.

5. The panoramic image synthesis method according to claim 1, characterized in that: The step of performing image correction on the stitched image to obtain a corrected image includes: Performing correction preprocessing on the stitched image to obtain a filtered stitched image; Using the Canny edge detection algorithm to extract the corrected edge features in the filtered stitched image; According to the corrected edge features, a least square method is used to establish a correction mapping relationship to obtain the corrected image.

6. The panoramic image synthesis method according to claim 1, characterized in that: The step of converting the corrected image into a top view to obtain a target image includes: Performing a top-down preprocessing on the corrected image to obtain a filtered corrected image; Using the Canny edge detection algorithm to extract the top-view edge features in the filtered and corrected image; According to the top-view edge features, a top-view mapping relationship is established using the least squares method to obtain the target image.

7. A high-precision turnout panorama synthesis system, characterized in that: It includes feature processing module, stitching module, correction module and top view conversion module. The feature processing module is used to extract and match features of the original image to construct a maximum set of stitchable images, wherein the original image is collected by a PTZ camera, and the field of view of the original image is a very small part of the railway turnout; The stitching module is used to stitch images according to the maximum stitchable image set to obtain a stitched image, wherein the stitching includes: The original images in the maximum stitchable image set are subjected to waveform correction to obtain a set of distortion-free images; the distortion-free images in the set of distortion-free images are subjected to distortion transformation to obtain a set of distortion-transformed images, including: First, all the undistorted images in the obtained undistorted image set are mapped to a spherical coordinate system to obtain a distorted image set, wherein all the distorted images in the distorted image set correspond one-to-one to the undistorted images in the undistorted image set; Then, all the distorted images in the obtained distorted image set are mapped to a rectangular plane to obtain a distorted transformed image set, wherein all the distorted transformed images in the distorted transformed image set correspond one-to-one to the distorted images in the distorted image set; Performing exposure compensation on the distorted transformed images in the distorted transformed image set to obtain a registered image set; performing multi-band fusion on the registered images in the registered image set to obtain the stitched image; The correction module is used to perform image correction on the stitched image to obtain a corrected image; The top view conversion module is used to perform top view conversion on the corrected image to obtain a target image.

8. A high-precision turnout panorama synthesis device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images