Train on-station running state monitoring and identifying method and system based on images

Abstract

The invention discloses an image-based train in-station operation state monitoring and identification method and system, and relates to the technical field of train state identification, and the method comprises the steps: collecting image data, carrying out the two-dimensional wavelet decomposition, carrying out the denoising processing through a soft threshold value, and regenerating the denoised alignment image data; gaussian pyramids for aligning the image data are constructed, equalization mapping is carried out through histogram equalization, and complete image data are combined based on each layer of Gaussian pyramids. According to the method, local details of the image are remarkably improved under different illumination conditions through CLAHE self-adaptive contrast adjustment, so that an original region with relatively low contrast becomes clearer, image levels with different resolutions are generated through multi-scale feature extraction and Gaussian pyramid decomposition, and the image resolution is improved. The structure information of the image under different scales is ensured to be reserved, and through the combination of the SGM and the RANSAC, the point cloud still has a strong anti-noise capability in a complex scene, and can adapt to environmental conditions such as illumination change and reflection interference.

Description

Technical Field

[0001] The present invention relates to the technical field of train status recognition, and in particular to an image-based train in-station running status monitoring and recognition method and system. Background Art

[0002] With the rapid development of modern railway transportation, train operation safety and efficiency have become the core issues of concern in the field of transportation. In the operation scenario of train stations, the dynamic status monitoring of trains and the accurate identification of their key components are directly related to operation safety and maintenance efficiency. Traditional train operation status monitoring methods usually rely on physical sensors or manual inspections, such as accelerometers, strain gauges and other physical devices. Although these methods have certain reliability; However, image data is usually significantly affected by illumination changes, background complexity, and noise interference, which directly affects the accuracy of subsequent feature extraction and anomaly judgment. On the other hand, existing methods mostly use single-scale feature extraction or global equalization processing, ignoring the effective information of multi-scale details and local contrast in the image. This single processing method is difficult to achieve efficient and stable feature extraction in complex scenes, thereby reducing the efficiency of recognizing the running status of the train. Summary of the invention

[0003] In view of the problems existing in the above-mentioned existing image-based train operation status monitoring and identification method and system, the present invention is proposed.

[0004] Therefore, the problem to be solved by the present invention is that image data is usually significantly affected by illumination changes, background complexity and noise interference, which directly affects the accuracy of subsequent feature extraction and abnormality judgment. On the other hand, existing methods mostly use single-scale feature extraction or global equalization processing, ignoring the effective information of multi-scale details and local contrast in the image. This single processing method is difficult to achieve efficient and stable feature extraction in complex scenes, thereby reducing the efficiency of recognizing the running status of the train.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for monitoring and identifying the running status of trains in stations based on images, comprising: Collect image data for two-dimensional wavelet decomposition, apply soft thresholding for denoising and regenerate aligned image data after denoising; Construct a Gaussian pyramid of aligned image data, perform equalization mapping through histogram equalization, and merge each layer of the Gaussian pyramid into complete image data; Based on the complete image data, the SGM algorithm is used to generate a disparity map, the random sampling consistency algorithm is used to remove matching points with large errors, and the spatial coordinates of each pixel are calculated to form point cloud data. A deep learning model is built to identify key point cloud data. Based on the determination of gradient features and texture features of Gaussian pyramid decomposition, the comprehensive eigenvalues ​​are calculated, and the comprehensive eigenvalues ​​corresponding to the key point cloud data are used for abnormal judgment. Based on the abnormal judgment, an early warning is issued and an abnormal point report is generated.

[0006] As a preferred solution of the image-based train operation status monitoring and identification method of the present invention, the collected image data is subjected to two-dimensional wavelet decomposition, denoised by applying a soft threshold and regenerating the denoised aligned image data, including: Collect image data from multiple perspectives of the train through a graphics acquisition camera; Select Daubechies wavelet as the wavelet basis function, perform two-dimensional wavelet decomposition on the image data, and the obtained sub-bands include low-frequency components and high-frequency components, representing different frequency details of the image data; Apply soft threshold to the high-frequency component sub-band for denoising to eliminate image noise, which is expressed as:

[0007] in represents the soft threshold, represents the standard deviation of the high-frequency subband, Indicates the total number of pixels in the subband; Apply a soft threshold to each high-frequency coefficient of the high-frequency component subband, expressed as:

[0008] in Represents the high-frequency coefficient value after soft threshold denoising. represents the soft threshold, Represents the sign function, indicating the positive or negative sign of x; Based on the high-frequency coefficient value after denoising, the high-frequency component subband after denoising is inversely transformed, and the decomposed frequency band data are merged into the aligned image data after denoising.

[0009] As a preferred solution of the image-based train in-station running status monitoring and identification method of the present invention, wherein: the Gaussian pyramid of aligned image data is constructed, equalized mapping is performed through histogram equalization, and each layer of the Gaussian pyramid is merged into complete image data, including: Construct a Gaussian pyramid of aligned image data and decompose the aligned image data into different levels of multiple scales, where each level represents the scale of the image at the kth level, expressed as:

[0010] in represents the image of the k+1th layer of the Gaussian pyramid, represents the Gaussian weight coefficient; The contrast-limited adaptive histogram equalization (CLAHE) method is used for each layer of the Gaussian pyramid image. Each layer of the image is divided into blocks, and the grayscale distribution of each sub-block is statistically analyzed to generate a grayscale histogram. , where g represents the gray level; Based on grayscale histogram Calculate the cumulative distribution function CDF, expressed as:

[0011] in represents the cumulative pixel frequency from the minimum gray level 0 to the current gray level g, Indicates grayscale The pixel frequency, Indicates that the gray level variable used for summation starts from 0 and accumulates step by step to the current gray level g; Determine a contrast limit threshold T based on the product of the clipping threshold ratio of the history histogram and the total number of pixels of the sub-block; Based on the contrast limit threshold T, if the pixel frequency of gray level g in the gray histogram is greater than the contrast limit threshold T, then Cut to T and calculate the grayscale histogram The grayscale pixel frequency that exceeds the contrast limit threshold T is evenly distributed to all gray levels, the adjusted grayscale pixel frequency is obtained, and the cumulative distribution function CDF is recalculated; The equalization mapping is calculated based on the recalculated cumulative distribution function, expressed as:

[0012] in represents the equalized pixel value of the image block pixel (i, j), represents the cumulative distribution function of the adjusted grayscale pixel frequency, represents the minimum value of the non-zero cumulative distribution, represents the total number of gray levels, and v represents the total number of pixels in the sub-block; Map the equalized values ​​back to the corresponding pixels in the sub-block; For the overlapping areas of adjacent sub-blocks, bilinear interpolation is used to smoothly transition, and all sub-blocks are combined to generate image data processed by CLAHE; Based on each layer of the Gaussian pyramid, starting from the lowest resolution image, it is upsampled layer by layer to a higher resolution through bilinear interpolation and finally merged into the complete image data.

[0013] As a preferred solution of the image-based train operation status monitoring and identification method of the present invention, wherein: based on the complete image data, the SGM algorithm is used to generate a disparity map, the random sampling consistency algorithm is used to remove matching points with large errors, and the spatial coordinates of each pixel are calculated to form point cloud data, including: According to the complete image data collected from the left and right perspectives and processed by CLAHE, the SGM algorithm is used to generate a disparity map, and the matching cost of each pixel is calculated by the absolute difference cost SAD, which is expressed as:

[0014] in represents the matching cost of the pixel at (x, y) when the disparity is d, and k represents the window size, which is used to control the matching range of the local area. represents the pixel value of the left view, The pixel value of the right view is offset by the disparity d; Optimize matching cost by accumulating cost; Based on the disparity d with the minimum optimized matching cost, a disparity value is assigned to each pixel to generate a preliminary disparity map. The random sampling consensus algorithm RANSAC is used to collect the pixel positions of the left and right view images based on the preliminary disparity map, detect and remove matching points with large errors, and generate an optimized disparity map. Based on the disparity values ​​at different positions of the optimized disparity map, the spatial coordinates of each pixel are calculated, expressed as:

[0015] in represents the depth of the pixel, f represents the focal length of the camera, B represents the baseline distance, Represents the disparity value of the pixel point (x, y); X and Y represent the horizontal and vertical coordinates of the spatial coordinates respectively. and Represents the horizontal and vertical coordinates of the camera's principal point coordinates; Traverse all pixels in the disparity map, calculate their corresponding spatial coordinates, and form point cloud data.

[0016] As a preferred solution of the image-based train operation status monitoring and identification method of the present invention, wherein: the construction of a deep learning model to identify key point cloud data includes: Build a PointNet++ deep learning model, including input layer, local feature extraction layer, feature propagation layer, hierarchical feature aggregation layer, and output layer; The input layer inputs point cloud data, the local feature extraction layer uses a distance-based grouping mechanism to select points in the local area, the hierarchical feature aggregation layer aggregates the extracted local features layer by layer, the feature propagation layer uses a feature propagation mechanism to pass high-level feature information back to the low-level point cloud data, and the output layer outputs the prediction results based on the segmentation label; Use the labeled point cloud samples as training data, select the cross entropy loss function to calculate the difference between the category probability predicted by the model and the actual label, and use the Adam optimizer for optimization. If the calculated loss no longer decreases significantly during the continuous iteration process, stop iterating and output the model. The newly calculated point cloud data is input into the model, and the label probabilities of being identified as different key train components and background are output. The label of the point cloud data with the largest probability is selected, and the point cloud data of the key train components is selected as the key point cloud data.

[0017] As a preferred solution of the image-based train operation status monitoring and identification method of the present invention, wherein: the determination of gradient features and texture features based on Gaussian pyramid decomposition to calculate comprehensive feature values, and abnormal judgment of the comprehensive feature values ​​corresponding to the key point cloud data, include: The gradient features and texture features are determined based on the image data decomposed by Gaussian pyramid, which can be expressed as:

[0018] in Represents the gradient feature of the k-th pixel point (x, y) of the Gaussian pyramid. Represents the image of the pixel (x, y) at the kth level of the Gaussian pyramid, represents the horizontal gradient, represents the vertical gradient, Represents the texture features of the k-th pixel point (x, y) of the Gaussian pyramid. Represents the frequency of occurrence of gray values ​​i and j in the neighborhood window centered at the pixel point (x, y) at the kth layer of the Gaussian pyramid. Represents the square of the gray value difference, which is used to measure the contrast of gray values; The gradient features and texture features of each layer are nonlinearly combined, expressed as:

[0019] in Represents the comprehensive feature value of the k-th layer image, and generates the full-resolution comprehensive feature value by weighted fusion of the comprehensive feature values ​​of all layers; Recalculate the pixel coordinates on the plane based on the coordinates of the key point cloud data, and extract the eigenvalues ​​of the plane pixel coordinates based on the full-resolution comprehensive eigenvalues; The historical distribution of the eigenvalues ​​of key point cloud data is statistically analyzed, and the difference between the eigenvalue mean and twice the standard deviation and the sum of the eigenvalue mean and twice the standard deviation are used as the abnormal threshold range; If the characteristic value of the extracted plane pixel coordinates is greater than or equal to the maximum value of the abnormal threshold range and less than the minimum value of the abnormal threshold range, it is recorded as the abnormal point coordinates.

[0020] As a preferred solution of the image-based train operation status monitoring and identification method of the present invention, wherein: the early warning based on abnormal judgment and the generation of abnormal point report include: The point cloud coordinates and corresponding eigenvalues ​​of the abnormal point coordinates are combined into an abnormal point list, and the plane image is color-marked according to the abnormal point coordinates; And according to the sound and light alarm, the maintenance personnel are warned, and the maintenance personnel make specific judgments and records on the abnormal types; Use the table generation tool to generate anomaly reports based on the anomaly list, color-coded plane images, and the maintenance personnel's maintenance records and anomaly type records.

[0021] Another object of the present invention is to provide a system for monitoring and identifying the train running status at a station based on an image, which comprises: The image preprocessing module collects and preprocesses multi-view image data, performs two-dimensional wavelet decomposition, separates low-frequency and high-frequency components, applies soft thresholding for denoising, and generates denoised aligned image data; The multi-scale feature enhancement module constructs a Gaussian pyramid of aligned image data, generates a multi-level image representation, performs histogram equalization on each layer of the image, improves the local contrast, and merges the Gaussian pyramid images of each layer into a complete image data; Point cloud construction module, based on the complete image data, uses the SGM algorithm to calculate the disparity map, calculate the spatial coordinates of each pixel, and generate point cloud data; Key point recognition module, builds a deep learning model to classify and segment point cloud data; The comprehensive eigenvalue calculation module calculates the gradient features and texture features based on Gaussian pyramid decomposition, performs nonlinear fusion on the eigenvalues ​​of each layer, calculates the comprehensive eigenvalues, and extracts the comprehensive eigenvalues ​​corresponding to the key point cloud data; The abnormal warning module determines the abnormal threshold range based on the historical distribution of the comprehensive characteristic values, marks the abnormal points and triggers the warning; The report generation module generates a report based on the abnormal point list and the corresponding point cloud coordinates, marks the abnormal points with colors in the corresponding plane images, and integrates the maintenance personnel's records of abnormal types.

[0022] A computer device comprises: a memory and a processor; the memory stores a computer program, and the processor implements the steps of the above-mentioned image-based train operation status monitoring and identification method when executing the computer program.

[0023] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned image-based train in-station running status monitoring and identification method.

[0024] The beneficial effects of the present invention are as follows: the local details of the image are significantly improved under different lighting conditions through CLAHE adaptive contrast adjustment, so that the originally low-contrast area becomes clearer; through multi-scale feature extraction, Gaussian pyramid decomposition generates image levels of different resolutions, ensuring that the structural information of the image at different scales is retained; through the combination of SGM and RANSAC, the point cloud still has strong noise resistance in complex scenes, can adapt to environmental conditions such as lighting changes and reflection interference, and improves the adaptability of the point cloud in various scenarios; based on the mapping of comprehensive eigenvalues ​​and point cloud data, the point cloud data of key components of the train can be efficiently extracted, thereby improving the efficiency of recognizing the running status of the train. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0026] Figure 1 The figure is a flow chart of the image-based train operation status monitoring and identification method in the station.

[0027] Figure 2 This is a structural diagram of the image-based train operation status monitoring and identification system. DETAILED DESCRIPTION

[0028] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0030] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0031] Example 1, reference Figure 1 , which is the first embodiment of the present invention, and which provides an image-based train in-station running status monitoring and identification method. The image-based train in-station running status monitoring and identification method includes: S1, collect image data for two-dimensional wavelet decomposition, apply soft threshold to perform denoising and regenerate denoised aligned image data; Preferably, collecting image data for two-dimensional wavelet decomposition, applying soft thresholding for denoising and regenerating denoised aligned image data, including: Collect image data from multiple perspectives of the train through a graphics acquisition camera; Select Daubechies wavelet as the wavelet basis function, perform two-dimensional wavelet decomposition on the image data, and the obtained sub-band includes low-frequency components and high-frequency components, which represent the different frequency details of the image data, expressed as:

[0032] in and denote low-frequency and high-frequency filters respectively, Indicates the distance between the filter center and the pixel coordinate (m, n) to perform low-frequency weighting on different positions of the image. Indicates the distance between the filter center and the pixel coordinate (m, n) to perform high-frequency weighting on different positions of the image. Represents the pixel value of the image at coordinate (m,n), Represents image coordinates, L represents the low-frequency component subband, which contains the main structural information in the image and is usually used to retain the general outline of the image in wavelet decomposition, and H represents the high-frequency component subband, which contains the detail information and edges in the image and is usually used to enhance the details of the image. Represents the summation operation of all pixel coordinates (m, n), traversing each pixel of the image, and is used to calculate the low-frequency or high-frequency components of all positions in the image; Apply soft threshold to the high-frequency component sub-band for denoising to eliminate image noise, which is expressed as:

[0033] in represents the soft threshold, represents the standard deviation of the high-frequency subband, Indicates the total number of pixels in the subband; A soft threshold is applied to each high-frequency coefficient of the high-frequency component subband. The high-frequency coefficient is the image signal value obtained from the high-frequency subband, expressed as:

[0034] in Represents the high-frequency coefficient value after soft threshold denoising. represents the soft threshold, Represents the sign function, indicating the positive or negative sign of x; Based on the high-frequency coefficient value after denoising, the high-frequency component sub-band after denoising is inversely transformed, and the decomposed frequency band data are merged into the aligned image data after denoising, which is expressed as:

[0035] in Represents the pixel value of the denoised image at position (x, y); Represents the high-frequency component subband after denoising, which contains the detailed information of the image.

[0036] Through Daubechies wavelet decomposition, random noise in the image is reduced, and the edge and detail quality of the image is enhanced without destroying the main structural information. Soft threshold denoising of high-frequency coefficients can retain detail information with larger amplitudes, suppress noise signals with smaller amplitudes, and avoid edge blurring caused by traditional filtering methods (such as mean filtering). Based on the denoised high-frequency component sub-bands and low-frequency component sub-bands, the denoised image is re-synthesized through inverse wavelet transform, providing high-quality input for the projection and comprehensive analysis of point cloud data.

[0037] S2, constructing a Gaussian pyramid of aligned image data, performing equalization mapping through histogram equalization, and merging each layer of the Gaussian pyramid into complete image data; Preferably, a Gaussian pyramid of aligned image data is constructed, and equalized mapping is performed through histogram equalization, and each layer of the Gaussian pyramid is merged into complete image data, including: Construct a Gaussian pyramid of aligned image data and decompose the aligned image data into different levels of multiple scales, where each level represents the scale of the image at the kth level, expressed as:

[0038] in represents the image of the k+1th layer of the Gaussian pyramid, Represents the Gaussian weight coefficient, which is determined by the Gaussian function based on the Gaussian filter. Indicates that the pixel position of the image at the kth layer is offset. The new image coordinates are offset by 2 units in the x and y directions respectively, so that the downsampled image will be reduced to half of the original image; The contrast-limited adaptive histogram equalization (CLAHE) method is used for each layer of the Gaussian pyramid image. Each layer of the image is divided into blocks, and the grayscale distribution of each sub-block is statistically analyzed to generate a grayscale histogram. , where g represents the gray level; Based on grayscale histogram Calculate the cumulative distribution function CDF, expressed as:

[0039] in represents the cumulative pixel frequency from the minimum gray level 0 to the current gray level g, Indicates grayscale The pixel frequency, Indicates that the gray level variable used for summation starts from 0 and accumulates step by step to the current gray level g; Determine a contrast limit threshold T based on the product of the clipping threshold ratio of the history histogram and the total number of pixels of the sub-block; Based on the contrast limit threshold T, if the pixel frequency of gray level g in the gray histogram is greater than the contrast limit threshold T, then Cut to T and calculate the grayscale histogram The grayscale pixel frequency that exceeds the contrast limit threshold T is evenly distributed to all gray levels, the adjusted grayscale pixel frequency is obtained, and the cumulative distribution function CDF is recalculated; The equalization mapping is calculated based on the recalculated cumulative distribution function, expressed as:

[0040] in represents the equalized pixel value of the image block pixel (i, j), represents the cumulative distribution function of the adjusted grayscale pixel frequency, represents the minimum value of the non-zero cumulative distribution, represents the total number of gray levels, and v represents the total number of pixels in the sub-block; Map the equalized values ​​back to the corresponding pixels in the sub-block; For the overlapping areas of adjacent sub-blocks, bilinear interpolation is used to smoothly transition, and all sub-blocks are combined to generate image data processed by CLAHE; Based on each layer of the Gaussian pyramid, starting from the lowest resolution image, it is upsampled layer by layer to a higher resolution through bilinear interpolation and finally merged into the complete image data.

[0041] Through CLAHE adaptive contrast adjustment, the local details of the image are significantly improved under different lighting conditions, making the originally low-contrast areas clearer. Through multi-scale feature extraction, Gaussian pyramid decomposition generates image levels of different resolutions to ensure that the structural information of the image at different scales is retained. The contrast and feature information are balanced in both high-frequency and low-frequency areas. Multi-scale and CLAHE processing enhances the edge and detail information in the image, making the features of corresponding pixels in different views more consistent, thereby making the calculation of SGM matching cost more accurate and facilitating the subsequent use of RANSAC. It can more efficiently identify and remove erroneous matching points, and ultimately optimize the quality of the disparity map and improve the accuracy and robustness of the depth map. Through the combination of CLAHE and Gaussian pyramid, the image information of the left and right views is made more uniform and detailed, which significantly improves the efficiency and accuracy of SGM matching cost calculation, thereby supporting more accurate 3D point cloud analysis and state recognition. The local adaptive adjustment mechanism of CLAHE can improve image quality under complex conditions such as uneven lighting and reflection interference. Combined with the multi-scale representation of the Gaussian pyramid, the information balance of the image in the global and local structures is ensured. In the multi-view acquisition of the train, it adapts to the image characteristics under different shooting conditions and provides stable and high-quality input data.

[0042] S3, based on the complete image data, uses the SGM algorithm to generate a disparity map, uses the random sampling consistency algorithm to remove matching points with large errors, and calculates the spatial coordinates of each pixel to form point cloud data, and builds a deep learning model to identify key point cloud data; Preferably, based on the complete image data, the SGM algorithm is used to generate a disparity map, the random sampling consistency algorithm is used to remove matching points with large errors, and the spatial coordinates of each pixel are calculated to form point cloud data, including: According to the complete image data collected from the left and right perspectives and processed by CLAHE, the SGM algorithm is used to generate a disparity map, and the matching cost of each pixel is calculated by the absolute difference cost SAD, which is expressed as:

[0043] in represents the matching cost of the pixel at (x, y) when the disparity is d, and k represents the window size, which is used to control the matching range of the local area. represents the pixel value of the left view, The pixel value of the right view is offset by the disparity d; The matching cost is optimized by accumulating the cost, which is expressed as:

[0044] in represents the optimized matching cost of the disparity d at position (x, y), , and Respectively represent the matching cost of disparity d at the adjacent position (x-1, y), the adjacent position (x, y-1), and the adjacent position (x+1, y); Based on the disparity d with the minimum optimized matching cost, a disparity value is assigned to each pixel to generate a preliminary disparity map. The random sampling consensus algorithm RANSAC is used to collect the pixel positions of the left and right view images based on the preliminary disparity map, detect and remove matching points with large errors, and generate an optimized disparity map. Based on the disparity values ​​at different positions of the optimized disparity map, the spatial coordinates of each pixel are calculated, expressed as:

[0045]

[0046] in represents the depth of the pixel, f represents the focal length of the camera, B represents the baseline distance, Represents the disparity value of the pixel point (x, y); X and Y represent the horizontal and vertical coordinates of the spatial coordinates respectively. and Represents the horizontal and vertical coordinates of the camera's principal point coordinates; Traverse all pixels in the disparity map, calculate their corresponding spatial coordinates, and form point cloud data.

[0047] The matching cost is optimized by the SGM algorithm, combined with the cumulative cost calculation of adjacent pixels, and the local and global information of the image is fully utilized to reduce the matching error. The generated disparity map is more accurate and can better reflect the depth information of objects in the image. The erroneous matching points in the disparity map are removed, and the quality and robustness of the depth map are improved. The random sampling consistency (RANSAC) algorithm is introduced to effectively remove the erroneous matching points in the preliminary disparity map. The generated optimized disparity map has higher accuracy and robustness, which significantly improves the reliability of subsequent depth map and point cloud generation, and reduces the matching errors caused by noise and inconsistent local features. The generated point cloud data is rich in details and high in precision, and can truly restore the three-dimensional shape of key train components (such as pantographs and contact wires), providing high-quality three-dimensional data input and supporting precise analysis in subsequent train status monitoring tasks. Through the combination of SGM and RANSAC, the point cloud still has strong noise resistance in complex scenes, and can adapt to environmental conditions such as lighting changes and reflection interference, which improves the adaptability of point clouds in various scenarios and provides stable data support for multi-view monitoring of trains, reduces the interference of erroneous points on subsequent analysis (such as anomaly detection), and improves the credibility of detection results. The SGM algorithm utilizes the local correlation and global constraints of image pixels in the cost calculation and optimization process, reduces computational redundancy, and significantly improves the overall system efficiency, which can meet the real-time requirements of train operation status monitoring.

[0048] Furthermore, a deep learning model is constructed to identify key point cloud data, including: Build a PointNet++ deep learning model, including input layer, local feature extraction layer, feature propagation layer, hierarchical feature aggregation layer, and output layer; The input layer inputs point cloud data, the local feature extraction layer uses a distance-based grouping mechanism to select points in the local area, the hierarchical feature aggregation layer aggregates the extracted local features layer by layer, the feature propagation layer uses a feature propagation mechanism to pass high-level feature information back to the low-level point cloud data, and the output layer outputs the prediction results based on the segmentation label; Use the labeled point cloud samples as training data, select the cross entropy loss function to calculate the difference between the category probability predicted by the model and the actual label, and use the Adam optimizer for optimization. If the calculated loss no longer decreases significantly during the continuous iteration process, stop iterating and output the model. The newly calculated point cloud data is input into the model, and the label probabilities of being identified as different key train components and background are output. The label of the point cloud data with the largest probability is selected, and the point cloud data of the key train components is selected as the key point cloud data.

[0049] The model can accurately identify key train components (such as pantographs, overhead wires, etc.) and background points, thereby generating point-by-point segmentation labels. The segmentation results can greatly reduce redundant calculations of non-critical points, providing high-quality input data for subsequent status monitoring and anomaly detection. The local feature extraction layer uses a distance-based grouping mechanism to dynamically select points in the local area, which can adapt to the sparsity and irregularity of point cloud data. The feature propagation layer propagates high-level features to low-level features, ensuring information flow and feature fusion between different levels of the point cloud. The PointNet++ model takes multi-scale characteristics into account in the process of hierarchical feature aggregation, and can adapt to point cloud data of different resolutions. It can significantly improve the accuracy, efficiency and applicability of the system, and provide comprehensive data support and technical guarantees for train operation safety.

[0050] S4, calculates comprehensive feature values ​​based on the determined gradient features and texture features of Gaussian pyramid decomposition, makes abnormal judgments on the comprehensive feature values ​​corresponding to the key point cloud data, issues warnings based on the abnormal judgments, and generates abnormal point reports; Preferably, the comprehensive feature value is calculated based on the determined gradient feature and texture feature of the Gaussian pyramid decomposition, and the comprehensive feature value corresponding to the key point cloud data is used for abnormal judgment, including: The gradient features and texture features are determined based on the image data decomposed by Gaussian pyramid, which can be expressed as:

[0051] in Represents the gradient feature of the k-th pixel point (x, y) of the Gaussian pyramid. Represents the image of the pixel (x, y) at the kth level of the Gaussian pyramid, represents the horizontal gradient, represents the vertical gradient, Represents the texture features of the k-th pixel point (x, y) of the Gaussian pyramid. Represents the frequency of occurrence of gray values ​​i and j in the neighborhood window centered at the pixel point (x, y) at the kth layer of the Gaussian pyramid. Represents the square of the gray value difference, which is used to measure the contrast of gray values; The gradient features and texture features of each layer are nonlinearly combined, expressed as:

[0052] in Represents the comprehensive feature value of the k-th layer image, and generates the full-resolution comprehensive feature value by weighted fusion of the comprehensive feature values ​​of all layers; Recalculate the pixel coordinates on the plane based on the coordinates of the key point cloud data, and extract the eigenvalues ​​of the plane pixel coordinates based on the full-resolution comprehensive eigenvalues; The historical distribution of the eigenvalues ​​of key point cloud data is statistically analyzed, and the difference between the eigenvalue mean and twice the standard deviation and the sum of the eigenvalue mean and twice the standard deviation are used as the abnormal threshold range; If the characteristic value of the extracted plane pixel coordinates is greater than or equal to the maximum value of the abnormal threshold range and less than the minimum value of the abnormal threshold range, it is recorded as the abnormal point coordinates.

[0053] By combining gradient features and texture features, the edge, details and texture information of key train components can be accurately captured, and the multi-dimensional expressiveness of image features is enhanced. The extraction of gradient and texture features enhances the recognition ability of details and edge features of local areas of the image (such as pantographs and contact wires), and improves the precise monitoring of key train components. By calculating the full-resolution comprehensive eigenvalues ​​and dynamically setting the abnormal threshold range, the abnormal points in the image can be accurately identified and located, reducing false alarms and missed alarms. The multi-level feature extraction of the Gaussian pyramid ensures that image information at different scales can be fully retained, adapting to the changes of train images under different resolutions and lighting conditions, and improving the robustness of the system. Based on the mapping of comprehensive eigenvalues ​​and point cloud data, the point cloud data of key train components can be efficiently extracted, providing accurate data support for subsequent status monitoring and anomaly detection. The CLAHE method and multi-scale processing improve the visibility of images under different lighting conditions, can effectively enhance the local details of the image under complex lighting environments, provide efficient and reliable key component status feedback, and help improve the safety of train operation.

[0054] Furthermore, based on abnormal judgment, early warning is carried out and abnormal point reports are generated, including: The point cloud coordinates and corresponding eigenvalues ​​of the abnormal point coordinates are combined into an abnormal point list, and the plane image is color-marked according to the abnormal point coordinates; And according to the sound and light alarm, the maintenance personnel are warned, and the maintenance personnel make specific judgments and records on the abnormal types; Use the table generation tool to generate anomaly reports based on the anomaly list, color-coded plane images, and the maintenance personnel's maintenance records and anomaly type records.

[0055] By combining the coordinates of the abnormal points with the corresponding eigenvalues ​​to form an abnormal point list, the position of the abnormal points can be accurately located, and the abnormal area can be intuitively displayed in the plane image through color marking. The color marking enables maintenance personnel to clearly identify the abnormal points on the image, which improves the intuitiveness and real-time nature of abnormality detection. The marking based on the coordinates and eigenvalues ​​of the abnormal points, combined with the sound and light alarm system, provides maintenance personnel with instant warnings, allowing the system to respond quickly when an abnormality is found. The sound and light alarm helps to quickly attract the attention of maintenance personnel, shorten the response time, and ensure that potential problems in train operation are discovered and resolved in a timely manner. The abnormal point report not only records the location of the abnormal point, but also includes detailed information on the eigenvalues ​​and abnormality types. Maintenance personnel can make more accurate judgments and processing based on these data. The abnormal point report is automatically generated through the table generation tool, combining the abnormal point information, plane images and maintenance records to provide detailed and easy-to-trace reports for management and recording.

[0056] Example 2, reference Figure 2 , which is the second embodiment of the present invention, and which is different from the previous embodiment, provides a system for monitoring and identifying the train running status in a station based on an image, comprising: The image preprocessing module collects and preprocesses multi-view image data, performs two-dimensional wavelet decomposition, separates low-frequency and high-frequency components, applies soft thresholding for denoising, and generates denoised aligned image data; The multi-scale feature enhancement module constructs a Gaussian pyramid of aligned image data, generates a multi-level image representation, performs histogram equalization on each layer of the image, improves the local contrast, and merges the Gaussian pyramid images of each layer into a complete image data; Point cloud construction module, based on the complete image data, uses the SGM algorithm to calculate the disparity map, calculate the spatial coordinates of each pixel, and generate point cloud data; Key point recognition module, builds a deep learning model to classify and segment point cloud data; The comprehensive eigenvalue calculation module calculates the gradient features and texture features based on Gaussian pyramid decomposition, performs nonlinear fusion on the eigenvalues ​​of each layer, calculates the comprehensive eigenvalues, and extracts the comprehensive eigenvalues ​​corresponding to the key point cloud data; The abnormal warning module determines the abnormal threshold range based on the historical distribution of the comprehensive characteristic values, marks the abnormal points and triggers the warning; The report generation module generates a report based on the abnormal point list and the corresponding point cloud coordinates, marks the abnormal points with colors in the corresponding plane images, and integrates the maintenance personnel's records of abnormal types.

[0057] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0058] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0059] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0060] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit with a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit with a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0061] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An image-based train operation status monitoring and identification method, characterized in that: include, Collect image data for two-dimensional wavelet decomposition, apply soft thresholding for denoising and regenerate aligned image data after denoising; Construct a Gaussian pyramid of aligned image data, perform equalization mapping through histogram equalization, and merge each layer of the Gaussian pyramid into complete image data; Based on the complete image data, the SGM algorithm is used to generate a disparity map, the random sampling consistency algorithm is used to remove matching points with large errors, and the spatial coordinates of each pixel are calculated to form point cloud data. A deep learning model is built to identify key point cloud data. Based on the determination of gradient features and texture features based on Gaussian pyramid decomposition, the comprehensive feature values ​​corresponding to the key point cloud data are used for abnormal judgment, and an early warning is issued based on the abnormal judgment and an abnormal point report is generated; The construction of a deep learning model to identify key point cloud data includes: Build a PointNet++ deep learning model, including input layer, local feature extraction layer, feature propagation layer, hierarchical feature aggregation layer, and output layer; The input layer inputs point cloud data, the local feature extraction layer uses a distance-based grouping mechanism to select points in the local area, the hierarchical feature aggregation layer aggregates the extracted local features layer by layer, the feature propagation layer uses a feature propagation mechanism to pass high-level feature information back to the low-level point cloud data, and the output layer outputs the prediction results based on the segmentation label; Use the labeled point cloud samples as training data, select the cross entropy loss function to calculate the difference between the category probability predicted by the model and the actual label, and use the Adam optimizer for optimization. If the calculated loss no longer decreases significantly during the continuous iteration process, stop iterating and output the model. Input the newly calculated point cloud data into the model, and output the label probabilities of being identified as different key train components and backgrounds. Select the label of the point cloud data with the largest probability, and select the point cloud data of the key train components as the key point cloud data. The method of calculating the comprehensive feature value by determining the gradient feature and the texture feature based on the Gaussian pyramid decomposition and making an abnormal judgment on the comprehensive feature value corresponding to the key point cloud data includes: The gradient features and texture features are determined based on the image data decomposed by Gaussian pyramid, which can be expressed as: ； ； in Represents the gradient feature of the k-th pixel point (x, y) of the Gaussian pyramid. Represents the image of the pixel (x, y) at the kth level of the Gaussian pyramid, represents the horizontal gradient, represents the vertical gradient, Represents the texture features of the k-th pixel point (x, y) of the Gaussian pyramid. Represents the frequency of occurrence of gray values ​​i and j in the neighborhood window centered at the pixel point (x, y) at the kth layer of the Gaussian pyramid. Represents the square of the gray value difference, which is used to measure the contrast of gray values; The gradient features and texture features of each layer are nonlinearly combined, expressed as: ； in Represents the comprehensive feature value of the k-th layer image, and generates the full-resolution comprehensive feature value by weighted fusion of the comprehensive feature values ​​of all layers; Recalculate the pixel coordinates on the plane based on the coordinates of the key point cloud data, and extract the eigenvalues ​​of the plane pixel coordinates based on the full-resolution comprehensive eigenvalues; The historical distribution of the eigenvalues ​​of key point cloud data is statistically analyzed, and the difference between the eigenvalue mean and twice the standard deviation and the sum of the eigenvalue mean and twice the standard deviation are used as the abnormal threshold range; If the characteristic value of the extracted plane pixel coordinates is greater than or equal to the maximum value of the abnormal threshold range and less than the minimum value of the abnormal threshold range, it is recorded as the abnormal point coordinates.

2. The image-based train operation status monitoring and identification method according to claim 1, characterized in that: The collected image data is subjected to two-dimensional wavelet decomposition, a soft threshold is applied to perform denoising processing and the denoised aligned image data is regenerated, including: Collect image data from multiple perspectives of the train through a graphics acquisition camera; Select Daubechies wavelet as the wavelet basis function, perform two-dimensional wavelet decomposition on the image data, and the obtained sub-bands include low-frequency components and high-frequency components, representing different frequency details of the image data; Apply soft threshold to the high-frequency component sub-band for denoising to eliminate image noise, which is expressed as: in represents the soft threshold, represents the standard deviation of the high-frequency subband, Indicates the total number of pixels in the subband; Apply a soft threshold to each high-frequency coefficient of the high-frequency component subband, expressed as: in Represents the high-frequency coefficient value after soft threshold denoising. represents the soft threshold, Represents the sign function, indicating the positive or negative sign of x; Based on the high-frequency coefficient value after denoising, the high-frequency component subband after denoising is inversely transformed, and the decomposed frequency band data are merged into the aligned image data after denoising.

3. The image-based train operation status monitoring and identification method according to claim 2, characterized in that: The Gaussian pyramid of the aligned image data is constructed, and the equalization mapping is performed through histogram equalization, and each layer of the Gaussian pyramid is merged into complete image data. include, Construct a Gaussian pyramid of aligned image data and decompose the aligned image data into different levels of multiple scales, where each level represents the scale of the image at the kth level, expressed as: in represents the image of the k+1th layer of the Gaussian pyramid, represents the Gaussian weight coefficient; The contrast-limited adaptive histogram equalization (CLAHE) method is used for each layer of the Gaussian pyramid image. Each layer of the image is divided into blocks, and the grayscale distribution of each sub-block is statistically analyzed to generate a grayscale histogram. , where g represents the gray level; Based on grayscale histogram Calculate the cumulative distribution function CDF, expressed as: in represents the cumulative pixel frequency from the minimum gray level 0 to the current gray level g, Indicates grayscale The pixel frequency, Indicates that the gray level variable used for summation starts from 0 and accumulates step by step to the current gray level g; Determine a contrast limit threshold T based on the product of the clipping threshold ratio of the history histogram and the total number of pixels of the sub-block; Based on the contrast limit threshold T, if the pixel frequency of gray level g in the gray histogram is greater than the contrast limit threshold T, then Cut to T and calculate the grayscale histogram The grayscale pixel frequency that exceeds the contrast limit threshold T is evenly distributed to all gray levels, the adjusted grayscale pixel frequency is obtained, and the cumulative distribution function CDF is recalculated; The equalization mapping is calculated based on the recalculated cumulative distribution function, expressed as: in represents the equalized pixel value of the image block pixel (i, j), represents the cumulative distribution function of the adjusted grayscale pixel frequency, represents the minimum value of the non-zero cumulative distribution, represents the total number of gray levels, and v represents the total number of pixels in the sub-block; Map the equalized values ​​back to the corresponding pixels in the sub-block; For the overlapping areas of adjacent sub-blocks, bilinear interpolation is used to smoothly transition, and all sub-blocks are combined to generate image data processed by CLAHE; Based on each layer of the Gaussian pyramid, starting from the lowest resolution image, it is upsampled layer by layer to a higher resolution through bilinear interpolation and finally merged into the complete image data.

4. The image-based train operation status monitoring and identification method according to claim 3, characterized in that: Based on the complete image data, the disparity map is generated using the SGM algorithm, the matching points with large errors are removed using the random sampling consistency algorithm, and the spatial coordinates of each pixel are calculated to form point cloud data, including: According to the complete image data collected from the left and right perspectives and processed by CLAHE, the SGM algorithm is used to generate a disparity map, and the matching cost of each pixel is calculated by the absolute difference cost SAD, which is expressed as: in represents the matching cost of the pixel at (x, y) when the disparity is d, and k represents the window size, which is used to control the matching range of the local area. represents the pixel value of the left view, The pixel value of the right view is offset by the disparity d; Optimize matching cost by accumulating cost; Based on the disparity d with the minimum optimized matching cost, a disparity value is assigned to each pixel to generate a preliminary disparity map. The random sampling consensus algorithm RANSAC is used to collect the pixel positions of the left and right view images based on the preliminary disparity map, detect and remove matching points with large errors, and generate an optimized disparity map. Based on the disparity values ​​at different positions of the optimized disparity map, the spatial coordinates of each pixel are calculated, expressed as: in represents the depth of the pixel, f represents the focal length of the camera, B represents the baseline distance, Represents the disparity value of the pixel point (x, y); X and Y represent the horizontal and vertical coordinates of the spatial coordinates respectively. and Represents the horizontal and vertical coordinates of the camera's principal point coordinates; Traverse all pixels in the disparity map, calculate their corresponding spatial coordinates, and form point cloud data.

5. The image-based train operation status monitoring and identification method according to claim 4, characterized in that: The method of providing an early warning based on abnormality judgment and generating an abnormality point report includes: The point cloud coordinates and corresponding eigenvalues ​​of the abnormal point coordinates are combined into an abnormal point list, and the plane image is color-marked according to the abnormal point coordinates; And according to the sound and light alarm, the maintenance personnel are warned, and the maintenance personnel make specific judgments and records on the abnormal types; Use the table generation tool to generate anomaly reports based on the anomaly list, color-coded plane images, and the maintenance personnel's maintenance records and anomaly type records.

6. A system based on the image-based train operation status monitoring and identification method according to any one of claims 1 to 5, characterized in that: include, The image preprocessing module collects and preprocesses multi-view image data, performs two-dimensional wavelet decomposition, separates low-frequency and high-frequency components, applies soft thresholding for denoising, and generates denoised aligned image data; The multi-scale feature enhancement module constructs a Gaussian pyramid of aligned image data, generates a multi-level image representation, performs histogram equalization on each layer of the image, improves the local contrast, and merges the Gaussian pyramid images of each layer into a complete image data; Point cloud construction module, based on the complete image data, uses the SGM algorithm to calculate the disparity map, calculate the spatial coordinates of each pixel, and generate point cloud data; Key point recognition module, builds a deep learning model to classify and segment point cloud data; The comprehensive eigenvalue calculation module calculates the gradient features and texture features based on Gaussian pyramid decomposition, performs nonlinear fusion on the eigenvalues ​​of each layer, calculates the comprehensive eigenvalues, and extracts the comprehensive eigenvalues ​​corresponding to the key point cloud data; The abnormal warning module determines the abnormal threshold range based on the historical distribution of the comprehensive characteristic values, marks the abnormal points and triggers the warning; The report generation module generates a report based on the abnormal point list and the corresponding point cloud coordinates, marks the abnormal points with colors in the corresponding plane images, and integrates the maintenance personnel's records of abnormal types.

7. A computer device comprising: Memory and processor; The memory stores a computer program, characterized in that: when the processor executes the computer program, the steps of the image-based train operation status monitoring and identification method in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image-based train operation status monitoring and identification method according to any one of claims 1 to 5 are implemented.

Images