Track foreign body detection method and system

Abstract

The present invention discloses a method and system for detecting foreign objects on a track, wherein the method comprises: obtaining a multispectral image under a track environment, wherein the multispectral image comprises a visible light image and an infrared light image. Acquiring the acceleration value and the angular velocity value during the operation of a train, and judging the motion state of the train according to the acceleration value and the angular velocity value. According to the motion state of the train, performing motion compensation and deblurring processing on the multispectral image to generate a motion-compensated multispectral image. Performing adaptive image enhancement processing on the motion-compensated multispectral image to generate an enhanced multispectral image. Extracting the geometric features and material features of the track area from the enhanced multispectral image. Thereby, the present invention significantly improves the accuracy and real-time performance of foreign object detection on the track, and provides strong support for ensuring the safe operation of rail transit.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of rail transit detection technology, and in particular to a method and system for detecting foreign objects on a track. Background Art

[0002] Foreign object detection in rail transit systems faces complex and ever-changing environmental challenges. High-speed trains cause the quality of collected images to deteriorate, resulting in severe blurring and distortion, making it difficult to accurately identify foreign objects on the track. At the same time, there are many types of potential foreign objects in the track environment, such as gravel, metal fragments, plastics, etc., which vary in shape, size and material, increasing the difficulty of detection.

[0003] Traditional foreign body detection methods are often designed for specific types and are difficult to adapt to a variety of foreign body situations. In addition, rail transit has extremely high requirements for safety and real-time performance. The detection system needs to respond quickly under the condition of high-speed train operation to detect potential dangers in a timely manner.

[0004] However, complex background environments, lighting changes, weather factors, etc. can interfere with detection results and easily lead to missed detections and false detections. How to overcome problems such as poor image quality, a wide variety of foreign objects, and severe environmental interference in such dynamic and complex scenes to achieve accurate detection and positioning of foreign objects on the track is a major technical challenge for ensuring the safe operation of rail transit. Summary of the invention

[0005] The embodiments of the present invention provide a method and system for detecting foreign objects on a track, which significantly improve the accuracy and real-time performance of detecting foreign objects on a track, and provide strong support for ensuring the safe operation of rail transit.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for detecting foreign matter on a track, comprising:

[0007] Acquire a multispectral image in a track environment, wherein the multispectral image includes a visible light image and an infrared light image;

[0008] Acquire the acceleration value and the angular velocity value during the running of the train, and judge the motion state of the train according to the acceleration value and the angular velocity value;

[0009] According to the motion state of the train, the multispectral image is subjected to motion compensation and deblurring processing to generate a motion-compensated multispectral image;

[0010] Performing adaptive image enhancement processing on the motion compensated multispectral image to generate an enhanced multispectral image;

[0011] extracting geometric features and material features of the track area from the enhanced multispectral image;

[0012] Inputting the geometric features of the track area and the material features into a pre-built foreign body feature database to obtain feature information of candidate foreign bodies;

[0013] The enhanced multispectral image and the feature information of the candidate foreign object are input into the foreign object recognition model to determine whether there is a foreign object in the multispectral image. If there is a foreign object, the category and location information of the foreign object are output.

[0014] In one embodiment of the present invention, the step of acquiring a multispectral image in a track environment, wherein the multispectral image includes a visible light image and an infrared light image, further includes: determining a fusion ratio of the visible light image and the infrared light image according to the lighting conditions of the track environment.

[0015] In one embodiment of the present invention, performing motion compensation and deblurring on the multispectral image according to the motion state of the train to generate the motion compensated multispectral image includes:

[0016] Obtaining the speed, acceleration and track curvature radius of the train;

[0017] Establishing a train motion model according to the speed, the acceleration and the track curvature radius;

[0018] The train motion model is used to perform motion compensation on the multispectral image to generate the motion compensated multispectral image.

[0019] In one embodiment of the present invention, the step of performing adaptive image enhancement processing on the motion compensated multispectral image to generate an enhanced multispectral image comprises:

[0020] Get the ambient light intensity;

[0021] According to the ambient light intensity, the contrast, brightness and saturation of the motion compensated multispectral image are adaptively adjusted to generate the enhanced multispectral image.

[0022] In one embodiment of the present invention, extracting geometric features and material features of the track area from the enhanced multispectral image includes:

[0023] Segmenting the track area from the enhanced multispectral image using a semantic segmentation algorithm;

[0024] Extracting geometric features of the track area, wherein the geometric features include edges and textures;

[0025] The material features of the track area are extracted, wherein the material features include color and glossiness.

[0026] In a second aspect, the present invention provides a track foreign body detection system, comprising: a first acquisition module, a first judgment module, a first generation module, a second generation module, an extraction module, a second acquisition module and a second judgment module. The first acquisition module is used to acquire a multispectral image under a track environment, and the multispectral image includes a visible light image and an infrared light image. The first judgment module is used to acquire the acceleration value and the angular velocity value during the operation of the train, and judge the motion state of the train according to the acceleration value and the angular velocity value. The first generation module is used to perform motion compensation and deblurring processing on the multispectral image according to the motion state of the train, and generate a motion-compensated multispectral image. The second generation module is used to perform adaptive image enhancement processing on the motion-compensated multispectral image to generate an enhanced multispectral image. The extraction module is used to extract the geometric features and material features of the track area from the enhanced multispectral image. The second acquisition module is used to input the geometric features and the material features of the track area into a pre-constructed foreign body feature database to obtain feature information of candidate foreign bodies. The second judgment module is used to input the enhanced multispectral image and the feature information of the candidate foreign object into the foreign object recognition model to determine whether there is a foreign object in the multispectral image, wherein if a foreign object exists, the category and location information of the foreign object is output.

[0027] In one embodiment of the present invention, the track foreign object detection system further includes a determination module for determining a fusion ratio of the visible light image and the infrared light image according to the lighting conditions of the track environment.

[0028] In one embodiment of the present invention, the first generation module includes: a first acquisition unit, an establishment unit, and a first generation unit. The first acquisition unit is used to acquire the speed, acceleration, and track curvature radius of the train. The establishment unit is used to establish a train motion model based on the speed, acceleration, and track curvature radius. The first generation unit is used to use the train motion model to perform motion compensation on the multispectral image to generate the motion compensated multispectral image.

[0029] In a third aspect, the present invention provides an electronic device, comprising:

[0030] at least one processor; and

[0031] a memory communicatively coupled to the at least one processor;

[0032] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the track foreign object detection method as described above.

[0033] In a fourth aspect, the present invention provides a computer-readable storage medium, comprising a computer program and instructions, which, when the computer program or the instructions are executed on a computer, enables the computer to execute the track foreign object detection method as described above.

[0034] Compared with the prior art, the track foreign object detection method and system of the present invention obtains multispectral images and train motion parameters, performs motion compensation and deblurring processing on the images, and then performs adaptive enhancement, thereby improving image quality. Then, a deep learning algorithm is used to extract track area features, combined with a pre-built foreign object feature database, and a foreign object recognition model trained by transfer learning is used to achieve accurate detection and positioning of track foreign objects. The present invention solves the problem of image quality degradation caused by complex track environment and high-speed train operation, and overcomes the limitation that traditional methods are difficult to accurately identify a variety of foreign objects. Through the integrated application of multiple advanced algorithms, the present invention significantly improves the accuracy and real-time performance of track foreign object detection, providing strong support for ensuring the safe operation of rail transit. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic flow chart of a track foreign body detection method in Embodiment 1 of the present invention;

[0036] Figure 2 is a structural schematic diagram of a track foreign body detection system in Embodiment 2 of the present invention;

[0037] Figure 3 It is a structural schematic diagram of an electronic device in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0038] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present invention, rather than to limit the embodiments of the present invention. It is also necessary to explain that, for ease of description, only parts related to the embodiments of the present invention are shown in the accompanying drawings, rather than all structures.

[0039] Embodiment 1

[0040] Figure 1 FIG. 1 is a flow chart of a method for detecting foreign matter on a track in Embodiment 1 of the present invention. Figure 1 As shown, embodiment 1 provides a track foreign body detection method, including:

[0041] Step S100, acquiring a multispectral image in a track environment, wherein the multispectral image includes a visible light image and an infrared light image;

[0042] Specifically, a multispectral imaging device is used to obtain the original image data in the track environment, and the original image data includes image information of the visible light band and the infrared band. According to the parameter setting of the multispectral imaging device, the band range of the visible light image and the infrared light image is determined, and the image information of the corresponding band is extracted from the original image data. Through image preprocessing technology, the extracted visible light image and infrared light image are denoised and corrected to eliminate image noise and distortion, and obtain high-quality multispectral images. According to the characteristics of the track environment, an adaptive image segmentation algorithm is used to divide the multispectral image into different regions, each of which corresponds to a different target or background in the track environment. Feature extraction is performed on the segmented multispectral image region to obtain feature information such as texture, shape, color, etc. of each region to form a regional feature vector. Pattern recognition technology is used to determine the target category of each region, such as track, vehicle, obstacle, etc., according to the regional feature vector, and the semantic information of the multispectral image is obtained. Combined with the prior knowledge of the track environment, the semantic information is optimized and corrected to eliminate errors and uncertainties in the recognition process and improve the accuracy of multispectral image understanding. According to application requirements, image fusion technology is used to fuse visible light images and infrared images to generate high-quality, information-rich multispectral fusion images. The acquired multispectral images, semantic information and fused images are stored and managed to build a multispectral image database in the orbital environment to provide data support for subsequent analysis and application.

[0043] In a specific embodiment, a hyperspectral camera with a wavelength range of 400-1000nm is used to obtain track environment images, and the spatial resolution reaches 0.1m. By setting the visible light band to 400-760nm and the infrared band to 760-1000nm, images of the two bands are extracted from the original image data. The median filtering algorithm is used to denoise the image and eliminate the salt and pepper noise, and then the affine transformation is used to correct the geometric distortion of the image. After correction, the pixel deviation is less than 0.5 pixels. Based on the adaptive segmentation algorithm of image gradient and regional growth, the image is divided into 12 regions according to the grayscale differences of targets such as ballast, sleepers, and rails in the track environment. Grayscale co-occurrence matrix features and Gabor texture features are extracted for each region to generate a regional feature vector containing 24-dimensional features. The support vector machine classifier is used to train and classify regional features, and the average recognition accuracy reaches more than 95%. The preliminary recognition results are post-processed and optimized in combination with the spatial topological relationship of the track environment to eliminate isolated areas and misclassified areas. The Laplacian pyramid fusion algorithm is used to fuse visible light and infrared images, and the information entropy of the fused image is increased by 1.5 bits / pixel. Finally, the acquired multispectral images, regional segmentation results, target recognition results and other information are stored in the MongoDB database according to the time and space position relationship, providing data support for track environment monitoring and analysis.

[0044] Step S200, obtaining the acceleration value and the angular velocity value during the running of the train, and judging the motion state of the train according to the acceleration value and the angular velocity value;

[0045] Specifically, the acceleration value and angular velocity value during the operation of the train are obtained, and the acceleration and angular velocity information during the operation of the train is collected in real time using an inertial measurement unit (IMU) sensor. The collected acceleration and angular velocity information is preprocessed, and the acceleration and angular velocity information is filtered by the Kalman filter algorithm to remove high-frequency noise interference. According to the filtered acceleration and angular velocity information, the kinematic model is used to calculate the train's instantaneous velocity, displacement, attitude angle and other motion parameters. According to the calculated train motion parameters, the motion state of the train is judged. If the train is in a uniform motion state, the acceleration value and angular velocity value are both zero. If the train is in an accelerated motion state, the acceleration value is not zero and the angular velocity value is zero; if the train is in a decelerated motion state, the acceleration value is a negative value and the angular velocity value is zero. If the train is in a turning motion state, the acceleration value is not zero and the angular velocity value is not zero. The turning direction and turning radius of the train are judged by the signs and magnitudes of the acceleration value and angular velocity value. According to the motion state of the train, the collected multispectral images are motion compensated and deblurred, and the images are motion compensated using a motion compensation algorithm based on motion vector estimation. The geometric features and material features of the track area are extracted from the motion-compensated multispectral images through a semantic segmentation algorithm based on deep learning, and the semantic segmentation results of the track area are obtained. The semantic segmentation results of the track area are combined with the motion state of the train to perform three-dimensional reconstruction of the track area and obtain a three-dimensional point cloud model of the track area, providing data support for train operation status monitoring and track defect detection.

[0046] In a specific embodiment, a 6-DOF IMU sensor consisting of a three-axis accelerometer and a three-axis gyroscope is used to collect acceleration and angular velocity information during the operation of the train at a frequency of 100 Hz. The collected acceleration and angular velocity information is preprocessed, and the data is filtered using a second-order Butterworth low-pass filter with a bandwidth of 10 Hz. Based on the filtered acceleration and angular velocity information, the Newton kinematic model is used to obtain the train's instantaneous velocity, displacement, attitude angle and other motion parameters through integral calculation. When the train runs at a uniform speed, the mean and variance of the acceleration and angular velocity values ​​are less than 0.01 m / s 2 and 0.01rad / s. When the train is moving at 0.2m / s 2 When the acceleration is uniformly accelerated, the average acceleration value is 0.2m / s 2 , variance is less than 0.01m / s 2, the mean and variance of the angular velocity are both less than 0.01 rad / s. When the train turns on a circular track with a radius of 1000m, the mean acceleration is 0.5m / s 2 , the average angular velocity value is 0.01rad / s. According to the motion state of the train, the motion compensation algorithm based on motion vector estimation is used to align the images, and the registration error is less than 1 pixel. Then, the track area is extracted from the motion-compensated multispectral image with a pixel accuracy of 95% through the trained FCN semantic segmentation model. Finally, the ICP algorithm is used to align the point cloud of the track area with the standard track model, and the registration error is less than 5mm, generating a 3D point cloud model of the track area.

[0047] Step S300, performing motion compensation and deblurring processing on the multispectral image according to the motion state of the train to generate a motion compensated multispectral image;

[0048] Specifically, a multispectral image under the track environment is obtained, including a visible light image and an infrared light image, and the motion parameters of the image are determined according to the motion state information of the train. According to the motion state of the train, a motion compensation algorithm is used to perform motion compensation on the multispectral image. By estimating the motion vector of the image, the image is geometrically transformed and pixel resampled to eliminate the image blur caused by motion. The multispectral image after motion compensation is deblurred, and an image deblurring algorithm such as Wiener filtering and Lucy-Richardson algorithm is used to restore the clarity of the image to obtain a multispectral image after motion compensation deblurring. The multispectral image after motion compensation deblurring is adaptively enhanced. By analyzing the histogram distribution of the image, the contrast, brightness and saturation of the image are dynamically adjusted to improve the visual quality of the image. According to the lighting conditions and imaging characteristics of the track environment, an image enhancement algorithm such as histogram equalization and Retinex algorithm is adaptively selected to enhance the details and texture information of the image in a targeted manner. The enhanced multispectral image is subjected to image fusion processing, and the high resolution of the visible light image and the thermal radiation information of the infrared light image are comprehensively utilized to generate a fused high-quality image. Image fusion algorithms based on multi-scale decomposition and sparse representation, such as wavelet transform and multi-scale geometric analysis, are used to extract multi-scale features of images and achieve lossless image fusion. Post-processing of the fused images, such as edge enhancement and noise suppression, is performed to further improve the image clarity and signal-to-noise ratio, and obtain the final high-quality multispectral image. The processed high-quality multispectral image is transmitted to the subsequent target detection and recognition module to provide high-quality image data support for target detection and status monitoring in the track environment.

[0049] In a specific embodiment, after obtaining a multispectral image including a visible light image and an infrared light image, according to the train speed of 80 km / h and the acceleration of 0.5 m / s2 The motion parameters of the image are determined by using the motion estimation algorithm based on block matching. The motion vector of the image is estimated by searching for the best matching position between image blocks, and the image is resampled by affine transformation and bilinear interpolation to eliminate the image blur caused by motion. The Lucy-Richardson iterative algorithm is used for the motion compensated image, and the number of iterations is set to 5 and the standard deviation of the point spread function is 1.5 to restore the clarity of the image. According to the image histogram distribution, the image contrast is adjusted by gamma transformation, and the gamma value is set to 0.8; the brightness is adjusted by linear transformation, and the offset is 20; the saturation is increased by the saturation adjustment algorithm, and the gain coefficient is 1.2. According to the dim lighting conditions of the orbital environment, the Retinex algorithm is selected to adaptively enhance the image, and the standard deviation of the Gaussian kernel is set to 15 and the dynamic range compression factor is 0.5. The image fusion algorithm based on non-subsampled Contourlet transform is used to perform a two-level decomposition of the multispectral image. The low-frequency subband is fused by the weighted average method, and the high-frequency subband is fused by the local energy maximum rule. The fusion weight is determined by local variance adaptively. The fused image is subjected to unsharp masking sharpening processing, with a Gaussian template size of 5×5 and a sharpening coefficient of 0.8; the wavelet threshold denoising method is adopted, the sym4 wavelet basis is selected, the number of decomposition layers is 3, and the threshold is adaptively determined by the BayesShrink criterion. Finally, the processed high-quality multispectral image is transmitted to the target detection module as input data for subsequent analysis.

[0050] Step S400, performing adaptive image enhancement processing on the motion compensated multispectral image to generate an enhanced multispectral image;

[0051] Specifically, according to the motion state of the train, motion compensation is performed on the multispectral image to eliminate the image blur caused by the train motion, and the motion compensated multispectral image is obtained. The motion compensated multispectral image is deblurred, and an adaptive filtering algorithm is used to remove the residual blur to obtain a clear multispectral image. The contrast, brightness and saturation information of the motion compensated deblurred multispectral image is obtained, and the contrast, brightness and saturation levels of the current image are determined by analyzing the histogram distribution of the image. According to the obtained contrast, brightness and saturation information, an adaptive enhancement algorithm is used to dynamically adjust the contrast, brightness and saturation of the multispectral image to make the image details clearer and the colors more vivid.

[0052] In a specific embodiment, according to the motion state of the train, a motion estimation algorithm based on feature point matching is used to perform motion compensation processing on the multispectral image. By calculating the displacement of feature points between adjacent frames, the motion parameters of the train are estimated, and the image is motion compensated to eliminate the image blur caused by the train moving at a speed of 80 km / h. Then, the multispectral image after motion compensation is deblurred, and an adaptive Wiener filtering algorithm is used to dynamically adjust the filter parameters according to the local variance of the image to remove the residual blur and improve the clarity of the image by 30%. Then, by analyzing the RGB three-channel histogram distribution of the multispectral image, the image contrast is obtained to be 0.6, the brightness is 120, and the saturation is 70%. According to these parameters, an adaptive histogram equalization algorithm is used to dynamically adjust the contrast, brightness and saturation of the image, so that the image details are clearer, the colors are more vivid, the contrast is increased by 20%, the brightness is increased by 10%, and the saturation is increased by 15%.

[0053] Step S500, extracting geometric features and material features of the track area from the enhanced multispectral image;

[0054] Specifically, a multispectral imaging technique is used to obtain a multispectral image of the track area, and the multispectral image is enhanced by an image preprocessing method to improve the image quality. According to the prior knowledge of the track area, the geometric features and material features of the track area are extracted from the enhanced multispectral image. The extracted geometric features and material features of the track area are used as prior knowledge and input into a pre-built foreign object feature database for matching to obtain the feature information of candidate foreign objects in the foreign object feature database. A semantic segmentation algorithm based on deep learning is used to segment the track area from the enhanced multispectral image to obtain accurate geometric features and material features of the track area. The enhanced multispectral image and the feature information of the candidate foreign objects are input into the pre-trained foreign object recognition model, and the model is fine-tuned by the transfer learning method to improve the accuracy of foreign object recognition. The foreign object recognition model determines whether there is a foreign object in the multispectral image. If there is a foreign object, the category and location information of the foreign object are output. According to the category and location information of the foreign object, the target detection algorithm is used to accurately locate the foreign object and obtain the accurate bounding box of the foreign object. The impact of foreign objects on track operation can be comprehensively judged by the category and precise bounding box information of foreign objects, combined with the geometric and material characteristics of the track area. According to the impact of foreign objects, foreign object handling solutions are automatically generated, including foreign object cleaning and track maintenance, to ensure the safe operation of the track.

[0055] In a specific embodiment, a hyperspectral imaging technique is used to obtain a hyperspectral image of the track area, and the image is enhanced by a histogram equalization algorithm to improve the image contrast. Based on the prior knowledge of the track area, an edge detection algorithm is used to extract the geometric features of the track area, such as the track width of 1.435 meters and the track height of 0.176 meters; a texture analysis algorithm is used to extract the track material features, such as the track material is steel rails with a density of 7.85g / cm 3 . The extracted track geometry and material features are input into the pre-built foreign object feature database for matching, and the K nearest neighbor algorithm is used to obtain the most similar candidate foreign object feature information, such as stones, animal carcasses, etc. The U-Net semantic segmentation algorithm is used to accurately segment the track area from the enhanced hyperspectral image, and the precise geometry and material features of the track area are extracted. The enhanced hyperspectral image and candidate foreign object feature information are input into the pre-trained YOLO foreign object recognition model, and the model is fine-tuned by the transfer learning method to improve the foreign object recognition accuracy to more than 95%. The foreign object recognition model determines whether there are foreign objects in the hyperspectral image. If there are foreign objects, the category and location information of the foreign objects are output, such as the stone is located at the center of the track. According to the category and location information of the foreign objects, the Faster R-CNN target detection algorithm is used to accurately locate the foreign objects and obtain the pixel-level bounding box of the foreign objects. The impact of foreign objects on track operation is comprehensively judged by the category, bounding box, track geometry and material features of the foreign objects. For example, if the diameter of the stone is greater than 5 cm, it will affect the normal operation of the train. According to the degree of impact of foreign objects, a foreign object handling plan is automatically generated, such as sending cleaning personnel to clean up foreign objects on the track within 30 minutes to ensure the normal operation of the train.

[0056] Step S600, inputting the geometric features and the material features of the track area into a pre-built foreign body feature database to obtain feature information of candidate foreign bodies;

[0057] Specifically, the motion state information of the train is obtained, and according to the motion state of the train, the collected multispectral image is subjected to motion compensation and deblurring processing to generate a motion-compensated multispectral image. The motion-compensated multispectral image is subjected to adaptive image enhancement processing, and the enhanced multispectral image is generated by dynamically adjusting the contrast, brightness and saturation of the image. The enhanced multispectral image is segmented by an image segmentation algorithm, and the candidate foreign body region in the image is extracted to obtain the position and contour information of the candidate foreign body. According to the position and contour information of the candidate foreign body, the texture, shape, color and other features of the candidate foreign body are extracted from the enhanced multispectral image to generate a feature vector of the candidate foreign body. The enhanced multispectral image and the feature vector of the candidate foreign body are input into a pre-trained foreign body recognition model, and the foreign body recognition model is trained on a large number of foreign body samples using a transfer learning method. The foreign body recognition model analyzes the input multispectral image and the candidate foreign body features, and determines whether there is a foreign body in the multispectral image through feature matching and classification algorithms. If the foreign body recognition model determines that there is a foreign body in the multispectral image, the category information and position coordinate information of the foreign body are output. The type and location information of foreign objects are associated with the location information of the train to determine the specific location of the foreign object relative to the train and generate foreign object alarm information. The foreign object alarm information is sent to the train control center and maintenance personnel, and the type, location and multi-spectral image of the foreign object are displayed through a visual interface to facilitate foreign object handling and line maintenance.

[0058] In a specific embodiment, the speed, acceleration and other motion state information of the train are obtained by the vehicle-mounted inertial navigation system, and the motion compensation algorithm is triggered according to the condition that the train speed is greater than 80km / h. The motion estimation method based on pixel block matching is used to calculate the motion vector between adjacent frames in the image sequence, and the image is motion compensated. The compensated image is deblurred using a Wiener filter. Then, the motion-compensated image is adaptively histogram equalized, and the contrast, brightness and saturation parameters of the image are dynamically adjusted to increase the contrast of the image by 20%, the brightness by 10%, and the saturation by 15%, thereby generating an enhanced multispectral image. Next, an image segmentation algorithm based on region growth is used, and the seed point threshold of the region growth is set to 1.2 times the average grayscale of the image, and the similarity threshold of the growth is 15, and the candidate foreign body region in the image is extracted to obtain the position and contour information of the foreign body. According to the contour of the foreign body, the texture features (such as LBP features), shape features (such as Hu moment invariant moments) and color features (such as color histograms) of the foreign body region are extracted to generate a 48-dimensional foreign body feature vector. The enhanced image and foreign object features are input into the pre-trained foreign object recognition model, which uses ResNet-50 as the backbone network, is pre-trained on the ImageNet dataset, and then fine-tuned on 5,000 foreign object sample images, with an accuracy rate of over 95%. The foreign object recognition model uses feature extraction, feature matching, and softmax classifiers to determine whether there are foreign objects in the image. If there are foreign objects, the foreign object category (such as stones, animals, etc.) and location coordinates (such as 2.5 meters from the center line of the rail and 50 meters from the train) are output. Finally, the foreign object information is combined with the GPS location information of the train to generate foreign object alarm information, which is sent to the train control center and the maintenance personnel's mobile terminal in real time through the on-board 4G network, and the multispectral image, category, and location information of the foreign object are displayed on the visual interface to provide decision support for foreign object handling and line maintenance.

[0059] Step S700, inputting the enhanced multispectral image and the feature information of the candidate foreign object into a foreign object recognition model to determine whether there is a foreign object in the multispectral image, and if there is a foreign object, outputting the category and location information of the foreign object;

[0060] Specifically, a multispectral image under a track environment is acquired to obtain a visible light image and an infrared light image. The motion-compensated multispectral image is subjected to adaptive image enhancement processing to generate image contrast, brightness and saturation data. According to the image contrast, brightness and saturation data, the image parameters are dynamically adjusted to obtain an enhanced multispectral image. The texture information of the enhanced multispectral image is extracted to obtain feature information of the candidate foreign object. If the feature information of the candidate foreign object meets the preset foreign object feature library comparison condition, the foreign object recognition model is called for processing, otherwise the candidate foreign object feature information is fed back to the feature library maintenance unit. The foreign object recognition model is trained by a transfer learning method to obtain an updated foreign object recognition model. The enhanced multispectral image is processed according to the updated foreign object recognition model to obtain a preliminary screening result of foreign objects. If there is a suspected foreign object in the preliminary screening result of the enhanced multispectral image, the foreign object contour detection algorithm is used for processing, otherwise the multispectral image data is archived. According to the processing result of the foreign object contour detection algorithm, the category and location information of the foreign object are determined.

[0061] In a specific embodiment, a multispectral image is obtained in a track environment, wherein the resolution of the visible light image is 1920×1080 pixels and the resolution of the infrared light image is 1024×768 pixels. The motion-compensated multispectral image is subjected to adaptive image enhancement processing, and the contrast is adjusted by using a histogram equalization algorithm to increase the contrast to 1.5 times and the brightness is adjusted to 120 cd / m 2 , the saturation is increased to 1.2 times, and an enhanced multispectral image is generated. The texture information of the enhanced image is extracted, and the Gabor filter is used to extract features to obtain the feature vector of the candidate foreign body, which contains 16 feature dimensions. If the Euclidean distance between the feature vector of the candidate foreign body and the feature vector in the preset foreign body feature library is less than 0.5, the foreign body recognition model is called for processing. The ResNet-50 network structure is used for transfer learning, and the pre-trained model is fine-tuned on the multispectral image dataset. The training set contains 5,000 annotated images, and it is iterated 50 times to obtain the updated foreign body recognition model. The enhanced multispectral image is processed according to the updated model, and the foreign body confidence is output, and the threshold is set to 0.8. If the confidence is greater than 0.8, the Canny edge detection algorithm is used to detect the foreign body contour, and the edge threshold is set to 50 and 150 to obtain the foreign body contour. Based on the contour information, the center coordinates and area of ​​the foreign body are calculated. Combined with the preset foreign body category library, the foreign body category is determined to be metal fragments. The location information is (950,540) in the image coordinate system, and the area is approximately 300 pixels2.

[0062] In this embodiment, before step S100, the following steps are further included:

[0063] Determine the fusion ratio of visible light image and infrared light image according to the lighting conditions of the orbital environment;

[0064] Specifically, in the process of foreign object detection in orbit, it is crucial to obtain high-quality images, because this directly affects the accuracy and reliability of subsequent foreign object identification. However, the lighting conditions in the orbit environment are often complex and changeable, ranging from bright daylight to dim night and possible shadow areas, and these changes will have a significant impact on image quality. Visible light images and infrared light images each have different advantages. Visible light images can provide rich details and clear images when there is sufficient light, but the image quality will be greatly reduced when there is insufficient light or shadows. In contrast, infrared light images do not rely on visible light and can capture the thermal radiation characteristics of objects. Therefore, infrared light images often provide more useful information when there is insufficient light or shadows. Therefore, before acquiring multispectral images in the orbit environment, dynamically adjusting the fusion ratio of visible light images and infrared light images according to the lighting conditions of the orbit environment can optimize the quality of the final fused image. The specific approach is as follows: First, the lighting conditions of the orbit environment are evaluated by sensors or image analysis technology, including factors such as light intensity and light uniformity. According to the evaluation results of the lighting conditions, the fusion ratio of visible light images and infrared light images is dynamically adjusted. For example, when there is sufficient light, the weight of the visible light image can be appropriately increased to take advantage of its high definition and rich detail information; when there is insufficient light or shadows, the weight of the infrared light image should be increased to take advantage of its characteristics that are not dependent on visible light. Finally, the visible light image and the infrared light image are fused according to the determined fusion ratio to generate a high-quality multispectral fusion image. This fused image will combine the advantages of visible light images and infrared images to provide more accurate and reliable image data for subsequent foreign body identification.

[0065] In a specific example, in a certain track foreign object detection scenario, the lighting conditions of the track environment gradually change from bright to dim. In this case, the fusion ratio of the visible light image and the infrared light image can be dynamically adjusted according to the change of light intensity: in an area with sufficient light and no shadow, the weight of the visible light image can be set to 80%, and the weight of the infrared light image can be set to 20%. In this way, the fused image will mainly utilize the high definition and rich details of the visible light image, while supplemented by the thermal radiation information of the infrared light image to improve the accuracy of foreign object recognition. In areas with insufficient light and shadows, the weight of the visible light image can be reduced to 40%, while the weight of the infrared light image can be increased to 60%. In this way, the fused image will rely more on the thermal radiation characteristics of the infrared light image to overcome the impact of insufficient light on image quality and ensure the reliability of foreign object recognition. In this way, the fusion ratio of the visible light image and the infrared light image can be dynamically adjusted according to the lighting conditions of the track environment, thereby optimizing the quality of the final fused image and improving the accuracy and reliability of track foreign object detection.

[0066] In this embodiment, step S300 includes:

[0067] Step S301, obtaining the speed, acceleration and track curvature radius of the train;

[0068] Specifically, these parameters are crucial to understanding the motion state of the train because they directly affect the blur and motion distortion of the image during image acquisition. The speed of the train determines the speed of movement of objects in the image (including the track and potential foreign objects). The faster the speed, the greater the displacement of the object in the image and the higher the image blur. Acceleration reflects the rate of change of the train speed. The presence of acceleration means that the train is speeding up or slowing down, which can cause non-uniform movement of objects in the image, further increasing the blur of the image. The track curvature radius describes the degree of curvature of the track. When traveling on a curve, the train will generate additional lateral acceleration, which not only affects the motion trajectory of objects in the image, but may also introduce additional image distortion.

[0069] In a specific example, the train is traveling at a speed of 80 km / h on a curve with a radius of 1000 m and is moving at a speed of 0.5 m / s 2 The acceleration is used to accelerate. Speed: 80km / h is converted to meters per second, which is about 22.22m / s. This is the speed of the train when it is traveling in a straight line. However, when traveling on a curve, due to the effect of centrifugal force, the actual speed component of the train on the curve will be slightly reduced. However, in order to simplify the calculation here, the straight-line speed is still used as a reference. Acceleration: 0.5m / s 2Indicates that the train is accelerating, which causes objects in the image to have a slight displacement perpendicular to the direction of motion, increasing the blur of the image. Track curvature radius: 1000m describes the curvature of the curve. When traveling on a curve, the train will be affected by centrifugal force, which will cause additional lateral displacement and distortion of objects in the image.

[0070] Step S302, establishing a train motion model according to the speed, the acceleration and the track curvature radius;

[0071] Specifically, the train motion model is a mathematical description that simulates the actual motion state of the train based on parameters such as the train's speed, acceleration, and track curvature radius. This model is crucial for subsequent motion compensation and deblurring because it helps us understand and predict the motion trajectory of objects in the image, thereby accurately compensating for image blur and distortion caused by the train's motion. When building a train motion model, the following key factors need to be considered: The speed of the train determines the speed of the objects in the image (including the track and potential foreign objects). The faster the speed, the greater the displacement of the object in the image and the higher the image blur. Acceleration reflects the rate of change of the train's speed. The presence of acceleration means that the train is accelerating or decelerating, which causes non-uniform motion of objects in the image, further increasing the blur of the image. The track curvature radius describes the curvature of the track. When traveling on a curve, the train will generate additional lateral acceleration, which not only affects the motion trajectory of objects in the image, but may also introduce additional image distortion. Based on these parameters, a two-dimensional or three-dimensional train motion model can be established. This model can be a simple uniform linear motion model or a more complex nonlinear model that considers the effects of acceleration and curves. The specific form of the model depends on the needs and accuracy requirements of the application scenario.

[0072] In a specific embodiment, based on these parameters, a simple two-dimensional train motion model can be established. This model can take into account the linear speed, acceleration and lateral acceleration caused by the curve of the train. Then, this model is used to predict the motion trajectory of the object in the image, so as to accurately compensate the collected multispectral image. Specifically, the motion of the train can be decomposed into linear motion along the track direction and lateral motion perpendicular to the track direction. The linear motion can be described by speed and acceleration, while the lateral motion can be calculated by the centrifugal acceleration of the track curvature radius and the train speed. Then, the motion vector of each pixel in the image is estimated based on these parameters, and the image is geometrically transformed and pixel resampled to eliminate or reduce the image blur and distortion caused by the train motion.

[0073] Step S303, using the train motion model, performing motion compensation on the multispectral image to generate the motion compensated multispectral image;

[0074] Specifically, during the high-speed operation of the train, due to the movement of the train, the collected multispectral images (including visible light images and infrared light images) will be blurred and distorted, which is mainly due to the movement of objects in the image and the shaking of the camera. In order to eliminate or reduce these effects, it is necessary to use the previously established train motion model (step S302) to perform motion compensation on the image. The basic idea of ​​motion compensation is to estimate the motion vector of each pixel in the image, and then perform geometric transformation and pixel resampling on the image to correct the image blur and distortion caused by the movement of the train. The specific steps are as follows: Motion vector estimation: According to the train motion model, calculate the displacement of each pixel in the image due to the movement of the train, that is, the motion vector. This displacement includes, for example, displacement along the track direction and displacement perpendicular to the track direction (when traveling on a curve). Geometric transformation: According to the estimated motion vector, the image is geometrically transformed to "move back" the objects in the image to where they should originally appear. For example, operations such as translation, rotation and scaling of the image are involved. Pixel resampling: Since the new pixel position may not be an integer coordinate after the geometric transformation, it is necessary to use an interpolation algorithm (such as bilinear interpolation, bicubic interpolation, etc.) to calculate the pixel values ​​at these non-integer coordinates to generate a motion-compensated image. Motion compensation can significantly eliminate or reduce image blur and distortion caused by train motion, improve image quality, and provide more accurate and reliable image data for subsequent image enhancement, feature extraction, and foreign object recognition.

[0075] In a specific embodiment, in a track foreign object detection scenario, a train is traveling at a speed of 80 km / h on a curve with a radius of 1000 m and is moving at a speed of 0.5 m / s 2The train is accelerated by the acceleration of the train. Based on this information, a train motion model has been established (step S302). Motion vector estimation: For displacement along the track direction, the displacement of each pixel due to the linear motion of the train can be calculated based on the speed and acceleration of the train. For displacement perpendicular to the track direction, the effect of centrifugal force when traveling on a curve needs to be considered. According to the track curvature radius and the train speed, the lateral displacement of each pixel due to traveling on a curve can be calculated. Geometric transformation: According to the estimated motion vector, the image is translated, rotated (when traveling on a curve) and other geometric transformations are performed to "move back" the objects in the image to where they should originally appear. Pixel resampling: Since the new pixel position may not be an integer coordinate after the geometric transformation, a bilinear interpolation algorithm is used to calculate the pixel values ​​at these non-integer coordinates to generate a motion-compensated multispectral image. Through the above steps, a motion-compensated multispectral image can be generated, which has significantly eliminated the blur and distortion caused by the train motion, providing clearer and more accurate image data for subsequent image enhancement, feature extraction and foreign object recognition.

[0076] In this embodiment, step S400 includes:

[0077] Step S401, obtaining ambient light intensity;

[0078] Specifically, in the detection of foreign objects on the track, the ambient light intensity has an important impact on the image quality. Too strong or too weak light may cause the image quality to deteriorate, thereby affecting the subsequent foreign object identification and positioning. Therefore, before performing adaptive image enhancement processing, it is necessary to obtain the current ambient light intensity so as to adjust the image parameters according to the lighting conditions and improve the image quality. There are many ways to obtain the ambient light intensity, including but not limited to: installing a light sensor in the track foreign object detection system to measure the ambient light intensity in real time. The light sensor can be a separate device or a part integrated in the imaging device. The ambient light intensity is estimated by analyzing the collected multispectral image data. For example, the average brightness or histogram distribution of the image can be calculated to infer the current lighting conditions. If the system has been running for a period of time and sufficient historical data has been collected, the data can be used to predict the current light intensity. For example, the light intensity can be estimated based on factors such as time (such as different periods of the day), weather conditions (such as sunny, cloudy, rainy, etc.) and geographical location. Regardless of which method is used, the acquired ambient light intensity will be used as one of the input parameters for the subsequent adaptive image enhancement processing.

[0079] Step S402, adaptively adjusting the contrast, brightness and saturation of the motion compensated multispectral image according to the ambient light intensity to generate the enhanced multispectral image;

[0080] Specifically, the purpose of this step is to adjust the contrast, brightness and saturation of the image so that the image can present the best visual effect under different lighting conditions, thereby improving the accuracy and reliability of subsequent foreign body recognition. Specifically, the execution process of step S402 is as follows:

[0081] Analyze the ambient light intensity: First, determine the current light condition according to the ambient light intensity value obtained in step S401. The light condition can be divided into strong light, weak light, normal light and other conditions.

[0082] Dynamically adjust image parameters: Contrast adjustment: Under strong light conditions, the image may be too bright, resulting in loss of details. At this time, the contrast can be appropriately reduced to make the difference between light and dark in the image more obvious, highlighting the outline and details of the object. On the contrary, under weak light conditions, the image may be too dim. At this time, the contrast can be appropriately increased to improve the clarity of the image. Brightness adjustment: Brightness adjustment is mainly used to compensate for the impact of ambient light on the overall brightness of the image. Under strong light conditions, the brightness can be appropriately reduced to avoid overexposure of the image; under weak light conditions, the brightness can be appropriately increased to make the image brighter. Saturation adjustment: Saturation adjustment is used to enhance or weaken the color information in the image. Under strong light conditions, the colors may be too bright and distorted, so the saturation can be appropriately reduced; under weak light conditions, in order to enhance the color expression of the image, the saturation can be appropriately increased.

[0083] Generate enhanced multispectral images: After the above adjustments, enhanced multispectral images are generated. These images are optimized in terms of contrast, brightness, and saturation, making them more suitable for subsequent foreign body recognition and positioning processing.

[0084] Through the above adjustments, enhanced multispectral images are generated. These images are optimized in terms of contrast, brightness and saturation, showing the track area and its potential foreign objects more clearly. These optimized images will serve as input data for subsequent foreign object identification and location processing, helping to improve the accuracy and reliability of foreign object detection.

[0085] In this embodiment, step S500 includes:

[0086] Step S501, segmenting the track area from the enhanced multispectral image using a semantic segmentation algorithm;

[0087] Specifically, the enhanced multispectral image is subjected to necessary preprocessing, such as normalization, denoising, etc., to improve the accuracy and efficiency of semantic segmentation. Features are extracted from the preprocessed image using a deep learning model (such as a convolutional neural network CNN). These features usually include information such as the edge, texture, and color of the image, which are essential for distinguishing different semantic regions. The extracted features are input into a semantic segmentation model, which is trained to identify and segment the track region in the image. Semantic segmentation models usually use architectures such as fully convolutional networks (FCN), U-Net, and DeepLab, which can effectively capture contextual information in the image, thereby achieving high-precision pixel-level segmentation. The results of semantic segmentation are post-processed, such as removing small noise areas, filling holes, etc., to obtain more accurate track region segmentation results. Through step S501, the system can accurately segment the track region from the enhanced multispectral image, providing a reliable basis for the subsequent extraction of geometric features and material features of the track region.

[0088] In a specific embodiment, the enhanced multispectral image is first normalized, and the pixel values ​​of the image are scaled to between 0 and 1 to eliminate the brightness difference between images of different bands. At the same time, the system also uses Gaussian filtering to denoise the image to improve the image quality. A pre-trained convolutional neural network (such as ResNet-50) is used to extract features from the pre-processed image. This network can capture low-level features such as edges and textures in the image through multi-layer convolution and pooling operations, and map these features to a high-dimensional space through a fully connected layer to form a feature representation with rich semantic information. The extracted features are input into a U-Net semantic segmentation model. This model adopts an encoder-decoder structure. The encoder gradually reduces the resolution of the image and extracts high-level features through multi-layer convolution and pooling operations, and the decoder gradually restores the resolution of the image through upsampling and convolution operations, and finally outputs a segmentation result of the same size as the input image. During the training process, the model learns how to correctly classify pixels in the image into track areas and non-track areas by minimizing the cross entropy loss function. The segmentation results output by the U-Net model are post-processed. First, the system uses morphological operations (such as opening and closing operations) to remove small noise areas and fill holes in the segmentation results. Then, the system uses contour detection algorithms to extract the precise boundaries of the track area so that the geometric features and material features of the track area can be extracted later.

[0089] Step S502, extracting geometric features of the track area, wherein the geometric features include edges and textures;

[0090] Specifically, an edge is a location in an image where the brightness or color changes significantly, usually corresponding to the boundary or outline of an object; a texture is a certain regular change in the grayscale or color of a pixel in a local area of ​​an image, which reflects the surface characteristics of an object. In track foreign body detection, extracting edge and texture features of the track area helps to more accurately identify the track and potential foreign bodies on it. Edge features can help determine the precise boundary of the track, thereby distinguishing between track and non-track areas; texture features can provide information about the material and state of the track surface, which is very useful for identifying anomalies or damage on the track. Methods for extracting edge features usually include edge detection algorithms, such as Canny edge detector, Sobel operator, Prewitt operator, etc. These algorithms detect edges by calculating the gradient strength and direction of each pixel in the image. The extraction of texture features may involve more complex image processing techniques, such as gray level co-occurrence matrix (GLCM), local binary pattern (LBP), Gabor filter, etc., which can capture the local statistical characteristics or frequency characteristics of the texture in the image.

[0091] In a specific embodiment, the track area has been accurately segmented from the enhanced multispectral image by a semantic segmentation algorithm. Next, it is necessary to extract the geometric features of the track area, including edges and textures. Edge feature extraction: The system uses, for example, the Canny edge detection algorithm to extract edge features of the track area. The Canny algorithm first smoothes the image through a Gaussian filter to reduce the influence of noise; then calculates the gradient intensity and direction of each pixel in the image; then applies non-maximum suppression to refine the edge; and finally uses double threshold detection and edge connection to determine the final edge. Through the Canny algorithm, the system can accurately detect the boundaries of the track area, including the edges of the rails, the edges of the sleepers, etc. These edge features will be used for subsequent foreign body identification and positioning. Texture feature extraction: For example, the gray level co-occurrence matrix (GLCM) is used to extract the texture features of the track area. GLCM is a statistical method that describes the spatial dependency of gray levels in an image, which can reflect the thickness, direction, contrast, and other information of the texture in the image. First, the GLCM of the track area is calculated, and then multiple statistics are extracted from it as texture features, such as contrast, correlation, energy, entropy, etc. These features can describe the roughness, regularity, uniformity and other properties of the track surface. For example, if there are cracks or wear on the track surface, these abnormal areas may show different texture features from the normal track surface on the GLCM, such as increased contrast, reduced energy, etc. Through step S502, rich geometric features, including edges and textures, can be extracted from the segmented track area. These features will provide important information support for subsequent foreign object identification and positioning, and help improve the accuracy and reliability of track foreign object detection.

[0092] Step S503, extracting material features of the track area, wherein the material features include color and glossiness;

[0093] Specifically, color is the perception result of a specific wavelength combination of light reflected or emitted from the surface of an object in the human eye, which reflects the spectral characteristics of the surface of the object; glossiness is the ability of the surface of an object to reflect light, which is usually related to the smoothness and material type of the surface of the object. In track foreign body detection, extracting the color and glossiness features of the track area helps to more accurately identify the track and potential foreign bodies on it. Color features can help distinguish objects of different material types, such as metal, plastic, gravel, etc., because their reflective spectra in the visible light band are different. Glossiness features can further distinguish objects with similar colors but different surface characteristics, such as newly laid tracks and worn tracks, or metal fragments and wet soil. Methods for extracting color features usually include color space conversion, color histogram statistics, color moments, etc. These methods can quantify the color information in the image into specific numerical features, which is convenient for subsequent analysis and processing. Extracting glossiness features may involve more complex image processing techniques, such as analysis based on image gradient or frequency domain characteristics, which can capture the intensity and directionality information of the light reflected from the surface of the object, thereby reflecting the glossiness of the object.

[0094] In a specific embodiment, the track area has been accurately segmented from the enhanced multispectral image by a semantic segmentation algorithm. Next, it is necessary to extract the material features of this track area, including color and glossiness. Color feature extraction: First, the image of the track area is converted from the RGB color space to the HSV color space (hue, saturation, brightness), because the HSV color space is more in line with the way humans perceive color and has a certain robustness to changes in illumination. Next, the color histogram of the track area in the HSV color space is calculated, and the number of pixels with different hues, saturation and brightness values ​​is counted. The color histogram can reflect the overall color distribution characteristics of the track area. In order to further quantify the color features, the system can also calculate the color moments of the track area, including the first-order moment (mean), second-order moment (variance) and third-order moment (skewness). These color moments can provide more statistical information about the color distribution of the track area. Gloss feature extraction: For example, a method based on image gradient is used to analyze the glossiness of the track area. First, the system calculates the gradient amplitude and gradient direction of the track area image. These gradient information can reflect the intensity and directionality of the light reflected from the surface of the object. Then, the gradient histogram of the track area image is statistically analyzed, especially the distribution characteristics of the gradient amplitude. The surface of an object with higher glossiness usually has a more consistent gradient amplitude and directionality, while the surface of an object with lower glossiness exhibits a more complex gradient distribution. In order to quantify the glossiness characteristics, indicators such as the gradient entropy or gradient consistency of the track area image can be calculated. Gradient entropy reflects the complexity of the gradient distribution, while gradient consistency measures the consistency of the gradient direction. These indicators can be used as a quantitative representation of the glossiness characteristics.

[0095] In this embodiment, inputting the enhanced multispectral image and the characteristic information of the candidate foreign object into the foreign object recognition model specifically includes: using a transfer learning method to apply the pre-trained foreign object recognition model to the enhanced multispectral image to realize the recognition of track foreign objects;

[0096] Specifically, transfer learning is a machine learning method that allows the knowledge learned in one domain (source domain) to be transferred to another related but different domain (target domain). In the case of foreign object detection on track, transfer learning can be used to transfer deep learning models pre-trained on large-scale general datasets (such as ImageNet) to the task of foreign object detection on track, thereby leveraging the rich feature representations that the pre-trained model has learned to accelerate the training process of the new task and improve the recognition performance. Pre-trained model selection: First, select a deep learning model pre-trained on a large-scale general dataset as a starting point. This model usually has strong feature extraction capabilities and can perform well in various image recognition tasks. For example, classic convolutional neural network models such as ResNet, VGG, and Inception can be selected. Model fine-tuning: Replace the last few layers of the pre-trained model (usually fully connected layers or classification layers) with new layers suitable for the task of foreign object detection on track. Then, fine-tune the model using the labeled data from the task of foreign object detection on track. During the fine-tuning process, most of the parameters of the pre-trained model are kept unchanged, and only the parameters of the newly added layers and some pre-trained layers are updated to adapt to the new recognition task. Data enhancement: In order to improve the generalization ability of the model, the training data can be enhanced, such as random cropping, rotation, flipping, adding noise, etc. These operations can increase the diversity of training data and help the model learn more robust feature representations. Model evaluation: During the fine-tuning process, the model is regularly evaluated using the validation set to monitor changes in the model's performance. If the performance of the model on the validation set begins to decline (i.e., overfitting occurs), training should be stopped in time, and regularization, early stopping and other strategies should be considered to prevent overfitting. Application identification: The fine-tuned model is applied to the enhanced multispectral image to identify track foreign objects. The model outputs the probability distribution of each pixel or image area belonging to different foreign object categories, and the final foreign object location and category are determined by setting a threshold or using methods such as non-maximum suppression.

[0097] In a specific embodiment, the enhanced multispectral image has been obtained through the previous steps and is ready for foreign object recognition. Pre-trained model selection: A ResNet-50 model pre-trained on the ImageNet dataset is selected as the starting point. This model has powerful feature extraction capabilities and has been widely used in various image recognition tasks. Model fine-tuning: The last two layers of the ResNet-50 model are replaced with two new fully connected layers, which are used for feature fusion and classification respectively. Then, the model is fine-tuned using a dataset containing 5,000 labeled track foreign object images. During the fine-tuning process, the system adopts strategies such as learning rate decay and batch normalization to accelerate the training process and improve model performance. Data enhancement: In order to increase the diversity of training data, each training image is randomly cropped, rotated, and flipped. These enhanced images together with the original images constitute the final training set. Model evaluation: During the fine-tuning process, the model will be evaluated on the validation set after each training round. The performance changes of the model are monitored by calculating indicators such as accuracy, recall, and F1 score. When the model's performance on the validation set began to decline, the system stopped training in a timely manner. Application identification: The fine-tuned ResNet-50 model was applied to the enhanced multispectral image. The model classified each pixel and output the probability distribution of each pixel belonging to different foreign object categories. By setting a threshold (such as 0.8), the system determined the final foreign object location and category, and generated a report containing the foreign object category and location information. Through the above steps, the pre-trained foreign object recognition model can be applied to the enhanced multispectral image using the transfer learning method to achieve accurate identification of track foreign objects. This method not only improves the accuracy and efficiency of foreign object recognition, but also reduces the dependence on large-scale labeled data.

[0098] Embodiment 2

[0099] Figure 2 is a schematic diagram of the structure of a track foreign body detection system in Embodiment 2 of the present invention. Figure 2As shown, embodiment 2 provides a track foreign body detection system, including: a first acquisition module 201, a first judgment module 202, a first generation module 203, a second generation module 204, an extraction module 205, a second acquisition module 206 and a second judgment module 207. The first acquisition module 201 is used to acquire a multispectral image under a track environment, and the multispectral image includes a visible light image and an infrared light image. The first judgment module 202 is used to acquire the acceleration value and the angular velocity value during the operation of the train, and judge the motion state of the train according to the acceleration value and the angular velocity value. The first generation module 203 is used to perform motion compensation and deblurring processing on the multispectral image according to the motion state of the train, and generate a motion-compensated multispectral image. The second generation module 204 is used to perform adaptive image enhancement processing on the motion-compensated multispectral image to generate an enhanced multispectral image. The extraction module 205 is used to extract the geometric features and material features of the track area from the enhanced multispectral image. The second acquisition module 206 is used to input the geometric features and the material features of the track area into a pre-built foreign body feature database to obtain feature information of candidate foreign bodies. The second judgment module 207 is used to input the enhanced multispectral image and the feature information of the candidate foreign bodies into a foreign body recognition model to judge whether there are foreign bodies in the multispectral image, wherein if there are foreign bodies, the category and location information of the foreign bodies are output.

[0100] In this embodiment, the track foreign object detection system further includes a determination module, which is used to determine the fusion ratio of the visible light image and the infrared light image according to the lighting conditions of the track environment.

[0101] In this embodiment, the first generation module 203 includes: a first acquisition unit, an establishment unit, and a first generation unit. The first acquisition unit is used to acquire the speed, acceleration, and track curvature radius of the train. The establishment unit is used to establish a train motion model according to the speed, acceleration, and track curvature radius. The first generation unit is used to use the train motion model to perform motion compensation on the multispectral image to generate the motion compensated multispectral image.

[0102] In this embodiment, the second generation module 204 includes: a second acquisition unit and a second generation unit. The second acquisition unit is used to acquire the ambient light intensity. The second generation unit is used to adaptively adjust the contrast, brightness and saturation of the motion compensated multispectral image according to the ambient light intensity to generate the enhanced multispectral image.

[0103] In this embodiment, the extraction module 205 includes: a segmentation unit, a first extraction unit and a second extraction unit. The segmentation unit is used to segment the track area from the enhanced multispectral image using a semantic segmentation algorithm. The first extraction unit is used to extract geometric features of the track area, wherein the geometric features include edges and textures. The second extraction unit is used to extract material features of the track area, wherein the material features include color and glossiness.

[0104] The various variations and specific examples of the track foreign object detection method provided in Example 1 are also applicable to the track foreign object detection system provided in this embodiment. Through the above detailed description of a track foreign object detection method, those skilled in the art can clearly know the implementation method of a track foreign object detection system in this embodiment, so for the sake of brevity of the specification, it will not be described in detail here.

[0105] Embodiment 3

[0106] Figure 3 is a schematic diagram of the structure of an electronic device in Embodiment 3 of the present invention, such as Figure 3 As shown, the third embodiment further provides an electronic device 300 , which may include: a processor 301 and a memory 302 .

[0107] The memory 302 is used to store programs; the memory 302 may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory 302 is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories 302. And the above computer programs, computer instructions, data, etc. can be called by the processor 301.

[0108] The above-mentioned computer programs, computer instructions, etc. may be stored in partitions in one or more memories 302 . And the above-mentioned computer programs, computer instructions, etc. may be called by the processor 301 .

[0109] The processor 301 is used to execute the computer program stored in the memory 302 to implement each step of the method involved in the above embodiment.

[0110] For details, please refer to the relevant description in the previous method embodiment.

[0111] The processor 301 and the memory 302 may be independent structures or integrated structures. When the processor 301 and the memory 302 are independent structures, the memory 302 and the processor 301 may be coupled and connected via a bus 303 .

[0112] The electronic device of this embodiment can execute the technical solution in the above method, and its specific implementation process and technical principle are the same, which will not be repeated here.

[0113] Embodiment 4

[0114] Embodiment 4 further provides a computer-readable storage medium, including a computer program and instructions. When the computer program or instructions are executed on a computer, the computer executes the track foreign matter detection method of any embodiment of the present invention.

[0115] Computer-readable storage media include: USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks, and other media that can store program codes.

[0116] This embodiment also provides a computer program product, which includes: a computer program, which is stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0118] In summary, the track foreign object detection method and system of the present invention obtains multispectral images and train motion parameters, performs motion compensation and deblurring processing on the images, and then performs adaptive enhancement, thereby improving image quality. Then, a deep learning algorithm is used to extract track area features, combined with a pre-built foreign object feature database, and a foreign object recognition model trained by transfer learning is used to achieve accurate detection and positioning of track foreign objects. The present invention solves the problem of image quality degradation caused by complex track environment and high-speed train operation, and overcomes the limitation that traditional methods are difficult to accurately identify a variety of foreign objects. Through the integrated application of multiple advanced algorithms, the present invention significantly improves the accuracy and real-time performance of track foreign object detection, providing strong support for ensuring the safe operation of rail transit.

[0119] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for detecting foreign matter on a track, characterized in that: include: Acquire a multispectral image in a track environment, wherein the multispectral image includes a visible light image and an infrared light image; Acquire the acceleration value and the angular velocity value during the running of the train, and judge the motion state of the train according to the acceleration value and the angular velocity value; According to the motion state of the train, the multispectral image is subjected to motion compensation and deblurring processing to generate a motion-compensated multispectral image; Performing adaptive image enhancement processing on the motion compensated multispectral image to generate an enhanced multispectral image; extracting geometric features and material features of the track area from the enhanced multispectral image; Inputting the geometric features of the track area and the material features into a pre-built foreign body feature database to obtain feature information of candidate foreign bodies; The enhanced multispectral image and the feature information of the candidate foreign object are input into the foreign object recognition model to determine whether there is a foreign object in the multispectral image. If there is a foreign object, the category and location information of the foreign object are output.

2. The track foreign body detection method according to claim 1, characterized in that: Before acquiring a multispectral image in a track environment, wherein the multispectral image includes a visible light image and an infrared light image, the method further includes: The fusion ratio of visible light images and infrared light images is determined according to the lighting conditions of the orbital environment.

3. The track foreign body detection method according to claim 1, characterized in that: The performing motion compensation and deblurring processing on the multispectral image according to the motion state of the train to generate the motion compensated multispectral image comprises: Obtaining the speed, acceleration and track curvature radius of the train; Establishing a train motion model according to the speed, the acceleration and the track curvature radius; The train motion model is used to perform motion compensation on the multispectral image to generate the motion compensated multispectral image.

4. The track foreign body detection method according to claim 1, characterized in that: The performing adaptive image enhancement processing on the motion compensated multispectral image to generate an enhanced multispectral image comprises: Get the ambient light intensity; According to the ambient light intensity, the contrast, brightness and saturation of the motion compensated multispectral image are adaptively adjusted to generate the enhanced multispectral image.

5. The track foreign body detection method according to claim 1, characterized in that: The step of extracting geometric features and material features of the track area from the enhanced multispectral image comprises: Segmenting the track area from the enhanced multispectral image using a semantic segmentation algorithm; Extracting geometric features of the track area, wherein the geometric features include edges and textures; The material features of the track area are extracted, wherein the material features include color and glossiness.

6. A track foreign body detection system, characterized in that: include: A first acquisition module is used to acquire a multispectral image in a track environment, wherein the multispectral image includes a visible light image and an infrared light image; A first judgment module is used to obtain the acceleration value and the angular velocity value during the running of the train, and judge the motion state of the train according to the acceleration value and the angular velocity value; A first generating module is used to perform motion compensation and deblurring processing on the multispectral image according to the motion state of the train to generate a motion-compensated multispectral image; A second generating module is used to perform adaptive image enhancement processing on the motion compensated multispectral image to generate an enhanced multispectral image; An extraction module, used for extracting geometric features and material features of the track area from the enhanced multispectral image; A second acquisition module is used to input the geometric features and the material features of the track area into a pre-built foreign body feature database to acquire feature information of candidate foreign bodies; The second judgment module is used to input the enhanced multispectral image and the feature information of the candidate foreign object into the foreign object recognition model to judge whether there is a foreign object in the multispectral image, wherein if there is a foreign object, the category and location information of the foreign object are output.

7. The track foreign object detection system according to claim 6, characterized in that: Also includes: The determination module is used to determine the fusion ratio of the visible light image and the infrared light image according to the lighting conditions of the track environment.

8. The track foreign object detection system according to claim 6, characterized in that: The first generation module comprises: A first acquisition unit, used to acquire the speed, acceleration and track curvature radius of the train; An establishing unit, used for establishing a train motion model according to the speed, the acceleration and the track curvature radius; The first generating unit is used to perform motion compensation on the multispectral image by using the train motion model to generate the motion compensated multispectral image.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; Wherein, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the track foreign object detection method according to any one of claims 1-5.

10. A computer-readable storage medium, characterized in that: The method comprises a computer program and instructions. When the computer program or the instructions are executed on a computer, the computer is enabled to execute the track foreign object detection method as claimed in any one of claims 1 to 5.

Images