A monocular vision-based method, system and device for detecting the ground clearance of a suspended object

Abstract

The application relates to the technical field of computer vision and deep learning, and discloses a method, system and device for detecting the ground clearance of a suspended object based on monocular vision, which comprises the following steps: acquiring continuous multiple frames of monocular images containing a target suspended object; processing the multiple frames of monocular images by using a trained deep learning network, wherein the processing comprises the following steps: detecting the target suspended object in each frame of monocular image to output 2D detection information; online estimating the camera parameters of the camera and the installation height of the camera relative to a reference plane based on the multiple frames of monocular images; inputting the 2D detection information, the camera parameters and the installation height of the camera into a differentiable geometric projection layer, using the known physical size of the target suspended object as a geometric constraint, and calculating the ground clearance of the target suspended object; and training the deep learning network by minimizing a loss function containing consistency constraints between the multiple frames of monocular images. The application makes full use of physical priori and timing information, and significantly improves the measurement accuracy.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, specifically to a method, system, and device for detecting the height of suspended objects off the ground based on monocular vision. Background Technology

[0002] Railways are one of the most important modes of transportation in modern society. As a crucial component of the railway traction power supply system, the overhead contact system's operational status directly affects the safe operation of electrified railways. The contact wire tension compensation device is installed at both ends of the anchor section, using weights to maintain a constant contact wire tension. Affected by temperature and vibration, the contact wire may slack and stretch, causing changes in the position of the weights. When the weights exceed the safe working distance, the tension compensation device fails, potentially leading to jamming, weight loss, or weight bottoming out, resulting in abnormal changes in contact wire tension. This affects the pantograph's sliding, and in severe cases, can cause major accidents such as pantograph strikes or wire breakage.

[0003] Therefore, how to design an online monitoring system for the compensation device to efficiently and accurately monitor its operating status (especially the height of the weight off the ground) has become an urgent technical problem to be solved.

[0004] Traditionally, manual inspections are relied upon, which are time-consuming, labor-intensive, and lack real-time performance. Some researchers have proposed building monitoring networks using sensors, but due to the large number of contact anchor sections, the detection system is complex and costly to construct. In recent years, with the development of machine vision technology, visual inspection has become a promising tool for non-contact anomaly detection in overhead contact lines. However, existing machine vision-based methods have the following shortcomings:

[0005] (1) Traditional monocular vision methods require manual calibration of camera parameters (such as focal length, installation height, pitch angle, etc.) beforehand. However, vibrations and bumps during train operation can cause slight changes in camera parameters, leading to rapid failure of calibration parameters and a significant decrease in measurement accuracy.

[0006] (2) Pure deep learning methods directly regress the height value from the image, treating deep learning as a black box model. They lack physical geometric constraints and have insufficient accuracy and poor generalization in complex railway scenarios.

[0007] (3) Existing methods mostly measure single-frame images and do not make full use of the multi-frame time sequence image information collected during the continuous movement of the train, making it difficult to solve the inherent scale ambiguity problem of monocular vision.

[0008] (4) There is little research on the detection of the weight of the overhead contact line compensation device. Existing methods require manual setting of key points and cannot achieve fully automatic detection.

[0009] Therefore, there is an urgent need for a high-precision, fully automatic method for detecting the height of suspended objects off the ground that can integrate physical geometric priors with deep learning, make full use of multi-frame temporal information, and eliminate the need for manual calibration of camera parameters. Summary of the Invention

[0010] The present invention aims to provide a method, system and train for detecting the height of suspended objects off the ground based on monocular vision, so as to solve the technical problems of low measurement accuracy, reliance on manual calibration, lack of physical constraints and inability to fully utilize temporal information in the prior art.

[0011] This invention is achieved through the following technical solution:

[0012] A method for detecting the height of a suspended object above the ground based on monocular vision, comprising:

[0013] Acquire continuous T-frame monocular images containing the target suspended object of the railway catenary compensation device, which are continuously collected by a monocular camera installed on the train during operation;

[0014] The trained deep learning network is used to process multiple frames of the monocular images, the processing including:

[0015] The system detects suspended objects in each frame of the monocular image and outputs 2D detection information. Based on multiple frames of the monocular image, the camera parameters and the camera's mounting height relative to the reference plane are estimated online. The 2D detection information, the camera parameters, and the camera mounting height are input into a differentiable geometric projection layer. The geometric projection layer uses the known standard physical width of the suspended object as a geometric constraint to calculate the height of the suspended object above the ground.

[0016] The deep learning network is trained by minimizing a loss function that includes consistency constraints between multiple frames of the monocular images. The consistency constraints include: the estimated ground clearance of the same suspended object remains consistent across different frames, and the camera parameters remain consistent across the same image sequence.

[0017] As an optimization, the continuous multiple frames of the monocular images are video sequences continuously acquired by the onboard monocular camera during train operation; wherein, the number of frames T in which the same suspended object is continuously captured in the video sequence is greater than or equal to 3.

[0018] As an optimization, the deep learning network is an end-to-end monocular height measurement network SeqM-Net, comprising:

[0019] The backbone network and feature fusion network are used to extract and fuse multi-scale features from the input image, and output first feature map P3, second feature map P4 and third feature map P5 at different scales, wherein the resolution of the first feature map P3, second feature map P4 and third feature map P5 gradually decreases.

[0020] The detection head is connected to the output of the backbone network and the feature fusion network, and outputs 2D detection information based on the first feature map P3, the second feature map P4 and the third feature map P5 respectively. The 2D detection information is a 2D bounding box containing the target suspended object.

[0021] The camera calibration head is connected to the output terminals of the second feature map P4 and the third feature map P5, and outputs two sets of camera parameters. These two sets of camera parameters are respectively input to the camera elevation head and the target elevation head. The two sets of camera parameters are, respectively, the camera pitch angle. and vertical field of view ;

[0022] The camera height head contains a TCN network, which receives camera parameters output from the camera calibration head and combines them with 2D detection information output from the detection head to estimate the camera's mounting height relative to a reference plane, i.e., the camera mounting height. ;

[0023] The target height head, which is the differentiable geometric projection layer, receives the 2D bounding box output by the detection head, all camera parameters output by the camera calibration head, and the camera installation height output by the camera height head, and calculates the ground height of the target suspended object using the perspective projection formula.

[0024] As an optimization, the camera calibration head includes:

[0025] The lightweight processing module performs depthwise convolution and pointwise convolution on the input third feature map P5 to compress the number of channels to 256.

[0026] The coordinate attention module performs horizontal pooling and vertical pooling on the features of the compressed third feature map P5 to capture the horizontal geometric features of the track and the vertical features of the catenary, respectively. After splicing and fusion and weight generation, the features of the third feature map P5 are enhanced by targeted weighting.

[0027] The feature reuse module bilinearly upsamples the coordinate attention weights derived from the third feature map P5 to the same resolution as the second feature map P4, and performs targeted weighted enhancement on the features of the second feature map P4.

[0028] The feature fusion module adds the enhanced features of the upsampled third feature map P5 to the enhanced features of the second feature map P4 element by element, and aggregates them into a global feature vector through global average pooling.

[0029] The first binning prediction module maps the global feature vector into scores for multiple discrete intervals through a convolutional layer. After Softmax normalization, the scores are weighted and summed to output at least two sets of camera parameters, including the camera pitch angle. (64 bins, -π / 12 to +π / 12 rad) and vertical field of view (32 bins, π / 6 to π / 3 rad), where the vertical field of view is... The camera pitch angle is used to calculate the camera focal length f. The camera focal length f is used to calculate the horizon position. .

[0030] Here, 64 bins and 32 bins refer to dividing the continuous parameter range into multiple discrete intervals, each interval being called a bin (container / box).

[0031] As an optimization, the camera height head (CamH Head) internally includes:

[0032] The temporal convolutional encoder takes an 8-dimensional geometric feature sequence of T consecutive frames as input. The 8-dimensional geometric features include the horizon ordinate, the left and right boundary x-coordinates of the detection box, the top and bottom boundary ordinates of the detection box, the offset of the detection box relative to the horizon, and the known physical width of the weight. The temporal convolutional encoder uses three layers of dilated temporal residual convolutional blocks with dilation rates set to d=1, 2, and 4, respectively, to gradually expand the receptive field to cover the entire input sequence. Each convolutional block is equipped with residual connections and 1×1 convolutional branches to encode the input sequence into high-dimensional temporal semantic features.

[0033] The attention pooling layer takes the multi-frame features output by the temporal convolutional encoder as input, calculates the importance score for each frame through a shared fully connected layer, and then performs a weighted summation of the features of each frame after Softmax normalization, and aggregates them into a global feature vector.

[0034] The second binning prediction module maps the global feature vector into scores for multiple discrete intervals through a fully connected layer. After Softmax normalization, the center values ​​of each interval are weighted and summed to output the camera's mounting height relative to the reference plane. .

[0035] As an optimization, the deep learning network is trained by minimizing the following loss function:

[0036] Detection box loss is used to monitor the 2D detection information of the target suspended object;

[0037] Reprojection loss is used to ensure that the estimated camera parameters are consistent with the 2D observation information;

[0038] Pixel width loss is achieved by using the known physical width of the target suspended object as a monitoring signal;

[0039] Target height consistency loss is used to ensure that the estimated ground clearance of the same suspended target remains consistent across different frames.

[0040] Camera parameter consistency loss is used to constrain camera parameters to remain consistent across the same image sequence.

[0041] Prior loss is used to impose reasonable physical range constraints on camera parameters and / or ground clearance.

[0042] As an optimization, the deep learning network further includes an instance segmentation branch for generating a pixel-level mask when multiple suspended objects overlap in an image; the instance segmentation branch includes:

[0043] The prototype branch takes the first feature map P3 as input and generates a set of prototype mask templates through upsampling and convolution operations.

[0044] The mask branch takes the first feature map P3, the second feature map P4, and the third feature map P5 output by the feature fusion network as input, and generates a mask coefficient vector through convolution dimensionality reduction and compression.

[0045] The weighted combination module uses the position and scale information of the target suspended object output by the detection head to assign a set of exclusive mask coefficients to each detected target suspended object. The mask coefficient vector is then weighted and combined with the prototype mask template to generate a pixel-level mask for each target suspended object.

[0046] As an optimization, the target suspended object is the weight of a railway catenary compensation device, and the reference plane is the ground or track reference plane; the method further includes:

[0047] The calculated ground clearance is compared with a preset safety threshold. When the ground clearance is lower than the safety threshold, the contact wire compensation device is determined to be in an abnormal state and an alarm signal is output.

[0048] This invention also discloses a monocular vision-based system for detecting the height of a suspended object off the ground, used to perform the aforementioned monocular vision-based method for detecting the height of a suspended object off the ground, comprising:

[0049] The image acquisition module is used to acquire continuous multi-frame monocular images containing the target suspended object;

[0050] A deep learning processing module, internally storing a trained deep learning network, is used to process multiple frames of the monocular images and output the ground clearance of the target suspended object relative to a reference plane. The processing includes: detecting the target suspended object in each frame of the monocular image and outputting 2D detection information; estimating the camera parameters and the camera's mounting height relative to the reference plane online based on multiple frames of the monocular images; and inputting the 2D detection information, the camera parameters, and the camera mounting height into a differentiable geometric projection layer, which uses the known physical dimensions of the target suspended object as geometric constraints to calculate the ground clearance.

[0051] The deep learning network is trained by minimizing a loss function that includes consistency constraints between multiple frames of images. The consistency constraints include: the estimated ground clearance of the same suspended object remains consistent in different frames, and / or the camera parameters in the same image sequence remain consistent.

[0052] The present invention also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a method for detecting the height of a suspended object off the ground based on monocular vision as described above.

[0053] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0054] This invention is the first to conduct a rigorous solvability analysis of monocular vision measurement of the height of a weight off the ground, proving the theoretical condition that a single frame image is unsolvable but multiple frames (T≥3) are solvable, providing a solid theoretical basis for using video sequences instead of single images for measurement.

[0055] This invention designs an end-to-end SeqM-Net network, embedding a differentiable geometric projection layer into a deep learning network, achieving a deep fusion of physical geometric models and data-driven approaches, and giving the SeqM-Net network clear physical interpretability.

[0056] This invention estimates camera parameters (pitch angle, focal length, installation height, etc.) online using a camera calibration head and a camera height head, eliminating the need for pre-calibration and enabling it to adapt to vibrations and bumps during train operation.

[0057] This invention designs a loss function that includes detection box loss, reprojection loss, pixel width loss, target height consistency loss, camera parameter consistency loss, and prior loss. It makes full use of physical prior (standard width of the weight) and temporal redundancy information, which significantly improves measurement accuracy.

[0058] On a real railway dataset, the method of this invention achieves a mean absolute error of 42.6 mm and a measurement accuracy of 91.3%@50 mm, which is significantly better than traditional geometric methods and pure regression methods.

[0059] This invention handles scenarios where weights overlap by segmenting branches based on instances, suppresses inter-frame fluctuations by using temporal consistency constraints, and avoids continuous regression instability in vehicle vibration scenarios by using a discrete interval prediction strategy, thus exhibiting good environmental adaptability.

[0060] The cross-line tests of this invention show that as the training data covers more railway lines, the model can maintain high measurement accuracy in different railway line scenarios and has good cross-line generalization performance. Attached Figure Description

[0061] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0062] Figure 1 This is a schematic diagram of an overhead contact line compensation device.

[0063] Figure 2 This is a schematic diagram of a monocular vision imaging model;

[0064] Figure 3 This is a diagram of the SeqM-Net network structure of the present invention, wherein, Figure 3 (a) is the structure of the prototype branch. Figure 3 (b) shows the structure of the mask branch;

[0065] Figure 4 This is a network structure diagram of the camera calibration head (CamCalib Head) of the present invention;

[0066] Figure 5 This is a diagram of the internal network structure of the camera height head (CamH Head) of the present invention;

[0067] Figure 6 This is a schematic diagram of the detection results of the present invention in different scenarios. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0069] To address the problems of existing technologies, Embodiment 1 discloses a method for detecting the height of suspended objects off the ground based on monocular vision. This method, based on the fact that the weight has a standard size, first establishes a monocular vision imaging model, then uses deep learning to determine model parameters and achieve target detection of the weight. Finally, a cost function incorporating multiple constraints is designed to accurately detect the height of the weight off the ground. The greatest advantage of this method is that the vision detection system can be installed on high-speed trains to collect image data of the overhead contact line in real time without interrupting railway operations. The detection of the compensator's b-value can then be achieved through video processing, greatly simplifying the system construction.

[0070] The specific process is as follows:

[0071] S1. Acquire continuous T-frame monocular images of the target suspended object of the railway catenary compensation device, which are continuously collected by the monocular camera installed on the train during operation.

[0072] In this step, the target suspended object is the weight. A monocular camera installed on the train continuously captures video sequences containing the weight of the overhead contact line compensation device while the train is in motion. A schematic diagram of the overhead contact line compensation device is shown below. Figure 1 As shown, the weight is composed of multiple weight blocks stacked on a weight frame. The physical width W of each weight block is a standardized known value (e.g., the outer diameter of a circular weight is 360mm). However, when multiple weight blocks are stacked, due to error accumulation, the total height of the weight deviates significantly from the standard value, thus the height of each weight is unknown. The train can continuously capture multiple frames of the same target during operation. To ensure the solvability of the subsequent height measurement problem, a theoretical analysis of the number of acquired frames T is required.

[0073] First, consider the single-frame case. Assume a frame contains only one weight, providing two constraint equations: the weight's base equation and its width equation. However, the unknown variable includes: camera height. (Globally unknown), Camera pitch angle Vertical field of view of the target (Related to focal length f), weight depth Z, weight height above the ground There are a total of 5 unknown variables. The number of constraint equations (2) is less than the number of unknowns (5), therefore a single frame image cannot be solved.

[0074] In practical applications, there are usually two weights: one for compensating for contact wire tension and one for compensating for the catenary. Assuming there are N weights, the number of unknown variables is 2N+3, and the number of constraint equations is 2N. Since the number of constraint equations is still less than the number of unknowns, the case of multiple weights in a single frame image is also unsolvable.

[0075] The train can continuously capture multiple frames of the same target while in operation. The analysis is based on the following facts: 1) the height of the falling weight above the ground. 1) The camera's intrinsic parameters are constant over a short period of time; 2) The camera's intrinsic parameters are unknown but constant.

[0076] For a sequence of falling weights containing T frames, the height of the falling weights above the ground is... Camera installation height The three unknown variables—camera focal length f, camera pitch angle, and camera angle—are the same for all frames of the drop; only the camera pitch angle is unknown. The number of unknowns varies depending on the frame, resulting in a total of T+4 unknowns. Each frame provides two constraints (bottom constraint and width constraint), allowing for the establishment of 2T equations. To ensure the problem has a solution, the following condition must be met: 2T ≥ T+4; that is, the problem can be solved when T ≥ 4. Considering engineering margins and actual detection accuracy requirements, this invention takes T ≥ 4.

[0077] If there are N weights and T frames, there are 3 + N + NT unknowns and 2NT constraint equations. For the problem to have a solution, 2NT ≥ 3 + N + NT and NT ≥ 3 + N must be satisfied. When N = 2, T ≥ 3 is sufficient. In practical applications, two weights are often very close together, so their Z-coordinates can be considered equal, requiring T ≥ 2 frames. However, in practice, because the contact wire compensation weight and the catenary compensation weight are very close, unless they are photographed directly opposite each other, the images of the two weights often overlap, resulting in only a complete image of one weight being visible. Therefore, in practical applications, we treat it as a single weight.

[0078] Through more rigorous mathematical derivation, it can be proven that the problem has a solution when the number of frames T≥3.

[0079] Now let's consider how to achieve T≥3. The number of frames the camera can detect of the same weight depends on three factors: the train speed, the camera's frame rate, and the spacing of the overhead contact line supports. Assuming a fixed frame rate and spacing of the overhead contact line supports, the number of frames the camera can detect of the same weight is primarily affected by the train speed. Assuming a frame rate of 25fps and an effective shooting distance of 10m, to ensure T≥3, we only need to ensure the train speed... A speed of km / h is sufficient to meet the requirements. For trains traveling at speeds exceeding this, the frame rate needs to be increased to meet the requirements.

[0080] The solvability analysis above shows that relying solely on a single frame image is insufficient to accurately determine the height of the weight off the ground. However, by utilizing a continuous multi-frame video sequence collected during train operation, combined with the prior conditions of the weight's standard physical dimensions and constant camera parameters, the constraints for height measurement can be satisfied, thus enabling the effective solution of unknown parameters.

[0081] S2. Process the multiple frames of the monocular images using the trained deep learning network. For example... Figure 3 As shown, the network includes: a backbone network and a feature fusion network (Backbone+Neck), an object bounding box detection head (ObjBBox Head), a camera calibration head (CamCalib Head), a camera height head (CamH Head), and an object height head (ObjH Head). The deep learning network is trained by minimizing a loss function that includes consistency constraints across multiple frames. These consistency constraints include: the estimated ground clearance of the same suspended object remains consistent across different frames, and the camera parameters remain consistent within the same image sequence.

[0082] This deep learning network is an end-to-end monocular height measurement network. It extracts and fuses multi-scale features through the backbone and neck, and feeds them in parallel into the detection head, camera calibration head, and target height head to achieve joint estimation of five key physical quantities: camera height, camera pitch angle, camera focal length (field of view), weight depth, and weight height above the ground. Specifically, the detection head outputs the 2D bounding box of the weight (identifying the weight and the ground reference target and outputting the bounding box) to provide geometric observation constraints. The camera calibration head predicts ground-based camera parameters such as pitch angle and focal length. The camera height head aggregates multi-target constraints based on the TCN network and outputs the camera installation height (i.e., camera height). The target height head calculates the weight height above the ground using imaging geometry formulas and outputs the weight's own geometric features. The deep learning network retains the multi-scale feature fusion capability of the YOLO11 network, extracts deep semantic features through downsampling paths, and fuses information at different scales using upsampling and stitching operations. Meanwhile, the SPPF module in the YOLOv11 network provides a global receptive field, while the C2PSA attention mechanism in the YOLOv11 network enhances the ability to extract features from the weight. This design ensures that the network can both capture local details for accurate localization and understand the global scene to correct measurement errors.

[0083] The core technological advantage of this network lies in its end-to-end integrated design, which combines target detection, key point localization, altitude regression, and online camera calibration. This design not only significantly improves measurement robustness in complex outdoor scenarios and enables real-time processing capabilities, but also greatly enhances the system's environmental adaptability while ensuring industrial-grade accuracy through the mutual promotion of multiple tasks.

[0084] Next, we will describe the implementation process of step S2 in detail.

[0085] S2.1 Detect suspended objects in each frame of monocular image and output 2D detection information.

[0086] This sub-step is implemented using the target bounding box detection head (hereinafter referred to as the "detection head").

[0087] First, the backbone network and the feature fusion network perform multi-scale feature extraction and fusion on the input image. Specifically, the backbone network performs preliminary feature extraction on the input image, while the feature fusion network (Neck) adopts an improved PAFPN structure, effectively fusing multi-scale features from the backbone network through a top-down upsampling path and lateral connections. Its core design includes an SPPF module to enhance multi-scale context awareness, followed by progressive upsampling and low-level feature concatenation (Concat) and convolutional fusion (C3K2), ultimately outputting a first feature map P3, a second feature map P4, and a third feature map P5 at different scales, with the resolution of P3, P4, and P5 gradually decreasing. This structure can both preserve the detailed information of shallow features and incorporate the rich semantics of deep features.

[0088] Then, the detection head (ObjBBox Head) is connected to the outputs of the backbone network and the feature fusion network, outputting 2D detection information based on the first feature map P3, the second feature map P4, and the third feature map P5, respectively. This 2D detection information consists of 2D bounding boxes containing the target suspended object. The three detection heads work in parallel: the P3 detection head has the highest resolution and is suitable for detecting small targets (distant objects); the P4 detection head has medium resolution and is suitable for detecting medium-distance objects; and the P5 detection head has the lowest resolution and is suitable for detecting large targets (near objects). After the prediction results from the three detection heads are processed by non-maximum suppression (NMS), an optimal 2D bounding box is output for each object, providing the bottom ordinate of the object in the image. Top vertical axis and pixel width Information such as...

[0089] Furthermore, when multiple weights overlap in an image (a common situation on railway sites, as contact wire compensation weights and catenary compensation weights are often installed very close together), the deep learning network also includes an instance segmentation branch. This branch generates pixel-level masks to distinguish overlapping weights and precisely locate their edges, thus differentiating individual weights. The instance segmentation branch includes:

[0090] Proto branch: such as Figure 3 As shown in (a), with the first feature map P3 as input, a set of prototype mask templates with a size of B×32×160×160 is generated through upsampling and convolution operations, providing refined spatial features for subsequent processing;

[0091] Mask branch: such as Figure 3As shown in (b), the first feature map P3, the second feature map P4 and the third feature map P5 output by the feature fusion network are used as inputs and compressed into a mask coefficient vector of size B×32×8400 after dimensionality reduction by convolution.

[0092] Weighted combination module: Using the target position and scale information output by the detection head, a set of exclusive mask coefficients is assigned to each detected target. Then, the mask coefficient vector is weighted and combined with the prototype mask template to generate a pixel-level mask for each target suspended object.

[0093] This instance segmentation branch provides pixel-level separation capabilities, avoiding feature confusion caused by bounding box intersections, thereby ensuring the accurate bottom ordinate of each weight. It can be accurately extracted, providing a reliable pixel-level input for subsequent height calculations.

[0094] It should be noted that the two branches of this invention are inherently coupled: the target location and scale information provided by the detection branch assigns a set of unique coefficients to each detected target; then, through a weighted combination of these coefficients and the prototype template, a pixel-level mask of the target is generated. This method not only relies on the detection branch to complete target localization, but also achieves high-precision spatial feature reconstruction through the prototype branch, providing reliable pixel-level support for subsequent point feature extraction and camera calibration.

[0095] S2.2. Based on multi-frame monocular images, estimate the camera parameters and the camera's mounting height relative to the reference plane online.

[0096] This step is achieved through the coordinated use of a camera calibration head and a camera height head.

[0097] (a) Camera calibration head estimates camera parameters:

[0098] The camera calibration head (CamCalib Head) connects to the outputs of the second feature map P4 and the third feature map P5 for online estimation of camera parameters. Designed for railway vehicle scenarios, this camera calibration head enhances the geometric features of the railway scene through coordinate attention (CA) and combines a discrete bin prediction strategy to achieve stable regression of camera calibration parameters, providing accurate geometric constraints for single-view measurement of the height of the bobber from the ground.

[0099] In some embodiments, the camera calibration head includes:

[0100] The lightweight processing module performs depthwise convolution and pointwise convolution on the input third feature map P5 to compress the number of channels to 256.

[0101] The coordinate attention module performs horizontal pooling and vertical pooling on the features of the compressed third feature map P5 to capture the horizontal geometric features of the track and the vertical features of the catenary, respectively. After splicing and fusion and weight generation, the features of the third feature map P5 are enhanced by targeted weighting.

[0102] The feature reuse module bilinearly upsamples the coordinate attention weights derived from the third feature map P5 to the same resolution as the second feature map P4, and performs targeted weighted enhancement on the features of the second feature map P4.

[0103] The feature fusion module adds the enhanced features of the upsampled third feature map P5 to the enhanced features of the second feature map P4 element by element, and aggregates them into a global feature vector through global average pooling.

[0104] The first binning prediction module maps the global feature vector into scores for multiple discrete intervals through a convolutional layer. After Softmax normalization, the scores are weighted and summed to output at least two sets of camera parameters, including the camera pitch angle. and vertical field of view The vertical field of view angle The camera pitch angle is used to calculate the camera focal length f. The camera focal length f is used to calculate the horizon position. .

[0105] like Figure 4 As shown, the camera calibration head first performs lightweight processing on the third feature map P5 (20×20×512): the channels are compressed to 20×20×256 through depthwise convolution and pointwise convolution to adapt to the input of the coordinate attention module (CAM).

[0106] Subsequently, in the coordinate attention layer, the compressed P5 features are subjected to horizontal pooling and vertical pooling to capture the horizontal geometric features of the track and the vertical features of the overhead contact system, respectively. After splicing and fusion and weight generation, the features of the original third feature map P5 are enhanced by directional weighting, and the coordinate attention weights are exported for reuse in the second feature map P4 branch.

[0107] For the second feature map P4 (40×40×256), the coordinate attention weights derived from the third feature map P5 are first bilinearly upsampled to the matching resolution. Then, the details of the weights and catenary are purified by depth convolution. Finally, the adapted coordinate attention weights are used to complete the directional weighted enhancement of the P4 features, ensuring that both branches have geometric priors of the railway scene.

[0108] After completing the dual-branch feature enhancement, the camera calibration head adds the enhanced features of the third feature map P5 (upsampled to 40×40×256) element-wise with the enhanced features of the second feature map P4, achieving a lightweight fusion of global geometric features and mesoscale detail features. The fused features are then aggregated into a 1×1×256 global feature vector through global average pooling, eliminating spatial dimensional redundancy and accurately adapting to parameter prediction requirements.

[0109] The parameter prediction stage employs a discrete interval (Bin) prediction strategy. Two branches are designed to regress the pitch angle separately. (64 discrete intervals, ranging from -π / 12 to +π / 12 rad), vertical field of view (32 discrete intervals, ranging from π / 6 to π / 3 rad). The scores of each discrete interval are output through two layers of 1×1 convolution, and the probability distribution is generated by Softmax. The scores are then weighted and summed to obtain continuous parameter values, thus avoiding the instability of continuous regression in vehicle vibration scenarios.

[0110] Finally, the camera calibration head completes parameter conversion based on the perspective projection formula: using the vertical field of view. The formula for calculating the camera focal length f is: , Image height (in pixels), combined with pitch angle Calculate the horizon pixel position The calculation formula is: , The coordinates are the optical center ordinates. At least two sets of camera parameters output by the camera calibration head. Both are input to the camera elevation sensor and the target elevation sensor respectively, and f is passed through The result is obtained through conversion.

[0111] (ii) Estimating the camera installation height using a camera height measurement tool:

[0112] like Figure 5As shown, the Camera Height Head (CamH Head) contains a TCN temporal convolutional network specifically designed to handle the processing needs of temporal features in continuous images. The core advantage of the TCN network lies in its temporal global modeling capability. It progressively expands the receptive field through multi-layer dilated temporal residual convolutions, effectively capturing inter-frame contextual relationships. Simultaneously, it introduces attention pooling to dynamically filter keyframe features, outputting global temporal features related to the sequence order. Ultimately, this achieves accurate estimation of the camera height, avoiding measurement bias caused by insufficient information in a single frame and improving robustness in train motion scenarios. This module receives camera parameters output from the camera calibration head and, combined with multi-frame 2D detection information output from the detection head, estimates the camera's installation height H relative to a reference plane (ground or track reference plane). The core objective of this module is to accurately estimate the 3D absolute height of the camera from the track reference from a continuous image sequence, providing a key scale reference for calculating the height of the bobber off the ground.

[0113] The camera elevation head input includes: 2D detection information of M suspended targets, and camera parameters output from the camera calibration head as geometric priors. Specifically, the TCN network input layer receives an 8-dimensional geometric feature vector:

[0114] ;

[0115] Where: v0 is the horizon ordinate (from the camera calibration head); and The x-coordinates of the left and right boundaries of the detection box; and The vertical coordinates of the upper and lower boundaries of the detection box; and The offset of the detection box relative to the horizon is a key quantity in perspective geometry, which is directly related to depth information; The physical width of the weight is known (as an absolute scale constraint). The design of this 8-dimensional feature vector embodies the core idea of ​​this invention: using a physical geometric model to guide feature engineering, rather than letting the network blindly learn from raw pixels.

[0116] The core network structure of the camera height head includes the following three parts:

[0117] The Temporal Convolutional Encoder (TCN Encoder) employs three layers of dilated temporal residual convolution blocks with dilation rates set to d=1, 2, and 4, progressively expanding the receptive field to cover the entire input sequence (from 3 frames to 15 frames). Each convolutional block is equipped with residual connections and 1×1 convolutional branches to ensure identity mapping even with dimensionality mismatches, effectively mitigating the vanishing gradient problem. The TCN encodes the input sequence into high-dimensional temporal semantic features.

[0118] Attention Pooling: The multi-frame features output by the TCN are input into the attention pooling layer. An importance score is calculated for each frame using a shared fully connected layer (256→1). After Softmax normalization, the features from each frame are weighted and summed to form a global feature vector. This mechanism dynamically selects frames that contribute significantly to height estimation, improving the model's robustness to disturbances such as vibration and occlusion.

[0119] The second bin prediction module (Discrete Bin Prediction) maps the global feature vector into scores for 256 discrete intervals through a fully connected layer. After Softmax normalization, the center values ​​of each interval are weighted and summed to obtain continuous estimates of the camera mounting height. The discrete interval covers a range of 3.5m to 45m, with an interval width of approximately 3.9mm, achieving a good balance between estimation accuracy and training stability.

[0120] This camera height head structure, while maintaining global temporal modeling capabilities, significantly reduces the number of parameters (from approximately 2.5M to 0.5M), making it more suitable for training on small to medium-sized datasets and able to adapt to camera parameter changes caused by vibration and bumps during train operation. The final output is the camera installation height. Combined with other camera parameters (pitch angle) ,focal length Enter the target height header together to calculate the height of the weight off the ground.

[0121] S2.3 Input the 2D detection information, camera parameters and camera installation height into the differentiable geometry projection layer to calculate the ground clearance of the target suspended object.

[0122] This sub-step is implemented using the Target Height Head (ObjH Head).

[0123] The target height head is the differentiable geometric projection layer. This layer is designed as a differentiable geometric projection layer, strictly following the perspective projection model while maintaining full differentiability to support end-to-end training. All operations in this layer are elementary arithmetic functions (subtraction, division, multiplication, and trigonometric functions with analytic derivatives), without performing any convolution operations or fully connected transformations, and without containing any learnable parameters. It remains a pure geometric computation module, but supports gradient backpropagation through the projection formula, thus allowing a highly consistent loss to supervise the camera parameter estimation and bob detection processes.

[0124] The target elevation head receives the following input:

[0125] The 2D bounding box output by the detection head (providing) , (e.g., parameters)

[0126] All camera parameters output by the camera calibration head ( (f, v0);

[0127] Camera mounting height output by the camera height head .

[0128] The target height head internally executes the perspective projection formula derived in this invention to calculate the height of the weight off the ground. .

[0129] The theoretical formula is derived as follows:

[0130] like Figure 2 As shown, a perspective projection model of a monocular camera is established. Let the camera pitch angle be... The camera mounting height (i.e., camera height) is (Vertical distance from the camera's optical center to the ground). The bottom of the weight is located at a height of... The plane, with the top located at the height The plane, i.e., the ordinate is , Let be the thickness (or height) of the weight. Let the world coordinate system coordinates be... The camera coordinate system coordinates are By using coordinate transformation and camera projection equations, the height of the boulder above the ground can be derived. The calculation formula.

[0131] Figure 1 There are two coordinate systems: the world coordinate system and the camera coordinate system. Initially, the two coordinate systems coincide, with the Z-axis pointing to the left, the Y-axis pointing upwards, and the X-axis perpendicular to the paper and pointing outwards. The transformation relationship between the two coordinate systems is as follows:

[0132] ;

[0133] Assume the camera's focal length is The coordinates of the optical center are Then the camera projection equation is:

[0134] ;

[0135] Height of the weight off the ground The calculation formula is:

[0136] ;

[0137] ;

[0138] ;

[0139] .

[0140] in: f is the height of the droplet off the ground (final output); f is the camera focal length, determined by the vertical field of view. The conversion yields: ; The physical width of the weight is known (prior knowledge, such as 360mm); This is the camera tilt angle (from the camera calibration head); The pixel width of the weight in the image ( ); Set the camera height (from the camera height head); The vertical coordinate of the bottom of the weight in the image (from the detection head); The ordinate of the camera's optical center is given.

[0141] The derivation of the formula is as follows: First, a camera imaging model is established, transforming points in the world coordinate system to the camera coordinate system. Then, the image coordinates are obtained through the camera projection equation. The known physical width W of the weight is used as a geometric constraint, combined with the bottom ordinate provided by the detection box. and pixel width The height of the weight off the ground can be determined by algebraic transformations. .

[0142] The target height head calculates the ground clearance of each weight using the formula described above. Since this layer is fully differentiable, the gradient can be backpropagated to all front-end modules (detection head, camera calibration head, camera height head), achieving end-to-end joint optimization.

[0143] S3, The training process of deep learning networks.

[0144] The deep learning network described above is trained by minimizing a loss function that includes consistency constraints across multiple frames. The total loss function is a weighted combination of the various losses:

[0145] ;

[0146] , , , , , These are the weights corresponding to the loss function.

[0147] (1) Detection frame loss :

[0148] The output of the supervisory detection head includes classification loss and bounding box regression loss. The detection head predicts both bounding boxes and classes simultaneously within the YOLO11 framework. For scenarios where two weights may overlap in an image, instance segmentation is introduced to provide pixel-level separation, avoiding feature confusion caused by bounding box intersections, thus ensuring the accurate determination of the bottom y-coordinate of each weight. It can be extracted precisely.

[0149] (2) Reprojection loss :

[0150] To ensure the predicted 3D geometric parameters To ensure consistency with 2D observation, a reprojection loss is introduced. The predicted 3D information is reprojected onto the image plane using the perspective projection formula, and the pixel error between the projected 2D bounding box and the target region derived from instance segmentation is calculated.

[0151] ;

[0152] This represents the predicted value of the vertical coordinate of the bottom of the weight in the t-th frame of the image. This represents the true value of the vertical coordinate of the bottom of the weight in frame t, which is usually obtained from the annotation. According to T represents the number of frames for the same falling weight.

[0153] The lower the loss, the better the network-predicted geometric parameters can explain the observations in the image, making it a key surrogate metric for evaluating the performance of the camera calibration module. In real-world railway scenarios where true camera parameter values ​​are unavailable, this is the most crucial and reliable surrogate metric for assessing the performance of the "camera calibration" module.

[0154] (3) Pixel width loss

[0155] The physical width W of the weight is a known standard value and can serve as an important geometric prior constraint for network learning. The pixel width loss is defined as the difference between the predicted pixel width and the actual width of the detection box.

[0156] ;

[0157] in, Based on the known physical width W (i.e. The predicted pixel width of the weight is obtained from the depth Z and focal length f, where the predicted pixel width of the weight is... It can be done through formula get, That is, in the formula Depth Z can be expressed by the formula get.

[0158] (4) Loss of high consistency of objectives

[0159] The predicted ground clearance of the same weight should be the same across multiple consecutive frames. Let the predicted ground clearance of the same weight across T consecutive frames be... The loss of target high consistency is defined as:

[0160] ;

[0161] Let be the predicted height of the falling weight above the ground in the t-th frame image. This loss function utilizes geometric constraints between multiple frames to effectively reduce inter-frame fluctuations, and is one of the key innovations of this invention in solving the problem of scale ambiguity in monocular vision.

[0162] (5) Consistency of camera parameters :

[0163] For images in the same sequence, the camera parameters should remain consistent. Let the camera parameter vector be... (After normalization), the camera parameter consistency loss is defined as:

[0164] This loss ensures that the camera height, focal length, and pitch angle parameters remain constant within the same video sequence, further aligning with physical priors. That is, the camera parameter vector consists of the pitch angle, vertical field of view, and camera height.

[0165] (6) Prior loss :

[0166] To prevent network predictions from exceeding reasonable physical limits, a prior loss is introduced to impose soft constraints on camera parameters and the height of the boulder. The ReLU function is used to limit the parameter boundaries, and an L1 loss is combined to make them closer to the prior values.

[0167] ;

[0168] This represents the maximum reasonable value for the camera installation height. The minimum reasonable installation height for the camera. Predicted camera mounting height. This is the camera focal length (predicted value). This represents the nominal / standard value of the camera's focal length.

[0169] All loss terms are differentiable L1 / L2 class losses, with stable gradient backpropagation, adaptable to end-to-end joint optimization. Through the above multi-constraint joint optimization, the SeqM-Net network of this invention can simultaneously achieve online camera calibration and accurate estimation of the boulder's height off the ground with only 2D bounding box annotation.

[0170] S4. Anomaly detection and alarm.

[0171] The height of the weight off the ground calculated in S2.3 The height of the falling weight off the ground is compared with a preset safety threshold. According to railway safety regulations, it is generally required that the height of the falling weight off the ground should not be less than 200mm under any circumstances. Therefore, this invention presets the safety threshold to 200mm.

[0172] when If the temperature drops below a certain threshold (mm), the contact wire compensation device is deemed to be in an abnormal state (the compensation function may fail due to a sinker hitting the bottom), and an alarm signal is output; otherwise, the state is deemed to be normal.

[0173] The judgment result can be presented to the operation and maintenance personnel in real time through the output unit (such as vehicle display screen, sound and light alarm or wireless communication module), or uploaded to the ground monitoring center to realize remote intelligent monitoring of the status of the entire contact network compensation device.

[0174] When the system detects an anomaly, it can also record information such as the time and location of the anomaly (via GPS / BeiDou positioning) and the height of the falling weight off the ground, forming an alarm log to provide data support for subsequent maintenance.

[0175] Next, specific experimental data will be used to demonstrate the effectiveness of the method of the present invention.

[0176] 1. Experimental setup

[0177] (1) Introduction to the dataset

[0178] The dataset for this study comes from real-time video streams of overhead contact line compensation devices captured by train-mounted front cameras. The video frame rate is fixed at 25 FPS, covering multiple main railway lines. Data acquisition is based on continuous video sequences as the smallest unit. The same compensation device weight is continuously captured in 7-12 frames of time-series images, forming video sequence samples in units of "support-weight", rather than independent scattered images.

[0179] The original video was approximately 7.5 hours long. After frame extraction, 35,720 valid images were obtained. These were clustered according to the targets of the overhead contact line supports and the falling weights, resulting in 3,610 video sequences. Videos from each line were processed separately, divided into training, validation, and test sets in a ratio of 7:2:1. The final division results were as follows: Training set: 2,527 image sequences, used for model parameter learning and optimization; Validation set: 722 image sequences, used for hyperparameter tuning, model selection, and overfitting monitoring during training; Test set: 361 image sequences, serving as "black box" data that was not involved in the training process, used to evaluate the model's generalization performance and practical value. All images in the test set contained the true height of the falling weights above the ground, measured using a laser rangefinder.

[0180] (2) Evaluation indicators

[0181] The evaluation metrics are divided into primary metrics and secondary metrics. The primary metrics are used to measure the height of the weight. Height measurement is the final output of this paper, and its accuracy directly determines the system's usability in practical engineering. Therefore, we use the following metrics to quantitatively evaluate the height prediction results:

[0182] 1) Mean Absolute Error: MAE

[0183] ;

[0184] This indicates the predicted height of the falling weight. This indicates the actual height of the weight.

[0185] MAE can intuitively reflect the average deviation between the predicted value and the actual value, has good interpretability, and is a core indicator for measuring the accuracy of height measurement.

[0186] 2) Root Mean Square Error: RMSE

[0187] RMSE is more sensitive to large errors and can effectively reflect the stability of the model and its performance under extreme conditions, making it suitable for evaluating the reliability of a system in practical applications. It is more sensitive to relatively large errors.

[0188] 3) The proportion of absolute errors less than the threshold T: Accuracy@T (e.g., Acc@5mm, Acc@10mm).

[0189] This paper selects two thresholds, T=50mm and T=100mm, which correspond to high-precision control and the allowable error range in engineering, respectively, and can intuitively reflect the applicability of the method in actual operation and maintenance.

[0190] Since it is difficult to obtain the true values ​​of camera parameters (such as focal length, pitch angle, and camera height) in real-world railway scenarios, this paper introduces geometric consistency metrics to indirectly evaluate the quality of the network's estimation of geometric parameters. These metrics can reflect whether the model "understands" the three-dimensional spatial structure in a geometric sense.

[0191] 4) Reprojection Error

[0192] The 3D bounding box of the falling object is reprojected onto the 2D image plane using camera parameters predicted by the network and the target depth. The pixel error between this reprojection and the detection box is then calculated. A lower error indicates that the network-predicted geometric parameters better explain the observations in the image, making it a key surrogate metric for evaluating the performance of the camera calibration module. The reprojection error is defined as follows:

[0193] ;

[0194] Here, N is the total number of samples.

[0195] 5) Width prediction error: using the predicted f and Given the known physical width W, calculate the predicted pixel width and compare it with the detection box width.

[0196] ;

[0197] and The lower the value, the better the network-estimated set of geometric parameters can explain the 2D observations, suggesting that its camera parameter estimates are more likely to be close to the true values. This is the most crucial and reliable alternative indicator for evaluating the performance of the "camera calibration" module when true values ​​are unavailable.

[0198] (3) Implementation details

[0199] The model is a customized sequential multi-task development based on the YOLO11m-seg architecture. The optimizer used is AdamW, and the initial learning rate is set to... The weight decays to The learning rate scheduling employs a strategy of three rounds of linear warm-up combined with cosine annealing, balancing stability in the early stages of sequence training with convergence performance in the later stages. Video sequences are used as the smallest training unit, with each batch containing only a single complete sequence of images to ensure globally constant camera parameters. The batch size is adaptively set based on GPU memory, with a total of 300 training rounds. An early stopping mechanism is enabled, terminating training when the mean absolute error of the validation set sequence shows no improvement for 20 consecutive rounds to avoid overfitting. Mixed-precision training is used during the training phase, sharing backbone network feature extraction results across multiple frames of the same sequence, reducing GPU memory usage while improving sequence data processing efficiency.

[0200] 2. Experimental Results and Analysis

[0201] (1) Comparison of results

[0202] This invention compares the proposed method with several representative baseline methods. PGM (Probabilistic Graphical Model) is a traditional geometric method. PGM first inputs a single-frame image of the falling weight into the model, using a probabilistic graphical model to jointly infer the image horizon position, camera mounting height, and the physical height of the falling weight. Then, based on perspective projection geometry, it converts the image coordinates into the actual physical height and calculates the weight's b-value. Monocular visual regression directly uses a deep network to regress the weight height from the image's ROI region, without explicitly modeling geometric relationships. Table 1 shows a comparison of the measurement accuracy metrics of the proposed method and the comparison methods on the test set.

[0203] Table 1 Comparison of Height Measurement Accuracy

[0204] PGM 95.2 138.5 0.39 0.64 0.1740 6.63 Regressing 52.4 67.2 0.77 0.83 － － SeqM-Net 42.6 48.14 0.91 0.97 0.0065 1.24

[0205] As can be seen, the proposed method significantly outperforms other comparative algorithms in all evaluation metrics: the mean absolute error is 42.6 mm and the root mean square error is 48.14 mm. Compared with the YOLO1 regression algorithm, the two errors are reduced by 18.7% and 28.2%, respectively; compared with the traditional projective geometric matching algorithm, the errors are reduced by 55.2% and 65.2%, respectively.

[0206] In terms of accuracy, the proposed method achieves an accuracy rate of 92.3% for 50 mm and 96.8% for 100 mm, meeting the engineering accuracy standards for overhead contact line inspection. In contrast, traditional projection geometry matching and regression algorithms only achieve 50 mm accuracy rates of 38.5% and 77.2%, respectively, making them unsuitable for practical engineering applications.

[0207] Table 1 shows that the proposed method outperforms the regression method, which in turn outperforms the traditional geometric method. This is because the traditional geometric method calculates height based on known parameters such as camera focal length, installation height, and pitch angle, and requires these parameters to remain constant throughout the entire process. In a railway vehicle scenario, the vibrations and bumps caused by train movement can lead to slight deviations in the camera pitch angle and installation height, causing the calibrated parameters to quickly become invalid and resulting in systematic errors in the height calculated by the formula. The regression network, however, autonomously learns the mapping relationship between "imaging features after slight changes in camera parameters" and "height" from a large number of labeled vehicle samples, naturally offsetting the errors caused by parameter changes. Compared to the regression method, the proposed method incorporates prior width constraints in the regression process, further improving detection accuracy.

[0208] (2) Module performance analysis

[0209] This invention employs YOLOv11 as the weight detection module, whose core function is to output the bounding box of the weight, providing pixel-level input for subsequent calculation of its height above the ground. This section comprehensively evaluates the performance of the detection module from two aspects: detection accuracy and error sensitivity, to verify its reliability as a system input.

[0210] 1) Detection accuracy assessment

[0211] To verify the positioning accuracy and reliability of the weight detection module, this paper uses common indicators in the target detection field for evaluation. As shown in Table 2, the YOLO11 detection model used in this paper achieves a Map@0.5 (mean accuracy with a threshold of 0.5) of 99.2% and an accuracy of 96% on the test set. It can achieve accurate positioning and stable detection of weight targets and provide reliable pixel-level input for subsequent height measurement.

[0212] Table 2 Target Detection Accuracy Evaluation

[0213] numerical values 99.2% 93.7% 95.7%

[0214] 2) Ablation Experiment Analysis

[0215] This invention designed ablation experiments with progressively increasing constraints to verify the effectiveness of each module. First, a single-frame model without consistency constraints was used as the baseline. Then, multi-frame high consistency constraints and camera parameter consistency constraints were sequentially added. All comparative experiments maintained consistency in network structure, dataset, and training strategy, adjusting only the constraint loss to ensure strict control of variables. Experimental results clearly show that high consistency constraints effectively reduce inter-frame fluctuations, and camera parameter consistency constraints further align with physical priors. Finally, the complete model achieves optimal performance, as shown in Table 3.

[0216] Table 3 Ablation Experiment Results

[0217] Baseline (detection bounding box only + fixed camera parameters) 78.76 24.82 61.2 Baseline + High Consistency of Objectives 67.32 17.15 73.6 Baseline + Target Height Consistency + Camera Parameter Consistency 42.64 7.4 91.3

[0218] (3) Generalization test

[0219] Figure 6 The test results are presented under different scenarios. Figure 6 (a) and Figure 6 (b) The routes are different, but the scenarios are similar. The two weights overlap. The two weights can be accurately detected by instance segmentation, thereby realizing the detection of the height of the weights off the ground. Figure 6 (c) The two weights almost completely overlap, but this has no effect on the test results; Figure 6 (d) shows the detection results at the track bend. In this case, the image needs to be corrected by first calculating the track inclination angle based on the vehicle's data file before detection to obtain accurate results.

[0220] To verify the cross-line generalization ability of the proposed SeqM-Net model, experiments were conducted to measure the height of the weights off the ground across the lines under different training data volumes. The mean absolute error (MAE, unit: mm) was used as the evaluation index, and the results are shown in Table 4. When trained using only data from line A, the model's mean squared errors (MAE) on test sets A, B, and C were 43.8mm, 55.0mm, and 53.0mm, respectively, with an average MAE of 49.8mm, demonstrating a certain cross-line detection capability. However, due to the limited training data, its adaptability was restricted. After adding data from line B to the training set, the model's MAE on all three test lines decreased to 40.6mm, 44.2mm, and 51.5mm, respectively, with an average MAE of 45.4mm, significantly improving cross-line detection accuracy. After training with all data from lines A, B, and C, the model achieved optimal measurement accuracy on each test set, with MAEs of 40.5mm, 42.8mm, and 44.5mm, respectively, and an average MAE of only 42.6mm. Experimental results show that as the number of training data lines increases, the model's adaptability to different railway line scenarios is significantly enhanced. Even in line scenarios not included in the training, it can still maintain high measurement accuracy, proving that the proposed method has good cross-line generalization performance and can adapt to the detection requirements of catenary compensation devices on different trunk railways.

[0221] Table 4 Cross-line generalization test (MAE, mm)

[0222] Line A 43.8 55.0 53.0 49.8 Line A+B 40.6 44.2 51.5 45.4 Line A+B+C (Full Quantity) 40.5 42.8 44.5 42.6

[0223] In summary, the end-to-end geometric perception framework based on monocular vision and deep learning in this invention differs from existing single-view measurement methods that primarily focus on the object's own height. For the first time, it redefines the problem as estimating the vertical distance from the bottom of a suspended object to the ground and establishes a complete geometric projection model that associates the 3D height above the ground with the 2D detection bounding box. By introducing a differentiable geometric projection layer, weakly supervised reprojection loss, and temporal consistency constraints into the YOLO11 detection backbone, our method can simultaneously achieve online camera calibration and accurate estimation of the weight's height above the ground, requiring only 2D bounding box annotation.

[0224] Experimental results show that the proposed method achieves a mean absolute error of 42.6 mm and an accuracy within 10 mm of 94.7% on a real railway inspection dataset, significantly outperforming traditional geometric methods. Ablation experiments validate the effectiveness of each core module. These experimental results demonstrate that embedding geometric priors into a deep learning framework is an effective approach to solving monocular measurement tasks in specific scenarios, ensuring both data efficiency and the model's generalization ability.

[0225] The loss function employs a weighted combination of detection box loss, reprojection loss, pixel width loss, multi-frame height consistency loss, camera parameter consistency loss, and prior loss. This strengthens the temporal constraints of consistent height across multiple consecutive frames of the same weight and constant camera height, focal length, and pitch angle parameters within the same sequence. All loss terms are differentiable and gradient backpropagation is stable, adapting to end-to-end joint optimization of sequences. Addressing the differences in camera installation height, focal length, and pitch angle across different acquisition devices, camera parameters are calculated online via the CamCalib Head and camera height estimation module, eliminating the need for pre-calibration. Independent parameter estimation is allowed between different sequences, with constraints only on constant parameters within the same sequence. Simultaneously, scale normalization, distortion correction, and motion compensation preprocessing are uniformly applied to the sequence images, avoiding enhancement methods that disrupt geometric relationships. This ensures the effectiveness of perspective projection constraints, enabling the model to exhibit good generalization capabilities across lines and devices.

[0226] Example 2 discloses a monocular vision-based system for detecting the height of a suspended object off the ground, used to execute the monocular vision-based method for detecting the height of a suspended object off the ground described in Example 1, including:

[0227] The image acquisition module is used to acquire continuous multi-frame monocular images containing the target suspended object;

[0228] A deep learning processing module, internally storing a trained deep learning network, is used to process the multi-frame images and output the height of the target suspended object relative to a reference plane. The processing includes: detecting the target suspended object in each frame of the monocular image and outputting 2D detection information; estimating the camera parameters and the camera's mounting height relative to the reference plane online based on the multi-frame monocular images; and inputting the 2D detection information, the camera parameters, and the camera mounting height into a differentiable geometric projection layer, which uses the known physical dimensions of the target suspended object as geometric constraints to calculate the height above the ground.

[0229] The deep learning network is trained by minimizing a loss function that includes consistency constraints between multiple frames of images. The consistency constraints include: the estimated ground clearance of the same suspended object remains consistent in different frames, and / or the camera parameters in the same image sequence remain consistent.

[0230] Example 3 discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a method for detecting the height of a suspended object from the ground based on monocular vision as described in Example 2.

[0231] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting the height of a suspended object above the ground based on monocular vision, characterized in that, include: Acquire continuous T-frame monocular images containing the target suspended object of the railway catenary compensation device, which are continuously collected by a monocular camera installed on the train during operation; The trained deep learning network is used to process multiple frames of the monocular images, the processing including: The system detects suspended objects in each frame of the monocular image and outputs 2D detection information. Based on multiple frames of the monocular image, it estimates the camera parameters and the camera's mounting height relative to a reference plane online. The 2D detection information, the camera parameters, and the camera mounting height are input into a differentiable geometric projection layer. The differentiable geometric projection layer uses the known standard physical width of the suspended object as a geometric constraint to calculate the height of the suspended object above the ground. The deep learning network is trained by minimizing a loss function that includes consistency constraints between multiple frames of the monocular images. The consistency constraints include: the estimated ground clearance of the same suspended object remains consistent across different frames, and the camera parameters remain consistent within the same image sequence.

2. The method for detecting the height of a suspended object above the ground based on monocular vision according to claim 1, characterized in that, The continuous multiple frames of the monocular image are video sequences continuously captured by the onboard monocular camera during train operation; wherein, the number of frames T in which the same suspended target is continuously captured in the video sequence is ≥3.

3. The method for detecting the height of a suspended object above the ground based on monocular vision according to claim 1, characterized in that, The deep learning network is an end-to-end monocular height measurement network, SeqM-Net, which includes: The backbone network and feature fusion network are used to extract and fuse multi-scale features from the input image, and output first feature map P3, second feature map P4 and third feature map P5 at different scales, wherein the resolution of the first feature map P3, second feature map P4 and third feature map P5 gradually decreases. The detection head is connected to the output of the backbone network and the feature fusion network, and outputs 2D detection information based on the first feature map P3, the second feature map P4 and the third feature map P5 respectively. The 2D detection information is a 2D bounding box containing the target suspended object. The camera calibration head is connected to the output terminals of the second feature map P4 and the third feature map P5, and outputs two sets of camera parameters. These two sets of camera parameters are respectively input to the camera elevation head and the target elevation head. The two sets of camera parameters are, respectively, the camera pitch angle. and vertical field of view ; The camera height head contains a TCN network, which receives camera parameters output from the camera calibration head and combines them with 2D detection information output from the detection head to estimate the camera's mounting height relative to a reference plane, i.e., the camera mounting height. ; The target height head, which is the differentiable geometric projection layer, receives the 2D bounding box output by the detection head, all camera parameters output by the camera calibration head, and the camera installation height output by the camera height head, and calculates the ground height of the target suspended object using the perspective projection formula.

4. The method for detecting the height of a suspended object above the ground based on monocular vision according to claim 3, characterized in that, The camera calibration head includes: The lightweight processing module performs depthwise convolution and pointwise convolution on the input third feature map P5 to compress the number of channels to 256. The coordinate attention module performs horizontal pooling and vertical pooling on the features of the compressed third feature map P5 to capture the horizontal geometric features of the track and the vertical features of the catenary, respectively. After splicing and fusion and weight generation, the features of the third feature map P5 are enhanced by targeted weighting. The feature reuse module bilinearly upsamples the coordinate attention weights derived from the third feature map P5 to the same resolution as the second feature map P4, and performs targeted weighted enhancement on the features of the second feature map P4. The feature fusion module adds the enhanced features of the upsampled third feature map P5 to the enhanced features of the second feature map P4 element by element, and then aggregates them into a global feature vector through global average pooling. The first binning prediction module maps the global feature vector into scores for multiple discrete intervals through a convolutional layer. After Softmax normalization, the scores are weighted and summed to output at least two sets of camera parameters, including the camera pitch angle. and vertical field of view The vertical field of view angle The camera pitch angle is used to calculate the camera focal length f. The camera focal length f is used to calculate the horizon position. .

5. The method for detecting the height of a suspended object above the ground based on monocular vision according to claim 3, characterized in that, The camera height sensor includes: The temporal convolutional encoder takes an 8-dimensional geometric feature sequence of T consecutive frames as input. The 8-dimensional geometric features include the horizon ordinate, the left and right boundary x-coordinates of the detection box, the top and bottom boundary ordinates of the detection box, the offset of the detection box relative to the horizon, and the known physical width of the weight. The temporal convolutional encoder uses three layers of dilated temporal residual convolutional blocks with dilation rates set to d=1, 2, and 4, respectively, to gradually expand the receptive field to cover the entire input sequence. Each convolutional block is equipped with residual connections and 1×1 convolutional branches to encode the input sequence into high-dimensional temporal semantic features. The attention pooling layer takes the multi-frame features output by the temporal convolutional encoder as input, calculates the importance score for each frame through a shared fully connected layer, and then performs a weighted summation of the features of each frame after Softmax normalization, and aggregates them into a global feature vector. The second binning prediction module maps the global feature vector into scores for multiple discrete intervals through a fully connected layer. After Softmax normalization, the center values ​​of each interval are weighted and summed to output the camera's mounting height relative to the reference plane. .

6. The method for detecting the height of a suspended object above the ground based on monocular vision according to claim 1, characterized in that, The deep learning network is trained by minimizing the following loss function: Detection box loss is used to supervise the 2D detection information of the target suspended object; Reprojection loss is used to ensure that the estimated camera parameters are consistent with the 2D observation information; Pixel width loss is achieved by using the known physical width of the target suspended object as a monitoring signal; Target height consistency loss is used to ensure that the estimated ground clearance of the same suspended target remains consistent across different frames. Camera parameter consistency loss is used to constrain camera parameters to remain consistent across the same image sequence. Prior loss is used to impose reasonable physical range constraints on camera parameters and / or ground clearance.

7. The method for detecting the height of a suspended object above the ground based on monocular vision according to claim 1, characterized in that, The deep learning network also includes an instance segmentation branch, which generates a pixel-level mask when multiple target suspended objects overlap in the image. The instance splitting branches include: The prototype branch takes the first feature map P3 as input and generates a set of prototype mask templates through upsampling and convolution operations. The mask branch takes the first feature map P3, the second feature map P4, and the third feature map P5 output by the feature fusion network as input, and generates a mask coefficient vector through convolution dimensionality reduction and compression. The weighted combination module uses the position and scale information of the target suspended object output by the detection head to assign a set of exclusive mask coefficients to each detected target suspended object. The mask coefficient vector is then weighted and combined with the prototype mask template to generate a pixel-level mask for each target suspended object.

8. The method for detecting the height of a suspended object above the ground based on monocular vision according to claim 1, characterized in that, The target suspended object is a weight of a railway catenary compensation device, and the reference plane is the ground or track reference plane; the method further includes: The calculated ground clearance is compared with a preset safety threshold. When the ground clearance is lower than the safety threshold, the contact wire compensation device is determined to be in an abnormal state and an alarm signal is output.

9. A system for detecting the height of a suspended object off the ground based on monocular vision, used to execute the method for detecting the height of a suspended object off the ground based on monocular vision as described in any one of claims 1-8, characterized in that, include: The image acquisition module is used to acquire continuous multi-frame monocular images containing the target suspended object; A deep learning processing module, internally storing a trained deep learning network, is used to process multiple frames of the monocular images and output the ground clearance of the target suspended object relative to a reference plane. The processing includes: detecting the target suspended object in each frame of the monocular image and outputting 2D detection information; estimating the camera parameters and the camera's mounting height relative to the reference plane online based on multiple frames of the monocular images; and inputting the 2D detection information, the camera parameters, and the camera mounting height into a differentiable geometric projection layer, which uses the known physical dimensions of the target suspended object as geometric constraints to calculate the ground clearance. The deep learning network is trained by minimizing a loss function that includes consistency constraints between multiple frames of images. These consistency constraints include: the estimated ground clearance of the same suspended object remains consistent across different frames, and the camera parameters remain consistent within the same image sequence.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a method for detecting the height of a suspended object off the ground based on monocular vision, as described in any one of claims 1 to 8.

Images