Railway multi-professional account mileage mapping method and device based on machine vision

Abstract

The application discloses a railway multi-specialty account mileage mapping method and device based on machine vision, wherein the method comprises the following steps: acquiring image data and mileage count along a railway line; identifying various professional equipment by using the image, and outputting identified various professional equipment information; the equipment information comprises a time stamp; based on the time stamp of the equipment information, the mileage count is matched and fused with the equipment, and a fused mileage is output; the fused mileage comprises a corresponding relationship between the mileage and various professional equipment; the equipment in the fused mileage is matched with the equipment in the existing account of various professional equipment, and a matching pair is output; the matching pair comprises a corresponding relationship between the mileage of the equipment in the existing account of various professional equipment and the mileage of the equipment in the fused mileage; based on the matching pair, a mapping relationship between the fused mileage and the equipment-level mileage of various professional equipment is established, and a multi-specialty mileage account under a unified reference system is output based on the mapping relationship. The application can realize accurate mileage alignment between different professional accounts.

Description

Technical Field

[0001] This invention relates to the field of railway engineering technology, and in particular to a method and apparatus for mapping mileage in multi-disciplinary railway ledgers based on machine vision. Background Technology

[0002] Railway system ledger data is a collection of various information recorded in the operation, management, and maintenance of railways. The railway system encompasses multiple specialties such as track maintenance, signaling, and power supply. Currently, each specialty establishes its own ledgers based on its operational requirements. However, the business focus and operational processes of different specialties differ significantly. The mileage of key equipment in each specialty is determined by its respective ledger, but due to the distinct specialization of each specialty, not only are the starting points for mileage in the ledgers inconsistent, but the measurement methods and error models for equipment locations also differ, resulting in fragmented mileage calculation systems across different specialty ledgers. The consequence is that the mileage corresponding to the same physical location varies across different specialty ledgers, making it difficult to interoperate and reuse ledgers between specialties. A specialty cannot quickly extrapolate similar mileage from another specialty's ledger. Furthermore, when conducting mileage statistics and comprehensive analysis of equipment across multiple specialties, data from each specialty is difficult to summarize and align within the same mileage reference system, severely restricting the efficiency and quality of cross-specialty collaboration, comprehensive judgment, and integrated operation and maintenance management.

[0003] Therefore, there is an urgent need for a multi-disciplinary ledger mileage mapping method to achieve accurate mileage alignment and comprehensive application between different professional ledgers. Summary of the Invention

[0004] This invention provides a machine vision-based method for mapping mileage in multi-disciplinary railway ledgers, used to construct a unified, objective, and cross-checkable spatial reference and mapping mechanism to achieve accurate mileage alignment between ledgers of different specialties. The method includes:

[0005] Acquire image data and mileage counts along the railway line;

[0006] The system uses images to identify various specialized equipment and outputs information about each identified piece of equipment, including a timestamp.

[0007] Based on the timestamp of the equipment information, the mileage count is matched and merged with the equipment to output the merged mileage; the merged mileage includes the correspondence between mileage and various professional equipment.

[0008] The system matches the equipment in the merged mileage with the equipment in the existing ledgers of each specialty, and outputs matching pairs. The matching pairs include the correspondence between the mileage of the equipment in the existing ledgers of each specialty and the mileage of the equipment in the merged mileage.

[0009] Based on the matching pairs, a mapping relationship is established between the fused mileage and the equipment-level mileage of each specialty, and a multi-professional mileage ledger under a unified reference system is output based on the mapping relationship.

[0010] This invention also provides a machine vision-based railway multi-professional ledger mileage mapping device to construct a unified, objective, and cross-checkable spatial reference and mapping mechanism, achieving accurate mileage alignment between different professional ledgers. The device includes:

[0011] The data acquisition module is used to acquire image data and mileage counts along the railway line;

[0012] The target recognition module is used to identify various specialized equipment using images and output information about the identified specialized equipment; the equipment information includes a timestamp.

[0013] The mileage calculation module is used to match and merge mileage counts with equipment based on the timestamps of equipment information, and output the fused mileage; the fused mileage includes the correspondence between mileage and various specialized equipment;

[0014] The equipment ledger matching and calculation module is used to match the equipment in the integrated mileage with the equipment in the existing ledgers of each specialty, and output matching pairs. The matching pairs include the correspondence between the mileage of the equipment in the existing ledgers of each specialty and the mileage of the equipment in the integrated mileage. Based on the matching pairs, a mapping relationship between the integrated mileage and the equipment-level mileage of each specialty is established, and a multi-professional mileage ledger under a unified reference system is output based on the mapping relationship.

[0015] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described machine vision-based railway multi-disciplinary ledger mileage mapping method.

[0016] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described machine vision-based railway multi-disciplinary ledger mileage mapping method.

[0017] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described machine vision-based railway multi-disciplinary ledger mileage mapping method.

[0018] Compared to existing technologies where each railway specialty uses its own ledger, resulting in fragmented ledger systems, this invention first utilizes image data and mileage technology to output a fused mileage that includes the correspondence between mileage and equipment of each specialty. Then, the fused mileage is matched with the ledgers of each specialty to establish a mileage mapping relationship between the fused mileage and the equipment level of each specialty. Based on this mapping relationship, a multi-specialty mileage ledger under a unified reference system is output, constructing a unified, objective, and mutually verifiable spatial reference and mapping mechanism. This achieves mileage unification and mutual conversion among specialties such as track maintenance, signaling, and power supply, eliminating the fragmentation of mileage across specialties. Ledgers of each specialty can be linked for retrieval, statistics, and joint analysis under a unified reference system, quickly aligning and confirming accurate mileage information between different specialty ledgers, supporting multi-specialty collaborative operation and maintenance and resource optimization. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0020] Figure 1 This is a flowchart illustrating the multi-disciplinary railway ledger mileage mapping method based on machine vision in an embodiment of the present invention.

[0021] Figure 2 This is a specific example of a machine vision-based railway multi-professional ledger mileage mapping method in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the processing flow of the target detection network in an embodiment of the present invention;

[0023] Figure 4 This is a structural diagram of the target detection network in an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of the fusion mileage calculation process in an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of the ledger equipment matching calculation process in an embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram showing the mapping result between the electrical transponder ledger and the overhead contact line ledger mileage in an embodiment of the present invention;

[0027] Figure 8 This is a schematic diagram of a railway multi-professional ledger mileage mapping device based on machine vision in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0029] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0030] Existing technologies suffer from fragmented mileage systems due to inconsistent starting points, varying measurement methods, and error models across multiple disciplines such as track maintenance, signaling, and power supply. This results in systematic deviations for the same physical equipment across different discipline ledgers, hindering cross-discipline alignment and reuse. To address the lack of a unified, objective spatial reference and robust equipment-level matching mechanism in existing methods, which prevents the convergence and application of multi-discipline data under a single mileage reference system, this invention provides a machine vision-based method and apparatus for mapping mileage across multiple railway discipline ledgers.

[0031] Figure 1 This is a flowchart illustrating the machine vision-based railway multi-disciplinary ledger mileage mapping method in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0032] Step 101: Acquire image data and mileage counts along the railway line;

[0033] Step 102: Use the image to identify various professional equipment and output the identified professional equipment information; the equipment information includes a timestamp;

[0034] Step 103: Based on the timestamp of the equipment information, match and merge the mileage count with the equipment to output the merged mileage; the merged mileage includes the correspondence between mileage and various professional equipment.

[0035] Step 104: Match the devices in the merged mileage with the devices in the existing ledgers of each specialty, and output the matching pairs; the matching pairs include the correspondence between the mileage of the devices in the existing ledgers of each specialty and the mileage of the devices in the merged mileage.

[0036] Step 105: Based on the matching pairs, establish a mapping relationship between the fused mileage and the equipment-level mileage of each specialty, and output a multi-specialty mileage ledger under a unified reference system based on the mapping relationship.

[0037] The following is a detailed explanation of the machine vision-based railway multi-disciplinary ledger mileage mapping method in the embodiments of the present invention.

[0038] First, acquire image data and mileage counts along the railway line. For example, acquire image data and mileage counts along the railway line using a railway inspection vehicle.

[0039] In one embodiment, the image data may include area array images and line array images, and the mileage count includes area array visual mileage count and wheel pulse encoder mileage count; the area array visual mileage count is based on visual mileage count formed from area array images.

[0040] Furthermore, step 102, which uses images to identify various professional equipment and outputs the identified professional equipment information, may include: using area array images and line array images to identify various professional equipment and outputting the identified professional equipment information.

[0041] Step 103, based on the timestamp of the device information, matches and fuses the mileage count with the device information to output the fused mileage, including: performing temporal domain alignment and fusion of the area array visual mileage count and the wheel pulse encoder mileage count, matching the timestamp of the device information, and outputting the fused mileage.

[0042] Then, in step 104, the devices in the fused mileage are matched with the devices in the existing ledgers of each specialty, and the matching pairs are output. In step 105, the mapping relationship between the fused mileage and the device-level mileage of each specialty is established, and a multi-specialty mileage ledger under a unified reference system is output based on the mapping relationship.

[0043] The embodiments of the present invention construct a unified, objective, and cross-checkable spatial reference and mapping mechanism, which can realize accurate mileage alignment and comprehensive application between different professional ledgers.

[0044] Figure 2 This is a specific example diagram of a machine vision-based railway multi-disciplinary ledger mileage mapping method in an embodiment of the present invention, as shown in the figure. Figure 2 As shown, the embodiments of the present invention can be mainly divided into input and preset, key equipment positioning, mileage fusion calculation, ledger equipment matching and ledger mapping.

[0045] Input and preset sections of this invention embodiment:

[0046] Simultaneously, area array images and linear array images are acquired, and wheel pulse encoder mileage counts and area array visual mileage meters are obtained; existing ledgers, equipment dictionaries and rules from various railway specialties such as track maintenance, signaling, and power supply are imported to establish a unified timeline and data structure, providing standardized input for subsequent detection, fusion and mapping.

[0047] Key equipment positioning part of this invention embodiment:

[0048] The system performs key equipment detection and OCR (Optical Character Recognition) in parallel with both array and linear images, outputting equipment type, text attribute information (number / direction / mileage marker, etc.), timestamp, and image coordinates. Within the same time window, it performs multi-source result association to generate a deduplicated "detection equipment event," which serves as the basic anchor point for subsequent mileage assignment and ledger matching.

[0049] Mileage fusion calculation part of this invention embodiment:

[0050] The visual odometry and encoder odometry are time-domain aligned and fused (e.g., Kalman filtering / factor graph smoothing) to form a continuous, monotonic, and anti-slip fused odometry curve S(t); based on event timestamps... Assign a fusion mileage m=S(t) to each detection device to obtain the "device-fusion mileage" anchor point set, and quantify the position of the device in the unified reference frame.

[0051] The ledger equipment matching part of this invention embodiment:

[0052] Using testing equipment information as a constraint, the testing equipment is matched with entries in the professional ledgers to generate matching pairs. .

[0053] in, For professional Mileage in the ledger To address the corresponding fusion milestones, scoring mechanisms and outlier removal methods can be employed to improve matching reliability and robustness.

[0054] The ledger mapping section of this invention embodiment:

[0055] Based on the reliable anchor points established in the previous step, construct a monotonic mapping function for "professional ledger mileage → integrated mileage". Piecewise linear or regular splines are preferred to balance accuracy and smoothness; inverse mapping is also provided. It supports cross-professional conversion and unified statistics, and writes back the results of "multi-professional equipment-integrated mileage" to form a corrected mapping ledger.

[0056] Output section of this invention embodiment:

[0057] It outputs mapping functions and parameters for each discipline and generates a multi-discipline mileage ledger under a unified reference system, supporting cross-discipline retrieval, statistical analysis, and collaborative operation and maintenance applications.

[0058] In this embodiment of the invention, the identification of various professional equipment is performed using area array images and line array images, and the information of the identified professional equipment is output. This may include: automatically locating the key equipment of each profession from the line array images and area array images through a target detection algorithm, and outputting information such as equipment category, confidence level, pixel coordinates and timestamp.

[0059] Railway engineering equipment includes, for example: turnouts, guard rails, insulating joints, rail joints, fishplates, fasteners, sleepers, track slabs, frogs / pivots, etc.

[0060] Electrical equipment includes, for example, transponders, switch machines, signals, turnout indicators, axle counting sensors, trackside equipment boxes, and compensation capacitors.

[0061] Power supply equipment includes, for example: supports, hanging columns, cantilever arms, positioners, segmented insulators, disconnect switches, compensation devices, guy wire / catering cable anchor points, etc.

[0062] To improve the detection accuracy of various specialized equipment, in one embodiment, area array images and linear array images are used to identify various specialized equipment and output the identified specialized equipment information. This can include: the area array images and linear array images are preprocessed and then input into a pre-built and trained target detection network to output the identified specialized equipment information; the target detection network architecture includes a backbone network, a neck network, and a head network; wherein, the backbone network includes a C2f module, an ELAN module, and a lightweight multi-core dilated pooling network; the lightweight multi-core dilated pooling network introduces dilated convolution in the spatial pyramid pooling structure SPP; the neck network uses a BiFPN network to perform multi-scale fusion from top to bottom and bottom to top, and introduces weighted fusion and channel attention mechanisms; the head network includes a classification network and a regression network.

[0063] Figure 3 This is a schematic diagram of the processing flow of the target detection network in an embodiment of the present invention. Figure 4 This is a structural diagram of the target detection network in an embodiment of the present invention, with reference to... Figure 3 , Figure 4 , Figure 4 Different colors represent different functional modules. This embodiment of the invention is an improvement on the YOLO algorithm. Before inputting the target detection network, the array image and the linear array image are preprocessed.

[0064] For area images: multi-scale scaling (e.g., minimum edge alignment to {640, 800, 1024}), small target copy-paste with random scaling / color enhancement, modal normalization;

[0065] For linear array images: row-oriented sliding window reconstruction and time-series stitching are used. The physical length of the window is determined by the encoder mileage (e.g., 5–15 m / window), and adjacent windows overlap by 50–70% to improve recall.

[0066] Finally, modality-specific normalization is performed on both the array and linear images. For example, DSBN (Domain Specific Batch Normalization) and GroupNorm (Group Normalization) are used to reduce distribution differences between different imaging modalities and reduce modality domain shift.

[0067] Row-to-row sliding window reorganization can include:

[0068] The line-by-line images continuously captured by the line scan camera are segmented and reassembled into a series of "strip images" according to a fixed "physical length". This is equivalent to piecing together several consecutive lines to form an image window covering a short section of the line.

[0069] Setting overlap between each window (such as more than half overlap) ensures that adjacent windows cover the same physical area, thereby avoiding missed detections and facilitating subsequent cross-window merging.

[0070] The timing splicing and result merging can include:

[0071] Perform an independent detection on each window to obtain information such as the device's location and category.

[0072] The detection results within the window are described using a unified mileage coordinate system (based on the known correspondence between the collection start point and the row sequence), so that the results from different windows fall on the same coordinate axis.

[0073] Results that “appear to be the same target” in adjacent windows are merged: if their mileage locations are very close, their categories are consistent, and their appearance locations match within overlapping areas, they are considered the same target and retained only once (usually retaining the one with higher confidence or a location-weighted average).

[0074] The backbone network employs a C2f module (lightweight feature fusion module) and an ELAN module (efficient layer aggregation module), while retaining high-resolution branches. CBAM (convolutional block attention module, channel attention followed by spatial attention) is inserted after shallow stages P2 and P3 to enhance small target features. SPPF (fast spatial pyramid pooling) is replaced with SPPF-Dilated (a dilated convolution variant of fast spatial pyramid pooling, lightweight multi-kernel dilated pooling) to expand the receptive field and control computational cost. Specifically, the Stem convolution (the first convolutional layer at the input) performs initial convolution processing on the input image, and features sequentially pass through Stage P2 to Stage P5 (a progressively increasing stride stage), finally entering SPPF-Dilated processing.

[0075] The neck network (feature fusion layer) receives the outputs of the backbone network Stage P2 to Stage P5 and SPPF-Dilated, respectively. It uses BiFPN (Bidirectional Feature Pyramid Network, multi-scale fusion from top to bottom and bottom to top) for feature fusion, introduces learnable weighting and lightweight channel attention, and adds P2_head (small target detection head of P2 layer, focusing on small-sized components).

[0076] The head network (detection head) employs a decoupled classification and regression head, including four scale outputs from Head P2 to Head P5. Classification uses Focal Loss or Varifocal Loss, while regression uses DFL (Distributed Focal Loss) and SIoU (Separable IoU Loss). Anchor boxes are adaptively clustered based on IoU (Intersection over Union) using K-means++ (improved K-means clustering) or k-medoids (K-medoids clustering), and generated by modality or category clusters. Positive and negative sample assignment uses Task-Aligned Assigner or SimOTA (Simplified Optimal Transfer Assignment).

[0077] The inference phase (post-processing) employs either DIoU-NMS (non-maximum suppression based on distance IoU) or Soft-NMS (soft non-maximum suppression). Temporal NMS and cross-window fusion are performed on the linear array sequence, and ByteTrack (lightweight target tracking) can be used for trajectory smoothing. Specifically, the DIoU-NMS method considers both box center distance and intersection-union ratio (IoU) in non-maximum suppression, reducing the false deletion of adjacent targets and improving localization accuracy. The Soft-NMS method is used in dense or small target scenes, replacing hard deletion with a smooth decay of candidate box scores based on overlap, improving recall and mitigating false negatives.

[0078] Finally, the recognition results are output, including device analogy, confidence score, pixel bounding box, timestamp, and other information.

[0079] In one embodiment, the target detection network can be trained as follows:

[0080] Historical image data along the railway line was collected; the collected data was stratified and sampled by section mileage and time to obtain sample data; the sampling rules satisfied the balance of data for intersecting lines, day and night, and different weather conditions; the area array images and line array images in the sample data were augmented and labeled to obtain training data; the training data were used to train the model; the optimizer was AdamW or SGD, and the learning strategy adopted was a preheating cosine annealing strategy.

[0081] In this embodiment, the data construction before training is performed by stratified sampling based on segment mileage and time to ensure a balanced distribution across different scenarios such as lines, day and night, and weather. Simultaneously, data from adjacent mileage segments or adjacent time windows are grouped into the same data partition to avoid overestimation due to "near-neighbor frame leakage." Resampling and instance equalization are performed on small targets and low-frequency classes (such as axle counters and segment insulators). Terminology Supplementary Explanation:

[0082] Small targets: refers to targets that are very small in scale on the image, which can be determined by absolute pixels or relative proportion. For example, the shorter side is less than about 16 pixels or the target area is less than about 32×32 pixels; or the target area occupies less than about 0.1% of the total image area (the threshold can be adjusted according to the distribution of project data).

[0083] Low-frequency classes: These are classes that account for a very small percentage of the total number of samples in the training set. For example, the number of instances in this class is less than 1%–5% or the absolute number of samples is less than a set lower limit (such as several hundred), which makes the model less easy to learn.

[0084] Resampling: A strategy to redistribute sampling probabilities to alleviate class imbalance. Common practices include oversampling minority / rare case samples (by copying or combining data augmentation to generate variants) and undersampling majority class samples (by reducing the probability of selection) to make the frequency of occurrence of each class during training more similar.

[0085] Instance balancing: Balancing training signals at the "instance" level rather than the "image" level. Typical practices include: constraining the upper and lower limits of the number of instances per class during batch construction; setting weights for images containing multiple instances based on the number of instances to avoid a large number of instances in a single image dominating the gradient; and using focus loss or class balancing loss at the loss level to give minority classes and small targets more reasonable training weights.

[0086] Data augmentation and annotation are performed on area and linear images in the sample data, including:

[0087] Area array image enhancement includes one or any combination of the following: Mosaic (intensity control) enhancement, MixUp enhancement, copying and pasting small targets with random scaling, overlaying random perspective, rotation, and cropping, overlaying HSV jitter, overlaying motion blur, JPEG compression, adding simulated noise, and covering high-speed and low-light scenes.

[0088] Linear image enhancement and annotation include: sliding window reconstruction, temporal stitching, mileage-aligned cross-window label mapping, random step-size jitter and row noise injection, temporal block stitching to simulate speed changes, and continuous cross-window annotation for ultra-long devices. The "step size" in the linear sliding window refers to the distance (or equivalent number of rows) that the windows advance when reconstructing consecutive rows into a "physical length window," used to segment the input; it is related to detection coverage density / overlap rate. A temporal block refers to a continuous linear sequence divided by time or mileage (e.g., 0.5–2 seconds or 10–30 meters), with adjacent rows / windows within and relatively independent of other blocks outside.

[0089] The optimizer can be either AdamW (weight decay optimizer, wd=0.05) or SGD (stochastic gradient descent with Nesterov momentum); the learning rate uses Warmup Cosine (warm-up + cosine annealing, warm-up for 5–10 training epochs), combined with EMA weight smoothing (parameter exponential moving average), FP16 mixed precision (mixed training of half-precision and single-precision), and gradient clipping (threshold 1.0) for stable training. The loss function combination is classification Focal Loss / Varifocal Loss (focal class loss / zoom focal loss) and regression SIoU+DFL (separable IoU loss + distributed focal regression loss), and introduces IoU-awareObjectness (awareness of the existence of objects in IoU); class and sample imbalance is alleviated through hard example mining (key sampling / weighted hard samples) and positive / negative ratio upper limit control (limiting the proportion of positive and negative samples).

[0090] The recommended training process is "general data pre-training → line domain incremental fine-tuning", and a phased freezing and unfreezing (first the head, then the whole network) is adopted to improve convergence stability.

[0091] Line domain incremental fine-tuning can include:

[0092] Use area and line array images of the target line to cover day and night, sunny, rainy, snowy, tunnel / open line, different speeds and operating conditions; equipment category includes all targets involved in the ledger such as transponders, supports, fasteners, etc.

[0093] Stratified sampling by mileage and time, reserving independent verification / test segments to avoid leakage between adjacent frames / adjacent mileages;

[0094] If the target route sample is insufficient, a small amount of "similar route data in the same domain" (same model, same camera parameters) can be introduced as a supplement, with the proportion not exceeding 20% ​​of the total.

[0095] Mileage fusion calculation is performed after the location and identification of the aforementioned key equipment.

[0096] In one embodiment, the area array visual odometry count is derived from the visual odometry formed by the area array images and may include: calculating the pose increment between adjacent area array image frames, and calculating the area array visual odometry count based on the pose increment and inertial navigation data.

[0097] In one embodiment, the wheel pulse encoder mileage count can be obtained as follows: by acquiring a preset number of rows and the corresponding mileage through a pre-calibrated line scan camera, and the number of image rows acquired by the line scan camera, the wheel pulse encoder mileage count is calculated; wherein, the line scan camera acquires data based on the wheel encoder pulse trigger.

[0098] Figure 5 This is a schematic diagram of the fusion mileage calculation process in an embodiment of the present invention, with reference to... Figure 5 In this embodiment of the invention, the relative mileage information of each key professional device is obtained by collaborative calculation using image and mileage sensors.

[0099] For area array images, since time-triggered shooting cannot directly accumulate mileage, visual odometry (VO) is used to estimate the pose increment between adjacent frames, and combined with IMU (Inertial Measurement Unit, which can collect inertial navigation data such as acceleration and angular velocity) to form visual-inertial odometry (VIO). While ensuring scale, it improves robustness to low-texture and motion-blurred scenes, and obtains a continuous visual odometry curve S_vo(t).

[0100] For linear scan images, the linear scan camera is triggered by pulses from the wheel encoder, and each line of images corresponds to a fixed travel distance, using a pre-calibrated pixel-mileage coefficient. (That is, the correspondence between the preset number of rows collected by the pre-calibrated line scan camera and the travel distance) After conversion, the segment mileage increment It can be represented as:

[0101] ;

[0102] in This represents the number of rows (or equivalent pulses) accumulated in this segment.

[0103] Continue to refer to Figure 5In this embodiment, odometer fusion is based on rigorous calibration and time synchronization. First, the array camera, line scan camera, IMU, and encoder are aligned to a unified time axis (hardware synchronization or post-processing alignment, with deviation controlled within milliseconds). This completes the external parameter calibration of the camera-IMU and encoder-vehicle body, and calibrates the scanning rate and pixel-odometer coefficient of the line scan camera, laying the foundation for the integrated fusion of multi-source data.

[0104] In one embodiment, the temporal alignment and fusion of the area array visual odometer count and the wheel pulse encoder odometer count, and the matching of the timestamp of the device information, can be performed to output the fused odometer. This can include: using an unscented Kalman filter method to temporally align and fuse the area array visual odometer count and the wheel pulse encoder odometer count, matching the timestamp of the device information, and outputting the fused odometer.

[0105] After continuous accumulation of pixel data for linear array odometry calculation and inter-frame pose data for area array visual odometry (VIO) with IMU pre-integration, an unscented Kalman filter method is used for fusion.

[0106] For example, using an unscented Kalman filter to temporally align and fuse area array visual odometer counts and wheel pulse encoder odometer counts, matching the timestamps of device information, and outputting the fused odometer count can include:

[0107] The vehicle's attitude is continuously calculated from the starting point using inertial navigation data; the attitude includes mileage, speed, and orientation.

[0108] Different weights are assigned to the area array visual odometer count and the wheel pulse encoder odometer count. Based on the unscented Kalman filter method, the vehicle attitude calculated from the inertial navigation data is corrected using the weighted area array visual odometer count and the wheel pulse encoder odometer count, and the fused odometer is output. Specifically, when wheel slippage is detected, the weight of the wheel pulse encoder odometer count is reduced, and when the information entropy of the area array image is detected to be lower than the preset information entropy value, the weight of the area array visual odometer count is reduced.

[0109] Continue to refer to Figure 5 During the fusion process, an unscented Kalman filter is used to fuse multi-source mileage information into a more reliable mileage curve through "prediction and correction on the same time". Specifically, an IMU is used as the "mileage predictor" to continuously calculate the vehicle's mileage, speed, and heading (the state includes mileage s, speed v, and heading angle). The system also uses IMU acceleration / gyroscope zero bias (b_a, b_g) to correct prediction errors. Simultaneously, it uses two observations to correct prediction errors: the encoder provides odometer readings (corresponding to s), and the VIO of the array channel provides visual inertial odometer readings (corresponding to s or the odometer increment at adjacent times). When wheel slippage or visual degradation is detected (e.g., blurred image, sparse texture, excessive darkness, or strong reflection, i.e., the array image information entropy is lower than the preset information entropy value), the system automatically reduces the weight of the corresponding observation and uses statistical thresholds and physical constraints (e.g., monotonic odometer readings, reasonable speed changes) to eliminate outliers. The final output is a monotonic, continuous, and slip-resistant fused odometer S(t), used for subsequent equipment positioning, ledger matching, and cross-disciplinary odometer mapping.

[0110] In a linear array channel, the linear array camera is triggered by pulses from the wheel encoder to acquire data. Each row of pixels corresponds to a fixed travel distance, and the segment mileage increment can be expressed as:

[0111] ;

[0112] in, This represents the cumulative number of rows / pulses in this segment.

[0113] For abnormal operating conditions such as slippage, dynamic weight reduction and median smoothing are performed by combining speed / acceleration thresholds to correct missing pulses and row noise; at the same time, a mapping from row index to time is established to ensure seamless splicing across windows and accurate fusion in subsequent time sequence.

[0114] In the area array channel, visual inertial odometry (VIO) is uniformly used for odometry estimation: by leveraging the tight coupling between the camera and the IMU, pose, velocity and zero bias are estimated simultaneously based on IMU pre-integration and sliding window optimization, ensuring scale consistency and significantly improving robustness under conditions such as low texture, motion blur and illumination changes; by tangential accumulation of pose increments in adjacent frames, a continuous and monotonic visual inertial odometry curve is obtained, and combined with degradation detection and static constraints to suppress drift, providing a highly reliable odometry input for subsequent multi-source fusion and ledger mapping.

[0115]

[0116] : No. The mileage increment estimated by VIO at any given moment. Indicates the initial time;

[0117] The displacement vector between two adjacent frames;

[0118] The tangential unit vector (dimensionless) is used to project displacement onto the direction of the line's movement;

[0119] From the initial moment to The cumulative visual inertial mileage.

[0120] The fusion layer employs unscented Kalman filtering to fuse multi-source odometer information into a more reliable odometer curve through "prediction and correction on the same side." Specifically, an IMU is used as the "odometer predictor," continuously calculating the vehicle's mileage, speed, and heading (state includes mileage s, speed v, and heading angle). And the IMU's acceleration / gyroscope bias Simultaneously, the two observation odometers are used to correct the prediction error—the encoder provides the odometer observation (corresponding to s), and the VIO of the array channel provides the visual inertial odometer observation (corresponding to s or the odometer increment at adjacent times). The formula is shown below:

[0121]

[0122] s: Cumulative mileage along the tangential direction of the line, in meters;

[0123] v: Tangential velocity along the track, in m / s;

[0124] Heading angle (the angle between the vehicle and the track tangent), unit: rad / s;

[0125] IMU accelerometer zero bias (along the tangential channel; in the three-axis case, it can be extended to a vector), unit m / s²;

[0126] IMU gyroscope zero bias (yaw / vertical axis channel; in the three-axis case, it can be extended to a vector), unit rad / s;

[0127] k: Index of discrete time step; Indicates time Predict the prior at time k;

[0128] Discrete sampling period, in seconds;

[0129] Tangential acceleration is obtained by projecting the IMU acceleration to the tangential direction of the line after zero bias removal and gravity and attitude compensation.

[0130] : Heading angular velocity (yaw angular velocity) measured by IMU gyroscope, unit: rad / s;

[0131] : , Process noise (random walk), zero-mean Gaussian noise;

[0132] Mileage prediction: s as a function of the speed at the previous moment according to integral;

[0133] Velocity prediction: v is determined by tangential acceleration Points update;

[0134] Course prediction: Measurement by gyroscope Zero bias Points will be updated later;

[0135] Zero-biased evolution: , It changes slowly according to the random walk model.

[0136] When wheel slippage or visual degradation (such as low texture or blur) is detected, the system automatically reduces the weight of the corresponding observation and uses statistical thresholds and physical constraints (such as monotonic mileage and reasonable speed changes) to eliminate outliers. The final output is a monotonic, continuous, and slip-resistant fused mileage, along with uncertainty, for subsequent equipment positioning, ledger matching, and cross-disciplinary mileage mapping.

[0137] In terms of constraints and quality control, the lateral and longitudinal drift is limited by the one-dimensional primary motion prior of the track, and the acceleration / angular velocity / rate of change of velocity are superimposed to limit the amplitude, and the mileage monotonicity is forced when traveling in the same direction. Finally, S(t) is used to backfill the values ​​of each row of the linear array and each frame of the area array to establish a unified mapping of "pixel / frame → mileage".

[0138] After completing the mileage fusion calculation, the ledger equipment matching calculation is performed.

[0139] In one embodiment, the devices in the merged mileage are matched with the devices in the existing ledgers of each specialty, and the matching pairs are output, including:

[0140] Using equipment type, text attribute information, adjacent equipment sequence relationship and spatial neighborhood topology as constraints, a coarse match is performed between the equipment in the fusion mileage and the equipment in the existing ledgers of each profession to form a candidate corresponding set;

[0141] The candidate matching set is scored, consistency is checked, and outliers are removed to obtain matching pairs.

[0142] The spatial neighborhood topology constraint refers to using "proximity" as an initial screening criterion: detected devices are placed on a unified mileage axis, and their positions relative to the ledger records are checked against a pre-defined, acceptable range; devices exceeding this range are eliminated or downweighted, while those within the range are retained as candidates. During coarse matching, the direction of increasing route mileage or the agreed-upon driving direction is used as a benchmark, and the results are arranged from smallest to largest (or from upstream to downstream); the detection results and the ledger records should maintain the same sequential order if they are in the same direction.

[0143] Figure 6 This is a schematic diagram of the ledger equipment matching calculation process in an embodiment of the present invention, such as... Figure 6 As shown, the first step is to perform a rough matching of the ledger equipment and establish the corresponding relationship.

[0144] Based on the detection results and the records of various specialties (including equipment entries, record mileage, and adjacency relationships), a coarse matching is first performed according to equipment type, OCR number / direction, adjacent equipment sequence relationship, and nearest neighbor constraint of fused mileage within the time window to form a candidate correspondence set. Then, scoring and consistency checks are used, combined with RANSAC (Random Sampling Consensus Algorithm) to eliminate outliers, ultimately determining the one-to-one or one-to-many correspondence between the detected equipment and the equipment in each specialties' records. For example, dual equipment at the same point: There are two transponder records for the same physical location, one uplink and one downlink. A single detection may simultaneously satisfy both records (marked as primary match + secondary match, with primary match given priority, and secondary match retained for verification). Combined records: The same support column has multiple records in the records for "main column" and "auxiliary components (hanging column / cantilever)," and a single "support column" detection will correspond to multiple records (treated as a group match, with order preserved within the group).

[0145] The scoring rules are as follows:

[0146] ① Score for type consistency: Full marks are given if the types are the same, otherwise 0 marks are given.

[0147] ② Text matching score: similarity between OCR number / direction and ledger (full score for exact match after edit distance / normalization, lower score for fuzzy match).

[0148] ③ Mileage nearest neighbor score: The mileage residual decreases linearly from 0 to full score (e.g., 5-10 meter tolerance for transponders, 8-15 meter tolerance for struts).

[0149] ④ Sequence Consistency Score: Bonus points are awarded for sequences that are consistent with the order of adjacent devices (the more neighbors a device passes through, the higher the score).

[0150] ⑤ Neighborhood topology score: Bonus points are awarded if common combinations of relationships (such as “signal → transponder” or “support → cantilever → segment insulator”) are satisfied.

[0151] ⑥ Observation quality score: bonus points for OCR confidence, detection confidence, and cross-window / cross-frame stability (consistency of repeated observations).

[0152] ⑦ Overall score = S(type) + S(text) + S(mileage) + S(sequence) + S(topology) + S(quality); those exceeding the threshold are included in the candidate set.

[0153] Consistency checks are as follows:

[0154] ① Sequence monotonicity: The order of candidate matches within a mileage window is consistent with the ledger (at least two adjacent constraints are satisfied).

[0155] ② Distance tolerance: The distance between adjacent devices falls within a reasonable range (devices exceeding the tolerance will be removed or downweighted).

[0156] ③ Two-way verification: There is no conflict between the detection → ledger and the ledger → detection; if the same ledger entry is hit by multiple detections, the one with the highest score and the most consistent global sequence will be retained first.

[0157] ④ RANSAC robustness: In this segment, RANSAC is used to estimate the local linear relationship between “ledger mileage → merged mileage”, retaining candidate interior points and removing outliers.

[0158] In one embodiment, before establishing a mapping relationship between fused mileage and device-level mileage for each specialty based on matching pairs, and outputting a multi-specialty mileage ledger under a unified reference system based on the mapping relationship, the method may further include:

[0159] Based on the fused mileage, the timestamps of the array image and the linear image are interpolated and extrapolated. Each device that uses the array image and the linear image for recognition is reassigned to obtain machine vision mileage under a unified reference system. The machine vision mileage includes the correspondence between mileage and device.

[0160] Based on matching pairs, a mapping relationship is established between fused mileage and device-level mileage for each specialty, which may include:

[0161] Based on the matching pairs, establish the mapping relationship between equipment-level mileage and machine vision mileage for each profession, and the inverse mapping relationship between machine vision mileage and equipment-level mileage for each profession.

[0162] refer to Figure 6 We then perform fusion-driven visual data mileage reassignment. This is done using the obtained fusion mileage curve. Interpolate / extrapolate the timestamps of the array and linear images, and reassign the machine vision mileage of each detection device event to a unified reference frame. This enables the joint suppression of visual scale drift and encoder slippage, resulting in highly consistent mileage markings for key equipment.

[0163] Finally, the ledger—machine vision correspondence and multi-disciplinary mapping construction.

[0164] Matching anchor set Based on this, the monotonic mapping of "ledger mileage → machine vision mileage" is estimated for each specialty. And give the inverse mapping. Provides cross-disciplinary conversion and joint statistics. Outputs mapping parameters, thereby obtaining a consistent mileage mapping relationship between various professional ledgers, supporting cross-disciplinary retrieval, statistics, and collaborative operation and maintenance.

[0165] In summary, this invention addresses the key facility images acquired by vehicle-mounted area scan cameras and line scan cameras. It utilizes artificial intelligence algorithms to reliably detect facility locations within the images and combines the visual odometer generated by the area scan camera with the wheel pulse-coded mileage from the line scan camera to calculate unified mileage coordinates for the equipment. Furthermore, it matches the detected equipment with corresponding equipment in various professional ledgers, establishing equipment-level mileage mapping relationships between different professions. This eliminates inconsistencies in mileage across professional ledgers at the source, supporting the alignment, coordinated statistics, and collaborative operation and maintenance of multi-professional data.

[0166] The following is an example of a method for mapping the mileage of electrical transponder ledgers to overhead contact line ledgers.

[0167] Step 1: Multi-source acquisition and detection.

[0168] Simultaneously acquire data from area array cameras and line array cameras, and perform target detection and attribute extraction (type, number / direction, timestamp, image coordinates) on the catenary support and transponder.

[0169] Step 2, Mileage Calculation and Integration.

[0170] The encoder mileage of the linear array channel and the visual inertial mileage of the area array channel are calculated separately to achieve time synchronization and fusion, forming a unified machine vision mileage curve S(t).

[0171] Step 3: Equipment-level coarse calibration and assignment (transponder).

[0172] Constrained by device type, OCR number / direction, adjacent sequence relationship, and mileage nearest neighbor, the detected transponders are coarsely matched with the transponder log to obtain matching anchor points; based on S(t), machine vision mileage is assigned to each transponder detection event. .

[0173] 4. Equipment-level coarse calibration and assignment (contact wire support).

[0174] Using the same constraints and procedures, the detected catenary supports are coarsely matched with the support ledger and anchor points are established; based on S(t), machine vision mileage assignment for support events is completed. .

[0175] 5. Construction of cross-disciplinary mapping relationships.

[0176] Machine vision mileage To unify the benchmark, based on the matching anchor point set of the two types of equipment, a monotonic mapping of "ledger mileage → machine vision mileage" and its inverse mapping are fitted to establish a consistent mileage conversion relationship between the transponder ledger and the catenary support ledger, and the mapping parameters and quality indicators are output.

[0177] Figure 7 This is a schematic diagram illustrating the mapping results between the electrical transponder ledger and the overhead contact line ledger mileage in an embodiment of the present invention, as shown below. Figure 7 As shown, the top list is the transponder register, listing the equipment number, line mileage, and attributes; the bottom yellow list is the overhead contact line register. For the support column number, the overhead contact line ledger can also include fields such as uniform mileage (e.g., 5.640) and type (anchor column); the middle image is a "machine vision mileage" view formed by stitching together inspection images, with sleepers, supports and transponders presented as icons.

[0178] Figure 7 The red lines in the middle represent "ledger entries". The matching anchor point of the "visual inspection results" is matched by type. Adjacent sequences and nearest-neighbor mileage constraints were used, and the results were confirmed after consistency and RANSAC outlier removal.

[0179] This invention also provides a machine vision-based railway multi-disciplinary ledger mileage mapping device, as described in the following embodiments. Since the principle behind this device's problem-solving is similar to that of the machine vision-based railway multi-disciplinary ledger mileage mapping method, its implementation can refer to the implementation of the machine vision-based railway multi-disciplinary ledger mileage mapping method; repeated details will not be elaborated further.

[0180] Figure 8 This is a schematic diagram of a machine vision-based railway multi-disciplinary ledger mileage mapping device in an embodiment of the present invention, as shown below. Figure 8 As shown, the device 800 includes:

[0181] Data acquisition module 801 is used to acquire image data and mileage counts along the railway line;

[0182] The target recognition module 802 is used to identify various professional equipment using images and output the identified professional equipment information, including a timestamp.

[0183] The fusion mileage calculation module 803 is used to match and merge mileage counts with equipment based on the timestamp of equipment information, and output the fused mileage; the fused mileage includes the correspondence between mileage and various professional equipment.

[0184] The equipment ledger matching and calculation module 804 is used to match the equipment in the integrated mileage with the equipment in the existing ledgers of each specialty, and output matching pairs; the matching pairs include the correspondence between the mileage of the equipment in the existing ledgers of each specialty and the mileage of the equipment in the integrated mileage; based on the matching pairs, a mapping relationship between the integrated mileage and the equipment-level mileage of each specialty is established, and a multi-professional mileage ledger under a unified reference system is output based on the mapping relationship.

[0185] In one embodiment, the image data includes area array images and line array images, and the mileage count includes area array visual mileage count and wheel pulse encoder mileage count; the area array visual mileage count is based on visual mileage formed from the area array images;

[0186] The target recognition module 802 is specifically used for:

[0187] Using area array images and line array images, various professional equipment are identified, and the identified professional equipment information is output.

[0188] The 803 fusion odometer module is specifically used for:

[0189] The area array visual mileage count and the wheel pulse encoder mileage count are time-domain aligned and fused, and the timestamps of the device information are matched to output the fused mileage.

[0190] In one embodiment, the target recognition module 802 is specifically used for:

[0191] Area array images and linear array images are preprocessed and then input into a pre-built and trained target detection network, which outputs the identified information of various professional equipment. The target detection network architecture includes a backbone network, a neck network, and a head network. The backbone network includes a C2f module, an ELAN module, and a lightweight multi-kernel dilated pooling network. The lightweight multi-kernel dilated pooling network introduces dilated convolution in the spatial pyramid pooling structure (SPP). The neck network uses a BiFPN network for top-down and bottom-up multi-scale fusion and introduces weighted fusion and channel attention mechanisms. The head network includes a classification network and a regression network.

[0192] In one embodiment, the target detection network is trained as follows:

[0193] Collect historical image data along the railway line;

[0194] The collected data were stratified and sampled according to segment mileage and time to obtain sample data; the sampling rules satisfied the balance of data from intersecting routes, day and night, and different weather conditions.

[0195] Data augmentation and annotation are performed on the area and line array images in the sample data to obtain training data;

[0196] The model is trained using training data; the optimizer is AdamW or SGD, and the learning strategy is a preheating cosine annealing strategy.

[0197] In one embodiment, the area array visual odometry is based on visual odometry formed from area array images, including:

[0198] Calculate the pose increment between adjacent area array image frames, and calculate the area array visual odometry based on the pose increment and inertial navigation data.

[0199] In one embodiment, the wheel pulse encoder mileage count is obtained as follows:

[0200] The wheel pulse encoder mileage count is obtained by converting the pre-calibrated line scan camera to collect the correspondence between the preset number of rows and the travel distance, and the number of image rows collected by the line scan camera; wherein, the line scan camera is triggered to collect data according to the wheel encoder pulse.

[0201] In one embodiment, the fusion mileage calculation module 803 is specifically used for:

[0202] The unscented Kalman filter method is used to perform temporal domain alignment and fusion of the area array visual mileage count and the wheel pulse encoder mileage count, match the timestamp of the device information, and output the fused mileage.

[0203] In one embodiment, the fusion mileage calculation module 803 is specifically used for:

[0204] The vehicle's attitude is continuously calculated from the starting point using inertial navigation data; the attitude includes mileage, speed, and orientation.

[0205] Different weights are assigned to the area array visual odometer count and the wheel pulse encoder odometer count. Based on the unscented Kalman filter method, the vehicle attitude calculated from the inertial navigation data is corrected using the weighted area array visual odometer count and the wheel pulse encoder odometer count, and the fused odometer is output. Specifically, when wheel slippage is detected, the weight of the wheel pulse encoder odometer count is reduced, and when the information entropy of the area array image is detected to be lower than the preset information entropy value, the weight of the area array visual odometer count is reduced.

[0206] In one embodiment, the device information further includes device type and text attribute information;

[0207] The equipment ledger matching and calculation module 804 is specifically used for:

[0208] Using equipment type, text attribute information, adjacent equipment sequence relationship and spatial neighborhood topology as constraints, a coarse match is performed between the equipment in the fusion mileage and the equipment in the existing ledgers of each profession to form a candidate corresponding set;

[0209] The candidate matching set is scored, consistency is checked, and outliers are removed to obtain matching pairs.

[0210] In one embodiment, the device 800 further includes:

[0211] The visual data mileage reassignment module is used to reassign the mileage based on the fusion mileage and the equipment-level mileage of each specialty before the equipment ledger matching and calculation module 804 establishes a mapping relationship between the fused mileage and the equipment-level mileage of each specialty based on the matching pairs, and outputs a multi-specialty mileage ledger under a unified reference system based on the mapping relationship. Based on the fused mileage, the module interpolates and extrapolates the timestamps of the array image and the line array image, and reassigns the mileage of each device identified by the array image and the line array image to obtain the machine vision mileage under the unified reference system. The machine vision mileage includes the correspondence between mileage and equipment.

[0212] The equipment ledger matching and calculation module 804 is specifically used for:

[0213] Based on the matching pairs, establish the mapping relationship between equipment-level mileage and machine vision mileage for each profession, and the inverse mapping relationship between machine vision mileage and equipment-level mileage for each profession.

[0214] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described machine vision-based railway multi-disciplinary ledger mileage mapping method.

[0215] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described machine vision-based railway multi-disciplinary ledger mileage mapping method.

[0216] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described machine vision-based railway multi-disciplinary ledger mileage mapping method.

[0217] The process and its coordination in this invention embodiment include: acquisition → detection → VIO / encoder UKF fusion → coarse matching → mileage reassignment → anchor-based monotonic mapping / inverse mapping → cross-professional conversion and backfilling. This invention embodiment proposes a mileage-detection linkage processing mechanism combining area array VIO strategy and linear array sliding window / temporal NMS, and proposes anchor point construction rules and a matching scoring system: type + OCR + sequence + nearest neighbor mileage + topology consistency, with robust RANSAC matching. It employs the construction and application of monotonic piecewise linear / monotonic spline mapping and inverse mapping for cross-professional mileage calculation, integrates a mileage-driven event-level mileage reassignment mechanism, and improves the structure and training deployment strategies of the target detection network in railway small / dense target scenarios (CBAM, BiFPN, P2_head, SIoU, anchor frame clustering, SimOTA, linear array-specific enhancement). This invention embodiment has the following beneficial effects:

[0218] Multi-professional mileage unification and inter-calculation: Based on anchor points, a monotonic mapping and inverse mapping of "ledger → machine vision" are constructed to realize the unification and inter-calculation of mileage between engineering, electrical engineering, power supply and other professions, and eliminate the fragmentation of mileage across professions.

[0219] Facility-level precise positioning: Using machine vision mileage as a unified reference, the detection results are accurately assigned to specific equipment (such as pillars, transponders, etc.), supporting one-click positioning "from mileage to equipment" and clear location description.

[0220] Enhanced reliability through integration: The encoder and VIO are deeply integrated, automatically identifying degradation scenarios such as slippage / low texture and adaptively reducing weights, outputting a monotonous, continuous, slip-resistant mileage curve and uncertainty, significantly improving positioning stability.

[0221] Significantly improved work efficiency: The entire process of detection, assignment, and mapping is completed automatically, reducing the workload of manual secondary positioning and ledger alignment, and speeding up fault location and work order flow.

[0222] The results are verifiable and traceable: the matching process incorporates consistency checks and RANSAC to remove outliers, and outputs quality indicators such as residuals and monotonicity, which facilitates acceptance and tracking.

[0223] Data reuse and collaboration: The ledgers of various specialties can be retrieved, statistically analyzed and jointly evaluated under a unified reference system, supporting multi-disciplinary collaborative operation and maintenance and resource optimization.

[0224] Compatibility and scalability: The solution is compatible with existing ledgers and existing testing equipment, and can be expanded to more equipment categories and line scenarios as needed, with good engineering implementation and scalability.

[0225] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0226] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0227] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0228] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0229] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A machine vision-based method for mapping mileage in multi-disciplinary railway ledgers, characterized in that, include: Acquire image data and mileage counts along the railway line; It uses images to identify various specialized equipment and outputs information about the identified specialized equipment. The device information includes a timestamp; Based on the timestamp of the device information, the mileage count is matched and merged with the device to output the merged mileage; The integration mileage includes the correspondence between mileage and various specialized equipment; The system matches the equipment in the merged mileage with the equipment in the existing ledgers of each specialty, and outputs matching pairs. The matching pairs include the correspondence between the mileage of the equipment in the existing ledgers of each specialty and the mileage of the equipment in the merged mileage. Based on the matching pairs, a mapping relationship is established between the fused mileage and the equipment-level mileage of each specialty, and a multi-professional mileage ledger under a unified reference system is output based on the mapping relationship. The image data includes area array images and line array images; the mileage count includes area array visual mileage count and wheel pulse encoder mileage count; the area array visual mileage count is based on the visual mileage count formed by the area array images. The system utilizes images to identify various specialized pieces of equipment and outputs information about each identified piece of equipment, including: Using area array images and line array images, various professional equipment are identified, and the identified professional equipment information is output. Based on the timestamp of the device information, the mileage count is matched and merged with the device information to output the merged mileage, including: The area array visual mileage count and the wheel pulse encoder mileage count are aligned and fused in the time domain, matched with the timestamp of the device information, and the fused mileage is output. Before establishing a mapping relationship between fused mileage and equipment-level mileage for each specialty based on matching pairs, and outputting a multi-specialty mileage ledger under a unified reference system based on the mapping relationship, the process also includes: Based on the fused mileage, the timestamps of the array image and the linear image are interpolated and extrapolated. Each device that uses the array image and the linear image for recognition is reassigned to obtain machine vision mileage under a unified reference system. The machine vision mileage includes the correspondence between mileage and device. Based on matching pairs, a mapping relationship is established between fused mileage and device-level mileage for each specialty, including: Based on the matching pairs, establish the mapping relationship between equipment-level mileage and machine vision mileage for each profession, and the inverse mapping relationship between machine vision mileage and equipment-level mileage for each profession.

2. The method as described in claim 1, characterized in that, Using area array images and line array images, various specialized equipment is identified, and the identified equipment information is output, including: Area array images and linear array images are preprocessed and then input into a pre-built and trained target detection network, which outputs the identified information of various professional equipment. The target detection network architecture includes a backbone network, a neck network, and a head network. The backbone network includes a C2f module, an ELAN module, and a lightweight multi-kernel dilated pooling network. The lightweight multi-kernel dilated pooling network introduces dilated convolution in the spatial pyramid pooling structure (SPP). The neck network uses a BiFPN network for top-down and bottom-up multi-scale fusion and introduces weighted fusion and channel attention mechanisms. The head network includes a classification network and a regression network.

3. The method as described in claim 2, characterized in that, The target detection network is trained as follows: Collect historical image data along the railway line; The collected data were stratified and sampled according to segment mileage and time to obtain sample data; the sampling rules satisfied the balance of data from intersecting routes, day and night, and different weather conditions. Data augmentation and annotation are performed on the area and line array images in the sample data to obtain training data; The model is trained using training data; the optimizer is AdamW or SGD, and the learning strategy is a preheating cosine annealing strategy.

4. The method as described in claim 1, characterized in that, The area array visual odometry is based on visual odometry formed from area array images, including: Calculate the pose increment between adjacent area array image frames, and calculate the area array visual odometry based on the pose increment and inertial navigation data.

5. The method as described in claim 1, characterized in that, The wheel pulse encoder mileage count is obtained as follows: The wheel pulse encoder mileage count is obtained by converting the pre-calibrated line scan camera to collect the correspondence between the preset number of rows and the travel distance, and the number of image rows collected by the line scan camera; wherein, the line scan camera is triggered to collect data according to the wheel encoder pulse.

6. The method as described in claim 4, characterized in that, The area array visual odometer and the wheel pulse encoder odometer are time-domain aligned and fused, matched with the timestamp of the device information, and the fused odometer is output, including: The unscented Kalman filter method is used to perform temporal domain alignment and fusion of the area array visual mileage count and the wheel pulse encoder mileage count, match the timestamp of the device information, and output the fused mileage.

7. The method as described in claim 6, characterized in that, The unscented Kalman filter method is used to perform temporal domain alignment and fusion of the area array visual odometer count and the wheel pulse encoder odometer count, matching the timestamp of the device information, and outputting the fused odometer, including: The vehicle's attitude is continuously calculated from the starting point using inertial navigation data; the attitude includes mileage, speed, and orientation. Different weights are assigned to the area array visual odometer count and the wheel pulse encoder odometer count. Based on the unscented Kalman filter method, the vehicle attitude calculated from the inertial navigation data is corrected using the weighted area array visual odometer count and the wheel pulse encoder odometer count, and the fused odometer is output. Specifically, when wheel slippage is detected, the weight of the wheel pulse encoder odometer count is reduced, and when the information entropy of the area array image is detected to be lower than the preset information entropy value, the weight of the area array visual odometer count is reduced.

8. The method as described in claim 1, characterized in that, The device information also includes device type and text attribute information; Match the devices in the merged mileage with the devices in the existing ledgers of each specialty, and output matching pairs, including: Using equipment type, text attribute information, adjacent equipment sequence relationship and spatial neighborhood topology as constraints, a coarse match is performed between the equipment in the fusion mileage and the equipment in the existing ledgers of each profession to form a candidate corresponding set; The candidate matching set is scored, consistency is checked, and outliers are removed to obtain matching pairs.

9. A machine vision-based railway multi-disciplinary ledger mileage mapping device, characterized in that, include: The data acquisition module is used to acquire image data and mileage counts along the railway line; The target recognition module is used to identify various specialized equipment using images and output information about the identified specialized equipment; the equipment information includes a timestamp. The mileage calculation module is used to match and merge mileage counts with equipment based on the timestamps of equipment information, and output the fused mileage; the fused mileage includes the correspondence between mileage and various specialized equipment; The equipment ledger matching and calculation module is used to match the equipment in the integrated mileage with the equipment in the existing ledgers of each specialty, and output matching pairs; the matching pairs include the correspondence between the mileage of the equipment in the existing ledgers of each specialty and the mileage of the equipment in the integrated mileage. Based on the matching pairs, a mapping relationship is established between the fused mileage and the equipment-level mileage of each specialty, and a multi-professional mileage ledger under a unified reference system is output based on the mapping relationship. The image data includes area array images and line array images; the mileage count includes area array visual mileage count and wheel pulse encoder mileage count; the area array visual mileage count is based on the visual mileage count formed by the area array images. The target recognition module is specifically used for: Using area array images and line array images, various professional equipment are identified, and the identified professional equipment information is output. The fusion odometer calculation module is specifically used for: The area array visual mileage count and the wheel pulse encoder mileage count are aligned and fused in the time domain, matched with the timestamp of the device information, and the fused mileage is output. It also includes: a visual data mileage reassignment module, used to establish a mapping relationship between fused mileage and equipment-level mileage of each specialty based on matching pairs in the equipment ledger matching and calculation module, and output a multi-specialty mileage ledger under a unified reference system based on the mapping relationship. Based on the fused mileage, the timestamps of array images and line array images are interpolated and extrapolated to reassign each device identified by array images and line array images to obtain machine vision mileage under a unified reference system; the machine vision mileage includes the correspondence between mileage and equipment; The equipment ledger matching and calculation module is specifically used for: Based on the matching pairs, establish the mapping relationship between equipment-level mileage and machine vision mileage for each profession, and the inverse mapping relationship between machine vision mileage and equipment-level mileage for each profession.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images