Railway multi-professional machine account mileage mapping method and device based on machine vision

Abstract

The invention discloses a railway multi-professional machine account mileage mapping method and device based on machine vision. The method comprises the following steps: acquiring image data and mileage count along a railway line; identifying each piece of professional equipment by using the image, and outputting the identified information of each piece of professional equipment; the equipment information comprises a timestamp; based on the timestamp of the equipment information, the mileage count and the equipment are matched and fused, and the fused mileage is output; the fused mileage comprises a corresponding relation between the mileage and each professional device; matching the equipment in the fused mileage with the equipment in the existing ledger of each specialty, and outputting a matching pair; the matching pair comprises a corresponding relation between the mileage of the equipment in the existing ledger of each specialty and the mileage of the equipment in the fused mileage; and based on the matching pairs, establishing a mapping relationship between the fusion mileage and the equipment-level mileage of each specialty, and based on the mapping relationship, outputting a multi-specialty mileage ledger under the unified reference system. According to the invention, accurate mileage alignment between different professional ledgers can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of railway engineering, and in particular to a railway multi-specialty account mileage mapping method and device based on machine vision. BACKGROUND

[0002] The account data of a railway system is a collection of various types of information recorded in the relevant work of railway operation, management and maintenance. The railway system covers multiple specialties such as engineering, signal and power supply. In the prior art, each specialty establishes an account according to its own work requirements. However, the business focus and work flow of different specialties are significantly different. The mileage of key equipment of each specialty is determined by its own account. However, due to the obvious specificity of the specialty, not only the starting points of the account mileage are inconsistent, but also the measurement methods and error models of the equipment positions are different, resulting in the fact that the mileage calculation systems of the accounts of different specialties are disconnected from each other. As a result, the mileage corresponding to the same physical location in different specialty accounts is deviated, which makes it difficult for the accounts of different specialties to be interconnected and reused, and a certain specialty cannot quickly calculate the similar mileage of its own by using the account of another specialty. When the mileage of multi-specialty equipment is counted and comprehensively analyzed, the data of each specialty is also difficult to be collected and aligned in the same mileage reference system, which seriously restricts the efficiency and quality of cross-specialty collaboration, comprehensive analysis and integrated operation and maintenance management.

[0003] Therefore, there is an urgent need for a multi-specialty account mileage mapping method to realize accurate mileage alignment and comprehensive application between different specialty accounts. SUMMARY

[0004] The embodiment of the present application provides a railway multi-specialty account mileage mapping method based on machine vision, which is used to construct a unified, objective and mutually checkable spatial reference and mapping mechanism, and realize accurate mileage alignment between different specialty accounts. The method comprises the following steps:

[0005] Obtaining image data and mileage count along a railway line;

[0006] Identifying each specialty equipment by using the image, and outputting the identified each specialty equipment information; the equipment information comprises a time stamp;

[0007] Matching and fusing the mileage count and the equipment based on the time stamp of the equipment information, and outputting the fused mileage; the fused mileage comprises the mileage and the corresponding relationship of each specialty equipment;

[0008] Matching the equipment in the fused mileage with the equipment in the existing account of each specialty, and outputting a matching pair; the matching pair comprises the corresponding relationship between the mileage of the equipment in the existing account of each specialty and the mileage of the equipment in the fused mileage;

[0009] Based on the matching pair, a mapping relationship between the fusion mileage and the device-level mileage of each professional is established, and a multi-professional mileage account under a unified reference system is output based on the mapping relationship.

[0010] The embodiment of the present application also provides a railway multi-professional account mileage mapping device based on machine vision, which is used to construct a unified, objective and mutually checkable spatial reference and mapping mechanism, and realize accurate mileage alignment between different professional accounts.

[0011] A data acquisition module is configured to acquire image data and mileage count along a railway line;

[0012] A target recognition module is configured to recognize each professional device by using the image, and output the recognized professional device information; the device information includes a time stamp;

[0013] A fusion mileage calculation module is configured to match and fuse the mileage count and the device based on the time stamp of the device information, and output fusion mileage; the fusion mileage includes the mileage and the corresponding relationship of each professional device;

[0014] A device account matching calculation module is configured to match the device in the fusion mileage with the device in the existing account of each professional, and output a matching pair; the matching pair includes the corresponding relationship between the mileage of the device in the existing account of each professional and the mileage of the device in the fusion mileage; based on the matching pair, a mapping relationship between the fusion mileage and the device-level mileage of each professional is established, and a multi-professional mileage account under a unified reference system is output based on the mapping relationship.

[0015] The embodiment of the present application also provides a computer device, which includes a memory, a processor and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the above-mentioned railway multi-professional account mileage mapping method based on machine vision is realized.

[0016] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program; when the processor executes the computer program, the above-mentioned railway multi-professional account mileage mapping method based on machine vision is realized.

[0017] The embodiment of the present application also provides a computer program product, which includes a computer program; when the processor executes the computer program, the above-mentioned railway multi-professional account mileage mapping method based on machine vision is realized.

[0018] Compared with the prior art in which each professional of the railway uses its own table account, the table account systems of each professional are isolated from each other, in the embodiment of the present application, a fused mileage including mileage and the corresponding relationship of each professional equipment is first output by using image data and mileage technology, then the fused mileage is matched with each professional table account, a mapping relationship between the fused mileage and the equipment level mileage of each professional is established, and a multi-professional mileage table account under a unified reference system is output based on the mapping relationship, a unified, objective and mutually checkable spatial reference and mapping mechanism is constructed, the mileage unification and mutual conversion among the professions of track maintenance, signal and power supply are realized, and the split of cross-professional mileage is eliminated. The table accounts of each professional can be linked, searched, counted and jointly judged under the unified reference system, the accurate mileage information alignment and confirmation between different professional table accounts are quickly performed, and the multi-professional collaborative operation and resource optimization configuration are supported. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings. In the drawings:

[0020] Figure 1 A flowchart of a railway multi-professional table account mileage mapping method based on machine vision in the embodiment of the present application;

[0021] Figure 2 A specific example diagram of the railway multi-professional table account mileage mapping method based on machine vision in the embodiment of the present application;

[0022] Figure 3 A processing flowchart of a target detection network in the embodiment of the present application;

[0023] Figure 4 A structure diagram of a target detection network in the embodiment of the present application;

[0024] Figure 5 A fused mileage calculation flowchart in the embodiment of the present application;

[0025] Figure 6 A table account equipment matching calculation flowchart in the embodiment of the present application;

[0026] Figure 7 A result diagram of the mileage mapping between the signal transponder table account and the overhead contact system table account in the embodiment of the present application;

[0027] Figure 8 A schematic diagram of a railway multi-professional table account mileage mapping device based on machine vision in the embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, further detailed description will be made to the embodiments of the present application with reference to the drawings. Herein, the illustrative embodiments of the present application and the description thereof are used to explain the present application but not to limit the present application.

[0029] The acquisition, storage, use, processing and the like of data in the technical solutions of the present application comply with the relevant provisions of the national laws and regulations.

[0030] The mileage system is fragmented due to the inconsistency of the mileage starting points in the work, electrical and power multi-specialty account books, and the different measurement methods and error models. Systematic deviations exist in the same physical device in different professional account books, and it is difficult to realize cross-specialty alignment and reuse. In order to solve the problem that the lack of unified and objective spatial reference and robust device-level matching mechanism in the prior art leads to the inability of multi-specialty data to converge and apply under the same mileage reference system, the embodiments of the present application provide a railway multi-specialty account mileage mapping method and device based on machine vision.

[0031] Figure 1 The flowchart of the railway multi-specialty account mileage mapping method based on machine vision in the embodiments of the present application is shown in FIG. 1, which comprises the following steps. Figure 1

[0032] Step 101, acquiring image data and mileage count along the railway line;

[0033] Step 102, identifying various professional devices using the image, and outputting the identified various professional device information; the device information comprises a time stamp;

[0034] Step 103, matching and fusing the mileage count and the device based on the time stamp of the device information, and outputting the fused mileage; the fused mileage comprises the mileage and the corresponding relationship of various professional devices;

[0035] Step 104, matching the device in the fused mileage with the device in the existing account book of each specialty, and outputting the matching pair; the matching pair comprises the corresponding relationship of the mileage of the device in the existing account book of each specialty and the mileage of the device in the fused mileage;

[0036] Step 105, establishing the mapping relationship of the fused mileage and the device-level mileage of each specialty based on the matching pair, and outputting the multi-specialty mileage account book under the unified reference system based on the mapping relationship.

[0037] The railway multi-specialty account mileage mapping method based on machine vision in the embodiments of the present application will be explained in detail.

[0038] ​Firstly, image data and mileage count along the railway line are acquired. For example, the image data and mileage count along the railway line are acquired based on a railway detection vehicle.

[0039] In an embodiment, the image data can include area array images and linear array images, and the mileage count includes area array visual mileage count and wheel pulse encoder mileage count; the area array visual mileage count is obtained based on visual mileage formed by the area array images.

[0040] Further, step 102 utilizes the images to identify each professional device, and outputs the identified professional device information, which can include: utilizing the area array images and the linear array images to identify each professional device, and outputting the identified professional device information.

[0041] Step 103 matches and fuses the mileage count with the device based on the time stamp of the device information, and outputs the fused mileage, including: time-domain alignment and fusion of the area array visual mileage count and the wheel pulse encoder mileage count, matching the time stamp of the device information, and outputting the fused mileage.

[0042] Then, step 104 matches the device in the fused mileage with the device in each professional existing account, and outputs the matching pair, and step 105 establishes a mapping relationship between the fused mileage and the device-level mileage of each professional, and outputs a multi-professional mileage account in a unified reference system based on the mapping relationship.

[0043] The embodiment of the present application constructs a unified, objective and mutually checkable spatial reference and mapping mechanism, and can realize accurate mileage alignment and comprehensive application between different professional accounts.

[0044] Figure 2 A specific example of the railway multi-professional account mileage mapping method based on machine vision in the embodiment of the present application is shown in FIG. Figure 2 The embodiment of the present application mainly includes input and preset, key device positioning, mileage fusion calculation, account device matching and account mapping.

[0045] The input and preset part of the embodiment of the present application:

[0046] The area array images and the linear array images are collected at the same time, and the wheel pulse encoder mileage count and the area array visual mileage count are obtained; the existing accounts and device dictionaries, rules of railway professionals such as maintenance, signal and power supply are imported, and a unified time axis and data structure are established, thereby providing standardized input for subsequent detection, fusion and mapping.

[0047] The key device positioning part of the embodiment of the present application:

[0048] The key equipment detection and OCR recognition (Optical Character Recognition) are performed on the area array image and the linear array image in parallel, and the device type, text attribute information (number / direction / mileage mark, etc.), timestamp and image coordinates are output; multi-source result association is performed in the same time window to generate a de-duplicated "detection equipment event" as a basic anchor point for subsequent mileage assignment and account matching.

[0049] The mileage fusion calculation part of the embodiment of the application:

[0050] The visual odometer and the encoder mileage are time-domain aligned and fused (such as Kalman filtering / factor graph smoothing) to form a continuous, monotonic and anti-slip fusion mileage curve S(t); according to the event timestamp The fusion mileage m=S(t) is assigned to each detection equipment to obtain an "equipment-fusion mileage" anchor point set, and the position of the equipment in the unified reference system is quantified.

[0051] The account equipment matching part of the embodiment of the application:

[0052] The detection equipment is matched with each professional account item by taking the detection equipment information as a constraint to output a matching pair 。

[0053] Among them, is the professional account mileage, is the corresponding fusion mileage. The scoring mechanism and the outlier rejection method can be used to improve the matching reliability and robustness.

[0054] The account mapping part of the embodiment of the application:

[0055] Based on the reliable anchor points in the previous step, a monotonic mapping function of "professional account mileage→fusion mileage" is constructed , and piecewise linear or regular spline is preferentially used to balance accuracy and smoothness; at the same time, the inverse mapping is provided Support cross-professional conversion and unified statistics, and write back the "multi-professional equipment-fusion mileage" result to form a mapping account after rectification.

[0056] The output part of the embodiment of the application:

[0057] The professional mapping functions and parameters are output, and a multi-professional mileage account in the unified reference system is generated to support cross-professional retrieval, statistical analysis and collaborative operation application.

[0058] In the embodiment of the present application, professional equipment recognition is performed using a surface array image and a linear array image, and the recognized professional equipment information is output, which can include: automatically positioning key equipment of each profession from the linear array image and the surface array image through a target detection algorithm, and outputting device category, confidence, pixel coordinates, and timestamp information.

[0059] The track maintenance professional equipment includes, for example, a turnout, a guard rail, an insulating joint, a rail joint, a fishplate, a fastener, a sleeper, a track slab, a center rail / switch frog, etc.

[0060] The signal maintenance professional equipment includes, for example, a balise, a switch machine, a signal machine, a turnout indicator, an axle counting sensor, a wayside equipment box, a compensation capacitor, etc.

[0061] The power supply professional equipment includes, for example, a support column, a suspension column, a cantilever, a positioner, a sectional insulator, a disconnector, a compensation device, a guy wire / anchor point of a bearing cable, etc.

[0062] In order to improve the detection accuracy of the professional equipment, in an embodiment, professional equipment recognition is performed using a surface array image and a linear array image, and the recognized professional equipment information is output, which can include: the surface array image and the linear array image are respectively preprocessed and then input into a pre-constructed target detection network trained, and the recognized professional equipment information is output; the architecture of the target detection network includes a backbone network, a neck network, and a head network; the backbone network includes a C2f module, an ELAN module, and a lightweight multi-core hollow pooling network; the lightweight multi-core hollow pooling network introduces a hollow convolution in a spatial pyramid pooling structure SPP; the neck network uses a BiFPN network for top-down and bottom-up multi-scale fusion, and introduces a weighted fusion and a channel attention mechanism; and the head network includes a classification network and a regression network.

[0063] Figure 3 The processing flowchart of the target detection network in the embodiment of the present application is shown in FIG. 1. Figure 4 The structure diagram of the target detection network in the embodiment of the present application is shown in FIG. 2. Figure 3 、 Figure 4 , Figure 4 Different colors in the figure represent different function module processing. The embodiment of the present application is improved on the basis of the YOLO algorithm. Before inputting into the target detection network, the surface array image and the linear array image are preprocessed.

[0064] For the surface array image: multi-scale scaling (for example, aligning the minimum side to {640, 800, 1024}), small target copying and pasting, and random scaling / color enhancement, and modal normalization;

[0065] For line array images: adopt row-wise sliding window reorganization and time series splicing, the physical length of the window is calibrated by the encoder mileage (such as 5-15 m / window), and the adjacent windows are 50-70% overlapped to improve the recall.

[0066] Finally, the modal specific normalization is performed on the area array image and the line array image. For example, DSBN (Domain Specific Batch Normalization) and GroupNorm (Group Normalization) are used to reduce the distribution difference between different imaging modalities and reduce the modal domain offset.

[0067] Among them, the row-wise sliding window reorganization can include:

[0068] The line-by-line images continuously collected by the line array camera are segmented and reorganized into a long strip image according to a fixed "physical length". This is equivalent to splicing several continuous rows into an image window covering a small section of the line distance.

[0069] An overlap (such as more than half overlap) is set between each window to ensure that adjacent windows cover the same physical area, thereby avoiding missed detection and facilitating subsequent cross-window merging.

[0070] Among them, the time series splicing and result merging can include:

[0071] A detection is performed on each window independently to obtain the position and category of the device and other information.

[0072] The detection results in the window are described using a unified mileage coordinate (based on the known correspondence between the collection starting point and the row sequence), so that the results of different windows fall on the same coordinate axis.

[0073] The results of "apparently the same target" in adjacent windows are merged: if their mileage positions are very close, the categories are consistent, and the appearance positions in the overlapping area are consistent, they are considered as the same target, and only one is retained (usually the one with higher confidence or weighted average position is retained).

[0074] The backbone network adopts a C2f module (lightweight feature fusion module) and an ELAN module (efficient layer aggregation module), and retains high-resolution branches; a CBAM (convolutional block attention module) is inserted after shallow stages P2 and P3 to strengthen small target features; SPPF (fast spatial pyramid pooling) is replaced by SPPF-Dilated (a dilated convolution variant of fast spatial pyramid pooling, lightweight multi-core hollow pooling) to expand the receptive field and control the amount of calculation. Among them, the Stem convolution (the first convolution layer at the input end) performs initial convolution processing on the input image, and the features pass through Stage P2 to Stage P5 (the step size increases stage by stage) in turn, and finally enter the SPPF-Dilated processing.

[0075] The neck network (Neck, feature fusion layer) receives the outputs of the backbone network Stage P2 to Stage P5 and SPPF-Dilated, respectively, uses BiFPN (bidirectional feature pyramid network, top-down and bottom-up multi-scale fusion) for feature fusion, introduces learnable weighting and lightweight channel attention, and adds P2_head (a small target detection head for P2 layer, focusing on small size components).

[0076] The head network (Head, detection head) adopts a decoupled classification head and regression head, including four scale outputs of Head P2 to Head P5; the classification uses Focal Loss (focal loss) or Varifocal Loss (variable focal loss), and the regression uses DFL (distributed focal loss) and SIoU (separable IoU loss). Anchor boxes are adaptively clustered based on IoU (intersection over union) by K-means++ (improved K-means clustering method) or k-medoids (K center point clustering method), and are generated by mode or class cluster; the positive and negative sample assignment uses Task-Aligned Assigner (task-aligned assigner) or SimOTA (simplified optimal transport assignment).

[0077] The inference stage (post-processing) uses DIoU-NMS (distance IoU non-maximum suppression) or Soft-NMS (soft non-maximum suppression); temporal NMS and cross-window fusion are performed on line array sequences, and ByteTrack (lightweight target tracking) can be optionally used for trajectory smoothing. Among them, the DIoU-NMS method is used to consider the frame center distance and the intersection over union in the non-maximum suppression at the same time, to reduce the false deletion of adjacent targets and improve the positioning accuracy; the Soft-NMS method is used in dense or small target scenarios to replace hard deletion with overlap degree smoothing attenuation of candidate frame scores, to improve the recall rate and alleviate the missed detection.

[0078] Finally, the recognition result is output, including device analogy, confidence, pixel box, timestamp and other information.

[0079] In an embodiment, the target detection network can be trained in the following way:

[0080] Collect historical image data along the railway line; stratified sampling is performed on the collected data according to the section mileage and time to obtain sample data; wherein the sampling rule meets the balance of cross-line, day and night, and different weather data; data augmentation and labeling are performed on the area array image and line array image in the sample data to obtain training data; model training is performed using the training data; wherein the optimizer is AdamW or SGD, and the learning strategy adopts a preheating cosine annealing strategy.

[0081] In the embodiment, the data construction before training is stratified sampling according to the section mileage and time, ensuring the balanced distribution of cross-line, day and night, and weather scenes; at the same time, the data of adjacent mileage sections or adjacent time windows are divided into the same data division to avoid the evaluation being too high due to “adjacent frame leakage”. Resampling and instance balancing are performed on small targets and low-frequency classes (such as axle counters and section insulators). The terms are supplemented as follows:

[0082] Small target: refers to a target that appears very small in size on an image, which can be determined by absolute pixels or relative proportion. For example, the short side is less than about 16 pixels or the target area is less than about 32*32 pixels; or the target area accounts for less than about 0.1% of the total image area (the threshold can be adjusted according to the project data distribution).

[0083] Low-frequency class: refers to a class with a very low sample proportion in the overall training set, for example, the proportion of the number of instances of this class is less than 1%-5% or the absolute number of samples is less than a certain lower limit (such as hundreds), resulting in a class that is difficult for the model to learn.

[0084] Resampling: a strategy for redistributing sampling probabilities to alleviate class imbalance. Common practices include oversampling (copying or combining data augmentation to generate variants) for minority classes / rare working condition samples, and undersampling (reducing the probability of being selected) for majority class samples, so that the occurrence frequency of each class in training is closer.

[0085] Instance balancing: balancing the training signal from the “instance” rather than the “image” level. Typical methods include: limiting the upper and lower limits of the number of instances of each class when constructing a batch; setting weights according to the number of instances for images containing multiple instances to avoid a single image with a large number of instances dominating the gradient; in the loss layer, focal loss or class balancing loss can be used to give minority classes and small targets more reasonable training weights.

[0086] Data augmentation and labeling are performed on the area array image and line array image in the sample data, including:

[0087] Face array image augmentation includes one or any combination of the following: Mosaic (control intensity) enhancement, MixUp enhancement, small target copy and paste and random scaling, superimposed random perspective, rotation, cropping, superimposed HSV jitter, superimposed motion blur, JPEG compression, increased analog noise, and covering high-speed and low-light scenes;

[0088] Linear array image augmentation and labeling includes: sliding window reorganization, time sequence splicing, cross-window label mapping of mileage alignment, random step jitter and row noise injection, time sequence block splicing simulating speed changes, and cross-window continuous labeling for super-long devices. In the linear array sliding window, the "step" refers to the distance (or equivalent number of rows) between the windows when reorganizing the continuous rows into a "physical length window" for input division, and is related to the detection coverage density / overlap rate. The time sequence block refers to a continuous sequence of linear arrays divided by time or mileage (for example, continuously 0.5-2 seconds, or continuously 10-30 meters), with internal rows / windows adjacent and external blocks relatively independent.

[0089] The optimizer can be AdamW (weight decay optimizer, wd=0.05) or SGD (stochastic gradient descent with Nesterov momentum); the learning rate uses Warmup Cosine (warmup + cosine annealing, warmup for 5-10 training rounds), combined with EMA weight smoothing (exponential moving average of parameters), FP16 mixed precision (half-precision and single-precision mixed training), and gradient clipping (threshold 1.0) to stabilize training. The loss function is a combination of classification Focal Loss / Varifocal Loss (focal class loss / varifocal loss) and regression SIoU+DFL (separable IoU loss + distributed focal regression loss), and IoU-aware Objectness (IoU-aware objectness) is introduced; class and sample imbalance is alleviated through difficult example mining (key sampling / weighted difficult samples) and positive / negative ratio upper limit control (limiting the positive / negative sample ratio).

[0090] The training process recommends "general data pre-training → linear domain incremental fine-tuning", and uses phased freezing and unfreezing (first head, then whole network) to improve convergence stability.

[0091] The linear domain incremental fine-tuning can include:

[0092] The face array image and linear array image of the target linear are used to cover day and night, sunny and rainy, tunnels / bright lines, different speeds and working conditions; the device class includes all targets involved in the account of transponders, pillars, fasteners, etc.;

[0093] Sample by mileage and time, leave independent verification / test sections, avoid adjacent frame / adjacent mileage leakage;

[0094] If the target line sample is insufficient, a small amount of "similar line data in the same domain" (same model, same camera parameters) can be introduced to supplement, and the proportion does not exceed 20% of the total.

[0095] After the above key equipment positioning recognition, the mileage fusion calculation is performed.

[0096] In an embodiment, the area array visual odometer count is obtained based on the visual odometer formed by the area array image, which can include: calculating the pose increment between adjacent area array image frames, and calculating the area array visual odometer count based on the pose increment and inertial navigation data.

[0097] In an embodiment, the wheel pulse encoder odometer count can be obtained in the following way: through the preset line number and the corresponding relationship between the travel mileage, and the image line number collected by the line array camera, the wheel pulse encoder odometer count is converted; wherein the line array camera is triggered by the wheel encoder pulse to collect.

[0098] Figure 5 For the fusion mileage calculation process schematic diagram in the embodiment of the application, refer to Figure 5 The relative mileage information of each professional key equipment in the embodiment of the application is obtained by image and mileage sensor cooperative calculation.

[0099] For area array images, since time-triggered shooting cannot directly accumulate mileage, the pose increment between adjacent frames is estimated by visual odometer (VO, Visual Odometry), and the visual-inertial odometer (VIO, Visual-Inertial Odometry) is formed by combining IMU (inertial navigation unit, which can collect inertial navigation data such as acceleration and angular velocity), which can guarantee the scale observability while improving the robustness to low-texture, motion blur and other scenes, and obtain continuous visual mileage curve S_vo(t).

[0100] For line array images, the line array camera is triggered by the wheel encoder pulse to collect, and each collected image row corresponds to a fixed travel distance, and the pixel-mileage coefficient (i.e. the preset line number and the corresponding relationship between the travel mileage collected by the line array camera) is converted, and the segment mileage increment can be expressed as:

[0101] ;

[0102] wherein is the accumulated line number (or equivalent pulse number) of the segment.

[0103] With reference to Figure 5In an embodiment, the mileage fusion takes rigorous calibration and time synchronization as a prerequisite. First, the unified time axis alignment (hardware synchronization or post-processing alignment, deviation controlled within milliseconds) of the area array camera, linear array camera, IMU, and encoder is performed, the external parameter calibration of the camera-IMU and the encoder-vehicle body is completed, and the scanning rate of the linear array camera and the pixel-mileage coefficient are calibrated, thereby laying a foundation for the integration of multi-source data.

[0104] In an embodiment, the time-domain alignment and fusion of the area array visual odometer count and the wheel pulse encoder mileage count, the matching of the timestamps of the device information, and the output of the fused mileage can include: using the unscented Kalman filtering method to perform the time-domain alignment and fusion of the area array visual odometer count and the wheel pulse encoder mileage count, match the timestamps of the device information, and output the fused mileage.

[0105] After the pixel accumulation of the linear array mileage calculation and the continuous accumulation of the area array visual odometer (VIO) interframe pose + IMU pre-integration, the unscented Kalman filtering method is used for fusion.

[0106] For example, using the unscented Kalman filtering method to perform the time-domain alignment and fusion of the area array visual odometer count and the wheel pulse encoder mileage count, match the timestamps of the device information, and output the fused mileage can include:

[0107] continuously calculating the vehicle's pose from the starting point using inertial navigation data; the pose includes mileage, speed, and orientation;

[0108] Different weights are assigned to the area array visual odometer count and the wheel pulse encoder mileage count, and based on the unscented Kalman filtering method, the vehicle's pose calculated by the inertial navigation data is corrected using the area array visual odometer count and the wheel pulse encoder mileage count after the weights are assigned, and the fused mileage is output; wherein when wheel slip is detected, the weight of the wheel pulse encoder mileage count is reduced, and when the information entropy of the area array image is lower than the preset information entropy value, the weight of the area array visual odometer count is reduced.

[0109] Continuing to refer to Figure 5 During the fusion process, the unscented Kalman filtering is used to fuse the multi-source mileage information into a more reliable mileage curve through "prediction and correction". The specific method is: using the IMU as a "mileage meter predictor" to continuously calculate the mileage, speed, and orientation of the vehicle (the state contains mileage s, speed v, and heading angle The system also uses IMU acceleration / gyroscope zero bias (b_a, b_g) to correct prediction errors. Simultaneously, it uses two observations to correct prediction errors: the encoder provides odometer readings (corresponding to s), and the VIO of the array channel provides visual inertial odometer readings (corresponding to s or the odometer increment at adjacent times). When wheel slippage or visual degradation is detected (e.g., blurred image, sparse texture, excessive darkness, or strong reflection, i.e., the array image information entropy is lower than the preset information entropy value), the system automatically reduces the weight of the corresponding observation and uses statistical thresholds and physical constraints (e.g., monotonic odometer readings, reasonable speed changes) to eliminate outliers. The final output is a monotonic, continuous, and slip-resistant fused odometer S(t), used for subsequent equipment positioning, ledger matching, and cross-disciplinary odometer mapping.

[0110] In a linear array channel, the linear array camera is triggered by pulses from the wheel encoder to acquire data. Each row of pixels corresponds to a fixed travel distance, and the segment mileage increment can be expressed as:

[0111] ;

[0112] in, This represents the cumulative number of rows / pulses in this segment.

[0113] For abnormal operating conditions such as slippage, dynamic weight reduction and median smoothing are performed by combining speed / acceleration thresholds to correct missing pulses and row noise; at the same time, a mapping from row index to time is established to ensure seamless splicing across windows and accurate fusion in subsequent time sequence.

[0114] In the area array channel, visual inertial odometry (VIO) is uniformly used for odometry estimation: by leveraging the tight coupling between the camera and the IMU, pose, velocity and zero bias are estimated simultaneously based on IMU pre-integration and sliding window optimization, ensuring scale consistency and significantly improving robustness under conditions such as low texture, motion blur and illumination changes; by tangential accumulation of pose increments in adjacent frames, a continuous and monotonic visual inertial odometry curve is obtained, and combined with degradation detection and static constraints to suppress drift, providing a highly reliable odometry input for subsequent multi-source fusion and ledger mapping.

[0115]

[0116] : No. The mileage increment estimated by VIO at any given moment. Indicates the initial time;

[0117] The displacement vector between two adjacent frames;

[0118] The tangential unit vector (dimensionless) is used to project displacement onto the direction of the line's movement;

[0119] From the initial moment to The cumulative visual inertial mileage.

[0120] The fusion layer employs unscented Kalman filtering to fuse multi-source odometer information into a more reliable odometer curve through "prediction and correction on the same side." Specifically, an IMU is used as the "odometer predictor," continuously calculating the vehicle's mileage, speed, and heading (state includes mileage s, speed v, and heading angle). And the IMU's acceleration / gyroscope bias Simultaneously, the two observation odometers are used to correct the prediction error—the encoder provides the odometer observation (corresponding to s), and the VIO of the array channel provides the visual inertial odometer observation (corresponding to s or the odometer increment at adjacent times). The formula is shown below:

[0121]

[0122] s: Cumulative mileage along the tangential direction of the line, in meters;

[0123] v: Tangential velocity along the track, in m / s;

[0124] Heading angle (the angle between the vehicle and the track tangent), unit: rad / s;

[0125] IMU accelerometer zero bias (along the tangential channel; in the three-axis case, it can be extended to a vector), unit m / s²;

[0126] IMU gyroscope zero bias (yaw / vertical axis channel; in the three-axis case, it can be extended to a vector), unit rad / s;

[0127] k: Index of discrete time step; Indicates time Predict the prior at time k;

[0128] Discrete sampling period, in seconds;

[0129] Tangential acceleration is obtained by projecting the IMU acceleration to the tangential direction of the line after zero bias removal and gravity and attitude compensation.

[0130] : Heading angular velocity (yaw angular velocity) measured by IMU gyroscope, unit: rad / s;

[0131] : , Process noise (random walk), zero-mean Gaussian noise;

[0132] Mileage prediction: s = s + v * dt According to Integration;

[0133] Velocity prediction: v = v + a * dt Integral update;

[0134] Heading prediction: Measured by the gyroscope Zero offset Post-integral update;

[0135] Zero offset evolution: , According to the random walk model, it changes slowly.

[0136] When the wheel slip or visual degradation (such as low texture, blur) is detected, the system will automatically reduce the weight of the corresponding observation; and use statistical threshold and physical constraints (such as mileage monotony, reasonable speed change) to remove outliers. The final output is a monotonic, continuous, anti-slip fusion mileage, while giving the uncertainty, which is used for subsequent device positioning, account matching and cross-professional mileage mapping.

[0137] In terms of constraints and quality control, the lateral and longitudinal drift is limited by the one-dimensional main motion prior of the track, the acceleration / angular velocity / velocity change rate is limited, the mileage monotonicity is forced during the same direction driving, and finally S(t) is backfilled to each row of the line array and each frame of the area array to establish a unified mapping from "pixel / frame to mileage".

[0138] After the above completion of the mileage fusion calculation, the account device matching calculation is performed.

[0139] In an embodiment, the devices in the fusion mileage are matched with the devices in the existing account of each profession, and the matching pairs are output, including:

[0140] The devices in the fusion mileage are coarsely matched with the devices in the existing account of each profession under the constraints of device type, text attribute information, adjacent device sequence relationship and spatial neighborhood topology, and a candidate corresponding set is formed;

[0141] The candidate corresponding set is processed by scoring, consistency checking and outlier removal to obtain the matching pairs.

[0142] Among them, the spatial neighborhood topology constraint means that "close or not close" is used for preliminary screening: the detected device is placed on the unified mileage axis, and whether its position with the account record is within a small acceptable range is determined; if it is out of range, it is first removed or weighted, and if it is within the range, it is retained as a candidate. In the coarse matching, the direction of increasing line mileage or the agreed driving direction is taken as the reference, and the values are arranged from small to large (or from top to bottom); the detection result and the account should maintain the same front and rear order in the same direction.

[0143] Figure 6 The schematic diagram of the account equipment matching calculation process in the embodiment of the present application is shown in FIG. 1. Figure 6 As shown in FIG. 1, first, the account equipment rough matching and the establishment of the corresponding relationship are performed.

[0144] Based on the detection results and each professional account (including the equipment item, the account mileage, and the adjacent relationship), first, the rough matching is performed according to the equipment type, the OCR number / direction, the adjacent equipment sequence relationship, and the time window within the fusion mileage near neighbor constraint to form a candidate corresponding set; then, the scoring and consistency checking are adopted and combined with RANSAC (random sample consensus algorithm) to remove outliers, and finally, the one-to-one or one-to-many corresponding relationship of the detected equipment and each professional account equipment is determined. For example, the same point double equipment: there are two transponder account records of uplink and downlink in the same physical position, and a single detection can satisfy the two records at the same time (marked as main matching + secondary matching, the main matching is preferred, and the secondary matching is reserved for verification). Combined record: there are multiple records of “main column” and “accessory member (suspender column / wrist arm)” in the account for the same column, and a single “column” detection corresponds to multiple accounts (as a group matching processing, the order is preserved within the group).

[0145] The scoring rules are, for example, as follows:

[0146] ① Type consistency score: full score for the same type, otherwise 0.

[0147] ② Text matching score: OCR number / direction and account similarity (edit distance / regularity, full score for complete consistency, fuzzy matching score reduction).

[0148] ③ Mileage near neighbor score: linear decay from 0 to full score according to the mileage residual (such as 5-10 meter tolerance for transponder, 8-15 meter tolerance for column).

[0149] ④ Sequence consistency score: add points for the same order of the adjacent equipment (the more neighbors passed, the higher the score).

[0150] ⑤ Neighborhood topology score: add points for common combination relationships (such as “signal machine→transponder” and “column→wrist arm→section insulator”).

[0151] ⑥ Observation quality score: OCR confidence, detection confidence, and cross-window / cross-frame stability (consistent repeated observation) add points.

[0152] ⑦ Comprehensive score = S(type) + S(text) + S(mileage) + S(sequence) + S(topology) + S(quality); above the threshold value enters the candidate set.

[0153] The consistency checking is, for example, as follows:

[0154] ① Sequence monotonicity: the order of candidate matches in a mileage window is consistent with the account (at least meet the 2 adjacent constraints).

[0155] ② Distance tolerance: the distance between adjacent devices falls within a reasonable range (outside the tolerance is excluded or reduced).

[0156] ③ Bidirectional verification: detection→account and account→detection do not conflict; if the same account entry is hit by multiple detections, the one with the highest score and the most consistent global sequence is retained.

[0157] ④ RANSAC robustness: estimate the local linear relationship between "account mileage→fusion mileage" in this section using RANSAC, and retain the inlier candidates and exclude outliers.

[0158] In an embodiment, before establishing the mapping relationship between the fusion mileage and the device-level mileage of each specialty based on the matching pairs, and outputting the multi-specialty mileage account in the unified reference system based on the mapping relationship, the method can further include:

[0159] Based on the fusion mileage, the timestamps of the area array image and the line array image are interpolated and extrapolated, and each device identified using the area array image and the line array image is revalued to obtain the machine vision mileage in the unified reference system; the machine vision mileage includes the correspondence between the mileage and the device;

[0160] Based on the matching pairs, the mapping relationship between the fusion mileage and the device-level mileage of each specialty can include:

[0161] Based on the matching pairs, the mapping relationship between the device-level mileage of each specialty and the machine vision mileage, and the inverse mapping relationship between the machine vision mileage and the device-level mileage of each specialty.

[0162] Reference Figure 6 , the visual data mileage revaluation driven by the fusion mileage is performed. Using the obtained fusion mileage curve , the timestamps of the area array and the line array image are interpolated / extrapolated, and each detection device event is revalued as the machine vision mileage in the unified reference system , the common suppression of visual scale drift and encoder slip is realized, and the key device mileage annotation with high consistency is obtained.

[0163] Finally, the account-machine vision correspondence and multi-specialty mapping are constructed.

[0164] Based on the matching anchor point set , the monotonic mapping of "account mileage→machine vision mileage" is estimated for each specialty p, and the inverse mapping For cross-professional conversion and joint statistics. Output mapping parameters, thereby obtaining consistent mileage mapping relationship between the account of each professional, supporting cross-professional retrieval, statistics and collaborative operation.

[0165] In summary, the embodiment of the application faces the key facility images obtained by the vehicle-mounted area array camera and linear array camera of the detection vehicle, reliably detects the facility position in the image by using an artificial intelligence algorithm, combines the visual odometer formed by the area array camera and the wheel pulse coding mileage driven by the linear array camera, and calculates the unified mileage coordinates of the equipment; further, the detected equipment is matched with the corresponding equipment in each professional account, and the equipment-level mileage mapping relationship between different professionals is established, thereby eliminating the inconsistency of the mileage in the cross-professional account from the source, supporting the alignment, joint statistics and collaborative operation of multi-professional data.

[0166] The following is an example of a mileage mapping method between a signal repeater account and a catenary account.

[0167] Step 1, multi-source acquisition and detection.

[0168] Synchronously acquire the data of the area array camera and the linear array camera, and perform target detection and attribute extraction (type, number / direction, timestamp, image coordinates) on the catenary support and the repeater.

[0169] Step 2, mileage calculation and fusion.

[0170] The encoder mileage of the linear array channel and the visual inertial mileage of the area array channel are calculated respectively, time synchronization and fusion are completed, and a unified machine vision mileage curve S(t) is formed.

[0171] Step 3, equipment-level coarse calibration and assignment (repeater).

[0172] The detected repeater is coarsely matched with the repeater account by taking the device type, OCR number / direction, adjacent sequence relationship and mileage near neighbor as constraints, and the matching anchor point is obtained; the machine vision mileage is assigned to each repeater detection event according to S(t) .

[0173] 4, equipment-level coarse calibration and assignment (catenary support).

[0174] The detected catenary support is coarsely matched with the support account by using the same constraints and processes, and the anchor point is established; the machine vision mileage of the support event is assigned according to S(t) .

[0175] 5, cross-professional mapping relationship construction.

[0176] The machine vision mileage To unify the benchmark, based on the matched anchor point set of the two types of equipment, a monotonic mapping of “account mileage → machine vision mileage” and its inverse mapping are fitted, a unified mileage conversion relationship between the balise account and the overhead line support account is established, and the mapping parameters and quality indicators are output.

[0177] Figure 7 A schematic diagram of the mapping result between the balise account and the overhead line support account mileage in the embodiments of the present application is shown in FIG. 1, where the upper list is the balise account, listing the equipment number, line mileage and attributes; the lower yellow list is the overhead line support account, listing the support number, and the overhead line support account can also include unified mileage (such as 5.640), type (anchor column) and other fields; the middle diagram is a “machine vision mileage” view formed by image stitching, where the sleepers, supports and balises are presented as icons. Figure 7 A schematic diagram of the mapping result between the balise account and the overhead line support account mileage in the embodiments of the present application is shown in FIG. 1, where the upper list is the balise account, listing the equipment number, line mileage and attributes; the lower yellow list is the overhead line support account, listing the support number, and the overhead line support account can also include unified mileage (such as 5.640), type (anchor column) and other fields; the middle diagram is a “machine vision mileage” view formed by image stitching, where the sleepers, supports and balises are presented as icons.

[0178] Figure 7 The middle red connecting line represents the matched anchor point of “account entry vision detection result”, and the matching is confirmed through type adjacent sequence and near mileage constraints, and after consistency and RANSAC rejection of outliers.

[0179] The embodiments of the present application also provide a railway multi-specialty account mileage mapping device based on machine vision, as described in the following embodiments. Since the device solves the problem by the same principle as the railway multi-specialty account mileage mapping method based on machine vision, the implementation of the device can be referred to the implementation of the railway multi-specialty account mileage mapping method based on machine vision, and the repeated parts will not be described herein.

[0180] Figure 8 A schematic diagram of the railway multi-specialty account mileage mapping device based on machine vision in the embodiments of the present application is shown in FIG. 2, where the device 800 includes: Figure 8

[0181] a data acquisition module 801, configured to acquire image data and mileage count along the railway line;

[0182] a target recognition module 802, configured to recognize each specialty equipment by using the image, and output the recognized each specialty equipment information; the equipment information includes a timestamp;

[0183] a fused mileage calculation module 803, configured to match and fuse the mileage count and the equipment based on the timestamp of the equipment information, and output a fused mileage; the fused mileage includes the mileage and the correspondence of each specialty equipment;

[0184] ​​The device account matching calculation module 804 is configured to match the devices in the fusion mileage with the devices in the existing accounts of various professions, and output matching pairs; the matching pairs include the correspondence between the mileage of the devices in the existing accounts of various professions and the mileage of the devices in the fusion mileage; based on the matching pairs, a mapping relationship between the fusion mileage and the device-level mileages of various professions is established, and a multi-professional mileage account in a unified reference system is output based on the mapping relationship.

[0185] In an embodiment, the image data includes area array images and line array images, and the mileage count includes area array visual mileage count and wheel pulse encoder mileage count; the area array visual mileage count is obtained based on visual mileage formed by the area array images;

[0186] The target identification module 802 is specifically configured to:

[0187] The area array images and the line array images are used to identify the devices of various professions, and the identified device information of various professions is output;

[0188] The fusion mileage calculation module 803 is specifically configured to:

[0189] The area array visual mileage count and the wheel pulse encoder mileage count are time-domain aligned and fused, the timestamps of the device information are matched, and the fusion mileage is output.

[0190] In an embodiment, the target identification module 802 is specifically configured to:

[0191] The area array images and the line array images are respectively input to a pre-constructed and pre-trained target detection network after being preprocessed, and the identified device information of various professions is output; the target detection network architecture includes a backbone network, a neck network, and a head network; the backbone network includes a C2f module, an ELAN module, and a lightweight multi-core hollow pooling network; the lightweight multi-core hollow pooling network introduces a hollow convolution in a spatial pyramid pooling structure SPP; the neck network uses a BiFPN network for top-down and bottom-up multi-scale fusion, and introduces a weighted fusion and a channel attention mechanism; the head network includes a classification network and a regression network.

[0192] In an embodiment, the target detection network is trained in the following manner:

[0193] Image data along a railway line is collected historically;

[0194] The collected data is stratified sampled according to section mileage and time to obtain sample data; wherein the sampling rule satisfies the balance of cross lines, day and night, and different weather data;

[0195] The area array images and the line array images in the sample data are data-augmented and labeled to obtain training data;

[0196] The model is trained using training data; wherein the optimizer is AdamW or SGD, and the learning strategy adopts a preheating cosine annealing strategy.

[0197] In an embodiment, the planar array visual odometry count is obtained based on visual odometry formed by the planar array image, comprising:

[0198] The pose increment between adjacent planar array image frames is calculated, and the planar array visual odometry count is calculated based on the pose increment and the inertial navigation data.

[0199] In an embodiment, the wheel pulse encoder odometry count is obtained as follows:

[0200] The wheel pulse encoder odometry count is obtained by converting the corresponding relationship between the preset number of rows and the travel mileage collected by the linear array camera and the number of rows of images collected by the linear array camera; wherein the linear array camera is triggered by the wheel encoder pulse to collect.

[0201] In an embodiment, the fusion mileage calculation module 803 is specifically configured to:

[0202] The planar array visual odometry count and the wheel pulse encoder odometry count are time-aligned and fused in time domain using an unscented Kalman filtering method, the timestamps of the device information are matched, and the fused mileage is output.

[0203] In an embodiment, the fusion mileage calculation module 803 is specifically configured to:

[0204] The attitude of the vehicle is continuously calculated from the starting point using the inertial navigation data; the attitude includes mileage, speed and orientation;

[0205] Different weights are assigned to the planar array visual odometry count and the wheel pulse encoder odometry count, the vehicle attitude calculated by the inertial navigation data is corrected using the planar array visual odometry count and the wheel pulse encoder odometry count after the weights are assigned based on the unscented Kalman filtering method, and the fused mileage is output; wherein the weight of the wheel pulse encoder odometry count is reduced when wheel slip is detected, and the weight of the planar array visual odometry count is reduced when the information entropy of the planar array image is lower than a preset information entropy value.

[0206] In an embodiment, the device information further includes device type, text attribute information;

[0207] The device account matching calculation module 804 is specifically configured to:

[0208] The device type, the text attribute information, the adjacent device sequence relationship and the spatial neighborhood topology are used as constraints to perform coarse matching of the device in the fused mileage with the devices in the existing accounts of each profession, to form a candidate corresponding set;

[0209] The matching pairs are obtained by scoring, consistency checking and outlier elimination on the candidate correspondence set.

[0210] In an embodiment, the apparatus 800 further comprises:

[0211] The visual data mileage reassignment module is configured to, before the device account matching calculation module 804 establishes the mapping relationship between the fusion mileage and the device-level mileage of each specialty based on the matching pairs and outputs the multi-specialty mileage account in the unified reference system based on the mapping relationship, perform interpolation extrapolation on the time stamps of the area array image and the linear array image based on the fusion mileage, reassign each device identified using the area array image and the linear array image, and obtain the machine vision mileage in the unified reference system; the machine vision mileage comprises the correspondence between the mileage and the device.

[0212] The device account matching calculation module 804 is specifically configured to:

[0213] The mapping relationship between the device-level mileage of each specialty and the machine vision mileage and the inverse mapping relationship between the machine vision mileage and the device-level mileage of each specialty are established based on the matching pairs.

[0214] The embodiment of the present application also provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned railway multi-specialty account mileage mapping method based on machine vision when executing the computer program.

[0215] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned railway multi-specialty account mileage mapping method based on machine vision.

[0216] The embodiment of the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the above-mentioned railway multi-specialty account mileage mapping method based on machine vision.

[0217] The process and its coordination in this invention embodiment include: acquisition → detection → VIO / encoder UKF fusion → coarse matching → mileage reassignment → anchor-based monotonic mapping / inverse mapping → cross-professional conversion and backfilling. This invention embodiment proposes a mileage-detection linkage processing mechanism combining area array VIO strategy and linear array sliding window / temporal NMS, and proposes anchor point construction rules and a matching scoring system: type + OCR + sequence + nearest neighbor mileage + topology consistency, with robust RANSAC matching. It employs the construction and application of monotonic piecewise linear / monotonic spline mapping and inverse mapping for cross-professional mileage calculation, integrates a mileage-driven event-level mileage reassignment mechanism, and improves the structure and training deployment strategies of the target detection network in railway small / dense target scenarios (CBAM, BiFPN, P2_head, SIoU, anchor frame clustering, SimOTA, linear array-specific enhancement). This invention embodiment has the following beneficial effects:

[0218] Multi-professional mileage unification and inter-calculation: Based on anchor points, a monotonic mapping and inverse mapping of "ledger → machine vision" are constructed to realize the unification and inter-calculation of mileage between engineering, electrical engineering, power supply and other professions, and eliminate the fragmentation of mileage across professions.

[0219] Facility-level precise positioning: Using machine vision mileage as a unified reference, the detection results are accurately assigned to specific equipment (such as pillars, transponders, etc.), supporting one-click positioning "from mileage to equipment" and clear location description.

[0220] Enhanced reliability through integration: The encoder and VIO are deeply integrated, automatically identifying degradation scenarios such as slippage / low texture and adaptively reducing weights, outputting a monotonous, continuous, slip-resistant mileage curve and uncertainty, significantly improving positioning stability.

[0221] Significantly improved work efficiency: The entire process of detection, assignment, and mapping is completed automatically, reducing the workload of manual secondary positioning and ledger alignment, and speeding up fault location and work order flow.

[0222] The results are verifiable and traceable: the matching process incorporates consistency checks and RANSAC to remove outliers, and outputs quality indicators such as residuals and monotonicity, which facilitates acceptance and tracking.

[0223] Data reuse and collaboration: The ledgers of various specialties can be retrieved, statistically analyzed and jointly evaluated under a unified reference system, supporting multi-disciplinary collaborative operation and maintenance and resource optimization.

[0224] Compatibility and scalability: The solution is compatible with existing ledgers and existing testing equipment, and can be expanded to more equipment categories and line scenarios as needed, with good engineering implementation and scalability.

[0225] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0226] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0227] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0228] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 Figure 1 The steps of the function specified in one or more boxes.

[0229] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A machine vision-based method for mapping mileage in multi-disciplinary railway ledgers, characterized in that, include: Acquire image data and mileage counts along the railway line; It uses images to identify various specialized equipment and outputs information about the identified specialized equipment. The device information includes a timestamp; Based on the timestamp of the device information, the mileage count is matched and merged with the device to output the merged mileage; The integration mileage includes the correspondence between mileage and various specialized equipment; The system matches the equipment in the merged mileage with the equipment in the existing ledgers of each specialty, and outputs matching pairs. The matching pairs include the correspondence between the mileage of the equipment in the existing ledgers of each specialty and the mileage of the equipment in the merged mileage. Based on the matching pairs, a mapping relationship is established between the fused mileage and the equipment-level mileage of each specialty, and a multi-specialty mileage ledger under a unified reference system is output based on the mapping relationship.

2. The method as described in claim 1, characterized in that, The image data includes area array images and line array images; the mileage count includes area array visual mileage count and wheel pulse encoder mileage count; the area array visual mileage count is based on the visual mileage count formed by the area array images. The system utilizes images to identify various specialized pieces of equipment and outputs information about each identified piece of equipment, including: Using area array images and line array images, various professional equipment are identified, and the identified professional equipment information is output. Based on the timestamp of the device information, the mileage count is matched and merged with the device information to output the merged mileage, including: The area array visual mileage count and the wheel pulse encoder mileage count are time-domain aligned and fused, and the timestamps of the device information are matched to output the fused mileage.

3. The method as described in claim 2, characterized in that, Using area array images and line array images, various specialized equipment is identified, and the identified equipment information is output, including: Area array images and linear array images are preprocessed and then input into a pre-built and trained target detection network, which outputs the identified information of various professional equipment. The target detection network architecture includes a backbone network, a neck network, and a head network. The backbone network includes a C2f module, an ELAN module, and a lightweight multi-kernel dilated pooling network. The lightweight multi-kernel dilated pooling network introduces dilated convolution in the spatial pyramid pooling structure (SPP). The neck network uses a BiFPN network for top-down and bottom-up multi-scale fusion and introduces weighted fusion and channel attention mechanisms. The head network includes a classification network and a regression network.

4. The method as described in claim 3, characterized in that, The target detection network is trained as follows: Collect historical image data along the railway line; The collected data were stratified and sampled according to segment mileage and time to obtain sample data; the sampling rules satisfied the balance of data from intersecting routes, day and night, and different weather conditions. Data augmentation and annotation are performed on the area and line array images in the sample data to obtain training data; The model is trained using training data; the optimizer is AdamW or SGD, and the learning strategy is a preheating cosine annealing strategy.

5. The method as described in claim 2, characterized in that, The area array visual odometry is based on visual odometry formed from area array images, including: Calculate the pose increment between adjacent area array image frames, and calculate the area array visual odometry based on the pose increment and inertial navigation data.

6. The method as described in claim 2, characterized in that, The wheel pulse encoder mileage count is obtained as follows: The wheel pulse encoder mileage count is obtained by converting the pre-calibrated line scan camera to collect the correspondence between the preset number of rows and the travel distance, and the number of image rows collected by the line scan camera; wherein, the line scan camera is triggered to collect data according to the wheel encoder pulse.

7. The method as described in claim 5, characterized in that, The area array visual odometer and the wheel pulse encoder odometer are time-domain aligned and fused, matched with the timestamp of the device information, and the fused odometer is output, including: The unscented Kalman filter method is used to perform temporal domain alignment and fusion of the area array visual mileage count and the wheel pulse encoder mileage count, match the timestamp of the device information, and output the fused mileage.

8. The method as described in claim 7, characterized in that, The unscented Kalman filter method is used to perform temporal domain alignment and fusion of the area array visual odometer count and the wheel pulse encoder odometer count, matching the timestamp of the device information, and outputting the fused odometer, including: The vehicle's attitude is continuously calculated from the starting point using inertial navigation data; the attitude includes mileage, speed, and orientation. Different weights are assigned to the area array visual odometer count and the wheel pulse encoder odometer count. Based on the unscented Kalman filter method, the vehicle attitude calculated from the inertial navigation data is corrected using the weighted area array visual odometer count and the wheel pulse encoder odometer count, and the fused odometer is output. Specifically, when wheel slippage is detected, the weight of the wheel pulse encoder odometer count is reduced, and when the information entropy of the area array image is detected to be lower than the preset information entropy value, the weight of the area array visual odometer count is reduced.

9. The method as described in claim 1, characterized in that, The device information also includes device type and text attribute information; Match the devices in the merged mileage with the devices in the existing ledgers of each specialty, and output matching pairs, including: Using equipment type, text attribute information, adjacent equipment sequence relationship and spatial neighborhood topology as constraints, a coarse match is performed between the equipment in the fusion mileage and the equipment in the existing ledgers of each profession to form a candidate corresponding set; The candidate matching set is scored, consistency is checked, and outliers are removed to obtain matching pairs.

10. The method as described in claim 2, characterized in that, Before establishing a mapping relationship between fused mileage and equipment-level mileage for each specialty based on matching pairs, and outputting a multi-specialty mileage ledger under a unified reference system based on the mapping relationship, the following steps are also included: Based on the fused mileage, the timestamps of the array image and the linear image are interpolated and extrapolated. Each device that uses the array image and the linear image for recognition is reassigned to obtain machine vision mileage under a unified reference system. The machine vision mileage includes the correspondence between mileage and device. Based on matching pairs, a mapping relationship is established between fused mileage and device-level mileage for each specialty, including: Based on the matching pairs, establish the mapping relationship between equipment-level mileage and machine vision mileage for each profession, and the inverse mapping relationship between machine vision mileage and equipment-level mileage for each profession.

11. A machine vision-based railway multi-disciplinary ledger mileage mapping device, characterized in that, include: The data acquisition module is used to acquire image data and mileage counts along the railway line; The target recognition module is used to identify various specialized equipment using images and output information about the identified specialized equipment; the equipment information includes a timestamp. The mileage calculation module is used to match and merge mileage counts with equipment based on the timestamps of equipment information, and output the fused mileage; the fused mileage includes the correspondence between mileage and various specialized equipment; The equipment ledger matching and calculation module is used to match the equipment in the integrated mileage with the equipment in the existing ledgers of each specialty, and output matching pairs; the matching pairs include the correspondence between the mileage of the equipment in the existing ledgers of each specialty and the mileage of the equipment in the integrated mileage. Based on the matching pairs, a mapping relationship is established between the fused mileage and the equipment-level mileage of each specialty, and a multi-specialty mileage ledger under a unified reference system is output based on the mapping relationship.

12. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 10.

Images