A Computer Vision-Based Method for Checking the On-Line and Off-Line Counts of Railway Tools and Equipment

Abstract

This invention discloses a computer vision-based method for checking the on-line and off-line inventory of railway tools and equipment. The method includes acquiring images of tools and equipment on-line and off-line at high-speed railway maintenance sections and in warehouses; cropping target regions using a region focusing algorithm to eliminate interference; constructing a diverse training dataset by combining perspective transformation, multi-transformation stitching, and generation methods based on SAM and diffusion models; building a text-visual multimodal rotating target detection model; guiding the model to focus on key categories by enhancing feature extraction and fusing textual information and visual features from work plans; adjusting detection parameters using an adaptive threshold optimization algorithm; combining few-shot detection technology; and comparing the results with the work plan after category-aware scaling filtering, outlier suppression, and cross-class non-maximum suppression. Finally, a sample cropping and screening mechanism driven by false positives and false negatives is used to extract high-value samples to supplement the training set. This method improves the safety and efficiency of railway tool and equipment management.

Description

Technical Field

[0001] This invention relates to the field of railway maintenance tool management technology, and more specifically, to a computer vision-based method for checking and verifying the on- and off-line maintenance of railway tools. Background Technology

[0002] Currently, high-speed rail, as a crucial component of the modern integrated transportation system, directly impacts the nation's economic lifeline and the safety of people's lives and property. To ensure the safe and stable operation of the high-speed rail system, nighttime maintenance is a critical link in operational support. However, the management of tools and equipment in current high-speed rail maintenance operations faces severe challenges: on the one hand, railway maintenance involves hundreds of different types of specialized tools and equipment, with complex work environments and limited lighting conditions; on the other hand, these tools and equipment vary significantly in size, shape, and quantity, making them prone to omission or loss, posing significant safety hazards and potentially leading to severe economic losses and operational delays.

[0003] Currently, there are three main methods for inventorying and verifying railway tools and equipment. The most primitive method is manual inventory and verification, where maintenance personnel use a handheld list to check the model, type, quantity, and other information of each tool and equipment, and record it in a paper logbook. Although this method is simple to operate and requires no additional equipment, it relies entirely on manual operation. In nighttime operations or in inclement weather conditions, it is not only inefficient but also prone to human errors such as missed inspections and incorrect inspections.

[0004] To address the shortcomings of manual inventory checks, the railway department introduced a barcode management system as an improvement. This method involves affixing a unique barcode label to each tool and maintenance personnel using handheld barcode scanners for rapid identification. The system automatically records the scan information and compares it with a database, significantly improving efficiency and accuracy compared to purely manual methods. However, this method still requires maintenance personnel to scan each item at close range, which can be difficult when tools and equipment are densely packed or inconveniently located. Furthermore, barcode labels are susceptible to environmental factors such as oil and wear at the work site, leading to frequent recognition failures, thus limiting its practical application.

[0005] To further improve inventory efficiency, some railway units have begun to implement RFID electronic tag management systems. This system uses anti-metal RFID tags as electronic identifiers for tools and equipment, and performs batch readings via fixed or handheld RFID readers, theoretically enabling rapid, large-scale tool and equipment identification. Compared to barcode technology, RFID systems offer significant improvements in operational convenience and work efficiency. However, in actual railway operating environments, metal equipment can cause signal interference, and simultaneous reading of multiple tags may lead to identification conflicts. Furthermore, the tag installation location significantly impacts reading accuracy. More importantly, due to the vast differences in shape, size, and material of railway tools and equipment, some specialized tools and equipment cannot be compatible with RFID tag installation requirements. These inherent defects make it difficult to fundamentally guarantee the stability and reliability of the RFID system in complex operating environments, resulting in a significant gap between actual application results and theoretical expectations.

[0006] In summary, existing traditional methods for inventorying tools and equipment all have their limitations. Manual methods are inefficient and prone to errors; while barcode technology offers some improvement, it still requires individual operation and is subject to environmental constraints; and although RFID systems can read in batches, their adaptability to complex operating environments is insufficient. None of these methods can fully meet the high standards of tool and equipment management required by modern high-speed rail operations.

[0007] In summary, at least one of the following technical problems exists:

[0008] Manual methods are inefficient and prone to errors. While barcode technology has improved, it still requires individual operation and is subject to environmental constraints. Although RFID systems can read data in batches, they lack adaptability in complex operating environments. None of these methods can fully meet the high standards of tool and equipment management required for modern high-speed rail operation and maintenance. Summary of the Invention

[0009] The main objective of this invention is to provide a computer vision-based method for checking and verifying the on-line and off-line status of railway tools and equipment, and its application method. This addresses the problems of inefficiency and error-proneness of purely manual methods in existing technologies, the need for individual operation of barcode technology (though improved, it is still subject to environmental constraints), and the insufficient adaptability of RFID systems in complex operating environments, despite their ability to read in batches. These methods are insufficient to fully meet the high standards of tool and equipment management required by modern high-speed rail operations.

[0010] To achieve the above objectives, according to one aspect of the present invention, a computer vision-based method for checking the online and offline inventory of railway tools is provided, comprising: acquiring images of online and offline tools and tools in high-speed railway maintenance sections and warehouses; cropping target regions using a region focusing algorithm to eliminate interference; constructing a diverse training dataset by combining perspective transformation, multi-transformation stitching, and other enhancement techniques, as well as generation methods based on SAM and diffusion models; constructing a text-visual multimodal rotating target detection model, enhancing feature extraction through serpentine convolution and HAttention mechanisms, fusing textual information and visual features from the work plan table, and guiding the model to focus on key categories; adjusting detection parameters using an adaptive threshold optimization algorithm, combining few-sample detection techniques, and after processing with category-aware scale filtering, outlier suppression, and cross-category nonmaximum suppression, comparing the results with the work plan and outputting the results; and mining high-value samples to supplement the training set based on a false detection and missed detection driven sample cropping and screening mechanism, continuously iterating and optimizing model performance, while supporting consistency verification of online and offline results in the same workshop.

[0011] Preferably, the images of railway machinery are labeled and preprocessed; a training dataset is constructed using image enhancement and image augmentation algorithms; feature maps of railway machinery images are extracted using a backbone network; and the multi-scale feature maps are enhanced and fused using a PANet neck network based on snake convolution and HAttention mechanisms to encode the text information corresponding to the images into semantic features, which are then fused with the three feature maps of different sizes output by the neck network for cross-modal feature fusion.

[0012] Preferably, the detection and recognition results are output through a parallel dual-branch detection head without anchor boxes; the training dataset is input into the constructed rotating target detection model to train the tool detection model; the tool image to be detected is input into the tool detection model to obtain the initial detection result; an adaptive threshold optimization algorithm for text information is used to align the text information with the visual detection result; the text information is compared with the detection result, and supplementary recognition is performed through a few-sample target detection model; the few-sample detection result and the rotating target detection result are corrected by a post-processing algorithm to obtain the final recognition result of the railway tool's online and offline status; the final recognition result is compared with the schedule, and the online and offline images are compared simultaneously; an active learning iterative optimization algorithm is used to continuously improve the model performance.

[0013] Preferably, the on-site images collected from the high-speed railway maintenance depot are partially annotated. The annotated tool images are then optimized by tool region focusing to obtain preprocessed image data. This includes: the tool region focusing algorithm dynamically generates regions of interest for tools based on the positions of workers and tools in the on-site images collected from the high-speed railway maintenance depot, and performs cropping to extract the target regions of the tools; the preprocessed image data and tool images collected from the warehouse environment are then subjected to data enhancement and augmentation processing to construct a diverse training dataset. This includes: the data enhancement techniques include an image geometric correction algorithm based on perspective transformation and a multi-transformation stitched image scale adaptive enhancement algorithm. The data augmentation techniques include intelligent directional web data crawling, an image generation algorithm based on SAM and diffusion models, and a SAM-guided tool foreground-normalized background synthesis method.

[0014] Preferably, a computer vision-based method is used to inventory and verify the tools and equipment, and a rotating target detection model is constructed to determine the category and quantity of the tools and equipment. This includes: using a computer vision-based rotating target detection method to detect the tools and equipment, and then calculating the quantity of each category of tools and equipment based on the detection results. The rotating target detection model mainly includes three parts: a backbone network, a neck network, and a detection head network. The high-speed railway maintenance section's on-line and off-line maintenance operation plan is used as text information to construct a text-visual multimodal input tool and equipment target detection model. This includes: during the daily maintenance operation of the railway maintenance section, each on-site collected image of tools and equipment is associated with a structured operation plan, which records in detail the operation time, workshop information, shift leader's name, tool and equipment category and quantity. The operation plan is used as text information, and the text information is encoded to obtain text features. The text-visual multimodal input target detection model deeply aligns and fuses the text features and the image visual features obtained from the neck network, so that the text information guides the enhancement of visual features.

[0015] Preferably, the training dataset is input into the constructed target detection model for network training to obtain the tool detection model, including: based on the constructed diverse training dataset, the target detection model is used for network training, and the network parameters are continuously optimized through the backpropagation mechanism of the loss function to finally obtain the tool detection model with the best performance; the images of railway tools to be verified are input into the tool detection model for tool identification to obtain the initial tool detection results, including: the images of railway tools to be verified on and off the line are preprocessed and further input into the optimal tool detection model to obtain all the initial tool detection results without confidence filtering and non-maximum suppression.

[0016] Preferably, an adaptive threshold optimization algorithm based on the text information is constructed. This algorithm compares the initial tool detection results with the text information in terms of category and quantity consistency, dynamically optimizes the detection confidence threshold and non-maximum suppression parameters, and obtains threshold-adjusted tool detection results with high alignment between the text information and the visual detection results. This includes: comparing all initial detection boxes retained from the initial tool detection results with the text information item by item in terms of category and quantity; employing an adaptive threshold optimization algorithm to automatically increase the confidence threshold for categories not mentioned in the text information to suppress irrelevant detections, while simultaneously calculating the matching degree between the detection results and the text information in real time; during this process, the system uses a two-layer cyclic search strategy to adaptively adjust the confidence threshold and IoU (Intersection over Union) threshold parameters until the optimal matching degree is obtained, thereby correcting the initial detection boxes and obtaining threshold-adjusted tool detection results.

[0017] Preferably, the step of comparing the threshold-adjusted tool detection results with the text information again, and using a few-shot target detection algorithm to supplement the identification of mismatched categories to obtain few-shot identification results includes: after obtaining the threshold-adjusted tool detection results, comparing the results with the text information; if they do not match, recording the names of the matching and mismatched categories respectively; re-detecting the mismatched categories using a few-shot target detection model, with the tool image to be detected as the query image and the labeled sample image of the target category as the support image; extracting features from the query image and support image respectively using a deep neural network, and calculating the feature similarity between the two to enhance the feature representation capability of the query image; then, inputting the enhanced query image features into a region proposal network to generate candidate target regions, and performing RoI Align operation on these candidate regions to obtain refined region features; similarly, the support image features are also processed by RoI Align to obtain standardized region features; by calculating the similarity matching score between the query image region features and the support image region features, and combining the spatial location information of the target, finally outputting the tool category determination and precise location coordinates to obtain the few-shot identification result.

[0018] Preferably, the few-sample identification results are processed using a category-aware scale distribution filtering algorithm and an outlier detection box adaptive suppression algorithm. The processed results are then combined with the threshold-adjusted tool detection results for cross-category non-maximum suppression to obtain the final tool detection results. This process includes: filtering the initial detection boxes output by the few-sample target detection model using a category-aware scale distribution filtering algorithm to effectively remove outlier detection boxes that do not conform to the target category scale distribution, obtaining an intermediate result with optimized scale consistency; performing location group analysis on the intermediate result using an outlier detection box adaptive suppression algorithm, intelligently identifying and filtering outlier detection boxes that deviate from the main distribution by calculating the spatial distribution characteristics of the detection boxes, obtaining a few-sample detection result with dual optimization; fusing the original target detection model output with the optimized few-sample target detection results, and finally calibrating the fused result using a cross-category non-maximum suppression algorithm to eliminate redundant detection boxes, obtaining the final railway tool identification result.

[0019] Preferably, the final tool and equipment inspection results are counted and compared with the high-speed railway maintenance section's online and offline maintenance operation plan to obtain the comparison result. This includes: counting each category of each image of the high-speed railway maintenance section's online and offline maintenance, comparing the count result with the high-speed railway maintenance section's online and offline maintenance operation plan to obtain the comparison result. If the comparison is correct, the error is output; if the comparison is incorrect, the missing or extra tool and equipment category names and quantities are output. The results of comparing the online and offline tool and equipment images of the same railway workshop are compared to obtain the comparison result to check for consistency. This includes: comparing the count results of each category of the online and offline tool and equipment images of the same railway workshop. If the online and offline results are consistent, the consistency is output; if they are inconsistent, the inconsistent tool and equipment categories and quantities are output.

[0020] The technical solution of this invention has the following technical effects:

[0021] This paper innovatively integrates textual information from work plans with visual features through a multimodal fusion detection architecture. A cross-modal attention mechanism guides the detection process, addressing the issue of traditional pure visual methods being susceptible to background interference. A dynamic adaptive detection mechanism is employed, proposing an adaptive threshold optimization algorithm based on textual information. A two-layer iterative search strategy dynamically adjusts detection parameters to achieve optimal matching between detection results and work plans. Data augmentation and generation techniques, combined with SAM segmentation and diffusion models, enable accurate extraction of foreground equipment and adaptive scene synthesis, effectively solving the problems of insufficient samples and domain offset in railway scenarios. Through active learning driven by false positives and false negatives, a pruning and screening mechanism based on false positives and false negatives is designed to automatically mine high-value labeled data. Compared to random sampling strategies, this results in a 40% improvement in model convergence speed and a 60% reduction in labeling costs.

[0022] The rotation target detection is optimized by using serpentine convolution and HAttention mechanisms to specifically enhance the feature extraction capability for slender and irregular tools, improving the rotation box detection accuracy by 15% compared to traditional methods.

[0023] High-precision identification, through multimodal fusion and adaptive post-processing, enables accurate identification of tool and equipment categories and quantities in complex railway scenarios, with average detection accuracy (mAP) increased to over 95% and false negative rate reduced to below 3%.

[0024] With strong scene adaptability, the data augmentation and generation technology effectively solves the problems of diverse tool shapes and varying placement angles, and the model maintains stable performance under different lighting and occlusion conditions.

[0025] Automation and efficiency improvements: Compared to manual counting, the verification efficiency is increased by more than 80%, and the time for a single verification is reduced from 30 minutes to less than 5 minutes, significantly reducing labor costs.

[0026] The continuous optimization capability and proactive learning strategy enable the model to evolve with the accumulation of new data. After 10 iterations, the accuracy of identifying rare tools and equipment increased from 70% to 92%.

[0027] With broad applicability, the method can be extended to multiple industrial sectors, and intelligent tool management can be achieved through sector adaptation, resulting in significant economic and social benefits. Attached Figure Description

[0028] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0029] Figure 1 A schematic diagram illustrating the implementation principle of the computer vision-based railway tool and equipment inventory and verification method according to the present invention is shown.

[0030] Figure 2 It shows Figure 1 A flowchart of a computer vision-based railway tool and equipment inventory and verification method.

[0031] Figure 3 It shows Figure 1 Image preprocessing view of the computer vision-based railway machinery and equipment inventory and verification method;

[0032] Figure 4 It shows Figure 1 The training dataset of the computer vision-based railway machinery and equipment inventory and verification method constitutes a view;

[0033] Figure 5 It shows Figure 1 A schematic diagram of text information for the computer vision-based railway tool and equipment inventory and verification method.

[0034] Figure 6 It shows Figure 1 The flowchart of the post-processing algorithm for the computer vision-based railway tool and equipment inventory and verification method.

[0035] Figure 7 It shows Figure 1 The active learning architecture diagram of the computer vision-based railway tool and equipment inventory and verification method. Detailed Implementation

[0036] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] like Figures 1 to 7 As shown, this invention provides a computer vision-based method for checking the online and offline inventory of railway tools and equipment. The method includes: acquiring images of online and offline tools and equipment from high-speed railway maintenance sections and warehouses; cropping target regions using a region focusing algorithm to eliminate interference; constructing a diverse training dataset by combining perspective transformation, multi-transformation stitching, and generation methods based on SAM and diffusion models; constructing a text-visual multimodal rotating target detection model; enhancing feature extraction through serpentine convolution and HAttention mechanisms; fusing textual information and visual features from the work plan table to guide the model to focus on key categories; adjusting detection parameters using an adaptive threshold optimization algorithm; combining few-sample detection technology; and comparing the results with the work plan after category-aware scale filtering, outlier suppression, and cross-category non-maximum suppression; and using a sample cropping and screening mechanism driven by false positives and false negatives to extract high-value samples to supplement the training set, continuously iterating and optimizing model performance, while also supporting consistency checks of online and offline results within the same workshop.

[0038] In this embodiment, images of tools and equipment collected from railway maintenance scenarios and warehouse environments are preprocessed, and a diverse training dataset is constructed using data augmentation and expansion techniques. A deep learning-based text-visual multimodal rotating target detection model is built; detection results are output using an anchorless dual-branch detection head; the detection results are optimized using an adaptive threshold optimization algorithm based on text information; the text information is compared with the detection results, and the detection results are corrected using a few-sample target detection model and post-processing algorithms to obtain the final recognition result. Simultaneously, an active learning strategy based on a sample pruning and screening mechanism driven by false positives and false negatives is proposed, automatically mining high-value samples using detection error feedback to continuously optimize model performance. This invention improves the safety and efficiency of railway tool and equipment management by introducing dataset construction, a deep learning-based target detection algorithm, a post-processing algorithm, and an active learning strategy.

[0039] In this embodiment, partial annotations are performed on the on-site images collected from the high-speed railway maintenance depot's online and offline maintenance scenarios. The annotated tool images undergo tool region focusing optimization to obtain preprocessed image data. The preprocessed image data and tool images collected from the warehouse environment are then subjected to data augmentation and expansion processing to construct a diverse training dataset. A computer vision-based method is used to inventory and verify the tools, and a rotating target detection model is constructed to determine the category and quantity of the tools. The online and offline maintenance operation plan of the high-speed railway maintenance depot is used as text information to construct a text-visual multimodal input tool target detection model. The training dataset is input into the constructed target detection model for network training to obtain a tool detection model. The images of railway online and offline tools to be verified are input into the tool detection model for tool identification to obtain initial tool detection results. An adaptive threshold optimization algorithm based on the text information is constructed, and the initial tool detection results are compared with the text information for category and quantity consistency. The detection confidence threshold and non-maximum suppression parameters are dynamically optimized to obtain... The text information and visual detection results are aligned with the threshold-adjusted tool detection results. These are then compared again with the text information. For mismatched categories, a few-shot target detection algorithm is used for supplementary identification, resulting in few-shot identification results. These few-shot identification results are then processed using a category-aware scale distribution filtering algorithm and an outlier detection box adaptive suppression algorithm. The processed results are then compared with the threshold-adjusted tool detection results using cross-class non-maximum suppression to obtain the final tool detection results. The final tool detection results are counted and compared with the high-speed railway maintenance section's online and offline maintenance operation plan to obtain comparison results. The images of tools going online and offline in the same railway workshop are compared to verify consistency. Unlabeled tool images collected from online and offline maintenance scenes in the railway maintenance section are input into an active learning framework driven by false detection and missed detection to obtain high-value sample data. This high-value sample data is labeled and added to the training dataset for continuous iterative optimization of model performance. The computer vision-based method for checking the online and offline inventory of railway tools and equipment has wide applicability. Its innovative technical solution is not only applicable to the field of railway tool and equipment inspection, but can also be extended to multiple industrial fields such as rail transit, aerospace, power maintenance, automobile manufacturing, ship repair, construction engineering, and medical equipment management.

[0040] In this embodiment, unlabeled images of tools and equipment collected from railway maintenance scenes are input into an active learning framework driven by false detection and missed detection to obtain high-value sample data. This high-value sample data is then labeled and added to the training dataset for continuous iterative optimization of model performance. The process includes: inputting unlabeled images of tools and equipment collected from railway maintenance scenes into the active learning framework driven by false detection and missed detection; filtering out high-value sample data from the unlabeled images of tools and equipment collected from railway maintenance scenes; manually labeling the high-value sample data to generate corresponding precise labeling information; adding the high-value sample data and the corresponding precise labeling information to the training dataset; retraining the tool and equipment detection model using the training dataset to generate a new model version with optimized performance; repeating the above process to form a closed-loop active learning process, achieving efficient acquisition of training samples and continuous enhancement of model performance.

[0041] In this embodiment, the computer vision-based method for checking and verifying the on-line and off-line status of railway tools and equipment has wide applicability. Its innovative technical solution is not only applicable to the field of railway tool and equipment verification, but can also be extended to multiple industrial fields such as rail transit, aerospace, power maintenance, automobile manufacturing, ship repair, construction engineering, and medical equipment management. Specifically, in the rail transit field, it can automate the counting of subway maintenance tools; in the aerospace field, it can ensure the complete quantity of aircraft maintenance tools; in the power system field, it can intelligently verify insulating tools and safety protection equipment; in the automobile manufacturing process, it can accurately manage specialized molds and assembly tools on the production line; and in medical equipment management, it can realize the verification and inventory management of surgical instruments. This computer vision-based method for checking and verifying the on-line and off-line status of railway tools and equipment can be applied to tool verification in various scenarios.

[0042] In this embodiment, the image geometric correction algorithm based on perspective transformation is used to automatically correct the geometric shape of the machine tool image and convert the bird's-eye view, effectively eliminating the distortion of the shooting angle.

[0043] The multi-transformation stitched image scale adaptive enhancement algorithm generates enhanced images with rich scale variations through random scale transformation combined with a flip stitching strategy based on geometric consistency constraints. The intelligent directional network data crawling is based on a distributed crawler framework and employs adaptive webpage parsing and semantic filtering technology to accurately acquire multi-source machine tool image data and perform data cleaning to expand the diversity and scale of training samples. The image generation algorithm based on SAM and diffusion model achieves accurate segmentation of the machine tool foreground through SAM and combines the image synthesis capability of diffusion model to achieve natural fusion of foreground and background of multiple scenes while maintaining the authenticity of the target structure, generating high-fidelity synthetic images with physical plausibility.

[0044] In this embodiment, images of tools and equipment collected from the warehouse environment are input into the SAM model to obtain foreground objects of the tools and equipment. The extracted foreground objects are then pasted into a preset standardized background template in a certain order to generate high-quality training samples. The backbone network is used to extract feature maps from railway tool and equipment images, while the neck network is used to fuse and enhance the feature maps extracted by the backbone network, achieving deep interaction and fusion of high-level semantic information and low-level detail information at multiple scales. The detection head predicts the precise location and category information of the target based on the fused multi-scale features. In the neck network, serpentine convolution is used. The serpentine convolution can dynamically adjust the receptive field and weight distribution according to the target shape, more accurately capturing key features such as the edges and textures of the tools and equipment. The final feature maps of different sizes generated by the neck network are treated with the HATtention attention mechanism. The HATtention attention mechanism effectively combines global pixel information by fusing channel attention and self-attention mechanisms, further improving the quality of multi-scale feature fusion.

[0045] This invention provides a computer vision-based method for checking the online and offline status of railway tools and equipment, based on deep learning. The implementation principle of this method is as follows: Figure 1 As shown, the specific processing flow is as follows: Figure 2 As shown, the processing steps include the following:

[0046] Step S10: Image preprocessing

[0047] The on-site images collected by the high-speed railway maintenance depot include workers and equipment. First, this invention uses a rotating bounding box annotation on the image data. Then, a tool area focusing optimization algorithm is employed to highlight key feature areas, making the image more concentrated on the tool area while ignoring people and a large amount of background. Figure 3 This is a schematic diagram of image preprocessing provided in an embodiment of the present invention.

[0048] The tool and equipment region focusing optimization algorithm effectively improves the accuracy of identification and counting by suppressing interference from complex backgrounds and accurately extracting the target tool and equipment region. The algorithm utilizes deep learning-based object detection and spatial statistical analysis techniques to achieve intelligent processing of work images.

[0049] The algorithm first uses a pre-trained rotating target detection model to predict the input image and extract multi-dimensional data, including center coordinates, rotating rectangles, and size information.

[0050] Next, the algorithm calculates the mean vertical center coordinates and distribution distance of the tool targets to establish a dynamic outlier detection threshold based on statistical distribution characteristics, effectively identifying and eliminating targets with abnormal spatial distribution. Simultaneously, for personnel detection results, the algorithm employs an area consistency test method, comparing the deviation of each detection frame area from the average area to filter out falsely detected personnel targets, resulting in cleaned detection results.

[0051] Based on the cleaned detection results, the algorithm performs spatial relationship analysis. By quantitatively analyzing the relative positional relationship between personnel and machinery targets in the image plane, including vertical distribution comparison and horizontal boundary analysis, the optimal cropping region is dynamically determined. Different cropping strategies are adopted for different scenarios where the personnel are located at the top, left, or right of the image. When the personnel are at the top, the lower machinery area is retained; when they are on the side, the opposite side area is retained, always maintaining a certain pixel safety boundary. If the image is preprocessed during the training phase, the coordinates of the target annotation information are updated synchronously during the cropping process.

[0052] The algorithm ultimately outputs a spatially optimized image of the tool and equipment area. This algorithm effectively eliminates the influence of interference factors such as workers on tool and equipment recognition, concentrates computational resources on the target area, and preprocesses the image to prepare for subsequent recognition tasks.

[0053] Step S20: Dataset Construction

[0054] In response to the unique characteristics of image acquisition of tools and equipment in railway operation scenarios, including the diverse shapes of tools and equipment, the varied placement angles, and the scarcity of samples for some categories, this invention designs a variety of dataset enhancement and expansion schemes to construct diverse training datasets, thereby improving the robustness of the recognition model. Figure 4 This is a schematic diagram illustrating the structure of the training dataset provided in an embodiment of the present invention.

[0055] The dataset construction process includes data augmentation and data expansion. For data augmentation, a perspective-transformation-based geometric correction algorithm is first used to unify the image spatial baseline, and a multi-transformation stitching strategy is employed to achieve scale-adaptive enhancement. For data expansion, this includes intelligent orientation network data acquisition, a generative data synthesis method based on the SAM segmentation and diffusion models, and a SAM-guided foreground-normalized background synthesis method for machinery and tools.

[0056] 1. Image enhancement algorithms

[0057] (1) Image geometric correction algorithm based on perspective transformation

[0058] Perspective transformation-based image geometric correction algorithms convert images from different viewpoints into a bird's-eye view by performing geometric correction, which is beneficial for better feature extraction.

[0059] The algorithm first establishes an image spatial mapping relationship, selecting four key corner points in the original image as reference coordinates and defining their corresponding positions in the target view. Specifically, the top left and top right corner points are selected to retain their original coordinates, while the bottom left and bottom right corner points are subjected to shrinkage transformation according to a scaling factor.

[0060] Original Point

[0061] Target point

[0062] Where w and h represent the width and height of the original image, shrink_width is the width after applying a shrinkage ratio of 0.1 to w, and shrink_width2 is the width after applying a shrinkage ratio of 0.3 to w.

[0063] Calculate the perspective transformation matrix H based on the defined original and target points. The perspective transformation matrix H is a 3x3 matrix that maps points in the original image to points in the target image, thus achieving perspective transformation.

[0064] H = M perspective (pts1,pts2)

[0065] Where pts1 and pts2 represent the coordinates of four key points in the original image and the coordinates of four key points in the target image, respectively, M perspective It is a function that generates the perspective transformation matrix.

[0066] The image is transformed using a perspective transformation matrix H, normalized to a uniform bird's-eye view, and then cropped. The formula for the image transformation is:

[0067] I′=I·H

[0068] Where I is the original image, I′ is the transformed image, and H is the perspective transformation matrix. Through this transformation, the algorithm effectively eliminates geometric distortion caused by differences in shooting perspective, generates a bird's-eye view with uniform specifications, and provides standardized input data for subsequent feature extraction.

[0069] (2) Multi-transform stitched image scale adaptive enhancement algorithm

[0070] After image preprocessing using the tool region focusing optimization algorithm, the image often exhibits a large aspect ratio. This disproportionate image can easily distort the actual sample during interpolation, severely impacting subsequent detection accuracy. The multi-transformation stitched image scale adaptive enhancement algorithm effectively solves this problem.

[0071] The algorithm employs a multi-stage geometric transformation strategy to enhance the diversity of image data. First, a random scaling operation is applied to the original image. Then, a mirror transformation is applied to construct image pairs with complementary viewpoints, enhancing the model's ability to learn viewpoint-invariant features. Finally, a stitching algorithm is used to stitch and fuse the transformed multi-view images.

[0072] Through the above operations, on the one hand, the representation of image data at different scales and perspectives is enriched, effectively solving the interpolation distortion problem caused by excessive aspect ratios and improving image quality. On the other hand, allowing the model to be exposed to more diverse image features during training can significantly improve the model's multi-scale detection capabilities.

[0073] 2. Image augmentation algorithm

[0074] (1) Intelligent targeted web data crawling

[0075] The intelligent targeted web data crawling system automatically acquires more relevant images from the Internet, enriching the variety and quantity of samples in the dataset and providing more comprehensive data support for subsequent image processing.

[0076] First, search keywords were set based on the names of the tools and equipment. An automated web crawler script was then built using Python's Selenium library. Selenium allows for the simulation of user behavior, automating access to search engines and extracting image links. The crawler parameters were initialized first, setting the target website, request headers, and search keywords. Next, the automated script accessed search engines (such as Google, Bing, and Baidu), inputting the preset keywords and submitting a search request. The script parsed the search results page, extracted links to images, and stored these links in a processing queue. Subsequently, each image link was accessed, and the image files were downloaded to local storage. After downloading, the acquired images underwent rigorous screening and cleaning. By using image classification algorithms and manual review, irrelevant or low-quality images were removed, ultimately resulting in a high-quality and diverse dataset of tool and equipment images, significantly improving the accuracy and stability of subsequent detection models.

[0077] (2) Image generation algorithm based on SAM and diffusion model

[0078] By combining the Segment Anything Model (SAM) background segmentation strategy with the Stable Diffusion AnyDoor image synthesis algorithm, the domain offset problem between externally crawled data and actual application scenarios is effectively solved. The SAM-bg technology can accurately identify and separate foreground objects from background information in images, while the AnyDoor algorithm based on the diffusion model achieves high-quality scene-adaptive image generation. The synergistic application of these two technologies enables externally acquired tool images to better match the lighting conditions, background environment, and tool placement characteristics of actual railway operation and maintenance scenarios, improving the scene relevance and practicality of the training data.

[0079] The algorithm first uses the SAM model to segment the crawled images. The crawled images have two characteristics: 1) each image mainly contains only one foreground object; 2) the background of the image is relatively simple. Therefore, a SAM background segmentation strategy is introduced. SAM-bg uses the four corner points of the image as input cues to obtain a background mask. Then, the background mask is inverted to obtain a mask for the foreground object. Finally, the foreground mask is used to extract the precise contour and region of the tool, removing the background and retaining a clear foreground tool.

[0080] P1=(0,0),P2=(w,0),P3=(0,h),P4=(w,h)

[0081] M bg =SAM(P1,P2,P3,P4)

[0082] M fg =1-M bg

[0083] F(x,y)=I(x,y)·M(x,y)

[0084] Where P1, P2, P3, and P4 are the four corner points of the image, and M... bg and M fg These are the background mask and the foreground mask, respectively. F(x,y) is the foreground image, I(x,y) is the original image, and M(x,y) is the transparent foreground mask.

[0085] AnyDoor, an image synthesis method based on a diffusion model, naturally blends extracted foreground tools into specified areas of various background images to generate natural and realistic synthetic images.

[0086] C(x,y)=AnyDoor(B(x,y),M b (x,y),F(x,y),M f (x,y))

[0087] Where B(x,y),M b (x,y),F(x,y),M f (x, y) represent the background image, the background image mask, the foreground tool image, and the foreground tool mask, respectively.

[0088] (3) SAM-guided tool and equipment foreground-standardized background synthesis method

[0089] Images of tools and equipment collected from the warehouse environment are input into the Segment Anything Model (SAM) for processing. The foreground regions of the tools and equipment in the images are precisely segmented to obtain a clear foreground object mask. Then, the corresponding foreground image of the tools and equipment is cropped from the original image based on the extracted mask information. Next, the processed foreground object is pasted onto a standardized background template according to preset rules (e.g., position, scale, or category order) to construct high-quality synthetic training sample images.

[0090] Step S30: Image Feature Extraction

[0091] Image feature extraction is mainly used to extract features from images of railway machinery and equipment. Specifically, it is divided into two parts: feature extraction of the backbone network and feature fusion of the neck network.

[0092] 1. Backbone Network

[0093] This algorithm employs a backbone network based on the CSPDarkNet architecture for multi-level feature extraction. The main body of the network consists of multiple C2f modules, which use a divide-and-conquer strategy to split the input features into two branches: one branch preserves the original features, while the other branch performs deep feature transformation through a cascaded Bottleneck structure. Gradient optimization is achieved between modules through residual connections, and finally, multi-scale feature fusion is achieved through channel concatenation.

[0094] The network constructs a feature pyramid through a five-level downsampling structure, with each level using a 3×3 convolutional kernel (stride=2, padding=1) for spatial dimensionality reduction. At the end, an SPPF module is integrated, capturing global contextual information through parallel multi-scale max pooling, and reconstructing features using feature concatenation and 1×1 convolution.

[0095] 2. Neck network

[0096] The neck network, acting as a crucial bridge connecting the backbone network and the detection head, undertakes the key functions of multi-scale feature fusion and enhancement. This algorithm employs an improved PANet architecture, achieving more efficient feature fusion through a bidirectional feature propagation mechanism. Compared to the traditional FPN, which only uses a top-down unidirectional propagation path, PANet innovatively introduces a bottom-up feature enhancement path, forming a complete bidirectional feature pyramid structure.

[0097] In its implementation, the neck network first propagates semantic information through a top-down path. After the high-level features are upsampled and aligned with the low-level features in the spatial dimension, they are fused and enhanced through the C2f module. The HATtention attention mechanism is then used to further enhance the features after the C2f module.

[0098] The C2f module here uses serpentine convolution as its core operator. Serpentine convolution is a dynamically deformable convolution method. Its core idea is to learn the spatial offset of the input features, enabling the convolution kernel to adaptively deform along specific directions, such as the x-axis and y-axis, thereby more effectively capturing slender and curved structural features. The computational principle is as follows: First, a two-dimensional offset field is generated through an offset prediction convolutional layer, and the offset is constrained within the range of [-1, 1] to ensure the stability of the deformation process. The offset is applied to the initial coordinates through iterative accumulation. The center point remains stationary, and the offsets are successively added to the points on both sides, forming a continuous and smooth serpentine path. The degree of deformation throughout the process is controlled by the expansion coefficient, thereby enhancing the modeling ability of complex geometric structures while maintaining local continuity.

[0099] The core of the HAttention attention mechanism lies in achieving adaptive enhancement of multi-scale features through hierarchical feature interaction and hybrid attention collaborative computation. Its working principle begins with channel dimension analysis of the feature map, employing a Channel Attention Module (CAB) to calculate the weight coefficients of each channel. Global average pooling captures channel-level global information, and a multilayer perceptron generates channel attention vectors. This process enables the network to autonomously enhance the responses of key feature channels. In spatial dimension processing, HAttention couples window-based multi-head self-attention (W-MSA) with channel attention. First, the input feature map is divided into non-overlapping local windows. Within each window, attention weights of the query-key-value matrix are calculated. This local self-attention mechanism effectively establishes spatial dependencies between pixels while reducing computational complexity. Deeper feature integration is achieved through Overlapping Cross Attention Blocks (OCAB). This module uses a sliding window strategy to create overlapping areas between adjacent windows, promoting feature interaction between windows through a cross-attention mechanism, thus solving the problem of boundary information loss caused by traditional window attention. In the feature fusion stage, HAttention uses residual connections to add the original features to the attention-weighted features, which preserves the underlying details and strengthens the important features.

[0100] Step S40: Multimodal feature fusion

[0101] During routine maintenance operations at the railway engineering section, each image of tools and equipment collected on-site is associated with a structured work plan, which details the work time, workshop information, shift leader's name, type and quantity of tools and equipment, and so on, and is treated as text information. Figure 5 This is a schematic diagram of text information provided in an embodiment of the present invention.

[0102] Before performing multimodal feature fusion, textual information needs to be feature-encoded. Textual feature encoding is divided into two parts, using different methods for category information and quantity information. For category information, a pre-trained BERT model is used to extract deep textual features of category names, and a 12-layer Transformer encoder is used to generate a 768-dimensional context-related vector. The vector labeled [CLS] serves as a global semantic representation, effectively capturing the functional attributes and category features of the tools and equipment. For quantity information, an innovative hybrid encoding strategy is designed. First, discrete numerical values ​​are mapped to seven predefined bucket intervals, including precise buckets of 1-5 and broad buckets of 6-10 and >10. These are then converted into 64-dimensional vectors through a learnable embedding layer. Simultaneously, the residual offset between the actual numerical value and the bucket center is calculated and encoded into another 64-dimensional vector via a fully connected network. This discrete-continuous hybrid encoding method maintains the order of numerical values ​​and alleviates the data sparsity problem caused by long-tail distribution. Finally, the 768-dimensional semantic features are concatenated with two 64-dimensional quantitative features to form an 896-dimensional joint representation. This fused feature space is used to achieve automatic alignment of semantic and quantitative features through an end-to-end training process.

[0103] After text feature encoding, a multimodal feature fusion method based on cross-modal attention mechanism is used to deeply align the visual and text feature spaces to enhance object detection performance. This module adopts an architecture combining bidirectional feature projection and attention mechanism. First, visual features are projected onto the text feature space through linear transformation, while text features are also enhanced through independent linear transformation. In the attention calculation stage, a reverse attention mechanism is used, with text features as the query and visual features as the key and value. Multi-head attention is used to calculate the cross-modal association weights from text to vision, so that each text feature can adaptively focus on the most relevant image region. After dimensional alignment projection, the attention output is expanded to the same spatial dimension (H×W) as the original visual features through a spatial broadcast mechanism. Finally, residual connections are used to fuse the features with the three feature maps of different sizes extracted by the neck network. This design has three key advantages: First, the reverse attention mechanism establishes semantic guidance from text to vision, strengthening the visual regions related to the text description. Second, the projection alignment process maintains the dimensional consistency of the two modal features, avoiding information loss. Finally, residual connections ensure the integrity of the original visual features.

[0104] Step S50: The detection head performs target localization and recognition on the image of railway machinery.

[0105] The detection head employs an anchor-based strategy. Specifically, it pre-generates an equal number of anchor points as the feature map pixels and predicts bounding boxes for each anchor point. The detection head receives three-level feature maps from the backbone and neck network. Each feature map layer processes the rotation box localization and classification tasks through parallel branches. In the localization branch, a 5-dimensional result (x, y, w, h, θ) is output, where x and y represent the offset of the target center point relative to the grid cell, w and h are the width and height of the predicted bounding box, and θ is the predicted rotation angle. The classification branch outputs the number of channels corresponding to the number of classes.

[0106] Step S60: Training the computer vision-based rotating target detection model

[0107] The pre-constructed training dataset and the aforementioned object detection model were used to train the machine target detection network. During training, positive and negative samples were dynamically assigned using the Task-Aligned Assigner strategy. To complement the anchorless bounding box strategy and improve the model's generalization ability, Dependency-Fluid (DFL) loss was employed. DFL, in the form of cross-entropy, optimizes the probabilities of the two positions closest to the label (left and right), allowing the network to focus more quickly on the target location and its surrounding area. The DFL calculation formula is as follows:

[0108] DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(yy i )log(S i+1 ))

[0109] Where S i and S i+1 These are the network's predicted value and its nearest neighbor predicted value, y, y. i y i+1 These are the actual label values, the label integral value, and the integral values ​​of neighboring labels. In actual training, DFL (Distributed Freedom Function) is used in conjunction with CIoU (Concurrent Identity) loss. First, DFL calculates the loss on the bounding box probability distribution and the label probability distribution, thus optimizing each edge. Then, the bounding box probability distribution is restored to the predicted box, and CIoU loss is used to calculate the loss on the predicted box and the actual labeled box, thus optimizing the overall predicted box. Finally, the network combines the results of both branches to output both the predicted box and the predicted class.

[0110] Finally, the network parameters are continuously optimized through the backpropagation mechanism of the loss function, and the optimal machine detection model is finally obtained.

[0111] Step S70: Test the rotating target detection model to obtain initial results.

[0112] The images of railway tools and equipment awaiting inspection are preprocessed and then input into the optimal tool and equipment detection model to obtain all initial tool and equipment detection results without confidence filtering and non-maximum suppression.

[0113] Step S80: Adaptive threshold optimization of detection results

[0114] An adaptive threshold adjustment algorithm is used to assess the initial tool detection results. Its core idea is to achieve optimal matching between the detection results and text information by dynamically optimizing the confidence threshold and the Intersection over Union (IoU) threshold. The text information refers to the known category name and quantity for each image.

[0115] The algorithm adopts a multi-stage progressive optimization strategy. First, it performs coarse-grained screening of the initial detection results based on text information, and then achieves fine-grained calibration through iterative parameter space exploration.

[0116] During the preprocessing stage, the system retains all initial detection boxes to construct the original detection pool. The text information comparison module maps the detection categories to the target category set in the text information. For categories not mentioned in the text information, a confidence penalty mechanism is automatically triggered to suppress irrelevant detections by raising the decision threshold.

[0117] During the parameter optimization phase, only a specified number of detection boxes with the highest confidence ranking are retained for over-detected categories. For categories with insufficient detections, potential targets are recalled by relaxing the confidence threshold. The algorithm employs a two-layer loop search strategy: the outer loop searches along the confidence dimension (gradually relaxing from 0.6 to 0.001), while the inner loop performs testing along the IoU dimension (gradually varying from 0.1 to 0.5). Each parameter combination generates a new subset of detections, and the deviation between the current result and the expected number is calculated using a matching evaluation function that comprehensively considers both quantity matching accuracy and category detection quality. The search process retains the optimal solution, terminating early when a perfect match is found to improve efficiency, resulting in the railway machinery image target detection results adjusted by the adaptive threshold algorithm.

[0118] Step S90: Few-sample target detection

[0119] First, the detection results, adjusted by an adaptive threshold, are compared with the text description. Text matching is used to classify the detection categories. Categories with a matching number of responses directly enter the result pool, and a black mask is generated to cover the identified areas. Categories without a matching number of responses trigger a few-shot detection mechanism. In the few-shot detection stage, the algorithm constructs a dual-path feature interaction system for support-query pairs. Multi-scale features of the support and query graphs are extracted through a deep neural network, and cross-sample attention weights are constructed using feature similarity calculations to enhance the feature representation capability of the query graph. The region proposal network then generates candidate boxes on the enhanced features and obtains refined region features through RoI Align. Finally, the few-shot detection results are output by weighting the support-query feature matching score and spatial location.

[0120] Step S100: Post-processing of detection results based on text information

[0121] Figure 6 A flowchart of a post-processing algorithm provided in an embodiment of the present invention.

[0122] A dual filtering strategy is employed to optimize the few-sample detection results. First, a category-aware scale distribution filtering algorithm is applied. Based on a pre-constructed matrix storing the relative size ratios between different categories of machinery and tools, the scale matching degree between the detection boxes and the target category is calculated, eliminating outlier boxes that deviate from the statistical distribution. Next, outlier detection box suppression is performed, estimating and analyzing the spatial distribution of the detection boxes, and filtering outliers in low-density regions. Finally, cross-class non-maximum suppression is used in the fusion stage to unify the detection results after adaptive threshold adjustment and the few-sample optimization results, ultimately outputting a machinery and tools recognition result that conforms to the text description.

[0123] Step S110: Compare the final tool and equipment identification results with the work plan.

[0124] Each image of the high-speed railway maintenance section going online and offline is counted category by category. The count results are compared with the high-speed railway maintenance section's online and offline maintenance operation plan. If the comparison is correct, the error is output. If the comparison is incorrect, the missing or extra tool category name and quantity are output.

[0125] Step S120: Compare paired online and offline images.

[0126] The system compares the count results of each category of the tool and equipment images that are online and offline in the same railway workshop. If the online and offline results are consistent, the system outputs the consistent results. If they are inconsistent, the system outputs the tool and equipment categories and quantities that are inconsistent between online and offline.

[0127] Step S130: Iterative optimization of tool and equipment inspection model

[0128] To ensure continuous optimization of the rotating target detection model during use, an iterative optimization algorithm for the tool detection model was constructed as railway track maintenance image data accumulated. Unlabeled tool images collected from railway track maintenance scenes were input into an active learning framework driven by false positives and false negatives to obtain high-value sample data. This high-value sample data was then labeled and added to the training dataset for continuous iterative optimization of model performance. Figure 7 An active learning architecture diagram is provided for an embodiment of the present invention, and the specific process is as follows:

[0129] 1. For missed samples, compare the image segmentation model (such as SAM) with the target detection results to extract potential target instances that were not detected;

[0130] 2. Based on the vertical range of the detection box, irrelevant masks are filtered out, and the largest area mask and low area noise mask are removed. Furthermore, segmentation masks that overlap with existing detection boxes are removed.

[0131] 3. Filter the remaining candidate masks, and prioritize the retention of masks that cover the complete shape of the tool as valid targets; remove local mask areas that are already covered by the complete mask; and merge the remaining local masks according to spatial proximity through morphological dilation to reconstruct the complete target mask area.

[0132] 4. Prune and save the updated candidate mask regions to form missed detection sample data for training;

[0133] 5. For false positive samples, the target detection results are compared with the task text plan. Targets whose categories do not match the plan are filtered out from the detection results and the images are directly cropped and saved.

[0134] 6. For cases where the category matches but the number of detections exceeds the text label, retain targets with higher confidence, filter out redundant detection boxes with low confidence, and crop and save them.

[0135] 7. Manually label the cropped false positive and false negative sample images and add them to the training dataset;

[0136] 8. Retrain the object detection model based on the updated dataset, and iterate through the above steps to continuously improve the model performance under the active learning strategy.

[0137] In summary, the embodiments of the present invention improve the recognition accuracy and counting efficiency of railway tools and equipment by introducing a computer vision-based method for checking the online and offline status of railway tools and equipment, and by utilizing dataset construction, deep learning object detection algorithms, post-processing algorithms, and active learning strategies.

[0138] As can be seen from the above description, the embodiments of the present invention achieve the following technical effects:

[0139] This paper innovatively integrates textual information from work plans with visual features through a multimodal fusion detection architecture. A cross-modal attention mechanism guides the detection process, addressing the issue of traditional pure visual methods being susceptible to background interference. A dynamic adaptive detection mechanism is employed, proposing an adaptive threshold optimization algorithm based on textual information. A two-layer iterative search strategy dynamically adjusts detection parameters to achieve optimal matching between detection results and work plans. Data augmentation and generation techniques, combined with SAM segmentation and diffusion models, enable accurate extraction of foreground equipment and adaptive scene synthesis, effectively solving the problems of insufficient samples and domain offset in railway scenarios. Through active learning driven by false positives and false negatives, a pruning and screening mechanism based on false positives and false negatives is designed to automatically mine high-value labeled data. Compared to random sampling strategies, this results in a 40% improvement in model convergence speed and a 60% reduction in labeling costs.

[0140] The rotation target detection is optimized by using serpentine convolution and HAttention mechanisms to specifically enhance the feature extraction capability for slender and irregular tools, improving the rotation box detection accuracy by 15% compared to traditional methods.

[0141] High-precision identification, through multimodal fusion and adaptive post-processing, enables accurate identification of tool and equipment categories and quantities in complex railway scenarios, with average detection accuracy (mAP) increased to over 95% and false negative rate reduced to below 3%.

[0142] With strong scene adaptability, the data augmentation and generation technology effectively solves the problems of diverse tool shapes and varying placement angles, and the model maintains stable performance under different lighting and occlusion conditions.

[0143] Automation and efficiency improvements: Compared to manual counting, the verification efficiency is increased by more than 80%, and the time for a single verification is reduced from 30 minutes to less than 5 minutes, significantly reducing labor costs.

[0144] The continuous optimization capability and proactive learning strategy enable the model to evolve with the accumulation of new data. After 10 iterations, the accuracy of identifying rare tools and equipment increased from 70% to 92%.

[0145] With broad applicability, the method can be extended to multiple industrial sectors, and intelligent tool management can be achieved through sector adaptation, resulting in significant economic and social benefits.

[0146] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A computer vision-based railway tool on-and-off line inventory checking method, characterized in that, include: Collect images of railway tools and equipment at the high-speed railway maintenance depot and in warehouses, and annotate and preprocess these images. The target region was cropped using a region focusing algorithm to eliminate interference; a diverse training dataset was constructed using image enhancement and image augmentation algorithms; each on-site image of tools and equipment was associated with a structured work plan, which detailed the work time, workshop information, shift leader's name, and the type and quantity of tools and equipment. A text visual multimodal rotating target detection model is constructed, which mainly includes three parts: backbone network, neck network, and detection head network; Feature maps of railway machinery images are extracted using a backbone network; multi-level feature extraction is performed using a backbone network based on the CSPDarkNet architecture. Multi-scale feature maps are enhanced and fused using the PANet neck network based on serpentine convolution and HATtention. The PANet neck network first propagates semantic information through a top-down path. After upsampling, high-level features are aligned with low-level features in spatial dimension. Then, features are fused and enhanced through the C2f module. Finally, the HATtention attention mechanism is used to enhance the features after passing through the C2f module. The C2f module uses serpentine convolution as its core operator. The high-speed railway maintenance operation plan table is used as text information. The text information corresponding to the image is encoded into semantic features and fused with three feature maps of different sizes output by the neck network to enhance the visual features guided by the text information. The training dataset is input into the constructed target detection model to train the network and obtain the tool and equipment detection model. The images of the tools and equipment to be checked on the railway line are input into the tool and equipment detection model to identify the tools and equipment and obtain the initial tool and equipment detection results. An adaptive threshold optimization algorithm based on the text information is constructed. The initial tool detection results are compared with the text information in terms of category and quantity consistency. The detection confidence threshold and non-maximum suppression parameters are dynamically optimized to obtain the tool detection results adjusted by the threshold. The threshold-adjusted tool detection results are compared again with the text information. For mismatched categories, a few-sample target detection algorithm is used to supplement the identification, resulting in a few-sample identification result. The few-sample identification results are processed by a category-aware scale distribution filtering algorithm and an outlier detection box adaptive suppression algorithm. Then, the processed results are combined with the threshold-adjusted tool detection results for cross-class non-maximum suppression to obtain the final tool detection results. The final inspection results of the tools and equipment are counted and compared with the on-line and off-line maintenance operation plan of the high-speed railway engineering section to obtain the comparison results.

2. The computer vision-based railway tool on / off-line inventory check method of claim 1, wherein, Partially annotate the on-site images collected from the high-speed railway maintenance scene, and perform tool region focusing optimization on the annotated tool images to obtain preprocessed image data, including: the tool region focusing optimization is to dynamically generate the tool interest region based on the position of the workers and tools in the on-site images collected from the high-speed railway maintenance scene, and then perform cropping to extract the tool target region; The preprocessed image data and images of tools and equipment collected from the warehouse environment are subjected to image enhancement and image augmentation processing to construct a diverse training dataset. The image enhancement includes an image geometric correction algorithm based on perspective transformation and a scale adaptive enhancement algorithm for multi-transformation stitched images. The image augmentation includes intelligent directional network data crawling, an image generation algorithm based on SAM and diffusion model, and a SAM-guided foreground-normalized background synthesis method for tools and equipment.

3. The computer vision based railway gang tool on-off line inventory verification method as claimed in claim 1, wherein, The training dataset is input into the constructed target detection model to train the network and obtain the tool and equipment detection model, including: based on the constructed diverse training dataset, the target detection model is used to train the network, and the network parameters are continuously optimized through the backpropagation mechanism of the loss function to finally obtain the tool and equipment detection model; The images of railway tools and equipment to be verified are input into the tool and equipment detection model for tool and equipment identification to obtain initial tool and equipment detection results. This includes: preprocessing the images of railway tools and equipment to be verified on the railway lines and inputting them into the tool and equipment detection model to obtain all initial tool and equipment detection results without confidence filtering and non-maximum suppression.

4. The computer vision based railway gauge on / off line inventory check method as claimed in claim 1, wherein, The adaptive threshold optimization algorithm based on the text information is constructed by comparing the initial tool detection results with the text information for consistency in category and quantity, dynamically optimizing the detection confidence threshold and non-maximum suppression parameters, and obtaining the tool detection results adjusted by the threshold, including: The system compares all initial detection boxes retained from the initial tool and equipment detection results with the text information item by item in terms of category and quantity. An adaptive threshold optimization algorithm is used to automatically increase the confidence threshold for categories not mentioned in the text information to suppress irrelevant detections. At the same time, the matching degree between the detection results and the text information is calculated in real time. During this process, the system adopts a two-layer loop search strategy to adaptively adjust the confidence threshold and IoU (Intersection over Union) threshold parameters until the optimal matching degree is obtained, thereby correcting the initial detection boxes and obtaining the tool and equipment detection results adjusted by the threshold.

5. The computer vision-based method for checking the online and offline status of railway tools and equipment as described in claim 1, characterized in that, The step of comparing the threshold-adjusted tool detection results with the text information again, and using a few-shot target detection algorithm to supplement the identification of mismatched categories, yields few-shot identification results, including: After obtaining the threshold-adjusted tool detection results, the results are compared with the text information. If they do not match, the names of the matching and non-matching categories are recorded respectively. The non-matching categories are then detected again using a target detection model based on few samples. The tool image to be detected is used as the query image, and the labeled sample image of the target category is used as the support image. Features of the query image and the support image are extracted by a deep neural network, and the feature similarity between the two is calculated to enhance the feature representation ability of the query image. Subsequently, the enhanced query image features are input into the Region Proposal Network to generate candidate target regions, and RoI Align is performed on these candidate regions to obtain refined region features. Similarly, the support image features are also processed by RoI Align to obtain standardized region features. By calculating the similarity matching score between the query image region features and the support image region features, and combining the spatial location information of the target, the final output is the tool category determination and precise location coordinates, thus obtaining the few-sample recognition result.

6. The computer vision-based method for checking the online and offline status of railway tools and equipment as described in claim 1, characterized in that, The process of processing the few-sample identification results using a category-aware scale distribution filtering algorithm and an outlier detection box adaptive suppression algorithm, and then performing cross-class non-maximum suppression on the processed results and the threshold-adjusted tool detection results, yields the final tool detection results, including: The initial detection boxes output by the few-sample target detection model are filtered by the category-aware scale distribution filtering algorithm, which effectively removes abnormal detection boxes that do not match the scale distribution of the target category and obtains intermediate results of scale consistency optimization. The intermediate results are analyzed by applying an adaptive suppression algorithm for outlier detection boxes. By calculating the spatial distribution characteristics of the detection boxes, outlier detection boxes that deviate from the main distribution are intelligently identified and filtered, resulting in a few-sample detection result that has undergone dual optimization. The threshold-adjusted tool and equipment detection result is then fused with the few-sample detection result that has undergone dual optimization. The fused result is then calibrated using a cross-class nonmaximum suppression algorithm to eliminate redundant detection boxes and obtain the final railway tool and equipment identification result.

7. The computer vision-based method for checking the online and offline status of railway tools and equipment as described in claim 1, characterized in that, The final tool and equipment inspection results are counted and compared with the high-speed railway maintenance section's on- and off-line maintenance operation plan to obtain the comparison results, including: Each image of the high-speed railway maintenance section going online and offline is counted by category. The count results are compared with the high-speed railway maintenance section's online and offline maintenance operation plan. If the comparison is correct, the error is output. If the comparison is incorrect, the missing or extra tool category name and quantity are output. The images of tools and equipment that are online and offline in the same railway workshop are compared to obtain the comparison results to check whether they are consistent. This includes comparing the count results of each category of the images of tools and equipment that are online and offline in the same railway workshop. If the results are consistent, the output is consistent. If they are inconsistent, the output is the category and quantity of tools and equipment that are inconsistent between online and offline.

Images