Railway on-off tool identification method and device based on multi-modal large model

Abstract

The application discloses a railway on-off tool identification method and device based on a multi-modal large model, and belongs to the field of image analysis. In order to solve the problems of insufficient positioning accuracy in complex scenes, excessive dependence on tool number identification and fixed model identification capability in the existing railway on-off tool counting technology, the SAM3 model fine-tuned by low-rank adaptation technology is used, the work tool list is converted into a text prompt word as prior guidance, pixel-level instance segmentation is performed on the on-site tool image, the tool type and quantity are counted based on the instance mask, and comparison with the work tool list is performed to complete the counting. The application applies image semantic segmentation technology to automatic counting of railway electric power operation tools, does not need to rely on tool surface numbers, has high identification accuracy in complex scenes, can dynamically adapt to operation requirements, improves counting efficiency and system robustness, and guarantees railway operation safety.

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation, specifically to a method and apparatus for identifying railway track entry and exit tools based on a multimodal large model. Background Technology

[0002] In railway electrical operations, the accurate counting of tools on and off the tracks is a crucial step in ensuring train operation safety and preventing accidents caused by tools left behind. Traditional manual counting methods are inefficient, prone to errors, and difficult to trace, making them unsuitable for the needs of modern railway safety management. In recent years, with the rapid development of computer vision technology, automated counting methods based on image recognition have gradually become a research and application hotspot.

[0003] Currently, mainstream visual inventory solutions for railway track maintenance tools typically employ a "detection-identification-matching" technical paradigm.

[0004] For example, prior art 1: Patent CN121481472A, a method for counting railway track-mounted tools based on image recognition. This method first acquires images of the tools on site, extracts edge and texture features to construct candidate regions for the tools; then, it applies the YOLO scheme to obtain tool category scores and location boxes, and uses the DETR scheme to achieve coordinate correction and category verification, obtaining accurate tool location and category information; finally, it reads the unique number on the tool through optical character recognition (OCR) technology, compares it with the ledger to confirm the uniqueness of the tool, and generates an inventory report.

[0005] The existing technology represented by technology 1 has achieved automation of tool counting to a certain extent, but it still has the following technical limitations in the complex scenarios of actual railway electrical operations:

[0006] First, existing bounding box-based target detection solutions suffer from poor scene adaptability and insufficient positioning accuracy, making them ill-suited for complex operational scenarios. Existing inventory solutions, exemplified by technology 1, all employ bounding box-based target detection, treating tools as a whole for detection and localization, failing to achieve pixel-level instance segmentation. In railway operation scenarios characterized by densely stacked, mutually obscuring, and complex backgrounds, issues such as overlapping bounding boxes, category confusion, and missed or false detections are highly likely to occur. The inability to accurately isolate each tool instance leads to a significant drop in inventory accuracy, failing to meet the safety management requirements of on-site operations.

[0007] Secondly, existing tool inventory methods rely excessively on the tool surface identification numbers, resulting in poor system robustness and insufficient field adaptability. Existing technologies, such as technology 1, heavily depend on the accurate reading of tool numbers using OCR technology to confirm tool uniqueness. However, the railway operating environment is complex and variable; tool numbers often wear out or become dirty due to long-term use, or the characters become blurred due to poor lighting conditions. In such cases, the OCR recognition success rate drops significantly, making it difficult for the system to complete number matching, leading to inventory failures or false alarms, and failing to meet the high reliability requirements of field operations.

[0008] Third, existing visual models have fixed recognition capabilities and lack dynamic interaction and adaptability. The recognition categories and capabilities of existing detection models are completely fixed after training. However, the list of tools required for different tasks in railway electrical work varies significantly. Existing models cannot perform targeted identification and verification based on the actual list of tools for a single task. If new work scenarios and tool types need to be adapted, data must be collected again and the model must be fully trained. This is not only costly in terms of computing power and time-consuming, but also cannot meet the flexible and efficient application requirements of on-site operations.

[0009] In summary, existing technologies cannot simultaneously address the core pain points of accurate positioning, unique tool identification, and dynamic business adaptation in complex environments during railway track maintenance. They also struggle to balance identification accuracy, system robustness, deployment costs, and on-site adaptability.

[0010] Therefore, there is an urgent need for a new intelligent identification and counting technology for railway track entry and exit tools based on a multimodal large model to meet the practical application needs of railway electrical operation safety management. Summary of the Invention

[0011] To alleviate or partially alleviate the above-mentioned technical problems, the solution of the present invention is as follows:

[0012] On one hand, this invention discloses a method for identifying railway track entry and exit tools based on a multimodal large model, comprising the following steps:

[0013] Step S1: Obtain on-site tool image data for the work nodes before and after the work begins, and obtain a list of work tools for this operation;

[0014] Step S2: Convert the list of work tools into text prompts, and input the text prompts and the on-site tool image data into the multimodal large model;

[0015] Step S3: Using the text prompt words as prior guiding conditions, the multimodal large model performs pixel-level instance segmentation on the on-site tool image data and outputs the instance mask of each tool;

[0016] Step S4: Obtain the actual tool types and quantities based on the instance mask, and compare them with the work tool list to obtain the inventory results; wherein,

[0017] The multimodal large model is a SAM3 model finely tuned using low-rank adaptation technology, and it has the ability to extract and learn the inherent visual features of railway electrical equipment.

[0018] In one embodiment, the SAM3 fine-tuned using low-rank adaptation techniques is constructed in the following manner:

[0019] Freeze all the original weight matrices of the pre-trained SAM3 backbone network, so that the backbone network weights are in an untrainable state during fine-tuning.

[0020] For each converter module in the backbone network, a trainable reduced-dimensional matrix and an increased-dimensional matrix are connected in parallel via bypass.

[0021] During forward propagation, the output of the multimodal large model is composed of the original output of the backbone network and the incremental output of the low-rank adaptive bypass.

[0022] During backpropagation, only the gradients of the reduced-dimensional matrix and the increased-dimensional matrix are updated until the performance index of the multimodal large model converges, thus completing the fine-tuning of the low-rank adaptation technique.

[0023] In one embodiment, the fine-tuning training of SAM3 employs a joint loss function, which is a weighted combination of cross-entropy loss and dice coefficient loss.

[0024] In one type of implementation, during forward propagation, the output h of the multimodal large model satisfies:

[0025] ;

[0026] Where W is the frozen original weight matrix, A is the dimension reduction matrix, B is the dimension increase matrix, and x is the input vector.

[0027] In one embodiment, the joint loss function is calculated as follows:

[0028] ;

[0029] Where CE is the cross-entropy loss function, DC is the dice coefficient loss function, ŷ is the predicted mask generated by the model, D(y) is the one-hot encoding result of the true mask, λ1 is the weight coefficient of the cross-entropy loss, and λ2 is the weight coefficient of the dice coefficient loss.

[0030] In one embodiment, the text prompt is a short English noun corresponding to a railway electrical tool, with each type of tool having a unique text prompt.

[0031] In one embodiment, step S3 specifically includes:

[0032] Using text prompts as prior guidance, pixel-level instance segmentation and category determination are performed on target tools in the image to obtain initial instance segmentation results;

[0033] The initial instance segmentation results are post-processed with non-maximum suppression to filter duplicate detection regions and output instance masks that are independent of each tool.

[0034] In one embodiment, step S1, the method for acquiring the on-site tool image data includes:

[0035] Use camera equipment to capture raw video or still images containing on-site tools;

[0036] The collected static images are screened and then directly used as on-site tool image data;

[0037] For the acquired raw video, frame extraction is performed on the raw video to extract key frame images as on-site tool image data.

[0038] In one embodiment, step S4, obtaining the actual tool types and quantities based on the instance mask, specifically includes:

[0039] The number of independent connected components in the instance mask corresponding to each type of tool is counted. The number of independent connected components is the actual number of occurrences of that type of tool. The actual occurrences of all categories are summed to obtain the actual types and quantities of tools.

[0040] In one embodiment, after obtaining the inventory results, the method further includes:

[0041] A standardized operation inventory report is generated based on the inventory results; when the comparison results show that the types or quantities of tools before and after the operation do not match, the abnormal tools are marked with differences and an alarm signal is output.

[0042] On the other hand, this invention discloses a railway track entry / exit tool identification device based on a multimodal large model, characterized in that it includes:

[0043] The data acquisition module is used to acquire on-site tool image data of the work nodes before and after the work begins, and to obtain a list of work tools for this operation.

[0044] A multimodal input module is used to convert the list of work tools into text prompts and input the text prompts and the on-site tool image data into a multimodal large model;

[0045] The instance segmentation module has a built-in multimodal large model, which is used to perform pixel-level instance segmentation on the on-site tool image data with the text prompt words as prior guiding conditions, and output the instance mask of each tool.

[0046] The statistical comparison module is used to obtain the actual tool types and quantities based on the instance mask, and compare them with the work tool list to obtain the inventory results;

[0047] The multimodal large model is a SAM3 model finely tuned using low-rank adaptation technology, and it has the ability to extract and learn the inherent visual features of railway electrical equipment.

[0048] The technical solution of this invention has one or more of the following beneficial technical effects:

[0049] (1) This invention transforms traditional bounding box target detection into pixel-level instance segmentation and uses the SAM3 model to achieve accurate stripping of tool instances in complex stacked and occluded scenes, greatly eliminating the interference caused by bounding box overlap and improving the recognition accuracy in complex scenes.

[0050] (2) The present invention enables the model to learn and rely on the visual features of the tool itself to complete the classification by using low-rank adaptation fine-tuning technology, eliminating the dependence on the OCR number on the tool surface, effectively overcoming the recognition failure problem caused by the number being dirty or missing, and significantly improving the robustness of the system.

[0051] (3) This invention uses a text-image multimodal input architecture and a list of work tools as text prompts to dynamically guide the model’s segmentation and recognition process, enabling the static model to adapt to and match dynamic work requirements, thus achieving the interactive capability of “on-demand inventory”. It eliminates the need to retrain the model for different work scenarios, significantly reducing application costs.

[0052] (4) This invention transforms the traditional “detection-recognition-matching” technology paradigm into a “segmentation-statistics-comparison” paradigm based on text prompts. It uses pixel-level instance segmentation to replace bounding box detection, fundamentally eliminating the interference caused by bounding box overlap, achieving accurate stripping of tool instances in complex stacking scenarios, and significantly improving recognition accuracy.

[0053] Furthermore, other beneficial effects of the present invention will be mentioned in the specific embodiments. Attached Figure Description

[0054] Figure 1 This is a flowchart of one embodiment of the present invention;

[0055] Figure 2 This is a flowchart illustrating the construction of an image dataset according to one embodiment of the present invention;

[0056] Figure 3 This is the overall architecture diagram of the present invention based on the LoRA method for fine-tuning SAM3;

[0057] Figure 4 This is a schematic diagram illustrating the principle of the LoRA method in this invention;

[0058] Figure 5 This is a visualization comparing the experimental results of the method of this invention with the basic pre-trained SAM3 model. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0060] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order.

[0061] Terminology Explanation:

[0062] On-site tool image data: refers to the original static images of all tools on site collected by camera equipment at two work nodes, before and after the workers go onto the track and after they leave the track, or key frame images extracted from the work video, which serve as the visual input source for multimodal large models.

[0063] Operational Tool List: This refers to the set of all tools and their corresponding quantities required for the railway electrical maintenance window, entered by the user before the start of the day's work, based on the work tasks, procedures, and safety control requirements. This list serves as the source of text prompts for the multimodal large model and is used for comparison with the actual tool statistics obtained from image recognition.

[0064] Instance mask: refers to the pixel-level segmentation result output by the multimodal large model after performing pixel-level instance segmentation on the image. It represents the precise outline and position of each tool in the image in the form of a binary mask or a class mask, and is used to count the types and number of tools.

[0065] Low-rank adaptation (LoRA) is a technique for lightweight fine-tuning of large models. By connecting a trainable low-rank matrix in parallel to the backbone network of a pre-trained model, only a small number of parameters need to be updated to adapt the model to specific vertical domain tasks, significantly reducing the computational cost of fine-tuning.

[0066] Inherent visual features: refer to the inherent visual attributes of the tool itself that do not depend on external identifiers (such as numbers or QR codes), including but not limited to the tool's category, shape, texture, and outline. This invention enables a multimodal large model to learn these features through fine-tuning to achieve accurate recognition independent of the number.

[0067] This invention discloses an intelligent identification and counting method for railway track entry and exit tools based on a multimodal large model, aiming to solve the technical defects of existing railway track entry and exit tool counting technologies, such as insufficient positioning accuracy in complex scenarios, over-reliance on tool number identification, and static and fixed model recognition capabilities. The core concept of this invention is to transform traditional bounding box object detection into pixel-level instance segmentation, and to identify tools based on their inherent visual features (shape, texture, contour), without relying on OCR numbers; to utilize a third-generation suggestible concept instance segmentation large model (Segment AnythingModel 3, SAM3) that supports text prompts, combined with low-rank adaptation (LoRA) fine-tuning technology, resulting in a multimodal large model for SAM3; to use the actual tool list input before operation as text prompts to guide the model to perform pixel-level instance segmentation on the images collected on-site, and to achieve accurate counting of tool types and quantities through statistical instance masks. For ease of description, the multimodal large model of this invention will be referred to as the "model" below.

[0068] Figure 1 This is a flowchart of one embodiment of the present invention. As shown in the figure, the method forms a complete closed loop from multimodal data input to inventory result output. Specifically, it is divided into four core processing stages executed sequentially: data acquisition and preprocessing, multimodal feature interaction and instance segmentation, mask statistics and logical comparison, and result output and verification closed loop. The specific implementation methods of each stage are described in detail below.

[0069] In the data acquisition and preprocessing stage, this phase is responsible for the synchronous input and standardized processing of multimodal front-end data. Specifically, before and after workers enter and exit the track, raw video or image data containing all tools on site is collected using video equipment, achieving complete data acquisition for both stages. To improve processing efficiency, the system performs frame extraction on the acquired video stream, extracting keyframe images as visual input sources. Simultaneously, the system receives a list of the types and quantities of tools actually required for the current track maintenance operation. This list is entered by the workers through their terminals according to the day's tasks and serves as prior text data for subsequent model processing. The standardized image data and text list data output in this phase together constitute the input foundation for the multimodal model.

[0070] In the multimodal feature interaction and instance segmentation stage, this stage utilizes a SAM3 model fine-tuned using LoRA technology. Based on standardized visual input sources and prior text data, it performs pixel-level instance segmentation and category determination for the tools, which is the core step in achieving accurate inventory. The specific implementation method for this stage is as follows: the list of operational tools entered in the aforementioned data acquisition and preprocessing stages is mapped to text prompts in a specific format. These prompts are short English terms specifically optimized for 33 categories of railway electrical tools, with each tool category corresponding to a unique text prompt. This effectively reduces the difficulty of model feature learning and improves the accuracy of target feature boundary extraction and inter-class discrimination. After the text prompts are converted, the standardized visual input source image and the optimized text prompts are input together into the SAM3 (multimodal large model) fine-tuned using LoRA technology. The model, through a multimodal feature fusion mechanism of text and image, uses the text prompts as prior guiding conditions to perform pixel-level instance segmentation and category determination for the target tools in the image, obtaining the initial instance segmentation results. For the initial segmentation results output by the model, this method further performs non-maximum suppression (NMS) post-processing to filter duplicate detection regions in the initial segmentation results, and finally outputs instance masks that are independent of each other and do not overlap or interfere with each other, thus avoiding inter-class confusion caused by semantic boundary ambiguity.

[0071] In the mask statistics and logical comparison stage, based on the independent instance mask data output by instance segmentation, the accurate count of tool quantities is completed, and a consistency comparison is performed with the work tool list to achieve automatic verification of the inventory results. The specific implementation method of this stage is as follows: First, the instance mask data output from the previous stage is received. For each type of tool, the number of independent connected components of each type of tool mask in a single image is counted. This number represents the actual number of tools of that type appearing in the image. After traversing all tool mask data and completing the quantity count of all tool categories, the results are summarized to form the actual tool types and quantities contained in the acquired images. After summarizing the visual statistical results, these results are compared one by one with the work tool list entered in the data acquisition stage. For the acquired data at both the previous and subsequent stages, the consistency verification of tool types and quantities is completed. Simultaneously, the differences in tool types and quantities between the two stages are calculated to provide data basis for the final inventory result determination.

[0072] In the result output and verification closed-loop stage, this method, based on the results of logical comparison, automatically generates the job count report and provides early warnings for abnormal situations, realizing a closed-loop process for tool count operations. The specific implementation of this stage is as follows: First, based on the output results of the mask statistics and logical comparison stage, a standardized job count report is automatically generated (for example, the report records detailed data such as the planned number of work orders for each tool, the actual number of counts before starting the process, the actual number of counts after finishing the process, the quantity difference value, and the segmentation recognition confidence level, indexed by tool category, and includes the original images collected before starting the process and the instance segmentation visualization images). This completes the full data traceability of the count process. In one embodiment, when the comparison results show that the types and quantities of tools before and after entering the track are completely consistent and match the work tool list, the work inventory report is marked with the conclusion that the work is normal and the tools are counted correctly. When the comparison results show that there is a mismatch in the types or quantities of tools, including missing or extra tools, or unknown tools not on the work order list, the abnormal tools are highlighted in the work inventory report, and at the same time, an audible and visual alarm signal is output through the on-site terminal to prompt the operators to check the tool status in time and eliminate the safety hazard of tools being left on the track. This completes the closed loop of intelligent identification and inventory of tools entering and leaving the railway track.

[0073] To enable those skilled in the art to fully reproduce the technical solution of this invention, the following provides a more detailed description of the specific implementation methods for constructing the image dataset, optimizing the multimodal cue word engineering, and deploying LoRA-based SAM3 fine-tuning and inference in the data preparation and preprocessing stages of this invention.

[0074] Regarding the construction of the image dataset, this embodiment of the invention constructs a dedicated tool segmentation dataset for railway electrical operation scenarios, providing high-quality training data for fine-tuning SAM3. Figure 2 This is a flowchart illustrating the construction of an image dataset according to one embodiment of the present invention. As shown in the figure, the specific implementation method for constructing the dataset is as follows: First, using on-site acquisition devices such as industrial cameras and handheld mobile terminals, raw on-site tool image data is acquired at two work nodes: before the workers enter the track and after they leave the track. This on-site tool image data includes directly captured still images or recorded work videos. For directly captured static images, images containing the target tools are selected and used directly as raw image samples. For recorded operation videos (e.g., collecting operation videos of 33 commonly used railway electrical operation tools under different railway operation sites, lighting conditions, tool placement angles, and stacking states), frame extraction is performed on the acquired videos to remove duplicate, blurry, and invalid images, and then raw image samples containing the target tools are obtained. Subsequently, the X-AnyLabeling annotation tool combined with the Segment Anything 2.1 model is used to perform semi-automatic polygon annotation on the 33 types of target tools in the raw image samples. After completing the annotation of a single image, all annotation data are uniformly converted into the CommonObjects in Context (COCO) format (e.g., the annotation file includes basic image information, instance mask coordinates of the target tool, category number, category name, and other core fields), and finally a dedicated dataset containing 10,680 annotated images is constructed for fine-tuning training of the model.

[0075] Next, we optimize the multimodal prompt words. Unlike the pure visual object detection models in existing technologies, this invention adopts a multimodal input mechanism that combines text and images. Extensive comparative experiments have verified that using short English nouns corresponding to tool categories as text prompt words for SAM3 effectively reduces the feature learning difficulty during the fine-tuning process in the railway vertical domain. Simultaneously, it makes the extracted target tool feature boundaries clearer and more accurate, improving the pixel-level accuracy of instance segmentation. If the semantic boundaries of the prompt words are vaguely defined, the model is prone to inter-class feature confusion, for example, misidentifying semantically similar words like "glove" as "dustcloth," directly affecting the accuracy of tool counting. Based on this, this invention establishes a one-to-one independent prompt word mapping relationship for each of the 33 categories of track-connecting tools. By optimizing and configuring the prompt words for each tool category, the model's inter-class feature discrimination and target recall rate for subdivided tools in the railway vertical domain can be significantly improved, effectively reducing the probability of inter-class misidentification. The optimized mapping relationship between some commonly used tools and their corresponding English prompt words is shown in Table 1.

[0076] Table 1: Tooltip Word List

[0077]

[0078] Figure 3 This is the overall architecture diagram of SAM3 fine-tuning based on the LoRA method of this invention. As shown in the figure, it illustrates the hierarchical structure and connection relationships of the architecture, namely "frozen backbone network - Transformer block parallel LoRA bypass - prompt word encoder - mask decoder," as well as the data flow. This invention constructs a multi-class instance segmentation network that includes low-rank adaptation. Addressing the issue of the large number of parameters in the Vision Transformer (ViT) architecture of SAM3 and the extremely high computational cost of full fine-tuning, this invention introduces LoRA technology to perform lightweight reconstruction and feature fine-tuning of the pre-trained model. This significantly reduces the computational cost of fine-tuning while significantly improving the model's segmentation and recognition accuracy for railway special tools. Specific implementation methods are as follows: Figure 4 This is a schematic diagram of the LoRA method in this invention. The diagram shows in detail the two-path structure of LoRA, the weight freezing logic, and the propagation principle of the low-rank matrix.

[0079] During the reconstruction of the fine-tuned network architecture, this invention freezes all the original weight matrices W of the pre-trained SAM3 backbone network, where the weight matrices... d and k are the output and input dimensions of the weight matrix, respectively, making the backbone network weights untrainable during fine-tuning. Simultaneously, two sets of trainable low-rank matrices are connected in parallel as bypasses for each Transformer module in the SAM3 network: a dimension-reduced matrix A and an increased-dimensional matrix B. The dimension-reduced matrix... Upgraded dimensional matrix Rank r satisfies The rank r is used to control the dimension of the low-rank matrix, enabling lightweight fine-tuning of the model.

[0080] During forward propagation, the model's output consists of the original output of the backbone network and the incremental output of the LoRA bypass. Given an input vector x, the formula for calculating the model's forward propagation output h is:

[0081] ,

[0082] Here, ΔW is the weight increment matrix, which is the product of the dimension reduction matrix A and the dimension increase matrix B. This formula clarifies the computational logic of parallel processing of the original backbone network and the LoRA bypass, and the sum of the results.

[0083] During backpropagation, this method does not require updating the original large weight matrix W; it only calculates and updates the gradients of the low-rank reduced-dimensional matrix A and the increased-dimensional matrix B. Its dimensionality reduction and increase mechanisms are as follows: Figure 4 As shown.

[0084] In the multi-class segmentation mapping mechanism of the model, given the input upper and lower tool images Where R represents the real number field, H is the image height, W is the image width, and C is the number of image channels. Let the true segmentation mask of the image be... In this paper, pixel value 0 represents the background class, and pixel values ​​1 to 33 correspond to 33 different categories of railway tools. The invention establishes a mapping function... Generate a prediction mask ŷ, where P is the optimized text prompt word embedding feature, θ is the frozen SAM3 pre-trained weights, and ∅ is the parameters of the trainable low-rank matrix accessed via the LoRA method.

[0085] In the mask decoder section, this invention does not introduce an additional LoRA layer, maintaining the native lightweight Transformer structure of SAM3. Fine-tuning is performed using the default cue embedding method, training only the low-rank matrix of the LoRA bypass and the classification head of the mask decoder, further reducing the number of training parameters and computational cost. During the model's fine-tuning training, to improve the edge smoothness and recognition accuracy of multi-class pixel-level segmentation, this invention uses a joint loss function composed of a weighted combination of cross-entropy loss (CE) and dice coefficient loss (DiCe loss, DC) as the target total loss function for training. The formula for calculating the total loss function L is:

[0086] ,

[0087] Where CE is the cross-entropy loss function, DC is the dice coefficient loss function, ŷ is the predicted mask generated by the model, and D(y) is the one-hot encoding result of the true mask. λ1 is the weight coefficient of the cross-entropy loss, and λ2 is the weight coefficient of the dice coefficient loss. In one embodiment, λ1 is set to 0.2 and λ2 is set to 0.8. By setting the weight coefficients, the proportions of cross-entropy loss and dice coefficient loss in the total loss are balanced.

[0088] During training, the trainable parameters ∅ of the LoRA bypass and the classification header parameters of the mask decoder are iteratively updated using gradient descent until the model's performance metrics on the test set converge, thus completing the model fine-tuning for the characteristics of railway special-purpose vehicles.

[0089] After completing the fine-tuning training of the model, in the actual business inference deployment phase, this invention adopts a reparameterization approach to merge the model weights. Specifically, the low-rank weight increment matrix ΔW (ΔW=BA) obtained after training is directly merged into the original pre-trained weights W of SAM3 to form the updated weights. This reparameterization operation makes the fine-tuned model completely consistent with the inference structure of the native SAM3 during the inference phase, without the need for additional bypass computation. It does not introduce any inference latency to the actual detection business, perfectly adapting to the characteristics of railway electrical equipment and the low-latency deployment requirements of edge devices at railway operation sites.

[0090] To further verify the effectiveness of the railway electrical work tool inventory method based on SAM3 and LoRA fine-tuning proposed in this invention, this embodiment conducted a detailed comparative experiment and effect analysis.

[0091] In a specific example, the hardware and software environment configuration for model fine-tuning in this embodiment is as follows: the operating system is Ubuntu 22.04, the CPU consists of two Intel Xeon Gold 6148 processors, the memory is 256GB (32GB×8) DDR5 memory, and the GPU consists of four NVIDIA A800 (80GB) graphics cards. The hyperparameters for the training process are set as follows: the training epoch is set to 100, the optimizer is AdamW, the initial learning rate is set to 0.005, and the total time for model fine-tuning is approximately 485 minutes. The dataset used in the experiment contains approximately 8267 training set images and 2413 test set images. The hyperparameters and hardware environment configuration are shown in Table 2.

[0092] Table 2: Hardware and software configuration and hyperparameter settings in an example of the present invention

[0093]

[0094] To verify the effectiveness and technical advantages of the method proposed in this invention, this embodiment conducted quantitative and qualitative comparative experiments on a constructed dedicated test set. The basic pre-trained model (SAM3 pre-trained) was used as the baseline, and pixel accuracy, intersection over union (IoU), and Dice coefficient were selected as evaluation metrics. Their calculation formulas are shown below:

[0095] ,

[0096] ,

[0097] ,

[0098] Where y represents the ground truth mask of the image, ŷ represents the predicted segmentation mask output by the model, and n and m are the width and height of the image, respectively.

[0099] The quantitative assessment results are shown in Table 3:

[0100] Table 3: Model Comparison Experiment

[0101]

[0102] Experimental results show that the SAM3 model fine-tuned with LoRA achieves significant improvements in all segmentation metrics. Specifically, the pixel accuracy of the method presented in this invention reaches 93.84%, the intersection-over-union ratio reaches 0.921, and the Dice coefficient reaches 0.935; compared with the untuned SAM3 base model, these three core metrics are improved by 3.71%, 0.064, and 0.055, respectively. Experimental results demonstrate that the dedicated dataset and lightweight fine-tuning strategy employed in this invention can effectively improve the segmentation accuracy and generalization ability of large models in the railway vehicle sub-domain.

[0103] Figure 5 This is a visualization comparing the experimental results of the method of this invention with the basic pre-trained SAM3. To intuitively demonstrate the segmentation advantages of the method of this invention, typical and challenging complex field images were selected for visualization comparison. The comparison objects include ground truth (GT), the base model (Base), and the fine-tuned model of this invention (LoRA). The comparison results are as follows: Figure 5 As shown.

[0104] The qualitative visualization results in the figure further reveal the technical advantages of the method of this invention. In the first set of comparative experiments, when faced with a safety hammer with similar material and background texture, the basic pre-trained SAM3 model failed to effectively extract target features, resulting in a serious false negative problem. However, the fine-tuned model of this invention accurately learned the specific visual features of the hammer through targeted training, successfully outputting a high-precision instance mask, effectively improving the target recall rate of the tool counting task. In the second set of comparative experiments, for white cloth and gloves, which are easily confused in form and semantics, the basic pre-trained model had a fuzzy understanding of the feature boundaries of the text prompt word "dustcloth," resulting in inter-class confusion by misclassifying gloves as white cloth. However, this invention, through deep alignment and targeted fine-tuning of multimodal features, enabled the model to define clearer, finer-grained decision boundaries in the feature space, accurately separating and distinguishing the two easily confused categories of gloves and white cloth, significantly reducing the false positive rate and further improving the accuracy of tool counting.

[0105] The technical solution of this invention is not limited to the specific embodiments described above. Without departing from the core concept of this invention, there are various feasible alternative implementation methods. In the LoRA fine-tuning parameter settings, the value of the rank r can be flexibly adjusted according to actual hardware resources and model performance requirements. In the weight settings of the loss function, the values ​​of λ1 and λ2 can be dynamically adjusted according to the actual convergence during training to adapt to different dataset distribution characteristics. In the image acquisition stage, in addition to using a monocular camera to acquire two-dimensional images, a binocular camera or a depth camera can also be used to acquire image data containing depth information. Combining depth information further optimizes the accuracy of instance segmentation and improves the tool discrimination capability in stacked scenarios.

[0106] In summary, the intelligent identification and counting method for railway track maintenance tools based on a multimodal large model described in this embodiment transforms traditional bounding box target detection into pixel-level instance segmentation. Based on SAM3, it achieves accurate tool instance separation in complex stacking and occlusion scenarios, significantly eliminating interference caused by bounding box overlap. Through LoRA lightweight fine-tuning technology, the model can complete identification and classification based on the inherent visual features of the tools, eliminating reliance on OCR number recognition and effectively overcoming recognition failures caused by damaged or missing numbers, significantly improving the system's robustness. Through a text-image multimodal input architecture, the tool list serves as a text prompt to dynamically guide the model's segmentation and recognition process, enabling the static model to adaptively match dynamic operational needs, achieving interactive "on-demand counting" capability. This eliminates the need to retrain the model for different operational scenarios, significantly reducing application costs. This invention addresses the core pain points and inherent defects of existing technologies, effectively improving the accuracy and efficiency of railway track maintenance tool counting, and providing reliable technical support for the safety management of railway electrical operations.

[0107] To implement the above method, the present invention also provides a railway track entry / exit tool identification device based on a multimodal large model. In one embodiment, the device includes a data acquisition module, a multimodal input module, an instance segmentation module, and a statistical comparison module.

[0108] The data acquisition module is used to perform the functions of the aforementioned data acquisition and preprocessing stages. Specifically, at two work nodes—before the workers enter the track and after they leave—the data acquisition module collects raw video or still images containing all tools on site using camera equipment. The collected still images are directly used as raw image samples after screening, while the collected raw video undergoes frame extraction to extract keyframe images as raw image samples, thereby obtaining on-site tool image data. Simultaneously, the data acquisition module receives a list of the types and quantities of tools actually required for this track maintenance window operation, entered by the workers through their terminals, as the work tool list.

[0109] The multimodal input module converts the list of operational tools acquired by the data acquisition module into text prompts in a specific format, and inputs the text prompts along with the on-site tool image data into the built-in multimodal large model. The text prompts are short English terms specifically optimized for 33 categories of railway electrical tools, with each category of tool corresponding to a unique text prompt.

[0110] The instance segmentation module incorporates SAM3, finely tuned using low-rank adaptation technology. Using text prompts from the multimodal input module as prior guidance, the module performs pixel-level instance segmentation and category determination on the field tool image data, outputting instance masks for each tool. In one embodiment, the instance segmentation module further performs non-maximum suppression post-processing to filter duplicate detection regions and output independent instance masks.

[0111] The statistical comparison module receives instance mask data output by the instance segmentation module, counts the number of independent connected components for each type of tool's mask to obtain the actual tool types and quantities, and compares this statistical result with the operational tool list obtained by the data acquisition module to generate an inventory result. When the comparison result shows a mismatch between the tool types or quantities before and after the process, the statistical comparison module also outputs a corresponding alarm signal.

[0112] The functions of the above modules work together to achieve intelligent identification and counting of railway track entry and exit tools. Those skilled in the art should understand that these modules can be implemented in software, hardware, or a combination of both, for example, by a processor executing a computer program stored in memory to achieve the functions of each module.

[0113] To better illustrate the present invention, numerous specific details have been provided in the detailed embodiments described above. Those skilled in the art should understand that the present invention can be practiced even without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of the present invention.

[0114] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-modal large model-based railway on-off tool identification method, characterized in that, Includes the following steps: Step S1: Obtain on-site tool image data for the work nodes before and after the work begins, and obtain a list of work tools for this operation; Step S2: Convert the list of work tools into text prompts, and input the text prompts and the on-site tool image data into the multimodal large model; Step S3: Using the text prompt words as prior guiding conditions, the multimodal large model performs pixel-level instance segmentation on the on-site tool image data and outputs the instance mask of each tool; Step S4: Obtain the actual tool types and quantities based on the instance mask, and compare them with the work tool list to obtain the inventory results; wherein, The multimodal large model is a SAM3 model finely tuned using low-rank adaptation technology, and it has the ability to extract and learn the inherent visual features of railway electrical equipment.

2. The railway on-off tool identification method based on a multi-modal large model according to claim 1, characterized in that, The SAM3, fine-tuned using low-rank adaptation technology, is constructed in the following manner: Freeze all the original weight matrices of the pre-trained SAM3 backbone network, so that the backbone network weights are in an untrainable state during fine-tuning. For each converter module in the backbone network, a trainable reduced-dimensional matrix and an increased-dimensional matrix are connected in parallel via bypass. During forward propagation, the output of the multimodal large model is composed of the original output of the backbone network and the incremental output of the low-rank adaptive bypass. During backpropagation, only the gradients of the reduced-dimensional matrix and the increased-dimensional matrix are updated until the performance index of the multimodal large model converges, thus completing the fine-tuning of the low-rank adaptation technique.

3. The railway track entry / exit tool identification method based on a multimodal large model according to claim 2, characterized in that: The fine-tuning training of SAM3 uses a joint loss function, which is a weighted combination of cross-entropy loss and dice coefficient loss.

4. The railway on-off tool identification method based on a multi-modal large model according to claim 2, characterized in that, During forward propagation, the output h of the multimodal large model satisfies: ； Where W is the frozen original weight matrix, A is the dimension reduction matrix, B is the dimension increase matrix, and x is the input vector.

5. The railway on-off tool identification method based on a multi-modal large model according to claim 3, characterized in that, The formula for calculating the joint loss function is as follows: ； Where CE is the cross-entropy loss function, DC is the dice coefficient loss function, ŷ is the predicted mask generated by the model, D(y) is the one-hot encoding result of the true mask, λ1 is the weight coefficient of the cross-entropy loss, and λ2 is the weight coefficient of the dice coefficient loss.

6. The railway track entry / exit tool identification method based on a multimodal large model according to claim 1, characterized in that: The text prompts are short English nouns corresponding to railway electrical tools, and each type of tool has a unique text prompt.

7. The railway track entry / exit tool identification method based on a multimodal large model according to claim 1, characterized in that, Step S3 specifically includes: Using text prompts as prior guidance, pixel-level instance segmentation and category determination are performed on target tools in the image to obtain initial instance segmentation results; The initial instance segmentation results are post-processed with non-maximum suppression to filter duplicate detection regions and output instance masks that are independent of each tool.

8. The railway track entry / exit tool identification method based on a multimodal large model according to claim 1, characterized in that, In step S1, the method for acquiring the on-site tool image data includes: Use camera equipment to capture raw video or still images containing on-site tools; The collected static images, after screening, are directly used as on-site tool image data; For the acquired raw video, frame extraction is performed on the raw video to extract key frame images as on-site tool image data.

9. The railway on-off tool identification method based on a multi-modal large model according to claim 1, characterized in that, In step S4, the actual tool types and quantities are obtained based on the instance mask, specifically including: The number of independent connected components in the instance mask corresponding to each type of tool is counted. The number of independent connected components is the actual number of occurrences of that type of tool. The actual occurrences of all categories are summed to obtain the actual types and quantities of tools.

10. A railway on-off tool identification device based on a multi-modal large model, characterized in that, include: The data acquisition module is used to acquire on-site tool image data of the work nodes before and after the work begins, and to obtain a list of work tools for this operation. A multimodal input module is used to convert the list of work tools into text prompts and input the text prompts and the on-site tool image data into a multimodal large model; The instance segmentation module has a built-in multimodal large model, which is used to perform pixel-level instance segmentation on the on-site tool image data with the text prompt words as prior guiding conditions, and output the instance mask of each tool. The statistical comparison module is used to obtain the actual tool types and quantities based on the instance mask, and compare them with the work tool list to obtain the inventory results; The multimodal large model is a SAM3 model finely tuned using low-rank adaptation technology, and it has the ability to extract and learn the inherent visual features of railway electrical equipment.

Images