Track foreign matter identification method and device

Abstract

The invention discloses a track foreign matter identification method and device, and the method comprises the steps: carrying out the processing of a collected track image through a target detection model, recognizing a foreign matter candidate region at a recall rate, and generating candidate box information; based on the candidate box information, cutting or highlighting the image, and forming an input pair of the image segment and a cue word in combination with a structured cue word which is constructed by scene metadata and contains coordinate and line information; analyzing the input pair based on an image-text multi-modal large model; the image-text multi-modal large model is used for introducing an external knowledge base on the basis of a retrieval enhancement generation technology, performing deep semantic understanding on each candidate region, and outputting structured results of whether foreign matters exist, foreign matter types, risk levels and disposal suggestions exist or not; and outputting the foreign matter category risk level and the corresponding disposal suggestion. The method is used for improving the efficiency and accuracy of track foreign matter recognition, risk assessment and treatment response.

Description

Technical Field

[0001] This invention relates to the field of railway operation and maintenance technology, and in particular to a method and device for identifying foreign objects on railway tracks. Background Technology

[0002] The long distances and complex, ever-changing conditions of railway lines place higher demands on the intelligent operation and maintenance of infrastructure. Broken parts or foreign objects are prone to occur along ballastless tracks, potentially being carried away by train airflow and causing accidents. Current detection methods mostly rely on images collected by specialized inspection vehicles for foreign object identification, but these methods have significant limitations: existing technologies primarily focus on locating the foreign object area, making it difficult to distinguish specific types of foreign objects, such as hard obstacles or lightweight floating objects, and failing to assess the hazard level. Furthermore, they are prone to misclassifying inherent facilities such as sleeper fasteners as foreign objects, leading to false alarms. Summary of the Invention

[0003] This invention provides a method for identifying foreign objects on orbits, thereby improving the efficiency and accuracy of foreign object identification, risk assessment, and response. The method includes:

[0004] The acquired track images are processed using a target detection model to identify foreign object candidate regions and generate candidate box information based on recall. The target detection model adopts an improved YOLOv11 architecture, which introduces a small target detection branch, a feature fusion mechanism, and a recall-first strategy to filter foreign object candidate regions and generate candidate box information.

[0005] Based on candidate box information, the image is cropped or highlighted and combined with scene metadata to construct structured prompts containing coordinate and line information, forming an input pair of image fragments and prompts;

[0006] The input pair is analyzed based on a large image-text multimodal model. The large image-text multimodal model is used to introduce an external knowledge base based on retrieval enhancement generation technology, perform deep semantic understanding on each candidate region, and output a structured result indicating whether it is a foreign object, the type of foreign object, the risk level, and the disposal suggestion. If the structured information determines that the input pair contains a foreign object, the foreign object category, risk level, and corresponding disposal suggestion are output.

[0007] This invention also provides a track foreign object identification device to improve the efficiency and accuracy of track foreign object identification, risk assessment, and response. The device includes:

[0008] The foreign object candidate region identification module is used to process the acquired track image using the target detection model, identify foreign object candidate regions with a high recall rate, and generate candidate box information. The target detection model adopts an improved YOLOv11 architecture, which introduces a small target detection branch, a feature fusion mechanism, and a recall priority strategy to filter foreign object candidate regions and generate candidate box information.

[0009] The input pair forming module is used to crop or highlight the image based on candidate box information and combine it with scene metadata to construct structured prompts containing coordinate and line information, forming input pairs of image fragments and prompts.

[0010] The structured results analysis module is used to analyze the input pair based on the image-text multimodal large model. The image-text multimodal large model is used to introduce an external knowledge base based on retrieval enhancement generation technology, perform deep semantic understanding on each candidate region, and output structured results such as whether it is a foreign object, the type of foreign object, the risk level, and disposal suggestions. If the structured information determines that the input pair contains a foreign object, the module outputs the foreign object category, risk level, and corresponding disposal suggestions.

[0011] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for identifying foreign objects on the track.

[0012] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for identifying foreign objects on the track.

[0013] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for identifying foreign objects on the track.

[0014] In this embodiment of the invention, a target detection model is used to process the acquired track images to identify foreign object candidate regions and generate candidate bounding boxes based on recall. The target detection model adopts an improved YOLOv11 architecture, which introduces a small target detection branch, a feature fusion mechanism, and a recall-first strategy to filter foreign object candidate regions and generate candidate bounding boxes. Based on the candidate bounding box information, the image is cropped or highlighted, and structured prompts containing coordinates and track information are constructed by combining scene metadata to form input pairs of image fragments and prompts. The input pairs are analyzed based on a large-scale image-text multimodal model. The large-scale image-text multimodal model is used to introduce an external knowledge base based on retrieval enhancement generation technology to perform deep semantic understanding on each candidate region and output a structured result indicating whether it is a foreign object, the foreign object category, the risk level, and the disposal suggestion. If the structured information determines that the input pair contains a foreign object, the foreign object category, risk level, and corresponding disposal suggestion are output. This invention employs an improved YOLOv11 model to achieve high recall in initial screening. Through a small target detection branch and feature fusion mechanism, it significantly enhances the ability to detect small-sized foreign objects, overcoming the missed detection problem of traditional methods under complex lighting and perspective changes. By combining a large-scale multimodal model of images and text with retrieval enhancement generation technology, and constructing input pairs containing panoramic and partially cropped images, the model can accurately distinguish real foreign objects from background facilities such as sleeper fasteners, reducing the false alarm rate for track foreign objects. Through the output of structured results, it achieves closed-loop management from detection to disposal, thereby improving the efficiency and accuracy of track foreign object identification, risk assessment, and disposal response. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0016] Figure 1 This is a flowchart illustrating a method for identifying foreign objects on a track according to an embodiment of the present invention;

[0017] Figure 2 This is a specific example diagram of a method for identifying foreign objects on a track according to an embodiment of the present invention;

[0018] Figure 3 This is a specific example diagram of a target detection network architecture in an embodiment of the present invention;

[0019] Figure 4 This is a specific example diagram of a large image and text model recognition and decision-making process in an embodiment of the present invention;

[0020] Figure 5 This is a schematic diagram of the structure of a foreign object identification device for tracks according to an embodiment of the present invention;

[0021] Figure 6 This is a schematic diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0023] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] In the description of this specification, the terms "comprising," "including," "having," and "containing" are open-ended terms, meaning that they include but are not limited to. The terms "an embodiment," "a specific embodiment," "some embodiments," and "for example," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example that are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. The order of steps involved in the various embodiments is used to illustrate the implementation of this application, and the order of steps is not limited and can be adjusted appropriately as needed.

[0025] The acquisition, storage, use, and processing of data in this application comply with relevant regulations. The information collected in this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation interfaces are provided for users to choose to authorize or refuse.

[0026] It should be noted that in the embodiments of this application, certain existing solutions in the industry, such as software, components, and models, may be mentioned. For example, some existing software tools, components, algorithm models, or solutions well-known in other technical fields may be cited. These should be considered exemplary, and their purpose is only to illustrate the feasibility of implementing the technical solution of this application. These mentions should be understood as typical examples, and their core purpose is to illustrate and verify the rationality and feasibility of implementing the technical solution proposed in this application. However, this does not mean that the applicant has already used or necessarily used the solution. Such citations do not imply that the applicant has actually adopted these existing solutions, or that it will necessarily adopt these methods in its technical implementation process in the future. In other words, these mentions are only illustrative in nature, helping to understand the connection and transcendence of the innovation points of this application with the prior art, and do not constitute an endorsement or reliance statement on a specific prior art product.

[0027] Railway lines are long, span vast distances, and are characterized by complex and ever-changing conditions, placing higher demands on the efficient and safe operation and maintenance of railway infrastructure. For the efficient and safe operation of high-speed railways, advanced technologies are needed for intelligent monitoring, analysis, early warning, and maintenance of railway infrastructure, improving the efficiency and quality of operation and maintenance while reducing costs and risks. In the actual operation of high-speed railways, broken parts or foreign objects are prone to appear near ballastless tracks. The strong airflow from high-speed trains can carry these foreign objects, causing collisions and potentially leading to structural damage. In actual inspections, specialized inspection vehicles continuously collect images of the tracks and their surroundings during operation. These track images can be used to identify foreign objects. Because the occurrence patterns of foreign objects on ballastless tracks are random and their numbers are small, the inspection work requires identifying non-inherent foreign objects from a large number of normal track images.

[0028] Currently, most methods for detecting foreign objects (FOOs) on high-speed railway ballastless tracks focus on locating the area where the FEO is located. Through image recognition and sensor monitoring, these methods can detect abnormal areas on the track to some extent. However, these methods have significant limitations: firstly, they cannot effectively distinguish the specific type of FEO, making it difficult to determine whether it is a rigid obstacle or a lightweight floating object, directly affecting the targeted nature of subsequent handling measures; secondly, they cannot assess the severity of the FEO, nor determine the potential hazard level to train operation, hindering the development of priority handling plans; and thirdly, existing technologies are prone to misidentifying inherent facilities in the track background (such as sleepers, fasteners, cable troughs, etc.) as FEOs, leading to numerous false alarms, increasing the workload of manual verification, and reducing the operational efficiency of the detection system.

[0029] Specifically, existing semantic segmentation-based approaches only output masks and outlines, making it difficult to provide specific categories, risk levels, and disposal recommendations for foreign objects, thus failing to support compliant and traceable business decisions. They rely on fixed outline thresholds, are sensitive to changes in viewing distance, scale, perspective, and lighting, and are prone to false positives and false negatives. Furthermore, they are insufficient for recalling small targets, low-contrast objects, and objects with metallic reflections. Pixel-level annotation is costly, has weak generalization across lines and devices, and is prone to over- or under-segmentation leading to distorted area determination. Inference overhead is significant, edge real-time performance is limited, there is a lack of false positive / false negative feedback and knowledge base update mechanisms, and interpretability is insufficient (lacking key evidence points and clause citations). Overall, they have shortcomings in accuracy, executability, maintainability, and engineering implementation.

[0030] Specifically, this invention aims to solve the key technical problems existing in the current foreign object detection technology for ballastless tracks in high-speed railways, as follows:

[0031] 1. Difficulty in distinguishing foreign object categories: Existing detection methods can only locate the foreign object area, but cannot accurately classify the detected foreign objects. It is difficult to distinguish different types of foreign objects such as stones, metal fragments, plastic waste, and track detachment parts, resulting in a lack of targeted subsequent disposal measures and affecting the efficiency of track safety assurance.

[0032] 2. Lack of foreign object severity assessment: Current technology cannot scientifically assess the potential hazard level of detected foreign objects, cannot determine whether foreign objects pose an emergency or threat to train operation, is not conducive to the formulation of priority handling plans, and may lead to key hidden dangers not being dealt with in a timely manner.

[0033] 3. Misjudgment of inherent facilities in the background: Existing detection methods are prone to misidentifying inherent facilities in the track background (such as sleepers, fasteners, cable troughs, etc.) as foreign objects, generating a large number of false alarms, increasing the workload of manual verification, reducing the overall operating efficiency of the detection system, and interfering with the effective identification and handling of real foreign objects;

[0034] 4. Insufficient in-depth analysis of detection results: Existing technologies lack the ability to deeply understand and analyze the semantics of detection results, and cannot transform positioning information into classification and risk assessment results with clear guiding significance, making it difficult to meet the needs of high-speed railway ballastless track safety monitoring for precise and intelligent decision support.

[0035] To address the aforementioned shortcomings, a technical solution capable of accurately classifying and identifying foreign objects is urgently needed. Based on this, and considering the powerful capabilities of image-text big data models in image content understanding, category differentiation, and semantic analysis, this paper proposes inputting the results obtained from previous detections as prompt words into the image-text big data model. Leveraging its deep analysis capabilities of image information, the model can accurately distinguish the categories of foreign objects, scientifically assess their severity, and effectively differentiate them from existing background facilities. Ultimately, this achieves accurate classification of foreign objects on ballastless tracks, providing more reliable technical support for the safe operation of high-speed railways.

[0036] To address the aforementioned issues, this invention provides a method for identifying foreign objects on railway tracks. By organically combining a high-efficiency target detection network with a large-scale image-text multimodal model, it achieves high recall, low false alarm detection, and the generation of actionable disposal suggestions for foreign objects in railway scenes. The method employs a cascaded process of "initial target detection screening + precise discrimination by the large-scale image-text model + result processing and feedback": first, the target detection network quickly locates suspected foreign object areas across the entire image; then, structured information such as coordinates, along with corresponding image fragments, is input into the large-scale image-text model for fine-grained recognition and semantic understanding, outputting the category and disposal suggestions; simultaneously, false alarms are suppressed, and the results are fed back for continuous optimization of model performance. This method can be applied to high-speed rail, conventional railway, and subway sections. This method aims to improve the efficiency and accuracy of foreign object identification, risk assessment, and disposal response on railway tracks. See [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart illustrating a method for identifying foreign objects on a track according to an embodiment of the present invention. The method may include:

[0037] Step 101: The acquired track image is processed using an object detection model to identify foreign object candidate regions and generate candidate box information based on recall rate; the object detection model adopts an improved YOLOv11 architecture, which introduces a small object detection branch, feature fusion mechanism and recall priority strategy to screen foreign object candidate regions and generate candidate box information.

[0038] Step 102: Based on the candidate box information, crop or highlight the image and combine it with the scene metadata to construct a structured prompt word containing coordinate and line information, forming an input pair of image fragments and prompt words;

[0039] Step 103: Analyze the input pair based on the image-text multimodal large model; the image-text multimodal large model is used to introduce an external knowledge base based on retrieval enhancement generation technology, perform deep semantic understanding on each candidate region, and output a structured result of whether it is a foreign object, the category of the foreign object, the risk level, and the disposal suggestion; if the structured information determines that the input pair contains a foreign object, then the foreign object category, risk level, and corresponding disposal suggestion are output.

[0040] In this embodiment of the invention, a target detection model is used to process the acquired track images to identify foreign object candidate regions and generate candidate bounding boxes based on recall. The target detection model adopts an improved YOLOv11 architecture, which introduces a small target detection branch, a feature fusion mechanism, and a recall-first strategy to filter foreign object candidate regions and generate candidate bounding boxes. Based on the candidate bounding box information, the image is cropped or highlighted, and structured prompts containing coordinates and track information are constructed by combining scene metadata to form input pairs of image fragments and prompts. The input pairs are analyzed based on a large-scale image-text multimodal model. The large-scale image-text multimodal model is used to introduce an external knowledge base based on retrieval enhancement generation technology to perform deep semantic understanding on each candidate region and output a structured result indicating whether it is a foreign object, the foreign object category, the risk level, and the disposal suggestion. If the structured information determines that the input pair contains a foreign object, the foreign object category, risk level, and corresponding disposal suggestion are output. This invention employs an improved YOLOv11 model to achieve high recall in initial screening. Through a small target detection branch and feature fusion mechanism, it significantly enhances the ability to detect small-sized foreign objects, overcoming the missed detection problem of traditional methods under complex lighting and perspective changes. By combining a large-scale multimodal model of images and text with retrieval enhancement generation technology, and constructing input pairs containing panoramic and partially cropped images, the model can accurately distinguish real foreign objects from background facilities such as sleeper fasteners, reducing the false alarm rate for track foreign objects. Through the output of structured results, it achieves closed-loop management from detection to disposal, thereby improving the efficiency and accuracy of track foreign object identification, risk assessment, and disposal response.

[0041] In specific implementation, the first step is to process the acquired track image using a target detection model to identify foreign object candidate regions and generate candidate box information based on recall rate. The target detection model adopts an improved YOLOv11 architecture, which introduces a small target detection branch, feature fusion mechanism and recall priority strategy to screen foreign object candidate regions and generate candidate box information.

[0042] In one embodiment, when processing the acquired track images using an object detection model, the input image is first preprocessed by maintaining its aspect ratio, filling edges, and normalizing to standardize the image input and improve model stability. The preprocessed image input is an improved YOLOv11 architecture backbone network. This network expands the receptive field by stacking specific modules and introducing locally large convolutional kernels in key layers, while enabling deformable convolutions to enhance robustness to perspective changes and non-rigid deformations. The network retains shallow features for small object detection and adds a spatial pyramid pooling module at the top to aggregate global contextual information and reduce interference from repetitive structures such as sleeper fasteners.

[0043] The multi-layer features output from the backbone network are weighted and converged by a neck feature fusion module. This module combines a path aggregation network, a bidirectional feature pyramid, and adaptive feature fusion, explicitly introducing shallow branches and fusing them with deep features across layers. This strengthens bottom-up detail reflow and top-down semantic transmission, thereby improving the separability of foreign objects of different sizes in dense scenes. The fused features are fed into an anchorless and decoupled detection head, which uses a three-branch structure to handle category judgment, target presence confidence, and bounding box regression. During training, a dynamic positive sample allocation strategy is used to adjust the weights of the loss function to prioritize recall, and diverse data augmentation and hard example learning are used to improve the model's generalization ability.

[0044] During the inference phase, a recall-first post-processing strategy is adopted, including setting a lower confidence threshold and a higher intersection-union ratio (IU) threshold, and applying soft or clustering-based nonmaximum suppression to reduce the missed detection of overlapping candidates. At the same time, test-time enhancement techniques such as multi-scale and mirror flipping are combined to further optimize the detection effect of small targets and low-contrast foreign objects. Finally, candidate box information with high recall rate and its associated confidence and hierarchical data are output.

[0045] In one embodiment, the target detection model is based on the YOLOv11 architecture, outputting multi-layer features through the backbone network, and then weighting and aggregating information at different scales via neck feature fusion, which is then fed into an anchorless decoupled detection head to generate candidate boxes; candidate box information is output by filtering under low confidence thresholds and high crossover ratio thresholds using soft or clustering variants of non-maximum suppression.

[0046] In the above embodiments, the target detection model is based on the YOLOv11 architecture to detect foreign object regions. Specifically, it includes preprocessing operations such as aspect ratio preservation, edge filling, and normalization of the input track image to standardize the image input and improve model stability. The preprocessed image input is used to improve the backbone network. This network expands the receptive field by stacking specific modules and introducing large convolutional kernels in key layers, while enabling deformable convolution to enhance the adaptability to perspective changes and deformations. The network retains shallow features for small target detection and uses spatial pyramid pooling to aggregate global contextual information to reduce interference from repetitive structures such as sleeper fasteners.

[0047] The multi-layer features output from the backbone network are weighted and converged by the neck feature fusion module. This fusion combines a path aggregation network, a bidirectional feature pyramid, and adaptive feature fusion, explicitly introducing shallow branches and performing cross-layer fusion with features of different scales to enhance detail reflow and semantic transmission, thereby improving the separability of multi-sized foreign objects in complex scenes. The fused features are then fed into an anchorless decoupled detection head, which handles category judgment, target presence confidence, and bounding box regression through independent branches. During training, a dynamic positive sample allocation strategy is used to adjust the sample matching range, and a dedicated loss function weight is configured to prioritize recall performance.

[0048] During the inference phase, a recall-first post-processing strategy is adopted, including setting a lower confidence threshold and a higher intersection-union ratio (IU) threshold, and applying soft or clustering-based nonmaximum suppression to reduce the missed detection of overlapping targets. At the same time, the detection results of multi-scale and mirror flip detection are integrated with test-time enhancement techniques, and finally, candidate box information with high recall rate and its associated confidence and feature level data are output.

[0049] In specific implementation, after step 101: using the target detection model to process the acquired track image, identifying the candidate region of foreign objects with recall rate and generating candidate box information, step 102: based on the candidate box information, cropping or highlighting the image and combining it with the structured prompt words containing coordinates and line information constructed by scene metadata, forming an input pair of image fragments and prompt words.

[0050] In this embodiment, based on candidate box information, the image is cropped or highlighted, and structured prompts containing coordinate and line information are constructed using scene metadata to form input pairs of image fragments and prompts, including:

[0051] A panoramic view of the orbital image and a cropped image obtained by cropping each suspected foreign object region are acquired; wherein the cropped image is based on a preset ratio of expansion of the candidate box to preserve the context, and the suspected foreign object region is highlighted on the panoramic image;

[0052] Structured prompts are constructed based on scene metadata; the scene metadata includes line name, section, time, weather, lighting conditions, camera height, pitch angle, track type, and acquisition speed; the structured prompts include the coordinate information of candidate boxes, area ratio, aspect ratio, scene metadata, and candidate attributes; the candidate attributes include candidate number, source level, initial screening confidence, neighborhood, and overlapping group number;

[0053] Combine the cropped image or the highlighted cropped image with structured prompts to form an input pair.

[0054] In the above embodiments, when constructing input pairs based on candidate box information, a panoramic view of the track image is first acquired, and a corresponding cropped image is generated for each candidate region. The cropped image is appropriately expanded outward based on the candidate box to retain surrounding contextual information, and the candidate region positions are highlighted on the panoramic image to establish a visual connection between the global and local contexts.

[0055] Subsequently, structured prompts are constructed based on scene metadata, including geometric features such as pixel coordinates, area ratio, and aspect ratio of candidate boxes, as well as environmental information such as line name, section location, acquisition time, weather and lighting conditions, camera parameters, and track type. Each candidate region also includes attribute data such as source level, initial screening confidence level, neighborhood relationship, and overlapping group number.

[0056] During data organization, highly overlapping candidate regions are grouped and merged to reduce redundant judgments. Image data can be processed using path referencing or inline encoding, and the cropping image size is optimized based on actual bandwidth conditions. Finally, the processed image data is combined with structured prompts to form a uniformly formatted input pair.

[0057] This input pair construction method, through a combination of panoramic and local visual presentations and detailed scene context descriptions, provides complete input information for the accurate discrimination of subsequent large-scale multimodal models. The rich metadata contained in the structured prompts effectively enhances the model's ability to perceive scale, adapt to the environment, and understand the scene.

[0058] In specific implementation, after step 102: based on candidate box information, cropping or highlighting the image and constructing structured prompts containing coordinates and line information in conjunction with scene metadata to form input pairs of image fragments and prompts, step 103: analyzing the input pairs based on a large-scale image-text multimodal model; the large-scale image-text multimodal model is used to introduce an external knowledge base based on retrieval enhancement generation technology, perform deep semantic understanding on each candidate region, and output structured results of whether it is a foreign object, the type of foreign object, the risk level, and disposal suggestions; if the structured information determines that the input pair contains a foreign object, the foreign object category, risk level, and corresponding disposal suggestions are output.

[0059] In this embodiment, when analyzing the input pairs based on a large-scale image-text multimodal model, a specific architecture of the large-scale image-text multimodal model is selected as the core for recognition and decision-making. This model is pre-tuned in supervised manner using railway inspection images and corresponding labeled data, and a parameter optimization strategy is adopted to achieve domain adaptation, focusing on enhancing the model's semantic understanding ability of the track scene. During the model fine-tuning process, cross-modal interaction effects are optimized through instruction alignment and preference alignment, while an efficient parameter fine-tuning method is used to maintain the integrity of the original knowledge system.

[0060] The inference process integrates a retrieval enhancement generation mechanism. By establishing a knowledge base containing railway maintenance rules, work instructions, and risk classification standards, rapid retrieval of regulations and clauses is achieved based on text vector indexing technology. Based on the attribute characteristics, location information, and context of candidate regions, the most relevant regulations and clauses are matched from the knowledge base, and their summaries and numbers are injected into the model input prompts as auxiliary decision-making criteria.

[0061] After receiving complete input including a panoramic view, cropped candidate region images, structured cue words, and regulatory context, the model performs deep semantic parsing on each candidate region. First, it determines whether the region is a foreign object. If confirmed, it further identifies the specific category, including common types such as stones, metal fragments, and plastic waste. Simultaneously, it assesses the risk level and generates corresponding disposal recommendations. The entire process outputs a rigorously structured result, including the foreign object determination conclusion, classification result, confidence level assessment, risk level classification, disposal measure recommendations, and the cited regulatory clause number.

[0062] For false alarms identifying a facility as a background facility, the model will output the corresponding facility category identifier and explain the reason for the judgment. All output results are presented in the form of a structured array with uniform Chinese key names, ensuring that subsequent systems can directly parse and use them. A continuous optimization mechanism is also established, incrementally fine-tuning the model by re-feeding back false alarm samples and regularly updating the knowledge base to maintain the system's adaptability and accuracy.

[0063] In one embodiment, the large-scale image-text multimodal model is constructed as follows:

[0064] A basic image-text multimodal large model was selected, and supervised fine-tuning was performed based on railway inspection images and corresponding structured label data; the fine-tuning process included optimizing the model through instruction alignment and preference alignment.

[0065] A parameter-efficient fine-tuning strategy is adopted to adapt the model; a railway maintenance rule knowledge base is established, which includes handling procedures and risk classification rules; based on text vector indexing and re-ranking technology, the attribute information of candidate regions is input into the model.

[0066] The model is deployed to receive inputs including images, candidate region information, and regulatory context, and outputs structured results including foreign object category, risk level, and disposal recommendations.

[0067] In the above embodiments, the construction of the large-scale image-text multimodal model is first based on the selected basic model architecture. Supervised fine-tuning is performed using image data specific to railway inspection scenarios and their corresponding structured labels to improve the model's adaptability to specific tasks in the track environment. The fine-tuning process focuses on optimizing the model's interactive performance through instruction alignment and preference alignment techniques to ensure that it can accurately understand the needs of the inspection task. At the same time, efficient parameter fine-tuning strategies are adopted, such as freezing some underlying network parameters or introducing adapter modules, to efficiently adjust the decision head and cross-modal alignment capabilities while maintaining the model's general knowledge, avoiding overfitting or catastrophic forgetting problems.

[0068] During the model building phase, a railway maintenance rules knowledge base is established concurrently. This knowledge base system integrates structured information such as standard operating procedures, emergency response processes, risk classification standards, and key points of historical cases, forming a searchable external knowledge source. An indexing system is built based on text vectorization technology, and combined with a re-ranking algorithm to achieve efficient retrieval. The retrieval mechanism dynamically matches the most relevant regulations from the knowledge base based on the attribute characteristics of candidate regions, such as suspected categories, spatial locations, and environmental conditions. The summary content and clause numbers are injected as contextual information into the model input prompts, enhancing the traceability of decision-making basis.

[0069] After fine-tuning and integration with the retrieval system, the model enters the deployment phase. Its input interface is designed to receive multimodal data packets containing panoramic images, candidate region cropping maps, coordinate information, scene metadata, and regulatory context. Internally, the model performs deep semantic parsing, identifying foreign objects, classifying them, assessing risks, and generating disposal suggestions for each candidate region. The output strictly adheres to a pre-defined structured format, including key fields such as foreign object presence determination, specific category attribution, risk level classification, disposal measure suggestions, and cited regulatory number, ensuring the output can be directly parsed and utilized by downstream systems. The entire construction process emphasizes the synergy of domain adaptation and knowledge injection, continuously re-feeding back misjudged samples for incremental optimization to maintain the model's accuracy and robustness in complex inspection environments.

[0070] In one embodiment, if the structured information determines that the input pair contains foreign objects, the output includes the foreign object category risk level and corresponding handling recommendations, including:

[0071] If the input pair is determined to contain foreign objects through structured information, a structured result array is output. This array includes a judgment on whether the object is foreign, the object category, confidence level, key reasons, risk level, disposal recommendations, recommended completion deadline, and cited regulation number. The object category is selected from a preset set of categories, including stones, metal fragments, plastic waste, rubber blocks, wood fragments, or detached track components. The risk level is divided into high, medium, and low levels. The disposal recommendations are determined based on the risk level and include immediate shutdown and cleanup, arranging inspection and handling, or periodic observation. The output format is a structured array with Chinese characters for the key names.

[0072] In the above embodiments, when the presence of a foreign object in the input pair is confirmed through structured information, the multimodal large model outputs a structured result array. This array comprehensively contains key information fields for foreign object identification, such as the judgment of whether it is a foreign object, the specific foreign object category, confidence level assessment, a list of reasons, risk level classification, suggested disposal measures, suggested completion deadline, and the cited regulatory clause number. The foreign object category is selected from a preset set, covering common types such as stones, metal fragments, plastic waste, rubber blocks, wood fragments, and track detachment parts. The risk level is divided into three levels: high, medium, and low. The disposal suggestions are dynamically matched according to the risk level. For example, in high-risk situations, it is recommended to immediately stop the machine for cleaning; in medium-risk situations, it is recommended to arrange inspection and handling; and in low-risk situations, it is recommended to observe regularly. The entire output strictly adopts a structured array format with Chinese characters as keys, ensuring that the field names are clear and easy for subsequent system parsing and integration, thereby supporting automated decision-making processes.

[0073] Two specific embodiments are given below to illustrate the specific application of the method of the present invention.

[0074] First specific embodiment:

[0075] like Figure 2 As shown, Figure 2 This is a specific example diagram of a method for identifying foreign objects on a track according to an embodiment of the present invention. This embodiment may include the following steps:

[0076] 1) Initial screening of target detection: The collected images of ballastless track of high-speed railway are input into the target detection network, which automatically identifies suspected foreign object areas and generates corresponding coordinate boxes, confidence scores and other information to form a candidate set.

[0077] 2) Prompt words and data organization: Based on the candidate boxes, the image is cropped or highlighted, and prompt words containing location coordinates, size quantization, and scene elements (such as sleepers, fasteners, cable troughs, etc.) are constructed to form an input pair of "image fragment + structured prompt".

[0078] 3) Large Image and Text Model Judgment: The above input pairs are fed into the large image and text model; if it is determined to be a foreign object, the specific category (such as stones, metal fragments, plastic waste, track detachment parts, etc.), size / location and other attributes are output, and disposal suggestions are given in combination with operational risks (such as "immediately stop the machine for cleaning", "arrange inspection and handling", "regular review and observation", etc.); if it is determined to be a false alarm (inherent facilities in the track background), it is filtered in the process.

[0079] 4) Result processing and system linkage: Generate event records, alarms and task assignments for "foreign object" results, and interface with the scheduling / inspection system; suppress "false alarm" results to avoid interference from false alarms.

[0080] 5) Feedback and continuous optimization: The annotation results and closed-loop data of the processing are fed back to the training / fine-tuning and threshold strategy optimization modules to continuously reduce false positives and false negatives and improve the overall detection rate and reliability.

[0081] Figure 2 The process described begins with image acquisition and input. After the inspection image is input, the target detection network performs initial screening, outputting the region, coordinates, and confidence level of suspected foreign objects. This is followed by the prompt word and data organization stage, where candidate regions are cropped or highlighted, and their coordinates, dimensions, and other scene information are assembled into structured prompts. The core processing unit is the image big model, which receives "image fragments + structured prompts" as input and performs accurate discrimination. Its core task is to determine "whether it is a foreign object" and identify its category (e.g., stones, metal, etc.). After discrimination, the process branches: if confirmed as a foreign object, attribute extraction is performed and disposal suggestions are generated. The results are stored in the event database and triggered for recording and alarm / task issuance, ultimately linking with downstream systems such as the scheduling platform. If determined to be a false alarm (e.g., sleepers, fasteners, etc.), it is filtered out to suppress false alarms. Finally, the system forms a continuous optimization closed loop: foreign object data, difficult cases, and false alarm samples generated in actual operation can be used as annotation feedback for fine-tuning the detection and recognition models and updating strategies, achieving system self-iteration and performance improvement.

[0082] Specifically, firstly, a high-efficiency target detection network rapidly screens the track images, locating all suspected foreign object regions and generating a candidate box set to ensure high recall. This step provides high-quality candidate targets for subsequent processing. Next, each candidate region undergoes refined data organization, combining image cropped segments with structured prompts containing coordinates and scene information into input pairs, providing complete context for subsequent in-depth analysis. The key discrimination step is completed by a large-scale image-text multimodal model. This model performs fine-grained identification of each candidate target, accurately distinguishing real foreign objects from track background facilities, and outputting specific classifications, risk levels, and actionable handling suggestions for confirmed foreign objects. The discrimination results are translated into practical action, generating alarms and coordinating dispatch for real foreign objects, while effectively filtering false alarms, significantly reducing false alarm interference. Finally, a continuous learning mechanism continuously optimizes performance, feeding back on-site handling results to the training module, forming a virtuous cycle of self-improvement. This solution, through the organic synergy of multiple technologies, significantly improves recognition accuracy and reliability while ensuring detection efficiency.

[0083] The following is a detailed description of this embodiment:

[0084] I. Foreign Object Region Target Detection Network Based on Improved YOLOv11:

[0085] Figure 3 This is a specific example diagram of a target detection network architecture in an embodiment of the present invention, such as... Figure 3 As shown, the object detection network architecture begins with the preprocessing of the input image, including aspect ratio preservation, edge padding, normalization, and data augmentation for complex scenes. Subsequently, the image enters an improved YOLOv11 backbone network for feature extraction. This network enhances feature representation by stacking specific modules and introducing locally large convolutions and deformable convolutions, outputting multi-scale feature maps from P2 to P5. The core component is the neck feature fusion module, which employs a path aggregation network, a bidirectional feature pyramid, and adaptive feature fusion techniques. It explicitly introduces shallow P2 features for small object detection and performs cross-layer weighted fusion with deep features to improve the detection capability of multi-scale objects. The fused features are then fed into a decoupled detection head for classification, confidence evaluation, and bounding box regression. The entire model employs a "recall-first" strategy for training and inference: during training, recall is optimized through dynamic sample matching, adjustment of the loss function, and the use of diverse data augmentation; during inference, a low confidence threshold, a specific non-maximum suppression strategy, and test-time augmentation techniques are used to ultimately output candidate boxes with high recall, providing candidate targets for subsequent fine-grained discrimination.

[0086] Figure 3In this paper, the network is designed for "high-recall initial screening proposals". The process is as follows: Input a ballastless track scene image (long side approximately 1280). After preprocessing, including aspect ratio preservation and edge filling, normalization and color perturbation, and aspect ratio grouping, it is fed into the YOLOv11 backbone network (C2f and C3 modules stacked, key layers introduce local large convolutions and deformable convolutions [second version, optional]). The network outputs P2 (stride 4) / P3 (stride 8) / P4 (stride 16) / P5 (stride 32) by layer. Then, global context aggregation is performed through spatial pyramid fast pooling or dilated spatial pyramid pooling. At the neck, a path aggregation network, bidirectional feature pyramid, and adaptive feature fusion weighting are used to explicitly introduce P2 for small targets, and P2 / P3 / P4 / P5 are weighted. Cross-layer fusion; decoupled detection heads are connected at each scale, outputting results in three branches: "classification / target presence confidence / regression," with an anchorless design and dynamic positive sample allocation; recall is prioritized during the inference phase, using a confidence threshold of approximately 0.1, an intersection-over-union (IoU) threshold of approximately 0.65, and a retention count of approximately 1000 per image, employing soft nonmaximum suppression or clustering nonmaximum suppression, while enabling test-time enhancement (multi-scale [0.8, 1.0, 1.3] and flipping); finally, candidate boxes are output, including coordinates, scale, and target presence confidence. Provides high-precision judgment for subsequent large-scale image and text models; the training side introduces "distillation / positive sample allocation", adopts dynamic matching and adaptive strategies and lowers the cross-union ratio threshold. The loss is composed of focus loss (exponential about 1.0) and label smoothing (about 0.05) combined to improve the cross-union ratio / complete cross-union ratio localization loss. It is combined with mosaic stitching, mixed overlay, copy and paste, blur / noise / compression / occlusion enhancement, as well as multi-scale (960–1536), automatic mixing accuracy and exponential moving average to ensure high recall of small targets and low-contrast foreign objects under complex working conditions.

[0087] The network in this embodiment is designed for the scenario of "high-recall initial screening proposals". The overall process is as follows: the input image is first preprocessed with aspect ratio preservation, edge filling and normalization, and then enters the improved YOLOv11 backbone network to output multi-layer features; the information at different scales is weighted and aggregated by the neck feature fusion module and sent to the anchorless and decoupled detection head to generate candidate boxes; finally, the soft or clustering variant of non-maximum suppression is used to filter at a low confidence threshold and a high crossover ratio threshold to output high-recall candidate boxes, which are then handed over to the image-text multimodal model for accurate discrimination.

[0088] The backbone network uses stacked C2f and C3 modules, and introduces locally large convolutions in key layers to expand the receptive field and enhance the visibility of low-contrast textures. Deformable convolutions are also enabled at appropriate locations to make the network more robust to perspective changes, bending, and non-rigid deformations. Shallow P2 features with a stride of 4 are retained for small object detection, and spatial pyramid fast pooling or hollow spatial pyramid pooling is added at the top to converge global context, thereby reducing confusion caused by repetitive structures such as sleepers and fasteners.

[0089] The neck feature fusion method combines a path aggregation network, a bidirectional feature pyramid, and adaptive feature fusion, explicitly introducing the P2 branch for cross-layer weighted fusion with P3, P4, and P5. This design enhances bottom-up detail reflow and top-down semantic delivery, improves the separability of foreign objects in different sizes and dense scenes, and reduces mutual masking between adjacent candidates.

[0090] The detection head employs an anchorless, decoupled three-branch structure, responsible for class determination, target presence confidence, and bounding box regression, respectively. During training, dynamic positive sample allocation is used to increase the number of matches and the center radius, while lowering the cross-union threshold for positive and negative sample determination, thereby allowing more "suspicious regions" to enter the candidate set. The classification branch uses focal loss combined with label smoothing, while the regression branch uses improved or full cross-union loss. At the same time, the weight of the target presence confidence branch is appropriately increased to ensure priority recall.

[0091] The training strategy is geared towards complex operating conditions and stable convergence: it employs diverse enhancements at the geometric, lighting, and quality levels, using techniques such as mosaic stitching, blending overlay, copy-paste, and perturbations including blurring, noise, compression, and occlusion; it enables random multi-scale and aspect ratio grouping to adapt to different cameras and mounting postures; it resamples false alarms and missed detections to emphasize learning from difficult examples; and it combines appropriate optimization and learning rate scheduling, automatic mixing of accuracy and exponential moving average, and, when necessary, introduces teacher model distillation to stabilize the confidence level of target existence.

[0092] The inference and post-processing adopt a recall-first strategy: setting the confidence threshold at a low level, setting the intersection-union ratio (IUGR) threshold for non-maximum suppression at a higher level, and employing soft or clustering-based non-maximum suppression to reduce false negatives caused by mutual inhibition, while increasing the number of candidates retained per image; based on this, enhanced multi-scale and mirror flipping are used during testing, and multi-view results are merged to further improve the detection of small and low-contrast targets. The final output candidate box coordinates, scale, and target existence confidence, along with hierarchical information, is constructed as an "image fragment + structured cue words," which is then fed into a large-scale image-text model for fine-grained category determination and processing suggestions, achieving a system-level balance between high recall and high precision.

[0093] In the above embodiments, the training process is optimized for complex working conditions, including diverse data augmentation at the geometric, lighting, and quality levels. It utilizes techniques such as mosaic stitching, blending overlay, copy-paste, and perturbations including blurring, noise, compression, and occlusion. Random multi-scale and aspect ratio grouping is enabled to adapt to different camera parameters, and false positives and false negatives are resampled to enhance hard example learning. Training combines optimizer scheduling, automatic mixed precision, and exponential moving average strategies, and, when necessary, introduces teacher model distillation to stabilize the confidence output. The inference phase employs a recall-first post-processing strategy, setting a lower confidence threshold and a higher intersection-union (IU) threshold. Soft or clustering-based non-maximum suppression is applied to reduce false negatives of overlapping candidates. Simultaneously, test-time augmentation techniques such as multi-scale and mirror flipping are used to merge detection results. Finally, the coordinates, scale, confidence, and hierarchy information of the candidate boxes are output, providing high-recall input data for subsequent large-scale image and text model processing.

[0094] II. Prompt words and data organization:

[0095] 1. Data organization (input packaging method for the text-image multimodal model):

[0096] Full and Local Views: Provides a panoramic view and a cropped view of each candidate region; the cropped view expands outward by 5%–10% from the original bounding box to preserve context, while the candidate region is highlighted in a semi-transparent manner on the panoramic view, which facilitates the model to establish global-local relationships.

[0097] Coordinates and Units: Image pixel coordinates are used uniformly, with the top left corner as the origin; the rectangle is represented by "top left corner x, top left corner y, width w, height h"; the area in pixels, the percentage of the whole image, and the aspect ratio are also given to make the model more sensitive to scale.

[0098] Candidate grouping: Candidates with high overlap (crossover ratio greater than 0.7) are grouped into the same group and provided with a group number to reduce duplicate determination of the same target.

[0099] Scene and camera information: Providing information such as track name, section, time, weather, lighting conditions, camera height and pitch angle, track type (ballastless track), and acquisition speed can significantly improve the model's ability to distinguish reflections, shadows, and repetitive structures.

[0100] Candidate attributes: Provide each candidate with source level (e.g., from P2 / P3 / P4 / P5), initial screening confidence, prior information on whether it is located in the neighborhood of fasteners / sleepers / cable troughs, and whether it contains boundary truncation, etc.

[0101] Image encoding: Images can be encoded using local paths or inline image data (such as a string representation of the image data); if bandwidth is limited, it is recommended to control the longer side of the cropped image to be in the range of 640–960 pixels.

[0102] Example input data:

[0103] {

[0104] "Panorama": "path_or_image_data_full",

[0105] Panoramic image resolution: {"Width": 1920, "Height": 1080}

[0106] Scene Information: {

[0107] "Line Name": "XX High-Speed ​​Railway",

[0108] Section: K120+300 ~ K120+500

[0109] "Shooting Time": "2025-08-18 14:32",

[0110] Weather: Sunny

[0111] "Light": "Strong light"

[0112] Camera height in meters: 1.6

[0113] Camera tilt: -10,

[0114] Track type: Ballastless track

[0115] },

[0116] "Candidate Region List": [

[0117] {

[0118] "Candidate ID": "c1",

[0119] "Source level": "P2",

[0120] Initial screening confidence level: 0.68

[0121] "Rectangle": {"x": 820, "y": 460, "w": 64, "h": 52},

[0122] "Outer Rectangle": {"x": 800, "y": 445, "w": 100, "h": 82},

[0123] "Area in pixels": 3328,

[0124] "Area percentage": 0.0016,

[0125] Aspect Ratio: 1.23

[0126] "Location": "Near the fastener"

[0127] "Overlapping group number": "g1",

[0128] "Cropped image": "path_or_image_data_crop_c1"

[0129] },

[0130] {

[0131] "Candidate ID": "c2",

[0132] "Source level": "P3",

[0133] Initial screening confidence level: 0.54

[0134] "Rectangle": {"x": 420, "y": 720, "w": 140, "h": 90},

[0135] "Outer Rectangle": {"x": 400, "y": 700, "w": 180, "h": 120},

[0136] "Area in pixels": 12600

[0137] "Area percentage": 0.0061,

[0138] Aspect Ratio: 1.56

[0139] "Location": "Edge of sleepers"

[0140] "Overlapping group number": "g2",

[0141] "Cropped image": "path_or_image_data_crop_c2"

[0142] } ]

[0144] }

[0145] In the above embodiments, the data organization process begins with the preparation and processing of image data. First, a panoramic view of the track image and a cropped image of each candidate region are provided. The cropped image expands outwards by a certain proportion from the original candidate bounding box to retain surrounding contextual information. Simultaneously, candidate regions are highlighted on the panoramic image, establishing a visual connection between the global and local regions, facilitating comprehensive analysis by the model. The coordinate system uses a unified image pixel coordinate representation, with the top-left corner as the origin. The bounding box is defined using the top-left corner's horizontal and vertical coordinates combined with its width and height. Scale-related parameters such as area pixel values, overall image proportion, and aspect ratio are calculated simultaneously to enhance the model's ability to perceive target size.

[0146] In the candidate region management phase, highly overlapping candidates are grouped. Similar regions are merged into the same group and assigned a group number based on the intersection-union ratio (IU) threshold, effectively reducing duplicate judgments of the same target and improving processing efficiency. The integrated scene information includes metadata such as line name, specific section, acquisition time, weather conditions, lighting conditions, camera altitude and pitch angle, track type, and acquisition speed. This contextual information significantly improves the model's accuracy in distinguishing reflections, shadows, and repetitive structures.

[0147] Each candidate region comes with a detailed attribute description, including its source level, initial screening confidence level, neighborhood relationships such as proximity to fasteners or sleepers, and prior information such as boundary truncation status. Image data encoding supports local path referencing or inline data formats, and the cropped image size is optimized based on actual bandwidth conditions to ensure that longer sides are within a reasonable pixel range. The final input data is organized in a structured format, covering panoramic primitive information, scene context, and a list of candidate regions, forming a standardized input for the image-text multimodal model parsing.

[0148] The entire data organization process ensures the integrity and consistency of input data through multi-level information integration and standardization, providing a reliable foundation for the accurate judgment of subsequent models.

[0149] 2. Prompt word template (used to guide the tasks and outputs of the text-image multimodal model):

[0150] Role and Objectives: The model is clearly defined as a "safety assistant for railway ballastless track inspection", emphasizing "first determining whether it is a foreign object, then providing classification and disposal suggestions", and the output is fixed as a structured result.

[0151] Task breakdown: Make an independent judgment for each candidate area; if it is classified as a false alarm, indicate "which type of background facility it belongs to"; if it is classified as a foreign object, provide "category, size details, risk level, disposal recommendations and reasons".

[0152] Categories and handling recommendations: Provide a clear set of categories and handling rules to avoid generalized descriptions.

[0153] Example:

[0154] Your role: You are a safety assistant for inspecting ballastless railway tracks. You will see a panoramic view and cropped views of several candidate areas. Your task is to determine whether each candidate area contains a foreign object; if it does, provide the specific category, risk level, and handling recommendations; if it does not contain a foreign object, provide the corresponding background facility category and mark it as a false alarm. Please only output structured results that conform to the specifications, without adding unnecessary text descriptions.

[0155] Scene and camera information:

[0156] Route and Section: [Route Name], [Section];

[0157] Time and Environment: [Shooting Time], [Weather], [Lighting];

[0158] Equipment and attitude: Camera height approximately [camera height in meters] meters, pitch approximately [camera pitch in degrees] degrees; track type: [track type]; inspection speed approximately [inspection speed in kilometers per hour] kilometers per hour;

[0159] Panoramic image resolution: [width] × [height];

[0160] Candidate region descriptions (processed one by one):

[0161] For each candidate, provide: whether it is a foreign object (yes / no), if it is a foreign object, its category (stone / metal fragment / plastic waste / rubber block / wood fragment / track detachment / other), if it is a false alarm, its background facility category (sleeper / fastener / cable trough / track slab / turnout component / other), key evidence points (no more than 5), risk level (high / medium / low), and disposal recommendations (arrange on-track treatment / regular observation).

[0162] Reference information: coordinates and size of the rectangle, area ratio, whether it is located in the vicinity of fasteners or sleepers, whether there is glare or shadow interference, and whether it may be a repeating structure.

[0163] Note: Small and low-contrast targets are more likely to be missed; therefore, it is necessary to combine the cropped image with the overall context for judgment. Obvious background features such as sleepers, fasteners, and cable troughs should be marked as false alarms and their categories should be provided.

[0164] Output format requirements:

[0165] Output only a structured array of "one object for each candidate"; keys must be in Chinese; numerical units must be pixels or percentages; no text other than the structured results should be output.

[0166] In the above embodiments, the construction process of the prompt word template first clearly defines the role of the graphic multimodal large model as a safety assistant for railway ballastless track inspection. Its core task is to determine foreign objects in each candidate area one by one and output structured results. The model needs to first determine whether the area is a foreign object. If it is confirmed to be a foreign object, it further identifies the specific category, including preset types such as stones, metal fragments, plastic waste, rubber blocks, wood fragments, or track detachment parts. At the same time, it assesses the risk level into three levels: high, medium, and low, and generates corresponding disposal suggestions based on the risk level, such as arranging on-track treatment or regular observation. If it is determined to be a false alarm, it needs to label the background facility category, such as sleepers, fasteners, cable troughs, track slabs, or turnout components.

[0167] Input data needs to integrate scene information including route name, section location, shooting time, weather conditions, lighting environment, camera height, tilt angle, track type, and acquisition speed, combined with reference information such as the rectangular coordinates of candidate regions, area proportion, aspect ratio, surrounding area, and the presence of reflections or shadows. Output is strictly required to be a structured array with Chinese characters for the key names, containing only necessary fields such as whether it is a foreign object, foreign object category, confidence level, key evidence points, risk level, and disposal suggestions, avoiding any redundant text and ensuring that the results can be directly parsed and used by downstream systems. The entire process guides the model to achieve accurate identification and executable decisions through standardized prompts.

[0168] 3. Expected output data format

[0169] The output is an "array of candidate results", with each element corresponding to a candidate number; it includes the judgment conclusion, category, risk and recommendation, key reasons, and backfilled geometric information to facilitate subsequent alignment and visualization.

[0170] Example: [

[0172] {

[0173] "Candidate ID": "c1",

[0174] "Is it a foreign object?": "Yes",

[0175] Foreign object category: "stone"

[0176] Confidence level: 0.82

[0177] Key reasons: ["Clear and irregular edges", "Distinct color contrast with track slab", "Not a fastener or sleeper type"]

[0178] Risk Level: Medium

[0179] Recommended course of action: Arrange for inspection and handling.

[0180] Suggested completion time (in hours): 4

[0181] "Rectangle": {"x": 820, "y": 460, "w": 64, "h": 52},

[0182] "Area percentage": 0.0016,

[0183] "Source level": "P2",

[0184] Overlapping group number: "g1"

[0185] },

[0186] {

[0187] "Candidate ID": "c2",

[0188] "Is it a foreign object?": "No",

[0189] "False Alarm Background Facility Category": "Rail Sleepers"

[0190] Confidence level: 0.76

[0191] Key reasons: ["Long straight edges and repeating texture", "Position consistent with sleeper arrangement", "Consistent with the appearance of surrounding sleepers"]

[0192] Risk Level: Low

[0193] Recommended course of action: Regular observation.

[0194] Suggested completion time (in hours): 72

[0195] "Rectangle": {"x": 420, "y": 720, "w": 140, "h": 90},

[0196] "Area percentage": 0.0061,

[0197] "Source level": "P3",

[0198] Overlapping group number: "g2"

[0199] }

[0200] ].

[0201] In the above embodiments, the technical process of the output data format is organized in the form of a candidate result array, with each array element corresponding to a candidate number, ensuring the standardization and parsability of the data structure. Each candidate result object contains core judgment fields, including the judgment conclusion of whether it is a foreign object, the classification information of the foreign object category or the false alarm background facility category, the confidence score, the risk level classification, and specific disposal recommendations. The reasoning field stores key evidence descriptions, such as morphological features or environmental comparisons, in array form to assist in the traceability of decision-making. The geometric information field is filled with the coordinates of the rectangle, the area ratio, the source level, and the overlapping group number to achieve spatial alignment with the input data. The output strictly follows the structured specification of Chinese characters for the key names, and all numerical units are uniformly set to pixels or percentages to avoid redundant text and ensure that downstream systems can directly integrate and process the data.

[0202] III. Large-Scale Image and Text Model Recognition and Decision Making

[0203] This solution uses a QVQ image-text multimodal large model as the core for recognition and decision-making, and performs supervised fine-tuning based on inspection image data to adapt the model to the specific data domain and task description of railway inspection. At the same time, it integrates the "Retrieval Enhanced Generation (RAG)" capability, which uses railway maintenance rules, work instructions and risk classification standards as external knowledge, retrieves them as needed and injects them into the prompts, and helps the model to provide actionable and traceable disposal suggestions while determining the type of foreign object.

[0204] Figure 4 This is a specific example diagram of a large-scale image and text model recognition and decision-making process in an embodiment of the present invention, such as... Figure 4 As shown, the system first receives a panoramic image, candidate region coordinates, and scene information as input. Guided by designed prompts, a finely tuned QVQ image-text model is then analyzed. The core innovation of the model lies in its integration of a retrieval-enhanced generation (RAG) mechanism, which can retrieve relevant regulations and clauses from a railway maintenance rules knowledge base in real time as a basis for decision-making, thereby outputting accurate and traceable identification results (such as foreign object category, risk level, and handling recommendations). These results are ultimately used to drive downstream systems such as alarms and task assignment, forming a closed loop. Handling data and difficult case samples are fed back for continuous optimization of the model and knowledge base. Figure 4 The execution flow of the following content is shown:

[0205] 1. Model and domain adaptation:

[0206] Use inspected images and structured labels for instruction and preference alignment; freeze the underlying visual encoder or use parameter-efficient fine-tuning (such as adapters, low-rank strategies) to focus on optimizing cross-modal alignment and decision heads to mitigate catastrophic forgetting; construct an input paradigm of "panoramic image only + candidate coordinates + scene metadata" to uniformly output Chinese structured results.

[0207] The technical process of image-text large-scale model recognition and decision-making begins with model selection and domain adaptation. A large-scale image-text multimodal model with a specific architecture is selected as the core, and it is adapted to the railway inspection scenario through supervised fine-tuning. The fine-tuning process uses inspection images and structured label data to align instructions and preferences. Cross-modal alignment and decision heads are optimized by freezing the visual encoder or using efficient parameter fine-tuning strategies. An input paradigm with panoramic images, candidate coordinates, and scene metadata as the core is constructed, and a unified Chinese structured result is output.

[0208] 2. Prompt words and input organization:

[0209] Package the panoramic image, candidate area coordinates, and scene information (time, weather, lighting, camera pose, route type, etc.) to generate Chinese task prompts: determine whether each candidate is a foreign object—if so, classify and risk-level it and provide disposal suggestions; otherwise, mark it as a background facility and explain the reasons; output structured results with Chinese key names and include traceable evidence points.

[0210] During the input data organization phase, the panoramic image, candidate area coordinates, and scene information (including time, weather, lighting, camera pose, route type, etc.) are integrated and packaged to generate clear Chinese task prompts. The model is required to determine whether each candidate is a foreign object, classify the foreign objects, assign risk levels, and generate disposal suggestions. For false alarms, the background facility category is marked and the reason is explained.

[0211] 3. Search Enhancement Generation (RAG):

[0212] Establish a "Railway Maintenance Rules Knowledge Base": including standards and specifications, handling procedures, thresholds and time limits, risk classification rules and case highlights; based on text vector indexing and reordering, retrieve the most relevant entries according to the candidate's "category prior, location neighborhood, and scenario meta-information", and inject the summary and clause number into the prompt as "rule context" to ensure that the suggestions are based on evidence and are traceable.

[0213] The enhanced retrieval generation mechanism establishes a railway maintenance rules knowledge base, integrating standards, procedures, risk classification rules, and other content. Based on text vector indexing and reordering technology, it retrieves the most relevant regulations and clauses according to the prior category, location neighborhood, and scenario metadata of candidate targets. The summary and clause number are injected as contextual prompts to ensure the traceability of decision-making basis.

[0214] 4. Reasoning and Output:

[0215] QVQ receives images and Chinese prompts (including regulatory context), and outputs the following for each candidate: whether it is a foreign object, the type of foreign object (such as stones, metal fragments, plastic waste, track debris, etc.), confidence level, risk level, disposal recommendations and completion deadline, key reasons, and the cited regulatory clause number; only a structured array is output for easy system consumption and visualization overlay.

[0216] 5. Continuous optimization and repatriation:

[0217] The system will fine-tune the incremental return of false alarms, missed detections, and high-risk samples; write the execution results and handling loop back to the data and knowledge base; re-rank the search engine and update the index to maintain compatibility with new regulations and new foreign objects on site; and monitor indicators such as recall rate, false alarm rate, and average handling time online.

[0218] Second specific embodiment:

[0219] This embodiment first processes the inspection image using an improved YOLOv11 object detection network to locate candidate foreign object regions and generate pixel coordinates and bounding box information. Then, the panoramic image, candidate bounding box data, and scene metadata are integrated into a structured input pair, which includes contextual information such as the road environment and camera parameters. Next, designed prompts guide the image-text model to determine the foreign object, requiring it to combine panoramic and local features to output classification results, risk levels, and handling suggestions. Finally, the model generates a structured result containing the foreign object category, confidence level, key reasons, and cited regulations, completing the closed-loop processing from image detection to decision-making suggestions. Specifically, the steps include:

[0220] I. Object Detection Network Recognition

[0221] The input inspection image is shown in the figure. It is fed into an improved YOLOv11 foreign object region target detection network to obtain the localization results. The candidate box pixel coordinates are the top left and bottom right corners: [1160, 688, 1349, 824]; therefore, the width is 189 pixels and the height is 136 pixels. To preserve context, the box is expanded by approximately 10%, resulting in an expanded box with a width of 208 and a height of 150, while the center remains unchanged.

[0222] If the panoramic image resolution is 1536×1024 (pixels), then the normalized coordinates are approximately:

[0223] Original frame: x=0.755, y=0.672, w=0.123, h=0.133

[0224] Outer frame: x=0.749, y=0.665, w=0.135, h=0.146

[0225] II. Data Organization for Model Input

[0226] {

[0227] "Panoramic image": "path / to / full_image.jpg",

[0228] Panoramic image resolution: {"Width": 1536, "Height": 1024}

[0229] Scene Information: {

[0230] "Line Name": "Example High-Speed ​​Railway",

[0231] "Section": "Kxxx+yyy",

[0232] "Shooting Time": "2025-08-18 14:32",

[0233] Weather: Sunny

[0234] "Light": "Strong light"

[0235] Camera height in meters: 1.6

[0236] Camera tilt: -10,

[0237] Track type: Ballastless track

[0238] "Inspection speed in kilometers per hour": 60

[0239] },

[0240] "Candidate Region List": [

[0241] {

[0242] "Candidate ID": "c1",

[0243] "Source level": "P2",

[0244] Initial screening confidence level: 0.57

[0245] "Rectangle_pixels": {"x": 1160, "y": 688, "w": 189, "h": 136},

[0246] "Rectangle_Normalized": {"x": 0.755, "y": 0.672, "w": 0.123, "h":0.133},

[0247] "Outer rectangle frame_pixels": {"x": 1151, "y": 681, "w": 208, "h": 150},

[0248] "Extended Rectangle_Normalized": {"x": 0.749, "y": 0.665, "w": 0.135, "h":0.146},

[0249] "Area in pixels": 25704

[0250] "Area percentage": 0.016,

[0251] Aspect Ratio: 1.39

[0252] "Location": "Near the fastener"

[0253] "Whether to truncate the boundary": "No",

[0254] "Overlapping group number": "g1",

[0255] "Cropped image": "path / to / crop_c1.jpg"

[0256] } ]

[0258] }

[0259] III. Prompts for the QVQ Graphical Model

[0260] 1. Your role: Safety assistant for railway ballastless track inspection. You will see a panoramic view and cropped images and coordinate information of candidate areas. Please determine whether each candidate area is a foreign object; if it is, please provide the specific category, risk level, and handling suggestions; if it is not a foreign object, please provide the corresponding background facility category and mark it as a false alarm. Only output a structured array of Chinese results, without adding any extra text.

[0261] 2. Processing requirements:

[0262] For each candidate, indicate whether it is a foreign object (yes / no); if it is a foreign object, distinguish its category (stone, metal fragment, plastic waste, rubber block, wood fragment, track detachment part, etc.); if it is a false alarm, indicate the category of background facility (sleeper, fastener, cable trough, track slab, turnout part, others).

[0263] Output confidence level, risk level (high / medium / low), handling recommendations (immediate shutdown and cleanup / arrange inspection and handling / regular observation), recommended completion time limit (hours), and provide no more than 5 "reasoning points".

[0264] Make full use of the panoramic context (positional relationship between sleepers / fasteners / track slabs, reflections and shadows) and the details of the cut (material, shape, outline).

[0265] If the candidates belong to the same objective, please state in the reasoning section that "they belong to the same objective as other candidates in the group".

[0266] Please strictly follow the field names below when outputting.

[0267] IV. Structured Output [

[0269] {

[0270] "Candidate ID": "c1",

[0271] "Is it a foreign object?": "Yes",

[0272] Foreign Object Category: "Trajectory Detachment Component (Fastening Spring)"

[0273] Confidence level: 0.86

[0274] "Key Reasons": [

[0275] "High-gloss, reflective metallic material with a bent, closed shape"

[0276] "The shape is the same as the fastener spring bar, but the position is abnormal."

[0277] "Located in the vicinity of the fastener and offset from the fixed position",

[0278] "The edges are sharp and the blocks are prominent, unlike the texture of the sleepers."

[0279] ],

[0280] Risk Level: Medium

[0281] Recommended course of action: Arrange for inspection and handling.

[0282] Suggested completion time (in hours): 2

[0283] "Referenced regulation clause number": "RAG-Rule-Fast-3.2.1",

[0284] "Rectangle_pixels": {"x": 1160, "y": 688, "w": 189, "h": 136},

[0285] "Rectangle_Normalized": {"x": 0.755, "y": 0.672, "w": 0.123, "h":0.133},

[0286] "Source level": "P2",

[0287] Overlapping group number: "g1"

[0288] } ]

[0290] The beneficial effects of the technical solution in this embodiment are as follows:

[0291] 1. Safety and economic benefits: Early detection of high-risk foreign objects reduces driving risks and costs caused by false stops and false alarms; improves inspection efficiency and line availability.

[0292] 2. High recall and low false negatives: The cascaded architecture uses YOLOv11 as a high-recall proposer, and adds small target branch, global context, soft nonmaximum suppression and test-time enhancement, which significantly improves the detection capability of small volume, low contrast and deformable objects.

[0293] 3. Significantly reduced false alarms: The large image and text model, combined with enhanced retrieval (access to railway maintenance rules), performs secondary discrimination and compliance verification, effectively filtering out false alarms caused by background structures such as sleepers, fasteners, and cable troughs.

[0294] 4. Executable results: Outputs "Is it a foreign object - Category - Risk level - Disposal suggestion - Suggested completion time", which facilitates direct dispatch and closed-loop management by the dispatch / inspection system, shortening response time.

[0295] 5. Real-time and resource-friendly: The front end only performs efficient initial screening, while complex semantics are handled by the back end, supporting edge-cloud collaboration and reducing overall computing power and deployment costs.

[0296] 6. Continuous evolution capability: incremental fine-tuning for false alarms, missed detections, and high-risk sample return; the knowledge base and index can be updated online to quickly adapt to new foreign objects and the latest regulations.

[0297] The key technical points and areas to be protected in this invention are as follows:

[0298] 1. Cascaded architecture: The closed loop from target detection and initial screening (high recall) → image and text model for fine-grained judgment (high accuracy) → handling and backflow is completed.

[0299] 2. Detection Network Modification (YOLOv11): Introduces P2 small target branch; local large convolution and deformable convolution; spatial pyramid pooling; path aggregation + bidirectional feature pyramid + adaptive fusion; anchor-free decoupled detection head; dynamic positive sample matching; recall-priority thresholding and post-processing (low confidence threshold, high crossover ratio, soft / clustering nonmaximum suppression, multi-scale and flipping during testing); optional temperature calibration and single-class suspected detection.

[0300] 3. Prompt words and data organization: Only panoramic images (cropping optional) are required, along with Chinese structured coordinates and scene metadata (time, weather, camera pose, neighborhood labels, etc.); unified Chinese key names and output format facilitate system integration.

[0301] 4. Large-scale image-text (QVQ) domain adaptation: Fine-tuning on the inspection image (instruction / preference alignment, efficient parameter fine-tuning) to enhance cross-modal understanding and Chinese structured output capabilities.

[0302] 5. Enhanced Retrieval Generation (RAG): Access to railway maintenance rules / work instructions / risk classification knowledge base, vector retrieval and reordering, inject clause summaries and numbers into prompts, and realize "traceable" classification and handling suggestions.

[0303] 6. Risk assessment and response generation: The model identifies risk levels, response recommendations, and recommended timeframes in conjunction with regulatory constraints.

[0304] 7. Difficult Case Backflow and Continuous Optimization: Incremental fine-tuning of backflow of false positives / false negatives / high-risk samples; online updates of rule base and index; indicator monitoring and threshold self-adaptation.

[0305] Of course, it is understood that there may be other variations of the above detailed process, and all such variations should fall within the protection scope of this invention.

[0306] In this embodiment of the invention, a target detection model is used to process the acquired track images to identify foreign object candidate regions and generate candidate bounding boxes based on recall. The target detection model adopts an improved YOLOv11 architecture, which introduces a small target detection branch, a feature fusion mechanism, and a recall-first strategy to filter foreign object candidate regions and generate candidate bounding boxes. Based on the candidate bounding box information, the image is cropped or highlighted, and structured prompts containing coordinates and track information are constructed by combining scene metadata to form input pairs of image fragments and prompts. The input pairs are analyzed based on a large-scale image-text multimodal model. The large-scale image-text multimodal model is used to introduce an external knowledge base based on retrieval enhancement generation technology to perform deep semantic understanding on each candidate region and output a structured result indicating whether it is a foreign object, the foreign object category, the risk level, and the disposal suggestion. If the structured information determines that the input pair contains a foreign object, the foreign object category, risk level, and corresponding disposal suggestion are output. This invention employs an improved YOLOv11 model to achieve high recall in initial screening. Through a small target detection branch and feature fusion mechanism, it significantly enhances the ability to detect small-sized foreign objects, overcoming the missed detection problem of traditional methods under complex lighting and perspective changes. By combining a large-scale multimodal model of images and text with retrieval enhancement generation technology, and constructing input pairs containing panoramic and partially cropped images, the model can accurately distinguish real foreign objects from background facilities such as sleeper fasteners, reducing the false alarm rate for track foreign objects. Through the output of structured results, it achieves closed-loop management from detection to disposal, thereby improving the efficiency and accuracy of track foreign object identification, risk assessment, and disposal response.

[0307] This invention also provides a track foreign object identification device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the track foreign object identification method, the implementation of this device can refer to the implementation of the track foreign object identification method; repeated details will not be elaborated further.

[0308] This invention also provides a track foreign object identification device to improve the efficiency and accuracy of track foreign object identification, risk assessment, and response. Figure 5 This is a schematic diagram of the structure of a track foreign object identification device according to an embodiment of the present invention, as shown below. Figure 5 As shown, the device includes:

[0309] The foreign object candidate region identification module 501 is used to process the acquired track image using a target detection model, identify foreign object candidate regions with a high recall rate, and generate candidate box information. The target detection model adopts an improved YOLOv11 architecture, which introduces a small target detection branch, a feature fusion mechanism, and a recall priority strategy to screen foreign object candidate regions and generate candidate box information.

[0310] The input pair forming module 502 is used to crop or highlight the image based on the candidate box information and combine it with the scene metadata to construct a structured prompt word containing coordinate and line information, forming an input pair of image fragments and prompt words;

[0311] The structured result analysis module 503 is used to analyze the input pair based on the image-text multimodal large model. The image-text multimodal large model is used to introduce an external knowledge base based on retrieval enhancement generation technology, perform deep semantic understanding on each candidate region, and output structured results such as whether it is a foreign object, the category of the foreign object, the risk level, and the disposal suggestions. If the structured information determines that the input pair contains a foreign object, the module outputs the foreign object category, risk level, and corresponding disposal suggestions.

[0312] In one embodiment, the target detection model is based on the YOLOv11 architecture, outputting multi-layer features through the backbone network, and then weighting and aggregating information at different scales via neck feature fusion, which is then fed into an anchorless decoupled detection head to generate candidate boxes; candidate box information is output by filtering under low confidence thresholds and high crossover ratio thresholds using soft or clustering variants of non-maximum suppression.

[0313] In one embodiment, based on candidate box information, the image is cropped or highlighted, and a structured prompt word containing coordinates and line information is constructed by combining scene metadata, forming an input pair of image fragments and prompt words, including:

[0314] A panoramic view of the orbital image and a cropped image obtained by cropping each suspected foreign object region are acquired; wherein the cropped image is based on a preset ratio of expansion of the candidate box to preserve the context, and the suspected foreign object region is highlighted on the panoramic image;

[0315] Structured prompts are constructed based on scene metadata; the scene metadata includes line name, section, time, weather, lighting conditions, camera height, pitch angle, track type, and acquisition speed; the structured prompts include the coordinate information of candidate boxes, area ratio, aspect ratio, scene metadata, and candidate attributes; the candidate attributes include candidate number, source level, initial screening confidence, neighborhood, and overlapping group number;

[0316] Combine the cropped image or the highlighted cropped image with structured prompts to form an input pair.

[0317] In one embodiment, the large-scale image-text multimodal model is constructed as follows:

[0318] A basic image-text multimodal large model was selected, and supervised fine-tuning was performed based on railway inspection images and corresponding structured label data; the fine-tuning process included optimizing the model through instruction alignment and preference alignment.

[0319] A parameter-efficient fine-tuning strategy is adopted to adapt the model; a railway maintenance rule knowledge base is established, which includes handling procedures and risk classification rules; based on text vector indexing and re-ranking technology, the attribute information of candidate regions is input into the model.

[0320] The model is deployed to receive inputs including images, candidate region information, and regulatory context, and outputs structured results including foreign object category, risk level, and disposal recommendations.

[0321] In one embodiment, if the structured information determines that the input pair contains foreign objects, the output includes the foreign object category risk level and corresponding handling recommendations, including:

[0322] If the input pair is determined to contain foreign objects through structured information, a structured result array is output. This array includes a judgment on whether the object is foreign, the object category, confidence level, key reasons, risk level, disposal recommendations, recommended completion deadline, and cited regulation number. The object category is selected from a preset set of categories, including stones, metal fragments, plastic waste, rubber blocks, wood fragments, or detached track components. The risk level is divided into high, medium, and low levels. The disposal recommendations are determined based on the risk level and include immediate shutdown and cleanup, arranging inspection and handling, or periodic observation. The output format is a structured array with Chinese characters for the key names.

[0323] This invention provides an embodiment of a computer device for implementing all or part of the above-described method for identifying foreign objects on tracks. The computer device specifically includes the following components:

[0324] The computer device comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between related devices; the computer device can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the computer device can be implemented with reference to the embodiments for implementing the method for identifying foreign objects on a track and the embodiments for implementing a device for identifying foreign objects on a track, the contents of which are incorporated herein by reference, and repeated details will not be described again.

[0325] Figure 6 This is a schematic diagram of a computer device provided in an embodiment of the present invention, which discloses a schematic block diagram of the system configuration of a computer device 1000 according to an embodiment of this application. Figure 6 As shown, the computer device 1000 may include a central processing unit 1001 and a memory 1002; the memory 1002 is coupled to the central processing unit 1001. It is worth noting that... Figure 6 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0326] In one embodiment, the foreign object detection function can be integrated into the central processing unit 1001. The central processing unit 1001 can be configured to perform the following control:

[0327] The acquired track images are processed using a target detection model to identify foreign object candidate regions and generate candidate box information based on recall. The target detection model adopts an improved YOLOv11 architecture, which introduces a small target detection branch, a feature fusion mechanism, and a recall-first strategy to filter foreign object candidate regions and generate candidate box information.

[0328] Based on candidate box information, the image is cropped or highlighted and combined with scene metadata to construct structured prompts containing coordinate and line information, forming an input pair of image fragments and prompts;

[0329] The input pair is analyzed based on a large image-text multimodal model. The large image-text multimodal model is used to introduce an external knowledge base based on retrieval enhancement generation technology, perform deep semantic understanding on each candidate region, and output a structured result indicating whether it is a foreign object, the type of foreign object, the risk level, and the disposal suggestion. If the structured information determines that the input pair contains a foreign object, the foreign object category, risk level, and corresponding disposal suggestion are output.

[0330] In another embodiment, the track foreign object identification device can be configured separately from the central processing unit 1001. For example, the track foreign object identification device can be configured as a chip connected to the central processing unit 1001, and the track foreign object identification function can be realized through the control of the central processing unit.

[0331] like Figure 6 As shown, the computer device 1000 may further include: a communication module 1003, an input unit 1004, an audio processor 1005, a display 1006, and a power supply 1007. It is worth noting that the computer device 1000 does not necessarily need to include... Figure 6 All components shown; in addition, the computer device 1000 may also include Figure 6 For components not shown, please refer to existing technologies.

[0332] like Figure 6 As shown, the central processing unit 1001, sometimes also referred to as a controller or operation control, may include a microprocessor or other processor device and / or logic device. The central processing unit 1001 receives input and controls the operation of various components of the computer device 1000.

[0333] The memory 1002 may be, for example, one or more of a cache, flash memory, hard drive, removable medium, volatile memory, non-volatile memory, or other suitable device. It can store the aforementioned device-related information, and may also store programs for executing that information. The central processing unit 1001 can execute the program stored in the memory 1002 to perform information storage or processing, etc.

[0334] Input unit 1004 provides input to central processing unit 1001. This input unit 1004 may be, for example, a keypad or touch input device. Power supply 1007 provides power to computer device 1000. Display 1006 displays images, text, and other display objects. This display may be, for example, an LCD display, but is not limited to this.

[0335] The memory 1002 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 1002 can also be some other type of device. The memory 1002 includes a buffer memory 1021 (sometimes referred to as a buffer). The memory 1002 may include an application / function storage unit 1022 for storing application programs and function programs or processes for executing operations of the computer device 1000 via the central processing unit 1001.

[0336] The memory 1002 may also include a data storage unit 1023 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the computer device. The driver storage unit 1024 of the memory 1002 may include various drivers for the computer device for communication functions and / or for performing other functions of the computer device (such as messaging applications, address book applications, etc.).

[0337] The communication module 1003 is a transmitter / receiver that transmits and receives signals via the antenna 1008. The communication module (transmitter / receiver) 1003 is coupled to the central processing unit 1001 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0338] Based on different communication technologies, multiple communication modules 1003 can be configured in the same computer device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 1003 is also coupled to a speaker 1009 and a microphone 1010 via an audio processor 1005 to provide audio output via the speaker 1009 and receive audio input from the microphone 1010, thereby realizing typical telecommunications functions. The audio processor 1005 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 1005 is also coupled to a central processing unit 1001, enabling on-device recording via the microphone 1010 and on-device playback of stored sound via the speaker 1009.

[0339] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for identifying foreign objects on the track.

[0340] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for identifying foreign objects on the track.

[0341] In this embodiment of the invention, a target detection model is used to process the acquired track images to identify foreign object candidate regions and generate candidate bounding boxes based on recall. The target detection model adopts an improved YOLOv11 architecture, which introduces a small target detection branch, a feature fusion mechanism, and a recall-first strategy to filter foreign object candidate regions and generate candidate bounding boxes. Based on the candidate bounding box information, the image is cropped or highlighted, and structured prompts containing coordinates and track information are constructed by combining scene metadata to form input pairs of image fragments and prompts. The input pairs are analyzed based on a large-scale image-text multimodal model. The large-scale image-text multimodal model is used to introduce an external knowledge base based on retrieval enhancement generation technology to perform deep semantic understanding on each candidate region and output a structured result indicating whether it is a foreign object, the foreign object category, the risk level, and the disposal suggestion. If the structured information determines that the input pair contains a foreign object, the foreign object category, risk level, and corresponding disposal suggestion are output. This invention employs an improved YOLOv11 model to achieve high recall in initial screening. Through a small target detection branch and feature fusion mechanism, it significantly enhances the ability to detect small-sized foreign objects, overcoming the missed detection problem of traditional methods under complex lighting and perspective changes. By combining a large-scale multimodal model of images and text with retrieval enhancement generation technology, and constructing input pairs containing panoramic and partially cropped images, the model can accurately distinguish real foreign objects from background facilities such as sleeper fasteners, reducing the false alarm rate for track foreign objects. Through the output of structured results, it achieves closed-loop management from detection to disposal, thereby improving the efficiency and accuracy of track foreign object identification, risk assessment, and disposal response.

[0342] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0343] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0344] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0345] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0346] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A rail foreign object recognition method characterized by comprising: The method comprises the following steps: processing the collected track images by using a target detection model to identify foreign object candidate regions and generate candidate box information with recall rate; the target detection model adopts an improved YOLOv11 architecture, which introduces a small target detection branch, a feature fusion mechanism and a recall priority strategy to screen foreign object candidate regions and generate candidate box information; based on the candidate box information, the image is cropped or highlighted, and a structured prompt word containing coordinates and line information is constructed by combining scene metadata to form an input pair of image segments and prompt words; the input pair is analyzed based on a graph-text multimodal large model; the graph-text multimodal large model is used to introduce an external knowledge base based on a retrieval enhancement generation technology to perform deep semantic understanding on each candidate region and output structured results including whether it is a foreign object, foreign object category, risk level and disposal suggestion; if the input pair contains a foreign object as determined by the structured information, the foreign object category, risk level and corresponding disposal suggestion are output.

2. The method of claim 1, wherein, The target detection model is based on the YOLOv11 architecture, which outputs multiple layers of features through the backbone network, and converges different scale information through neck feature fusion weighting, and then inputs the candidate box generated by the anchor point decoupling detection head; the candidate box information is output by screening with soft or clustering variants of non-maximum suppression under low confidence threshold and high intersection over union threshold.

3. The method of claim 1, wherein, Based on the candidate box information, the image is cropped or highlighted, and a structured prompt word containing coordinates and line information is constructed by combining scene metadata to form an input pair of image segments and prompt words, comprising: obtain a panoramic image of the track image and a cropped image obtained by cropping each suspected foreign object region; wherein the cropped image is based on a candidate box expanded by a preset proportion to retain context, and the suspected foreign object region is marked on the panoramic image in a highlighted manner; construct a structured prompt word based on scene metadata; the scene metadata includes line name, section, time, weather, lighting condition, camera height, pitch angle, track type and collection speed; the structured prompt word contains coordinate information, area ratio, aspect ratio, scene metadata and candidate attributes of the candidate box; the candidate attributes include candidate number, source level, primary screening confidence, neighborhood and overlapping grouping number; combine the cropped image or highlighted cropped image with the structured prompt word as an input pair.

4. The method of claim 1, wherein, The graph-text multimodal large model is constructed as follows: select a basic graph-text multimodal large model and perform supervised fine-tuning based on railway inspection images and corresponding structured label data; wherein the fine-tuning process includes optimizing the model through instruction alignment and preference alignment; adapt the model by using a parameter-efficient fine-tuning strategy; establish a railway maintenance rule knowledge base; the knowledge base contains disposal process and risk classification rules; based on text vector indexing and reordering technology, input the model according to the attribute information of the candidate region; deploy the model so that it can receive inputs containing images, candidate region information and regulatory context, and output structured results containing foreign object category, risk level and disposal suggestion.

5. The method of claim 1, wherein, If the structured information determines that the input pair has a foreign matter, output the foreign matter category risk level and the corresponding treatment suggestion, including: If the structured information determines that the input pair has a foreign matter, output a structured result array; the structured result array includes a judgment conclusion of whether it is a foreign matter, a foreign matter category, a confidence level, a key point of reason, a risk level, a treatment suggestion, a suggestion completion time limit and a reference regulation number; wherein the foreign matter category is selected from a preset category set, including stone, metal fragments, plastic garbage, rubber blocks, wooden fragments or track shed parts; the risk level is divided into three levels of high, medium and low; the treatment suggestion is determined based on the risk level, including immediate shutdown and cleaning, arrangement of inspection and processing or regular observation; the output format is a structured array with Chinese key names.

6. A rail foreign object detection device, characterized by, Comprising: A foreign matter candidate area identification module for processing the collected track image using a target detection model to identify foreign matter candidate areas with a recall rate and generate candidate box information; The target detection model adopts an improved YOLOv11 architecture, which introduces a small target detection branch, a feature fusion mechanism and a recall priority strategy to screen foreign matter candidate areas and generate candidate box information; An input pair forming module for cropping or highlighting the image based on the candidate box information and combining scene metadata to construct structured prompt words containing coordinates and line information, forming an input pair of image segments and prompt words; A structured result analysis module for analyzing the input pair based on a graph-text multimodal large model; the graph-text multimodal large model is used to introduce an external knowledge base based on retrieval enhancement generation technology to perform deep semantic understanding on each candidate area, and output structured results of whether it is a foreign matter, a foreign matter category, a risk level and a treatment suggestion; if the structured information determines that the input pair has a foreign matter, output the foreign matter category risk level and the corresponding treatment suggestion.

7. The apparatus of claim 6, wherein, The input pair forming module is specifically used for: Obtaining a panoramic image of the track image and a cropped image obtained by cropping each suspected foreign matter area; wherein the cropped image is based on a preset proportion of candidate box expansion to retain context, and the suspected foreign matter area is marked on the panoramic image in a highlighted manner; Constructing structured prompt words based on scene metadata; The scene metadata includes line name, section, time, weather, lighting conditions, camera height, pitch angle, track type and collection speed; The structured prompt words include coordinate information, area ratio, aspect ratio, scene metadata and candidate attributes of the candidate box; the candidate attributes include candidate number, source level, preliminary screening confidence, neighborhood and overlapping grouping number; The cropped image or highlighted cropped image and the structured prompt words are combined into an input pair.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the method of any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method of any one of claims 1-5.

10. A computer program product, characterised in that, The computer program product includes a computer program, and the computer program is executed by the processor to realize the method of any one of claims 1-5.

Images