Method for carrying out on-line and off-line intelligent checking and checking on railway tools and tools

Abstract

The invention provides a method for carrying out on-line and off-line intelligent checking and checking on railway tools and tools. The method comprises the following steps: inputting original feature maps of a query graph and a support graph into a depth estimation network, calculating cosine similarity between depth features of the support graph and the query graph, generating candidate regions by utilizing a candidate region generation network, and applying RoI Align operation on each candidate region to generate support RoI features and query RoI features; extracting an attribute word related to the current to-be-detected railway tool from the query RoI feature through an attribute analyzer, and obtaining a text feature of the updated railway tool category name through a Star-Transform; and performing similarity calculation on the CLIP RoI feature of the query graph and the text feature of the railway tool category name by using cosine similarity to obtain a recognition result of the tools in the railway work depot. According to the invention, by introducing the deep learning-based on-line and off-line intelligent checking method for the railway tools, the checking efficiency of the tools is improved, and the manual operation time is shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of railway tool maintenance, and in particular to a method for performing intelligent counting and verification of railway tools on and off-line. Background Art

[0002] High-speed rail is an important means of transportation in modern society. It is high-speed, efficient, safe and convenient, and is widely used in travel and logistics transportation. In order to ensure the safe operation of high-speed rail, the operation and maintenance of high-speed rail is crucial. Considering the safety factor, most high-speed rail lines will be shut down for maintenance at night. The daily maintenance of railways is divided into many types of operations. Each type of operation requires the use of different types and quantities of tools. The number of tools involved in the daily maintenance of high-speed rail can reach hundreds of types. Therefore, it is easy to lose tools at the operation site during operation and maintenance, which will cause safety hazards to the high-speed rail and economic losses to the railway department.

[0003] At present, the existing tool management methods mainly include manual registration, barcode scanning and RFID (Radio Frequency Identification) tag recognition, etc. However, these methods have exposed many shortcomings in practical applications and cannot meet the growing management needs of high-speed railway maintenance sections.

[0004] Barcode technology is a traditional method widely used in tool management. Each tool is affixed with a unique barcode label, which is identified and recorded by a handheld barcode scanner. The disadvantages of this method include: barcode labels are exposed to the outside and are easily affected by the environment, such as stains, wear, moisture, etc. Once the barcode labels are worn or contaminated, the scanning equipment may not be able to read them correctly, resulting in data entry errors, which in turn affects management decisions. Barcode recognition requires scanning one by one, and batch recognition cannot be achieved. When the number of tools is large, the inventory workload is huge and the efficiency is low.

[0005] In order to solve some limitations of barcode technology, RFID technology was introduced into tool management. RFID tags are identified through radio waves, and have advantages such as batch reading and not easy to damage. RFID technology can read multiple tags at the same time, which improves the recognition efficiency. The disadvantages of the RFID method include: RFID signals are easily interfered in metal environments or other complex physical environments, resulting in large recognition errors. Many tools in the railway engineering section are metal products, and the metal environment has a large interference with RFID signals, affecting the recognition accuracy. The installation and maintenance costs of the RFID system are high, and the tags need to be regularly maintained and replaced, especially in harsh environments, where the tags are easily damaged, further increasing the maintenance costs. The hardware equipment of the RFID system (such as readers and antennas) also needs regular maintenance, which increases the management costs. RFID technology requires matching reading and writing equipment, which increases the complexity and cost of the system. RFID reading and writing equipment is expensive and requires professional personnel to install and maintain, which increases the cost of initial investment and subsequent maintenance.

[0006] Manual registration is the most traditional tool management method, which relies on the manual recording and management of each tool by the staff. The disadvantages of this method include: manual recording is prone to omissions and errors, and the accuracy and reliability of data are poor. Errors caused by human factors are inevitable in actual operations, affecting the accuracy of tool management. Summary of the invention

[0007] The embodiment of the present invention provides a method for intelligently counting and checking railway tools and equipment on and off the line, so as to effectively improve the efficiency of counting the tools and equipment on and off the line in the maintenance section.

[0008] In order to achieve the above object, the present invention adopts the following technical scheme.

[0009] A method for intelligently counting and checking railway tools and equipment on and off the line, comprising:

[0010] The railway tool images taken during the offline maintenance process are used as query images, and the railway tool images from the tool warehouse are used as support images. The backbone network is used to extract features from the support image and the query image to obtain the original feature images of the support image and the query image.

[0011] Inputting the original feature map of the query graph and the original feature map of the support graph into a depth estimation network, wherein the depth estimation network extracts depth features of the query graph and the support graph using a depth perception algorithm;

[0012] Calculating the cosine similarity value between the depth features of the support graph and the query graph through a deep matching fusion module, and obtaining a feature graph of the query graph with depth information enhanced according to the cosine similarity value;

[0013] The original feature map of the support graph and the feature map of the query graph enhanced by the depth information are sent to the candidate region generation network to generate a series of candidate regions, and the RoI Align operation is applied to each candidate region to generate a corresponding query RoI feature. At the same time, the RoI Align operation is applied to the support graph feature to generate a support RoI feature, and the initial score is obtained by calculating the similarity between the query RoI feature and the support RoI feature;

[0014] Inputting the query graph into a CLIP image encoder, the CLIP image encoder outputs CLIP features of the query graph, and generating CLIP RoI features from the CLIP features by using the candidate region through a RoI Align operation;

[0015] Extract attribute words related to the current railway tools to be detected from the query RoI features through an attribute parser, input the attribute words into a CLIP text encoder, the CLIP text encoder outputs attribute word features related to the current railway tools to be detected, and obtains updated text features of the railway tools category name through a Star-Transformer;

[0016] The cosine similarity is used to calculate the similarity between the CLIP RoI features of the query graph and the text features of the railway tool category name to obtain a new matching score, and the matching score is weighted and corrected with the initial score to obtain a final matching score. The target is determined and identified based on the final matching score and the tool coordinates to obtain the identification result of the railway tools on the maintenance section.

[0017] Preferably, the railway tool images taken during the offline maintenance process are used as query graphs, and the railway tool images from the tool warehouse are used as support graphs. The support graph and the query graph are subjected to feature extraction through a backbone network to obtain original feature graphs of the support graph and the query graph, including:

[0018] The railway tool images taken during the actual offline maintenance process are used as query graphs, and the railway tool images from the tool warehouse are used as support graphs. The support graph and the query graph are subjected to feature extraction through a backbone network to obtain multi-level feature graphs of the support graph and the query graph, respectively. The multi-level feature graphs contain spatial information and semantic information. The spatial information describes the position and structure of objects in the image, and the semantic information describes the category of objects in the image. The multi-level feature graphs of the support graph and the query graph are used as the original feature graphs of the support graph and the query graph.

[0019] Preferably, the inputting of the original feature map of the query graph and the original feature map of the support graph into a depth estimation network, wherein the depth estimation network extracts depth features of the query graph and the support graph using a depth perception algorithm, comprises:

[0020] The original feature map of the query map and the original feature map of the support map are input into a depth estimation network, and the depth estimation network uses a Depth Anything algorithm to process the original feature map of the query map and the original feature map of the support map to generate preliminary depth maps of the query map and the support map, and performs noise filtering and edge preservation post-processing on the preliminary depth maps of the query map and the support map to obtain depth features of the query map and the support map.

[0021] Preferably, the step of calculating the cosine similarity value between the depth features of the support graph and the query graph by a deep matching fusion module, and obtaining the feature graph of the query graph with enhanced depth information according to the cosine similarity value comprises:

[0022] The cosine similarity value between the deep features of the support graph and the query graph is calculated through the deep matching fusion module, and the depth information related to the target of the support graph in the query graph is selected according to the cosine similarity value. The selected depth information is highly correlated with the target area. The selected depth information is spliced ​​with the original feature graph of the query graph in the channel dimension to obtain a feature graph of the query graph with depth information enhancement. The feature graph of the query graph with depth information enhancement includes the original feature information of the query graph extracted by the backbone network, and the depth information of the query graph related to the target of the support graph.

[0023] Preferably, the original feature map of the support graph and the feature map of the query graph enhanced by the depth information are sent to a candidate region generation network to generate a series of candidate regions, a RoI Align operation is applied to each candidate region to generate a corresponding query RoI feature, and at the same time, a RoI Align operation is applied to the support graph feature to generate a support RoI feature, and an initial score is obtained by calculating the similarity between the query RoI feature and the support RoI feature, including:

[0024] The original feature map of the support image and the feature map of the query image enhanced by the depth information are fed into a candidate region generation network, which generates a series of candidate regions in the query image based on the original feature map of the support image and the feature map of the query image enhanced by the depth information using a sliding window technique, and calculates the possibility of each box containing the target and bounding box regression;

[0025] The RoI Align operation extracts corresponding features from the query graph features enhanced by depth information according to a series of candidate regions to generate query RoI features, and directly performs RoI Align operation on the feature map of the support graph to generate support RoI features;

[0026] The initial score is obtained by calculating the similarity between the query RoI feature and the supporting RoI feature.

[0027] Preferably, the step of inputting the query graph into a CLIP image encoder, the CLIP image encoder outputting CLIP features of the query graph, and generating CLIP RoI features from the CLIP features by using the candidate regions through a RoI Align operation comprises:

[0028] Inputting the query graph into a CLIP image encoder, the CLIP image encoder extracts the overall features of the query graph, and adjusting the overall features of the query graph through a multi-scale Adapter to obtain CLIP features of the query graph;

[0029] Divide the CLIP features of the query graph into regions using the series of candidate regions, apply a RoI Align operation to each candidate region, and generate a CLIP RoI feature for each candidate region;

[0030] The CLIP RoI feature of each candidate region is updated using a multi-level refinement network to obtain the CLIP RoI feature.

[0031] Preferably, the attribute words related to the railway tools to be detected are extracted from the query RoI features by the attribute parser, and the attribute words are input into the CLIP text encoder, and the CLIP text encoder outputs the attribute word features related to the railway tools to be detected, and the text features of the updated railway tools category name are obtained by Star-Transformer, including:

[0032] Extract multiple attribute words related to the railway tools to be detected from the query RoI features through the attribute parser, and the multiple attribute words include the area, aspect ratio, mean and variance of the box;

[0033] Use CLIP text encoder to convert each attribute word into corresponding attribute word features;

[0034] Put the text features of the railway tool category name into the root node of Star-Transformer, and put the features of each attribute word into the external node of Star Transformer;

[0035] The text features of the railway tool category names are updated using other attribute word features, and the updated text features of the railway tool category names are output.

[0036] Preferably, the method of using cosine similarity to calculate the similarity between the CLIP RoI feature of the query graph and the text feature of the railway tool category name to obtain a new matching score, weighting and correcting the matching score and the initial score to obtain a final matching score, determining and identifying the target based on the final matching score and the tool coordinates to obtain the identification result of the railway tool on the maintenance section includes:

[0037] Use cosine similarity to calculate the similarity between CLIP RoI features and the updated text features of railway tool category names to obtain a new matching score;

[0038] Compare the new matching score with the initial score, make weighted corrections, and get the final matching score;

[0039] The query RoI features are detected through the regression network to obtain the coordinates of railway tools;

[0040] The target is determined and identified based on the final matching score and the coordinates of the railway tools and machinery, and the identification result of the railway tools and machinery on the railway maintenance section is obtained. The identification result includes the coordinates of the detected tools and machinery and the corresponding confidence scores.

[0041] It can be seen from the technical solutions provided by the above embodiments of the present invention that the present invention aims to solve multiple problems existing in the prior art by introducing an intelligent counting method for railway tools and equipment on and off the line based on deep learning, thereby improving the counting efficiency of tools and equipment and reducing manual operation time.

[0042] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0044] Figure 1 A schematic diagram of a method for intelligently counting and checking railway tools and equipment on and off the line provided by an embodiment of the present invention;

[0045] Figure 2A processing flow chart of a method for performing intelligent inventory and verification of railway tools and equipment on and off the line provided by an embodiment of the present invention;

[0046] Figure 3 A multi-level refinement feature update graph is provided in an embodiment of the present invention.

[0047] Figure 4 A schematic diagram of updating a Star-Transformer attribute graph provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0049] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0050] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0051] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0052] Meta-learning refers to a method in machine learning that improves the performance of an algorithm on a new task by learning from the experience of multiple tasks. It aims to solve the problem of small sample learning by quickly adapting to data from a small number of samples.

[0053] The present invention provides a method for intelligently counting and checking railway tools on and off the line based on deep learning. The implementation principle of the method is as follows: Figure 1 The specific processing flow is as shown in Figure 2 As shown, the processing steps include the following:

[0054] Step S10: feature map extraction.

[0055] The support graph and query graph represent the standardized reference image and the image to be detected in the actual application scenario, respectively. The support graph usually comes from images taken in the railway tool warehouse. Each image contains only one tool, and each type of tool has a support image taken from the warehouse. These support graphs contain accurate feature information about the target. The query graph is an image of the tool taken during the actual offline maintenance process, which may contain multiple tools. By utilizing the feature information in the support graph, the tools in the query graph can be detected and identified for effective management. First, the support graph and query graph are feature extracted through the backbone network. The backbone network can use the ResNet101 neural network to ensure the depth and breadth of feature extraction. ResNet101 is a deep residual network that solves the gradient vanishing problem of deep neural networks by introducing residual connections. After the support graph and query graph pass through the backbone network, multi-level feature graphs are obtained respectively. These feature graphs contain rich spatial and semantic information, which is conducive to accurate matching and deep fusion in subsequent processing steps.

[0056] Spatial information describes the location and structure of objects in an image. Through convolution operations, neural networks can capture features at different locations in an image, such as edges, corners, and textures. Spatial information helps locate and identify railroad tools. For example, through spatial information, the shape, location, and relative size of a wrench can be determined to distinguish it from other tools. Semantic information describes the category of objects in an image. Through feature extraction at a deep network level, neural networks are able to understand high-level semantic concepts of objects in an image, such as recognizing that this is a wrench instead of a screwdriver.

[0057] Specifically, the feature maps extracted from the support graph and query graph through the ResNet101 network include multiple feature layers from shallow to deep. Shallow feature maps usually contain more spatial details, while deep feature maps focus more on high-level semantic information. Such multi-level feature representation can more comprehensively describe the image content and lay the foundation for subsequent deep information extraction and matching fusion.

[0058] Step S20: Extract deep features of the query graph and the support graph.

[0059] The above multi-level feature maps are input into the depth perception algorithm to extract depth information. Specifically, the Depth Anything algorithm is used to obtain pixel-level depth information of the image through the depth estimation network. The Depth Anything algorithm is based on a deep learning model and can infer the depth information of the scene from a monocular image. The acquired depth information is processed by noise filtering and edge preservation to improve the accuracy and robustness of the depth feature.

[0060] Specifically, the depth estimation network first processes the feature map to generate a preliminary depth map. Then, a noise filtering algorithm is used to remove noise from the depth map, such as median filtering and bilateral filtering. At the same time, edge-preserving post-processing techniques are used to ensure that important edge information is preserved in the depth map. These steps help improve the accuracy of the depth information, making subsequent depth matching and fusion more reliable.

[0061] First, the Depth Anything algorithm uses a depth estimation network to process the input image to generate a preliminary depth map. The depth estimation network is a deep learning-based model that can predict the depth information of each pixel from a monocular image. Next, the generated depth map usually contains some noise, which may come from interference or other factors during the image capture process. In order to improve the accuracy of the depth information, the algorithm uses noise filtering technology to process the depth map. After noise filtering, the algorithm also performs edge preservation post-processing. The purpose of this step is to ensure that important edge information in the depth map is preserved, which is crucial for accurate depth information. Through the above steps, high-quality depth information is obtained, which provides reliable data support for subsequent depth matching and feature fusion.

[0062] Step S30: Depth matching fusion

[0063] The cosine similarity value between the deep features of the support image and the query image is calculated through the deep matching fusion module. The depth information related to the target of the support image in the query image is selected according to the cosine similarity value to ensure that the selected depth information is highly correlated with the target area. The selected depth information is spliced ​​with the original feature map of the query image in the channel dimension to obtain the feature map of the query image with enhanced depth information.

[0064] The feature map enhanced with depth information refers to the feature map of the query map. This feature map contains the original feature information of the query map extracted by the backbone network, as well as the depth information of the query map related to the support map object. Since the feature map enhanced with depth information only processes the query map and does not directly interact with the feature information and depth information of the support map, the feature map enhanced with depth information does not contain any information about the support map.

[0065] Example: Assume that the support image is an image of a wrench, and the query image contains a variety of tools, including wrenches. Deep Anything is used to extract the depth information of the support image and the query image respectively, where the depth information of the support image only contains the relevant features of the wrench, while the depth information of the query image covers the depth features of various tools. By calculating the cosine similarity between the depth information of the support image and the query image, the depth information of the wrench (related to the support image target) can be selected from the query image. Then, the selected depth information is concatenated with the original feature information of the query image in the channel dimension to obtain a feature map with enhanced depth information.

[0066] Step S40: Candidate region generation and RoI feature extraction

[0067] The input data of this step includes the original feature map of the support map obtained by step S10, and the query map feature map enhanced by depth information generated by step S30. The output data of this step includes a series of candidate regions generated by the candidate region generation network, and the support RoI features and query RoI features generated by the RoI Align operation.

[0068] The original feature map of the support graph and the feature map of the query graph enhanced with depth information are fed into the candidate region generation network to generate a series of candidate regions, and then the RoI Align operation is used to extract features from these candidate regions to generate query RoI features. The specific process is as follows:

[0069] 1.RPN (Region Proposal Network) generates a series of candidate regions in the query graph based on the query graph features enhanced with support graph features and depth information. These regions may contain targets. The RPN network uses a sliding window technique to generate candidate boxes on the feature map and calculates the possibility of each box containing the target and the precise bounding box regression.

[0070] 2. The RoI Align operation accurately extracts the corresponding features from the query map features enhanced by depth information based on a series of candidate regions to generate query RoI features. RoI Align uses bilinear interpolation technology to ensure that the feature values ​​of the feature map in the RoI area are not affected by boundary effects, ensuring the accuracy of feature extraction.

[0071] 3. Similarly, perform RoI Align operation directly on the feature map of the support map to generate supporting RoI features.

[0072] 4. Obtain the initial score by calculating the similarity between the query RoI feature and the supporting RoI feature. The cosine similarity method can be used for similarity calculation.

[0073] Step S50: CLIP (Contrastive Language-Image Pre-Training) feature processing.

[0074] The input data of this step includes the query graph, and the output data of this step is the CLIP RoI feature updated by the multi-level refinement network.

[0075] The query graph features are extracted through the CLIP image encoder and adjusted through the multi-scale Adapter to adapt to the current application scenario. The specific process is as follows:

[0076] 1. Use the CLIP image encoder to extract the overall features of the query graph and generate a preliminary high-dimensional feature representation. The CLIP model is a multimodal model that can map images and text into the same feature space, thereby achieving cross-modal feature comparison.

[0077] 2. The overall features of the extracted query graph are adjusted through a multi-scale adapter. The multi-scale adapter consists of multiple parallel convolution branches, each of which contains convolution kernels of different sizes (such as 3x3, 5x5, 7x7, etc.) to enhance the expressiveness of the features. Multi-scale convolution can capture feature information of different scales in the image and improve the richness of feature representation.

[0078] 3. The query graph features extracted by CLIP are divided into regions using a series of candidate regions generated in step S40, and a RoI Align operation is applied to each candidate region, thereby generating a CLIP RoI feature for each candidate region.

[0079] 4. Use the multi-level refinement network to update the CLIP RoI features. The multi-level refinement network uses multiple layers of graph convolution (GCN) to gradually refine and enhance the features. Figure 3 A multi-level refinement feature update graph is provided in an embodiment of the present invention.

[0080] Step S60: Create and update the property graph

[0081] The input data of this step is the query RoI feature generated by step S40. The output data of this step is the text feature of the tool category name after being updated by the Star-Transformer network.

[0082] The attribute parser is used to statistically query multiple attributes of the RoI feature (such as the area of ​​the box, aspect ratio, mean and variance, etc.), and form an attribute word set {railway tool category name, box area, box aspect ratio, mean and variance}. The specific process is as follows:

[0083] 1. The attribute parser extracts relevant attribute words from the query RoI features, such as the area, aspect ratio, mean, and variance of the box. These attribute words can be used to describe the characteristics of the target area in a more detailed way.

[0084] 2. Use the CLIP text encoder to convert each attribute word into the corresponding attribute word feature.

[0085] 3. Put the text features of the railway tool category names into the root node of Star-Transformer, and put the features of each attribute word into the external node of Star-Transformer. Star-Transformer is a network used to process graph structured data, which can efficiently transfer and update information between graph nodes.

[0086] 4. Use other attribute word features to update the text features of the railway tool category name, thereby improving the feature representation of the railway tool category name. Figure 4 A schematic diagram of updating a Star-Transformer attribute graph provided by an embodiment of the present invention.

[0087] Step S70: Similarity calculation and score correction

[0088] The input data of this step includes the updated CLIP RoI feature generated by step S50, the text feature of the updated tool category name generated by step S60, and the initial score Score obtained by step S40. The output data of this step is the weighted and corrected score Score. The similarity of the updated feature map and the text feature is calculated to obtain a new score, and the initial score is corrected. The specific process is as follows:

[0089] 1. Use cosine similarity to calculate the similarity between the updated CLIP RoI features and the text features of the tool category name to obtain a new matching score.

[0090] 2. Compare the new matching score with the initial score and make weighted corrections to get the final matching score.

[0091] 3. Query the RoI features and obtain the detected tool coordinates through the regression network

[0092] 4. Determine and identify the target based on the final matching score and tool coordinates to ensure high accuracy and robustness of intelligent inventory verification.

[0093] The final result is the identification result of the railway tools and equipment in the maintenance section. These identification results include the coordinates of the detected tools and equipment and the corresponding confidence scores. Figure 1This process was demonstrated to ensure accurate identification of various railway tools and machinery within the maintenance section.

[0094] In summary, the embodiment of the present invention introduces a method for intelligent counting of railway tools and equipment on and off the line based on meta-learning and CLIP, and uses deep learning algorithms to greatly improve the accuracy of recognition and counting efficiency. In particular, when processing a large number of tools and equipment in batches, it can maintain efficient operation and significantly reduce manual operation time and bit error rate.

[0095] The present invention aims to solve several problems existing in the prior art by introducing an intelligent inventory method for railway tools and equipment on and off the line based on deep learning. First, the present invention greatly improves the efficiency of tool inventory and significantly reduces manual operation time. At the same time, the deep learning algorithm significantly reduces the bit error rate and improves the accuracy of recognition, especially in complex environments. The limitations of barcode and RFID technology are effectively overcome. Secondly, the present invention speeds up the recognition speed to ensure that efficient operation can be maintained when processing a large number of tools and equipment. In addition, by reducing the reliance on physical tags, the present invention effectively reduces maintenance costs and reduces the need for equipment maintenance and replacement. Finally, the present invention reduces the reliance on manual experience, reduces training costs, ensures the continuity and consistency of tool management, and ultimately realizes the modernization, efficiency and intelligence of tool management in railway engineering sections.

[0096] The present invention reduces maintenance requirements and costs by reducing reliance on physical labels. Barcodes and RFID tags have high maintenance and replacement costs, especially in harsh environments where they are easily damaged. By utilizing the image recognition technology of the present invention, there is no need to frequently replace and maintain labels, which significantly reduces the operating costs of the equipment. At the same time, the intelligent inventory method reduces reliance on manual experience, reduces training costs, and ensures the continuity and consistency of management work. Even in the case of high personnel mobility, efficient operation of tool management can be guaranteed. Through the above advantages, the present invention has demonstrated significant technical advantages and application value in the field of railway tool management, and has realized the modernization, efficiency, and intelligence of tool management.

[0097] First, in terms of depth information extraction methods, in addition to using the Depth Anything algorithm, other depth estimation methods can also be used. For example, Monodepth is an algorithm that uses monocular images for depth estimation, and DPT (DensePrediction Transformer) performs dense prediction based on Transformers. LiDAR or binocular cameras can also be used to directly obtain depth information to reduce computational complexity and errors.

[0098] In terms of network refinement, in addition to using graph convolutional networks (GCN), you can also use Graph Attention Network (GAT, a graph convolutional network based on attention mechanism), GraphSAGE (graph convolution through samples and aggregated node features) and Spectral GCN (graph convolution method based on spectral domain convolution) for feature refinement and enhancement.

[0099] Regarding the attribute graph establishment and update method, in addition to using CLIP text encoder and Star Transformer, you can also use BERT or GPT-3 (Transformer-based text encoding model) to generate attribute graphs, use GNN (Graph Neural Network) to process graph structure data, and update and fuse attribute features through the Attention mechanism.

[0100] Through these alternatives, we can expand the scope of protection of the patent, ensure the diversity and flexibility of the technical solutions, and prevent others from circumventing the invention to achieve the same purpose.

[0101] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0102] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0103] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0104] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for intelligently counting and checking railway tools and equipment, characterized in that: include: The railway tool images taken during the offline maintenance process are used as query images, and the railway tool images from the tool warehouse are used as support images. The backbone network is used to extract features from the support image and the query image to obtain the original feature images of the support image and the query image. Inputting the original feature map of the query graph and the original feature map of the support graph into a depth estimation network, wherein the depth estimation network extracts depth features of the query graph and the support graph using a depth perception algorithm; Calculating the cosine similarity value between the depth features of the support graph and the query graph through a deep matching fusion module, and obtaining a feature graph of the query graph with depth information enhanced according to the cosine similarity value; The original feature map of the support graph and the feature map of the query graph enhanced by the depth information are sent to the candidate region generation network to generate a series of candidate regions, and the RoI Align operation is applied to each candidate region to generate a corresponding query RoI feature. At the same time, the RoI Align operation is applied to the support graph feature to generate a support RoI feature, and the initial score is obtained by calculating the similarity between the query RoI feature and the support RoI feature; Inputting the query graph into a CLIP image encoder, the CLIP image encoder outputs CLIP features of the query graph, and generating CLIP RoI features from the CLIP features by using the candidate region through a RoI Align operation; Extract attribute words related to the current railway tools to be detected from the query RoI features through an attribute parser, input the attribute words into a CLIP text encoder, the CLIP text encoder outputs attribute word features related to the current railway tools to be detected, and obtains updated text features of the railway tools category name through a Star-Transformer; The cosine similarity is used to calculate the similarity between the CLIP RoI features of the query graph and the text features of the railway tool category name to obtain a new matching score, and the matching score is weighted and corrected with the initial score to obtain a final matching score. The target is determined and identified based on the final matching score and the tool coordinates to obtain the identification result of the railway tools on the maintenance section.

2. The method according to claim 1, characterized in that The railway tool images taken during the offline maintenance process are used as query images, and the railway tool images from the tool warehouse are used as support images. The support image and the query image are extracted through the backbone network to obtain the original feature images of the support image and the query image, including: The railway tool images taken during the actual offline maintenance process are used as query graphs, and the railway tool images from the tool warehouse are used as support graphs. The support graph and the query graph are subjected to feature extraction through a backbone network to obtain multi-level feature graphs of the support graph and the query graph, respectively. The multi-level feature graphs contain spatial information and semantic information. The spatial information describes the position and structure of objects in the image, and the semantic information describes the category of objects in the image. The multi-level feature graphs of the support graph and the query graph are used as the original feature graphs of the support graph and the query graph.

3. The method according to claim 1, characterized in that The step of inputting the original feature map of the query graph and the original feature map of the support graph into a depth estimation network, wherein the depth estimation network extracts depth features of the query graph and the support graph using a depth perception algorithm, comprises: The original feature map of the query map and the original feature map of the support map are input into a depth estimation network, and the depth estimation network uses a Depth Anything algorithm to process the original feature map of the query map and the original feature map of the support map to generate preliminary depth maps of the query map and the support map, and performs noise filtering and edge preservation post-processing on the preliminary depth maps of the query map and the support map to obtain depth features of the query map and the support map.

4. The method according to claim 1, characterized in that: The method of calculating the cosine similarity value between the depth features of the support graph and the query graph through the deep matching fusion module, and obtaining the feature graph of the query graph with enhanced depth information according to the cosine similarity value, includes: The cosine similarity value between the deep features of the support graph and the query graph is calculated through the deep matching fusion module, and the depth information related to the target of the support graph in the query graph is selected according to the cosine similarity value. The selected depth information is highly correlated with the target area. The selected depth information is spliced ​​with the original feature graph of the query graph in the channel dimension to obtain a feature graph of the query graph with depth information enhancement. The feature graph of the query graph with depth information enhancement includes the original feature information of the query graph extracted by the backbone network, and the depth information of the query graph related to the target of the support graph.

5. The method according to claim 1, characterized in that The original feature map of the support map and the feature map of the query map enhanced by the depth information are sent to the candidate region generation network to generate a series of candidate regions, and the RoI Align operation is applied to each candidate region to generate a corresponding query RoI feature. At the same time, the RoI Align operation is applied to the support map feature to generate the support RoI feature, and the initial score is obtained by calculating the similarity between the query RoI feature and the support RoI feature, including: The original feature map of the support image and the feature map of the query image enhanced by the depth information are fed into a candidate region generation network, which generates a series of candidate regions in the query image based on the original feature map of the support image and the feature map of the query image enhanced by the depth information using a sliding window technique, and calculates the possibility of each box containing the target and bounding box regression; The RoI Align operation extracts corresponding features from the query graph features enhanced by depth information according to a series of candidate regions to generate query RoI features, and directly performs RoI Align operation on the feature map of the support graph to generate support RoI features; The initial score is obtained by calculating the similarity between the query RoI feature and the supporting RoI feature.

6. The method according to claim 5, characterized in that The step of inputting the query graph into a CLIP image encoder, wherein the CLIP image encoder outputs CLIP features of the query graph, and generating CLIP RoI features from the CLIP features by using the candidate region through a RoI Align operation, comprises: Inputting the query graph into a CLIP image encoder, the CLIP image encoder extracts the overall features of the query graph, and adjusting the overall features of the query graph through a multi-scale Adapter to obtain CLIP features of the query graph; The CLIP feature of the query graph is divided into regions by the series of candidate regions, and a RoI Align operation is applied to each candidate region to generate a CLIP RoI feature of each candidate region; The CLIP RoI feature of each candidate region is updated using a multi-level refinement network to obtain the CLIP RoI feature.

7. The method according to claim 6, characterized in that The attribute parser extracts attribute words related to the current railway tools to be detected from the query RoI features, inputs the attribute words into the CLIP text encoder, and the CLIP text encoder outputs attribute word features related to the current railway tools to be detected, and obtains the updated text features of the railway tools category name through the Star-Transformer, including: Extract multiple attribute words related to the railway tools to be detected from the query RoI features through the attribute parser, and the multiple attribute words include the area, aspect ratio, mean and variance of the box; Use CLIP text encoder to convert each attribute word into corresponding attribute word features; Put the text features of the railway tool category name into the root node of Star-Transformer, and put the features of each attribute word into the external node of Star Transformer; The text features of the railway tool category names are updated using other attribute word features, and the updated text features of the railway tool category names are output.

8. The method according to claim 7, characterized in that The method of using cosine similarity to calculate the similarity between the CLIP RoI feature of the query graph and the text feature of the railway tool category name to obtain a new matching score, weighting and correcting the matching score and the initial score to obtain a final matching score, determining and identifying the target based on the final matching score and the tool coordinates, and obtaining the identification result of the railway tool on the maintenance section includes: Use cosine similarity to calculate the similarity between CLIP RoI features and the updated text features of railway tool category names to obtain a new matching score; Compare the new matching score with the initial score, make weighted corrections, and get the final matching score; The query RoI features are detected through the regression network to obtain the coordinates of railway tools; The target is determined and identified based on the final matching score and the coordinates of the railway tools and machinery, and the identification result of the railway tools and machinery on the railway maintenance section is obtained. The identification result includes the coordinates of the detected tools and machinery and the corresponding confidence scores.

Images