Intelligent track foreign matter identification method based on Segment Anything and open set classification

By combining the Segment Anything model and the open set classification model, the problem of insufficient segmentation accuracy and unknown foreign object processing capabilities of orbital foreign object recognition in complex backgrounds is solved, and efficient and accurate orbital foreign object recognition is achieved, reducing false positives and missed reports The rate of the rail transit system improves the efficiency and accuracy of safety inspection.

CN118982706APending Publication Date: 2024-11-19SHANDONG ZHIYANG HUITONG DIGITAL TECH CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411050663.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The existing orbital foreign object recognition technology has insufficient segmentation accuracy in complex backgrounds and cannot effectively deal with unknown foreign objects. The resource occupancy and false alarm rate are high, which affects the accuracy and efficiency of security detection.

Method used

The image segmentation and open set classification model based on the Segment Anything model are used to identify significant differences by comparing the segmentation mask, and the unknown foreign objects are identified in combination with the OpenMax algorithm to reduce dependence on pre-trained models and reduce the missed and false alarm rates.

Benefits of technology

It improves the accuracy and efficiency of track foreign objects recognition, reduces labor costs, realizes effective identification of complex backgrounds and unknown foreign objects, and enhances the safety detection capabilities of the rail transit system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982706A_ABST
    Figure CN118982706A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of track safety monitoring, and particularly relates to a track foreign matter intelligent identification method based on Segment Anything and open set classification, a storage medium, a program product and computer equipment. The method comprises the following steps: segmenting a track image without hidden danger according to a preset frame prompt by using an SAM model, and obtaining a background image segmentation mask; segmenting the track image to be identified by using the same frame prompt to obtain a segmentation mask of the image to be identified; comparing the two segmentation masks, if an obvious difference is found, taking a local image of a corresponding area as a suspected hidden danger image, and sending the suspected hidden danger image into a track foreign matter open set classification model for further judgment; the classification model identifies whether the suspected hidden danger map belongs to a background, a known hidden danger or an unknown foreign matter class, and gives an alarm to the suspected hidden danger map classified as a hidden danger or a foreign matter in time. By combining the image segmentation capability of the SAM model and the foreign matter classification capability of the open set classification model, the accuracy and efficiency of foreign matter recognition can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of track safety monitoring, and in particular relates to a track foreign body intelligent recognition method based on Segment Anything and open set classification, a storage medium and a computer device. Background Art

[0002] In rail transit systems, a key link to ensure the safe operation of trains is to accurately identify foreign objects on the track. Traditional methods of identifying foreign objects on the track are not only labor-intensive and inefficient, but also have obvious shortcomings in terms of accuracy. In recent years, with the rapid development of computer vision technology and machine learning technology, track foreign object identification has ushered in a new opportunity. Despite this, there are still some problems with current methods: (1) Segmentation accuracy: The accuracy of image segmentation in existing technologies is limited, especially for complex backgrounds or when the color and texture of foreign objects are similar to those of the track background, the segmentation effect is poor. This will affect subsequent foreign object identification and hidden danger detection, reducing the overall safety detection accuracy. (2) Limited processing capabilities for unknown foreign objects: In actual applications, there are many types of foreign objects that may appear on the track, and it is impossible for all of them to be included in the training data set. However, existing technologies rely too much on pre-trained models and data sets. If the type of foreign object is not in the training data set, or the foreign object feature is not similar to the category in the training set, it may lead to a high false alarm rate, which will affect the accuracy and efficiency of track safety detection. (3) Resource usage and false alarm rate issues: Due to changes in image quality and the instability of the segmentation model, existing technologies will increase the usage of computing resources and the workload of manual compounding due to slight segmentation differences. They will also trigger false alarms due to unnecessary slight differences, thus interfering with the normal security monitoring process. Summary of the invention

[0003] The purpose of the present invention is to provide a method for intelligently identifying foreign objects on tracks based on Segment Anything and open set classification. By combining the image segmentation capability of the SAM model and the foreign object classification capability of the open set classification model, the present invention aims to overcome the limitations of traditional foreign object identification methods on tracks, improve the accuracy and efficiency of foreign object identification, thereby ensuring the safe operation of the rail transit system, reducing labor costs, and providing more reliable technical support for track safety detection.

[0004] The embodiment of the present invention discloses a method for intelligently identifying track foreign objects based on Segment Anything and open set classification, comprising the following steps:

[0005] S1. Obtain a track image without hidden dangers, input the track image without hidden dangers into the SAM model, and input the region of interest as a box prompt into the SAM model. The SAM model performs image segmentation on the track image without hidden dangers according to the box prompt to obtain a background image segmentation mask;

[0006] S2, obtain the track image to be identified, input the track image to be identified into the SAM model, and input the box prompt consistent with S1 into the SAM model. The SAM model performs image segmentation on the track image to be identified according to the box prompt to obtain the segmentation mask of the image to be identified;

[0007] S3, comparing the segmentation mask of the image to be identified with the segmentation mask of the background image, if the difference rate of the proportion of a certain color pixel area located in the same region of interest in the two reaches a preset threshold, then the local image of the region of interest in the image to be identified is used as a suspected hidden danger image and input into the trained track foreign body open set classification model;

[0008] S4, the track foreign body open set classification model classifies the received suspected hidden danger image to be identified to determine whether it belongs to the background class, the known hidden danger class or the unknown foreign body class;

[0009] S5: For suspected hidden danger images classified as known hidden dangers or unknown foreign objects, push alarms.

[0010] By means of the above technical scheme, the present invention has at least the following advantages: (1) The present invention uses the SAM model (SegmentAnything Model) for image segmentation, which can maintain a high segmentation accuracy in the case of complex background and foreign objects close to the track background. (2) The track image may contain a large amount of background information, which is relatively stable and consistent in multiple recognition processes. By first comparing the segmentation mask, the area with significant difference from the background can be quickly screened out, thereby avoiding unnecessary classification processing of a large number of unchanged background areas. This can significantly improve the processing efficiency of the recognition method and reduce the waste of computing resources. (3) Changes in image quality and instability of the SAM model segmentation results may lead to slight differences in the segmentation results. Changes in image quality may be caused by factors such as differences in acquisition equipment, different lighting conditions or distortion in image transmission, which will cause fluctuations in the results during the segmentation process; even if the image quality remains unchanged, the results of the SAM segmentation model still have certain instability. In order to address these problems, the present invention first compares the segmentation mask to accurately identify the parts with significant differences, and then excludes the slight differences caused by image quality or model instability, effectively reducing the false alarm rate triggered by these slight differences. (4) The present invention combines an open set classification model, which can not only effectively identify known hidden danger classes, but also identify unknown foreign objects that are not in the training data set. This method reduces the dependence on pre-trained models and data sets, improves the system's ability to handle unknown foreign objects, and thus reduces the false negative rate. (5) Through an automated intelligent recognition method, the present invention can monitor the foreign object situation on the track in real time, promptly detect and deal with safety hazards, thereby improving the efficiency and accuracy of track safety detection. At the same time, it reduces labor costs and improves work efficiency.

[0011] Preferably, the embodiment of the present invention further discloses that the step of inputting the region of interest as a frame prompt into the SAM model comprises:

[0012] Based on the known track protection zone information, a mask of the track area is generated, and the upper and lower boundaries of the track area are determined accordingly;

[0013] By calculating the width of the track area at different y coordinates, the linear relationship between the track width and the y coordinate is fitted;

[0014] Traverse each row of the track area, calculate the width of the track based on the fitted linear relationship, and deduce the width of the candidate box based on it;

[0015] In each track region segment, a candidate frame is generated according to a preset step length to obtain a candidate frame covering the track region;

[0016] The candidate boxes are fed into the SAM model as box hints.

[0017] The above candidate box generation method has the following advantages: (1) Improving the accuracy of the box prompt: By generating a mask of the track area based on the known track protection zone information and determining the upper and lower boundaries of the track area based on this, it is possible to ensure that the box prompt is closely aligned with the actual track area. This precise box prompt can significantly improve the accuracy of the SAM model in image segmentation and reduce the occurrence of mis-segmentation and missed segmentation. (2) Optimizing the efficiency of box prompt generation: By fitting the linear relationship between the track width and the y coordinate and traversing each row of the track area to deduce the width of the candidate box, it is possible to efficiently generate a candidate box covering the track area. This method avoids complex image processing and calculations, simplifies the box prompt generation process, and improves the overall algorithm operation efficiency. (3) Enhancing the flexibility and adaptability of the method: The size and position of the candidate box can be automatically adjusted according to different track areas to adapt to changes in various track shapes and sizes. This flexibility and adaptability enable the method to be widely used in different types of rail transit systems without the need for a large number of customized adjustments. (4) Improving the reliability of foreign object recognition: Accurate box prompts can ensure that the SAM model can more accurately identify foreign objects in the track area and reduce background interference and misidentification. This further improves the reliability of foreign object identification and provides strong guarantees for the safe operation of the rail transit system.

[0018] Preferably, the embodiment of the present invention further discloses that the steps of constructing the track foreign matter open set classification model include:

[0019] Get orbital image dataset;

[0020] The track image in the data set is input into the SAM model, and the box prompt consistent with S1 is input into the SAM model. The SAM model performs image segmentation on the track image in the data set according to the box prompt to obtain a segmentation mask;

[0021] The SAM model extracts the local image of the segmentation mask area on the track image in the dataset through the segmentation mask;

[0022] Manually screen and classify the extracted partial images, and mark the partial images as background and hidden danger categories, wherein the background category refers to an area without any foreign matter, and the hidden danger category refers to an area with foreign matter;

[0023] The pre-trained ResNet-34 model is used as the basis, and the closed-set classification training of the ResNet-34 model is performed using a dataset containing background and hidden danger labeling information;

[0024] After completing the closed-set classification training of the model, the OpenMax algorithm is introduced to allow the model to learn to distinguish known hidden danger categories and unknown foreign object categories, and obtain an open-set classification model for track foreign objects.

[0025] Preferably, the embodiment of the present invention further discloses that, in S1, after obtaining a track image without hidden dangers, the track image without hidden dangers is preprocessed and then input into the SAM model; in S2, after obtaining a track image to be identified, the track image to be identified is preprocessed and then input into the SAM model; the preprocessing step of the track image includes image denoising, image enhancement and image adjustment.

[0026] Preferably, an embodiment of the present invention further discloses that the SAM model includes an image encoder, a prompt encoder and a mask decoder, wherein the image encoder is used to digitize the input track image into a feature tensor, the prompt encoder is used to encode the box prompt into a form combined with the image encoding, and the mask decoder is used to generate a segmentation mask of the track image based on the image encoding and the prompt encoding.

[0027] Preferably, the embodiment of the present invention further discloses that the step S5 includes:

[0028] For suspected hidden danger images classified as known hidden dangers or unknown foreign objects, coordinate transformation is performed to find their positions in the track image to be identified;

[0029] Use OpenCV's cv2.rectangle function to draw the minimum circumscribed rectangular frame in the track image to be identified, highlight the area corresponding to the suspected hidden danger map, and push an alarm.

[0030] Based on the same inventive concept, an embodiment of the present invention also discloses a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the above-mentioned intelligent identification method of track foreign objects based on Segment Anything and open set classification.

[0031] Based on the same inventive concept, an embodiment of the present invention also discloses a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the above-mentioned intelligent identification method of track foreign objects based on Segment Anything and open set classification is implemented.

[0032] Based on the same inventive concept, an embodiment of the present invention also discloses a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the above-mentioned intelligent identification method of track foreign objects based on Segment Anything and open set classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the intelligent method for identifying foreign objects on tracks based on Segment Anything and open set classification in Example 1.

[0034] Figure 2 This is the track image recognition result in the first scene in Example 1.

[0035] Figure 3 This is the track image recognition result in the second scene in Example 1.

[0036] Figure 4 This is a processing flow chart of the image encoder in Example 1.

[0037] Figure 5 This is a processing flow chart of the prompt encoder in Example 1.

[0038] Figure 6 This is a processing flow chart of the mask decoder in Example 1. DETAILED DESCRIPTION

[0039] The embodiment of the present invention proposes a method for intelligent identification of foreign objects on tracks based on Segment Anything and open set classification. The SAM model is used for image segmentation. By comparing the segmentation masks of the track images without hidden dangers and those to be identified, the accuracy of foreign object identification is significantly improved. At the same time, combined with the open set classification model, it can not only effectively identify known hidden dangers, but also identify unknown foreign objects, reduce dependence on training data sets, and reduce false positives and missed negatives. This automated intelligent identification method monitors track safety in real time, improves detection efficiency and accuracy, and saves labor costs.

[0040] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0041] like Figure 1 As shown, in Example 1, a track foreign body intelligent identification method based on Segment Anything and open set classification includes the following steps:

[0042] S1. Get the background image segmentation mask:

[0043] S11. Use image acquisition equipment to obtain images of the track without hidden dangers, including the track itself and the surrounding environment.

[0044] In order to enhance the robustness and generalization of subsequent image processing and analysis, the time nodes for acquiring hazard-free orbital images include day, night, and different seasons.

[0045] S12. In order to improve the image quality and ensure the accuracy of subsequent segmentation, comparison and classification, it is necessary to preprocess the track image without hidden dangers. The preprocessing steps of the track image include image denoising, image enhancement and image adjustment.

[0046] Image denoising: The acquired orbital images may contain noise or interference, which will have a negative impact on subsequent image processing and analysis. Therefore, denoising is required first. Common denoising methods include median filtering, Gaussian filtering, mean filtering, etc., which can effectively reduce the noise in the image and improve the image quality and clarity.

[0047] Image enhancement: To enhance the visual effect of the track image and the ability to identify track foreign objects, enhancement processing can be performed. Common methods include contrast enhancement, brightness adjustment, histogram equalization, etc., to improve the grayscale distribution and contrast of the image, making track foreign objects more obvious and easy to identify in the image.

[0048] Image adjustment: Before image segmentation and classification, it is usually necessary to adjust the image to meet the requirements of subsequent processing. Adjustment includes unifying the image size, converting the color space, etc., to facilitate subsequent image segmentation and feature extraction operations.

[0049] S13, input the preprocessed track image without hidden dangers into the SAM model, and input the region of interest as a box prompt into the SAM model. The SAM model performs image segmentation on the track image without hidden dangers according to the box prompt to obtain a background image segmentation mask.

[0050] In this embodiment, candidate boxes are generated based on known track protection zone information, and these candidate boxes are regarded as regions of interest. When selecting, the track area is given priority, and other local visual positioning methods can be adaptively cascaded with the segmentation algorithm effect of the SAM model.

[0051] The steps to import a track region into a SAM model as a box cue are:

[0052] Based on the known track protection zone information, a mask of the track area is generated, and the upper and lower boundaries of the track area are determined accordingly;

[0053] By calculating the width of the track area at different y coordinates, the linear relationship between the track width and the y coordinate is fitted;

[0054] Traverse each row of the track area, calculate the width of the track based on the fitted linear relationship, and deduce the width of the candidate box based on it;

[0055] In each track region segment, candidate boxes are generated according to a preset step size to obtain candidate boxes covering the track region, and the sizes and positions of these candidate boxes are ensured to be reasonable, that is, the width of the candidate box matches the calculated track width, and the candidate box is completely located in the track region.

[0056] The above candidate box generation method has the following advantages: (1) Improving the accuracy of the box prompt: By generating a mask of the track area based on the known track protection zone information and determining the upper and lower boundaries of the track area based on this, it is possible to ensure that the box prompt is closely aligned with the actual track area. This precise box prompt can significantly improve the accuracy of the SAM model in image segmentation and reduce the occurrence of mis-segmentation and missed segmentation. (2) Optimizing the efficiency of box prompt generation: By fitting the linear relationship between the track width and the y coordinate and traversing each row of the track area to deduce the width of the candidate box, it is possible to efficiently generate a candidate box covering the track area. This method avoids complex image processing and calculations, simplifies the box prompt generation process, and improves the overall algorithm operation efficiency. (3) Enhancing the flexibility and adaptability of the method: The size and position of the candidate box can be automatically adjusted according to different track areas to adapt to changes in various track shapes and sizes. This flexibility and adaptability enable the method to be widely used in different types of rail transit systems without the need for a large number of customized adjustments. (4) Improving the reliability of foreign object recognition: Accurate box prompts can ensure that the SAM model can more accurately identify foreign objects in the track area and reduce background interference and misidentification. This further improves the reliability of foreign object identification and provides strong guarantees for the safe operation of the rail transit system.

[0057] S2. Get the segmentation mask of the image to be identified:

[0058] S21. Using an image acquisition device, obtain an image of the track to be identified, including the track itself and the surrounding environment.

[0059] S22, performing the same preprocessing as S1 on the track image to be identified.

[0060] S23, input the preprocessed track image to be identified into the SAM model, and input the box prompt consistent with S1 into the SAM model. The SAM model performs image segmentation on the track image to be identified according to the box prompt to obtain a segmentation mask of the image to be identified.

[0061] S3. Identify suspected hidden dangers:

[0062] The segmentation mask of the image to be identified is compared with the segmentation mask of the background image. If the difference rate of the proportion of a certain color pixel area located in the same area of ​​interest in the two reaches a preset threshold, the local image of the area of ​​interest in the image to be identified is used as a suspected hidden danger image and input into the trained track foreign object open set classification model.

[0063] In this embodiment, the preset threshold is 30%, and the specific color is white. In a certain comparison scene, the proportion of white pixels in the track area candidate frame of the segmentation mask of the image to be identified is 60%, and the proportion of white pixels in the track area candidate frame of the background image segmentation mask is 30%. After comparison, the difference rate is 30%, reaching the preset threshold. Then, the local image of the track area candidate frame in the image to be identified is used as a suspected hidden danger image and input into the trained track foreign body open set classification model.

[0064] S4. Classification of suspected hidden danger pictures:

[0065] The track foreign object open set classification model classifies the received suspected hidden danger map to determine whether it belongs to the background class, the known hidden danger class or the unknown foreign object class.

[0066] The steps of constructing the track foreign body open set classification model include:

[0067] The image acquisition equipment is used to obtain the track image dataset, including the track itself and the surrounding environment.

[0068] In order to enhance the robustness and generalization of subsequent image processing and analysis, the time nodes of the acquired orbital images include day and night.

[0069] The track images in the dataset are preprocessed in the same way as S1.

[0070] The track images in the dataset are input into the SAM model, and the box prompts consistent with S1 are input into the SAM model. The SAM model performs image segmentation on the track images in the dataset according to the box prompts to obtain a segmentation mask. The segmentation mask is a binary image of the same size as the original image, where the foreground target pixel value is 1 and the background target pixel value is 0.

[0071] The SAM model extracts the local image of the segmentation mask area on the track image in the dataset through the segmentation mask. Specifically, the segmentation mask is matched with its original image in the dataset to ensure that the segmentation mask has the same size and coordinate system as the original image, so that each pixel in the segmentation mask can be accurately mapped to the corresponding position in the original image. Subsequently, the segmentation mask is used to select pixels in the original image. For pixels with a value of 1 in the mask, the corresponding pixels in the original image are selected (or retained); for pixels with a value of 0 in the mask, the corresponding pixels in the original image are ignored (or not selected). It is particularly important to extract only local images of the region of interest, which contain targets where track foreign objects may exist.

[0072] The extracted local images are manually screened and classified, and the local images are marked as background and hidden danger categories, where the background category refers to the area that does not contain any foreign objects, and the hidden danger category refers to the area where foreign objects exist, such as fallen rocks, people, relics on the track, animals, branches, light objects, etc.

[0073] The pre-trained ResNet-34 model is used as the basis for fine-tuning on the dataset, that is, the ResNet-34 model is further trained using the dataset to meet the requirements of the specific task. During the fine-tuning process, the last few fully connected layers of the ResNet-34 model are replaced with new fully connected layers adapted to the specific classification task. Next, the ResNet-34 model is trained for closed-set classification using a dataset containing background and hidden danger labeling information. The model parameters are adjusted through the back-propagation algorithm and optimization method, so that the model can more accurately classify the background and hidden danger classes in the track scene.

[0074] After completing the closed set classification training of the model, the OpenMax algorithm is introduced to enable the model to distinguish between known hidden danger categories and unknown foreign object categories, and obtain the track foreign object open set classification model. OpenMax is an algorithm for open set classification. Its characteristic is that it can take into account the existence of unknown categories during the training process, thereby improving the model's generalization ability for unknown categories. When the OpenMax algorithm is applied for training, the model can learn how to distinguish between known hidden danger categories and unknown foreign object categories. In the process of training the OpenMax model, we can not only use the labeled hidden danger category data, but also introduce unlabeled data. The purpose of this is to improve the model's ability to recognize unknown foreign objects.

[0075] S5: For suspected hidden danger images classified as known hidden dangers or unknown foreign objects, coordinate transformation is performed to find their positions in the track image to be identified, that is, the coordinate information in the mask is used to locate the corresponding area in the original image; then, the cv2.rectangle function of OpenCV is used to draw a minimum circumscribed rectangular frame in the track image to be identified, highlighting the area corresponding to the suspected hidden danger image, and an alarm is pushed.

[0076] Figure 2 This is the track image recognition result in the first scene in this embodiment, and the tool bag on the track is classified as a known hidden danger class “bag”. Figure 2 This is the track image recognition result in the second scenario in this embodiment, and the intruding object on the track is classified as an unknown foreign object (“unknown”).

[0077] The SAM model (Segment Anything Module) is an image segmentation model based on VIT (Vision Transformer) and is used to accurately segment objects and backgrounds in images. The SAM model includes an image encoder, a hint encoder, and a mask decoder.

[0078] The image encoder is used to convert the input track image into a feature tensor. Specifically, Figure 4 The processing flow of the image encoder is shown:

[0079] 1. Input processing:

[0080] The original image is first scaled and padded to a uniform size of 1024x1024.

[0081] Next, the image is processed through a convolutional layer with a kernel size of 16 and a stride of 16. This step discretizes the image into an image tensor with dimensions batch_size x 64x 64x 768.

[0082] 2. Flattening of feature maps:

[0083] The image tensor is sequentially flattened in the width (W) and channel (C) dimensions, converting the original two-dimensional feature map (64x64) into a one-dimensional sequence. This is done to adapt to the input requirements of the Transformer encoder because the Transformer processes sequence data.

[0084] 3. Transformer encoder processing:

[0085] The flattened tensor is fed into a multi-layer Transformer encoder for feature extraction and mapping.

[0086] Each layer of the Transformer encoder consists of a self-attention and feed-forward neural network layer. The self-attention mechanism enables the model to consider the information of all other positions when processing each position in the sequence, thereby capturing global dependencies.

[0087] The feedforward neural network further processes and transforms the output of self-attention to enhance the expressiveness of the features.

[0088] 4. Convolution compression and normalization:

[0089] After being processed by the Transformer encoder, the output vector is compressed through two layers of convolution.

[0090] The kernel size of the first convolution layer is 1, which does not change the spatial size of the feature map, but may adjust the number of feature channels as needed. It is followed by a LayerNorm2d layer to normalize the features, which helps speed up the training process and improve the generalization ability of the model.

[0091] The kernel size of the second convolution layer is 3, which further aggregates the features locally to extract higher-level feature information. Similarly, a LayerNorm2d layer is also connected after this convolution layer for normalization.

[0092] 5. Output:

[0093] Finally, the feature dimension is compressed to 256, outputting a feature tensor of size batch_size x 64x64x 256. This feature tensor contains a high-level feature representation of the image and can be used by subsequent hint encoders and mask decoders.

[0094] The hint encoder is used to encode the frame hint in a form combined with image encoding.

[0095] Figure 5 The processing flow of the prompt encoder is shown:

[0096] 1. Input stage:

[0097] Receive input prompt information, including points, boxes, text, and input masks.

[0098] These cues are encoded into two types of embeddings: sparse embeddings corresponding to discrete elements such as points and boxes, and dense embeddings corresponding to continuous or regional information.

[0099] 2. Dealing with sparse embeddings:

[0100] Sparse embeddings go through a specific processing flow, which may involve operations such as linear transformations and activation functions.

[0101] Make sure the processed sparse embeddings have a specific output shape: batch_size x N x embed_dim, where batch_size represents the batch size, N is determined by the number of input points and boxes, and embed_dim is the embedding dimension (256 in this case).

[0102] 3. Dealing with dense embeddings:

[0103] Dense embeddings go through a series of processing steps that may include convolution, pooling, normalization, etc.

[0104] The processed dense embedding shape is: batch_size x embed_dim x embed_H x embed_W, where embed_H and embed_W are both equal to 64, representing the height and width of the embedded feature map.

[0105] 4. Output stage:

[0106] After the sparse embedding process is completed, it is output in shape batch_size x N x embed_dim.

[0107] After the dense embedding processing is completed, it is output in shape batch_size x embed_dim x embed_H x embed_W.

[0108] In this embodiment, the hint encoder is used to encode the box hint in a form combined with the image encoding, and the box-based hint encoding is mainly applied. The "marked box encoding" module is used to implement the encoding of the box hint. This module performs position encoding on the coordinates of the upper left corner and the lower right corner of the box, and adds the corresponding corner point embedding, so as to obtain a richly expressive box hint embedding.

[0109] The mask decoder is used to generate a segmentation mask of the track image according to the image encoding and the hint encoding. Specifically, Figure 6 The processing flow of the mask decoder is shown:

[0110] The processing steps of the mask decoder are:

[0111] 1. Receive and integrate input data

[0112] Receives the output of the image encoder (image_embeddings), which is a feature map of size 256x64x64, where each 64x64 grid point corresponds to a 256-dimensional feature vector.

[0113] Takes in positional encodings of the same shape and size as image_embeddings, which provide the model with spatial location information.

[0114] Receives sparse embeddings of points and boxes, which represent information about specific points and boxes in the image, from the hint encoder.

[0115] Takes in dense embeddings of mask inputs, which are used to indicate image regions that the model should pay special attention to or ignore.

[0116] 2. Process input data through Transformer blocks

[0117] The integrated input data is fed into two Transformer blocks.

[0118] Inside each Transformer block, self-attention calculation is first performed, allowing the model to pay attention to different parts of the input data and weight them according to their importance.

[0119] Then, a cross-attention calculation is performed to allow token embeddings (composed of sparse and dense embeddings) to interact with image features. This process allows the model to emphasize or weaken specific features in the image based on additional cues (such as points, boxes, masks, etc.).

[0120] Cross-attention is also performed between token embeddings to capture and model the relationships and dependencies between them.

[0121] Finally, the features processed by the attention mechanism are further transformed and learned through a feedforward neural network (MLP) layer.

[0122] 3. Feature upsampling and Mask prediction

[0123] After being processed by the Transformer block, the obtained feature map is upsampled to increase its resolution to match the original image. This step is to prepare for the subsequent fine mask prediction.

[0124] The upsampled feature map is fed into a series of fully connected layers (Mask_mlp) to calculate the probability of each pixel belonging to each segmentation category and generate a preliminary mask prediction.

[0125] 4. Generate final output

[0126] Token embedding of segmentation results: After the above processing steps, the mask decoder outputs token embeddings representing different segmentation results. These embeddings will be used for subsequent segmentation map generation and confidence evaluation.

[0127] Final segmentation map generation: To obtain the final segmentation map, the token embedding of the segmentation result is further processed through the MLP and dot-producted with the original image embedding. This step ensures that the high-level segmentation prediction is aligned with the original image features, thereby generating an accurate segmentation image.

[0128] Confidence evaluation and output: At the same time, the mask decoder also outputs a special iou token. After the token is processed by MLP (iou_mlp), the confidence score of each segmentation result is given. These scores reflect the confidence level of the model for different segmentation results and are provided to users as part of the output for reference.

[0129] By means of the above technical solution, the present embodiment has at least the following advantages: (1) The present embodiment uses the SAM model (Segment Anything Model) for image segmentation, which can maintain a high segmentation accuracy in the case of complex backgrounds and foreign objects that are close to the track background. (2) The track image may contain a large amount of background information, which is relatively stable and consistent in multiple recognition processes. By first comparing the segmentation mask, the area with significant difference from the background can be quickly screened out, thereby avoiding unnecessary classification processing of a large number of unchanged background areas. This can significantly improve the processing efficiency of the recognition method and reduce the waste of computing resources. (3) Changes in image quality and instability of the SAM model segmentation results may lead to slight differences in the segmentation results. Changes in image quality may be caused by factors such as differences in acquisition equipment, different lighting conditions, or distortion in image transmission, which will cause fluctuations in the results during the segmentation process; even if the image quality remains unchanged, the results of the SAM segmentation model still have certain instability. In order to deal with these problems, the present embodiment first compares the segmentation mask to accurately identify the parts with significant differences, and then excludes the slight differences caused by image quality or model instability, effectively reducing the false alarm rate triggered by these slight differences. (4) This embodiment combines an open set classification model, which can not only effectively identify known hidden dangers, but also identify unknown foreign objects that are not in the training data set. This method reduces the dependence on pre-trained models and data sets, improves the system's ability to handle unknown foreign objects, and thus reduces the false negative rate. (5) Through the automated intelligent recognition method, this embodiment can monitor the foreign objects on the track in real time, promptly detect and deal with safety hazards, thereby improving the efficiency and accuracy of track safety detection. At the same time, it reduces labor costs and improves work efficiency.

[0130] This embodiment also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned intelligent identification method for track foreign objects based on Segment Anything and open set classification is implemented.

[0131] An embodiment of the present invention further discloses a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned intelligent identification method for track foreign objects based on Segment Anything and open set classification.

[0132] This embodiment also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the above-mentioned intelligent identification method for track foreign objects based on Segment Anything and open set classification.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A track foreign body intelligent identification method based on Segment Anything and open set classification, characterized in that: The following steps are involved: S1. Obtain a track image without hidden dangers, input the track image without hidden dangers into the SAM model, and input the region of interest as a box prompt into the SAM model. The SAM model performs image segmentation on the track image without hidden dangers according to the box prompt to obtain a background image segmentation mask; S2, obtain the track image to be identified, input the track image to be identified into the SAM model, and input the box prompt consistent with S1 into the SAM model. The SAM model performs image segmentation on the track image to be identified according to the box prompt to obtain the segmentation mask of the image to be identified; S3, comparing the segmentation mask of the image to be identified with the segmentation mask of the background image, if the difference rate of the proportion of a certain color pixel area located in the same region of interest in the two reaches a preset threshold, then the local image of the region of interest in the image to be identified is used as a suspected hidden danger image and input into the trained track foreign body open set classification model; S4, the track foreign object open set classification model classifies the received suspected hidden danger image to determine whether it belongs to the background class, the known hidden danger class or the unknown foreign object class; S5: For suspected hidden danger images classified as known hidden dangers or unknown foreign matter, push alarms.

2. The track foreign body intelligent identification method based on Segment Anything and open set classification as claimed in claim 1, characterized in that: The steps to enter the region of interest as a box cue into the SAM model include: Based on the known track protection zone information, a mask of the track area is generated, and the upper and lower boundaries of the track area are determined accordingly; By calculating the width of the track area at different y coordinates, the linear relationship between the track width and the y coordinate is fitted; Traverse each row of the track area, calculate the width of the track based on the fitted linear relationship, and deduce the width of the candidate box based on it; In each track region segment, a candidate frame is generated according to a preset step length to obtain a candidate frame covering the track region; The candidate boxes are fed into the SAM model as box hints.

3. The track foreign body intelligent identification method based on Segment Anything and open set classification as claimed in claim 1, characterized in that: The steps of constructing the track foreign body open set classification model include: Get orbital image dataset; The track image in the data set is input into the SAM model, and the box prompt consistent with S1 is input into the SAM model. The SAM model performs image segmentation on the track image in the data set according to the box prompt to obtain a segmentation mask; The SAM model extracts the local image of the segmentation mask area on the track image in the dataset through the segmentation mask; Manually screen and classify the extracted partial images, and mark the partial images as background and hidden danger categories, wherein the background category refers to an area without any foreign matter, and the hidden danger category refers to an area with foreign matter; The pre-trained ResNet-34 model is used as the basis, and the closed-set classification training of the ResNet-34 model is performed using a dataset containing background and hidden danger labeling information; After completing the closed-set classification training of the model, the OpenMax algorithm is introduced to enable the model to distinguish between known hidden dangers and unknown foreign objects, and obtain an open-set classification model for track foreign objects.

4. The track foreign body intelligent identification method based on Segment Anything and open set classification as claimed in claim 1, characterized in that: In S1, after obtaining a track image without hidden dangers, the track image without hidden dangers is preprocessed and then input into the SAM model; in S2, after obtaining a track image to be identified, the track image to be identified is preprocessed and then input into the SAM model; the preprocessing steps of the track image include image denoising, image enhancement and image adjustment.

5. The track foreign body intelligent identification method based on Segment Anything and open set classification as claimed in claim 1, characterized in that: The SAM model includes an image encoder, a hint encoder and a mask decoder. The image encoder is used to digitize the input track image into a feature tensor, the hint encoder is used to encode the box hint into a form combined with the image encoding, and the mask decoder is used to generate a segmentation mask of the track image based on the image encoding and the hint encoding.

6. The track foreign body intelligent identification method based on Segment Anything and open set classification as claimed in claim 1, characterized in that: The step of S5 comprises: For suspected hidden danger images classified as known hidden dangers or unknown foreign objects, coordinate transformation is performed to find their positions in the track image to be identified; Use OpenCV's cv2.rectangle function to draw the minimum circumscribed rectangular frame in the track image to be identified, highlight the area corresponding to the suspected hidden danger map, and push an alarm.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the track foreign body intelligent identification method based on SegmentAnything and open set classification according to any one of claims 1 to 6 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the intelligent track foreign body identification method based on Segment Anything and open set classification as described in any one of claims 1 to 6 is implemented.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the intelligent track foreign body identification method based on Segment Anything and open set classification as described in any one of claims 1 to 6.

Citation Information

Cited By

  • Track foreign matter intelligent detection system and method based on image recognition

    CN120375099A

  • Abnormality detection method, system, device and medium based on multi-modal visual large model

    CN120726398B

  • Traffic image labeling method and device based on D-S evidence theory and medium

    CN121170796A