Railway foreign matter detection method, device and equipment based on semantic fusion and medium

Abstract

The invention discloses a railway foreign matter detection method, device and equipment based on semantic fusion and a medium, and the method comprises the following steps: carrying out the image coding of an obtained railway real-time image, and obtaining image coding features; according to the target query, decoding the image coding features to obtain target region features; according to the category semantic features corresponding to the railway real-time image, a preset historical image visual library is updated, and a multi-image visual library is obtained; and matching the target area features with the foreign matter category features in the multi-image vision library to obtain a foreign matter detection result. According to the method, through semantic intersection learning of multi-image prompt, the capability of detecting unknown category foreign matters in a complex scene is improved, the calculation cost is remarkably reduced while high accuracy is kept, and real-time adaptation to an open set environment is achieved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, deep learning and intelligent transportation technology, and in particular to a railway foreign object detection method, device, equipment and medium based on semantic fusion. Background Art

[0002] In railway scenarios, rapid detection and identification of foreign objects on the track is crucial to ensure driving safety. However, traditional obstacle detection methods usually rely on models trained with closed set data, which are difficult to cope with complex open set environments and various unknown categories of obstacles. These methods show low detection accuracy and response speed when faced with unexpected situations (such as falling objects or animal intrusions).

[0003] In order to improve the robustness and real-time performance of detection, some open-set object detection (OSOD) techniques have been proposed in recent years. In many OSOD models, text descriptions are used as semantic anchors to help visual models understand object categories, but the abstractness of text leads to poor detection results in fine-grained visual features. Secondly, a single image prompt cannot fully express all feature changes of a category, especially in railway scenes, where there are many types of objects and diverse features, and a single prompt is difficult to fully summarize. In addition, the limitation of computing resources is another challenge. Multimodal fusion methods rely on the joint training and reasoning of large language models and visual models, which has high computational overhead, especially when processing a large number of categories or high-resolution images, and it is difficult to meet the needs of real-time detection. In summary, due to modality differences, prompt limitations and computing resource constraints, existing OSOD methods are difficult to achieve efficient and robust foreign object detection in an open set environment. Summary of the invention

[0004] The purpose of the present invention is to provide a railway foreign object detection method, device, equipment and medium based on semantic fusion, so as to solve the problem that the existing OSOD method is difficult to achieve efficient and robust foreign object detection in an open set environment due to modal differences, prompt limitations and computing resource constraints.

[0005] To solve the above technical problems, a technical solution adopted by the present invention is to provide a railway foreign object detection method based on semantic fusion, the method comprising the following steps: performing image encoding on the acquired railway real-time image to obtain image encoding features; decoding the image encoding features according to the target query to obtain target area features; updating the preset historical image visual library according to the category semantic features corresponding to the railway real-time image to obtain a multi-image visual library; matching the target area features with the foreign object category features in the multi-image visual library to obtain foreign object detection results.

[0006] In some embodiments, encoding the acquired real-time railway image to obtain image coding features includes the steps of: extracting features from the real-time railway image to obtain multi-scale features; performing attention calculation on the multi-scale features to obtain efficient hybrid coding; and performing feature fusion of the efficient hybrid coding and the multi-scale features to obtain the image coding features.

[0007] In some embodiments, the attention calculation of the multi-scale features to obtain efficient hybrid coding includes the steps of: performing deformable attention calculation on deep features in the multi-scale features to obtain image features; and superimposing preset position codes with the image features to obtain the efficient hybrid coding.

[0008] In some embodiments, the method further comprises the step of: constructing the historical image visual library, wherein the historical image visual library stores and updates foreign object category features from different time points.

[0009] In some embodiments, the updating of a preset historical image visual library according to the category semantic features corresponding to the real-time railway image to obtain a multi-image visual library includes the steps of: performing prompt encoding on the real-time railway image to obtain the category semantic features; updating the historical image visual library according to the similarity between the category semantic features and the foreign object category features in the historical image visual library to obtain a multi-image visual library.

[0010] In some embodiments, the target area features are matched with foreign object category features in a multi-image visual library to obtain foreign object detection results, including the steps of: calculating a similarity matrix between the target area features and the foreign object category features; converting the similarity matrix into a probability matrix using a probability conversion formula; and determining, based on the probability matrix, the foreign object category label of the foreign object in the target area feature corresponding to the area of ​​the real-time railway image.

[0011] In some embodiments, the probability conversion formula is:

[0012]

[0013] Among them, S i,j represents the similarity score between the i-th target area feature and the average feature of all foreign object category features in the j-th foreign object category, wherein the i-th target area feature corresponds to the foreign object in the i-th target area of ​​the railway real-time image; exp(S i,j ) indicates S i,j The index value; |C| represents the number of foreign body categories; P(S i,j ) represents the probability that the foreign object in the i-th target area belongs to the j-th foreign object category.

[0014] The present invention also provides a railway foreign object detection device based on semantic fusion, the device comprising: an image encoding unit, used to perform image encoding on the acquired railway real-time image to obtain image encoding features; a target decoding unit, used to decode the image encoding features according to the target query to obtain target area features; a visual library updating unit, used to update the preset historical image visual library according to the category semantic features corresponding to the railway real-time image to obtain a multi-image visual library; a feature matching unit, used to match the target area features with the foreign object category features in the multi-image visual library to obtain a foreign object detection result.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0016] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0017] The beneficial effects of the present invention are as follows: the present invention discloses a railway foreign body detection method, device, equipment and medium based on semantic fusion, the method comprising the following steps: image encoding the acquired railway real-time image to obtain image encoding features; decoding the image encoding features according to the target query to obtain the target area features; updating the preset historical image visual library according to the category semantic features corresponding to the railway real-time image to obtain a multi-image visual library; matching the target area features with the foreign body category features in the multi-image visual library to obtain the foreign body detection results. The method of the present invention improves the ability to detect foreign bodies of unknown categories in complex scenes through semantic intersection learning of multi-image prompts, significantly reduces the computational cost while maintaining a high accuracy rate, and realizes real-time adaptation to the open set environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of an embodiment of a railway foreign body detection method based on semantic fusion of the present invention;

[0019] Figure 2 It is an overall principle diagram of an embodiment of a railway foreign body detection method based on semantic fusion of the present invention;

[0020] Figure 3 It is a schematic diagram of step S1 in an embodiment of a railway foreign body detection method based on semantic fusion of the present invention;

[0021] Figure 4 It is a composition diagram of each submodule in the feature fusion module used in step S1 in an embodiment of a railway foreign body detection method based on semantic fusion of the present invention;

[0022] Figure 5 It is a specific flow chart of step S1 in an embodiment of a railway foreign body detection method based on semantic fusion of the present invention;

[0023] Figure 6 It is a specific flow chart of step S3 in an embodiment of a railway foreign body detection method based on semantic fusion of the present invention;

[0024] Figure 7 It is a specific flow chart of step S4 in an embodiment of a railway foreign body detection method based on semantic fusion of the present invention;

[0025] Figure 8 It is a detection result diagram of an embodiment of a railway foreign body detection method based on semantic fusion of the present invention;

[0026] Fig. 9 It is a structural block diagram of an embodiment of a railway foreign body detection device based on semantic fusion according to the present invention;

[0027] Fig.10 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention;

[0028] Fig.11 It is a schematic block diagram of an embodiment of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0029] In order to facilitate the understanding of the present invention, the present invention is described in more detail below in conjunction with the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are provided in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0030] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more related listed items.

[0031] like Figure 1 As shown, a railway foreign body detection method based on semantic fusion of the present invention is shown, and the method comprises the following steps:

[0032] Step S1: performing image coding on the acquired real-time railway image to obtain image coding features.

[0033] like Figure 2 and Figure 3 As shown, an image encoder 1 is used to encode a railway real-time image 10 to obtain an image encoding feature 11. The image encoder 1 includes a backbone network 12, a deformable attention module 13 (DeformableAttention Module, DAM) and a feature fusion module 14 (Cross-Stage Feature Fusion Module, CCFM), which sequentially perform feature extraction, attention calculation and feature fusion operations on the railway real-time image 1 to obtain the image encoding feature 11.

[0034] Specifically, if Figure 5 As shown, step S1 includes the following sub-steps:

[0035] Step S11: extracting features from the railway real-time image to obtain multi-scale features.

[0036] In this embodiment, ResNet-34 is used as the backbone network 12, so that the railway real-time image 10 is subjected to multi-layer convolution and downsampling operations to obtain a set of multi-scale features. Among them, a set of multi-scale features includes five layers of features, and the last three layers of features are S3, S4 and S5 respectively. S3-S5 usually refers to the features obtained from the third to fifth feature extraction stages of the network, that is, the part that gradually transitions to the deep layer. Each stage consists of several residual blocks, which are mainly used to extract high-level features of the input image.

[0037] As an example, if the size (width*height) of the input railway real-time image 10 is 640*640, the corresponding size of S3 is 80*80, the size of S4 is 40*40, and the size of S5 is 20*20.

[0038] Step S12: Perform attention calculation on multi-scale features to obtain efficient hybrid coding.

[0039] It should be noted that in the traditional self-attention mechanism, each query position needs to calculate the attention weight with all the positions of the input features, so the computational complexity is high, especially when the spatial dimension of the input features is large, the amount of calculation will increase significantly. In contrast, the deformable attention mechanism allows the model to select a set of key spatial positions at each query position to calculate attention. These positions are sparse, that is, small in number. By selecting a few important positions for calculation, the spatial relationship of the features can be efficiently captured while greatly reducing the amount of calculation. At the same time, this sparse selection usually generates an offset through dynamic learning to determine where to perform the attention operation, thereby achieving a more adaptable and flexible deformable attention mechanism.

[0040] like Figure 3As shown, the deep features (the last layer features S5) output by the backbone network 12 are sent to the deformable attention module 13. The deformable attention module 13 adopts a deformable attention mechanism to select the key position of the attention calculation of S5 by learning a set of offsets, so as to more effectively focus on the image area related to the current task. The calculation formula in the deformable attention module 13 is as follows:

[0041]

[0042] Where q, K, q are the query vector, key and value in the attention mechanism respectively, M is the number of attention heads, and W m is the attention weight matrix, A m,k is the attention weight, p is the position index, Δp m,k is a learnable offset. Through the offset, we can adaptively select which positions in the last layer of features S5 to participate in the attention calculation, which makes the image encoding process focus on more discriminative feature areas and improves the effectiveness of feature expression.

[0043] In this embodiment, the attention operation is not limited to calculation at all positions, but some specific and relatively important spatial positions are selected to calculate the attention weights, which can reduce the computational cost and enhance the flexibility of attention calculation.

[0044] Secondly, in order to enhance the capture of spatial structural information, learnable position encoding is also added. Position encoding is an important component of the Transformer architecture. Because the self-attention mechanism itself does not retain position information, by superimposing position encoding with image features, it is possible to better understand the relative positions of various parts in the railway real-time image 10, thereby more accurately capturing the spatial structural relationship of the target. Among them, position encoding is achieved by learning a set of embedding vectors that match the size of the railway real-time image 10. The position encoding is added element by element with the image features to obtain an efficient hybrid code F5.

[0045] Step S13: performing feature fusion on the efficient hybrid coding and multi-scale features to obtain image coding features.

[0046] like Figure 2 and Figure 3As shown, the middle-level features (S3 and S4) from the main network 12 are fused with the efficient hybrid code F5 obtained by the deformable attention module 13 through the feature fusion module 14. Among them, the efficient hybrid code F5 (consistent with the dimension of S5) is first upsampled through the upsampling layer 41, and then fused with S4 through the fusion layer 43 to obtain the first fusion feature E1; then the first fusion feature E1 is upsampled through the upsampling layer 41 again, and then fused with S3 through the fusion layer 43 to obtain the second fusion feature E2; the second fusion feature E2 is downsampled through the downsampling layer 42, and fused with the upsampled result of the first fusion feature E1 through the fusion layer 43 to obtain the third fusion feature E3; the third fusion feature E3 is downsampled through the downsampling layer 42, and fused with the upsampled result of the efficient hybrid code F5 through the fusion layer 43 to obtain the fourth fusion feature E4; finally, the second fusion feature E2, the third fusion feature E3 and the fourth fusion feature E4 are spliced ​​(corresponding to the figure). ), obtain image coding feature 11.

[0047] like Figure 4 As shown, both the upsampling layer 41 and the downsampling layer 42 include a convolution layer, a BN layer (Batch Normalization) and a SiLU (Sigmoid Linear Unit) activation function. Among them, the convolution layer in the upsampling layer 41 adopts Conv 1*1s1, and the convolution layer in the downsampling layer 42 adopts Conv 3*3s2. The BN layer is a technology for normalizing data during the training process of a neural network. The SiLU (Sigmoid Linear Unit) activation function is an improved version of Sigmoid and ReLU. SiLU has the characteristics of no upper bound, lower bound, smoothness, and non-monotonicity. The effect of SiLU on deep models is better than ReLU, and it can be regarded as a smooth ReLU activation function.

[0048] like Figure 4 As shown in FIG. 4 , the fusion layer 43 includes a 1*1 convolution layer, a multi-branch convolution layer and an activation function F. The input of the fusion layer 43 is two initial features (C1 and C2). The two initial features are first concatenated and then input into two 1*1 convolution layers respectively for 1*1 Conv convolution. The output of one 1*1 convolution layer corresponds to the input of the multi-branch convolution layer. After the multi-branch convolution, the output is element-wise added with the output of the other 1*1 convolution layer (corresponding to FIG. ), and then through the activation function F (corresponding to the ) is activated to obtain the output of the fusion layer 43.

[0049] In this embodiment, the feature fusion module 14 aggregates features of different scales through a multi-scale feature fusion strategy, retaining rich low-level spatial information and high-level semantic information. The output of this fusion provides a set of improved image features with stronger feature expression capabilities and higher detection accuracy.

[0050] In step S1, the image encoder 1 optimizes feature extraction and information representation through various mechanisms, thereby achieving efficient target detection in the railway real-time image 10. Among them, through various optimization strategies, the image encoder 1 can efficiently extract and fuse multi-scale and multi-level image features, laying a solid foundation for generating accurate target prediction frames and target area features in the subsequent decoding process, while maintaining high detection accuracy and significantly improving the inference speed.

[0051] Step S2: According to the target query, the image encoding features are decoded to obtain the target area features.

[0052] like Figure 2 As shown, an improved DETR decoder 2 is used to decode the image coding feature 11. At the same time, through target query, the target object is identified and located in the image coding feature 11, and the format of the final output feature of the improved DETR decoder 2 is defined, thereby obtaining multiple target area features 21, and each target area feature 21 corresponds to a target area in the railway real-time image 10, and each target area contains an unknown foreign object. The foreign object category of the unknown foreign object in the railway real-time image 10 can be identified through subsequent steps.

[0053] It should be noted that the original DETR decoder has some limitations in open set target detection, such as the lack of the ability to dynamically fuse new features and over-reliance on global feature modeling. In response to these problems, the present application introduces multi-image visual cues and multi-image fusion mechanisms in the improved DETR decoder architecture. The introduction of multi-image visual cues is intended to enhance the semantic expression ability of the model, especially in the detection of unknown categories and the improvement of model robustness and generalization. On the one hand, under open set conditions, single image cues or fixed object queries cannot fully express the feature changes of different categories, while multi-image visual cues allow the decoder to learn the diversity features of categories at multiple time points and different perspectives, thereby enhancing the semantic understanding ability of the decoder. On the other hand, in the railway scene, the primary task is to determine whether there are unknown foreign objects in the real-time railway image 10, rather than just identifying foreign objects of known categories. Multi-image visual cues learn object features from different angles, and even in the face of unseen foreign objects, they can make effective judgments based on the similarity with known cues, thereby improving detection performance.

[0054] Furthermore, the target query is one of the inputs of the improved DETR decoder 2, and the role of the target query is to help the improved DETR decoder 2 identify and locate the target object in the image coding feature 11. Specifically, the role of the target query is to guide the improved DETR decoder 2 to learn the target representation, learn the relationship between the targets, and alleviate the position invariance problem.

[0055] In this embodiment, target query plays a key role in object query and feature aggregation in the improved DETR decoder 2, enabling the improved DETR decoder 2 to effectively identify and locate objects in the image coding features 11 while avoiding the complex post-processing steps in traditional target detection methods.

[0056] In this embodiment, based on the original DETR decoder, the improved DETR decoder 2 adds multi-image visual cues and new feature fusion strategies. These improvements improve the detection capability and real-time performance in an open set environment, and can maintain a high detection accuracy when facing unknown categories.

[0057] It should be noted that before step S3, it is necessary to build a historical image visual library, which is responsible for storing and updating image prompt features from different historical time points to ensure that key semantic information of each category is efficiently preserved. When initially building the historical image visual library, only historical data (railway historical images) can be used to complete the construction of the historical image visual library.

[0058] like Figure 2 As shown, a prompt encoder 3 is used to obtain multiple ROI (Region of Interest) features according to historical visual prompts 30 of railway historical images at different times, and input them into the historical image visual library 4.

[0059] In this embodiment, a CLIP encoder is used as a prompt encoder 3 to extract ROI features of railway historical images, all the acquired ROI features are input into the historical image visual library 4, and all the ROI features are divided into multiple foreign object categories, each foreign object category corresponds to a feature set, and each feature set retains multiple ROI features as foreign object category features, that is, core visual prompt features, which have a high correlation with the corresponding type of foreign objects and are the core of accurately identifying the type of foreign objects, ensuring the completeness and accuracy of the semantic information of each category in the historical image visual library 4 under different railway scenes.

[0060] Among them, the historical image visual library 4 contains feature sets corresponding to multiple feature categories, each feature set corresponds to a foreign object category and also corresponds to a category label, and each feature set contains at most n foreign object category features, and n is the maximum number of foreign object category features corresponding to each feature set in the historical image visual library 4.

[0061] In this embodiment, the historical image visual library 4 contains 9 feature sets, each of which contains at most 5 foreign object category features. Feature matching can then be performed based on the historical image visual library 4 to accurately identify the types of foreign objects on the railway track to ensure driving safety.

[0062] In some other embodiments, the feature set in the historical image visual library 4 and the number of foreign object category features in each feature set may be other values ​​to adapt to different railway scenarios and foreign object recognition requirements.

[0063] Step S3: According to the category semantic features corresponding to the railway real-time image, the preset historical image visual library is updated to obtain a multi-image visual library.

[0064] like Figure 2 As shown, the real-time feature fusion module 5 significantly improves the real-time detection performance of railway foreign objects through a fusion strategy based on cosine similarity. The input of the real-time feature fusion module 5 includes the new features to be fused (ROI features), the category to which the new features belong (ROI category), and the historical image visual library 4. The output of the real-time feature fusion module 5 is the multi-image visual library 6 obtained by fusing and updating the historical image visual library 4. The data sources of the multi-image visual library 6 cover three parts: real-time collected image data, historical data and video streams, and synthetic and enhanced data.

[0065] Specifically, if Figure 6 As shown, step S3 includes the following sub-steps:

[0066] Step S31: Prompt encoding is performed on the railway real-time image to obtain category semantic features.

[0067] like Figure 2 As shown, the real-time visual prompt 20 of the railway real-time image 10 at the corresponding moment is prompted and encoded according to the prompt encoder 3 to obtain the corresponding category semantic feature 31 (ROI feature), and the category semantic feature 31 is input into the real-time feature fusion module 5 together with the historical image visual library 4.

[0068] Step S32: According to the similarity between the category semantic features and the foreign object category features in the historical image visual library, the historical image visual library is updated to obtain a multi-image visual library.

[0069] like Figure 2 As shown, when a new feature (category semantic feature 31 corresponding to the real-time railway image 10) is input, the real-time feature fusion module 5 calculates the cosine similarity between the new feature and the foreign object category feature of the corresponding category in the historical image visual library 4, identifies the most similar foreign object category feature, and fuses and updates the historical image visual library 4.

[0070] The specific process of the real-time feature fusion module 5 fusing and updating the historical image visual library 4 is as follows:

[0071] First, traverse each foreign body category in the historical image visual library 4, that is, for each foreign body category in the historical image visual library 4, perform the following operations: first obtain the category label of the feature set, detect whether the category label of the current foreign body category (category n) is consistent with the category label of the new feature p Match; if the category label matches the category I of the new feature p Match, then obtain the feature set G of the foreign object category n Then, traverse the feature set G n For each feature position (from 1 to 5) in , if a feature position i is found to be empty (G n [i] = f ni =0), then the new feature F p Insert it to this position and return the fused updated multi-image vision library 6; if all feature positions are full, calculate the new feature F p and the feature set G of the current foreign body category n The cosine similarity of all foreign object category features in G, that is, for the feature set G n For each feature position m (from 1 to 5) in p With the feature set G n Each feature position m in the foreign body category feature f nm The cosine similarity s m . Find all cosine similarities s m The index feature position k corresponding to the maximum value in the image is used as the new feature F. p and feature set G n The average FV of all foreign body category features in use Replace the original feature f at this position nk After completing the traversal and fusion update of all foreign object categories, a multi-image visual library 6 is obtained.

[0072] In this embodiment, the real-time feature fusion module 5 adopts an efficient feature fusion mechanism to integrate new features into the feature set of the specified foreign object category. If the feature position of the foreign object category is full, the most relevant foreign object category feature is quickly selected based on the cosine similarity between the new feature and the visual prompt, and the new feature is fused with the average value of the feature set to replace the foreign object category feature. This process uses the new features corresponding to the railway real-time image 10 to continuously update the historical image visual library 4, which can respond to changing scenes in a timely manner. It is particularly suitable for railway foreign object detection tasks that require high response speed and accuracy, so that the foreign object category features corresponding to each foreign object category are more consistent with the actual features of different foreign objects in the railway scene, effectively supporting the application of this application in complex railway scenes.

[0073] Among them, the historical image visual library 4 and the multi-image visual library 6 adopt an efficient storage and fusion mechanism, which dynamically manages and optimizes the semantic information of the category through cosine similarity calculation and feature average fusion, effectively reduces the storage space occupied, and improves the performance and efficiency of foreign body detection. The real-time feature fusion module 5 quickly selects and replaces the foreign body category features most relevant to the new features based on the cosine similarity between the new features and the visual prompts, and enhances the accuracy and response speed of detection. This dynamic fusion method ensures the efficient detection capability of foreign objects in complex railway scenes without increasing the computational burden. Afterwards, the same method can be used to continuously update the multi-image visual library 6 in real time, continuously optimize the perception capability of foreign objects, and ensure that unknown foreign objects can be accurately detected even in complex and dynamically changing railway scenes.

[0074] In some other embodiments, a similarity threshold is set. If the cosine similarity between the new feature and all foreign object category features in the feature set of all foreign object categories is less than the probability threshold, it means that the foreign object does not belong to any category in the existing historical image visual library 4, and it is necessary to set the feature set corresponding to the new foreign object category in the historical image visual library 4, so as to obtain a multi-image visual library 6.

[0075] In some other embodiments, step S3 may be executed synchronously with step S1 and step S2, as long as the target area features 21 and the latest multi-image visual library 6 are obtained before step S4.

[0076] Step S4: Match the target area features with the foreign object category features in the multi-image vision library to obtain the foreign object detection results.

[0077] like Figure 2 As shown, the multiple target area features 21 output by the improved DETR decoder 2 are matched with the foreign object category features in the multi-image visual library 6 respectively through the alignment head 7, so as to obtain the foreign object detection result.

[0078] Specifically, if Figure 7As shown, step S4 includes the following sub-steps:

[0079] Step S41: Calculate the similarity matrix between the target region features and the foreign object category features.

[0080] In this embodiment, the extracted target region features 21 and the average features of all foreign object category features in each foreign object category in the multi-image visual library 6 are first calculated, and these features represent the semantic information of the image region and the foreign object category, respectively. Then, the similarity matrix between the target region features 21 and the average features of the corresponding foreign object category is calculated to match the target region features 21 corresponding to the railway real-time image 10 with the foreign object category features in the multi-image visual library 6. For each target region in the railway real-time image 10, by calculating the similarity score between the target region features 21 and the average features corresponding to each foreign object category, the similarity matrix can be obtained as follows:

[0081]

[0082] Among them, F r ∈R bs×nq×D represents the target region feature matrix composed of all target region features 21, F V ∈R bs ×|C|×D Represents the average feature matrix of foreign object category features of all foreign object categories in the multi-image vision library 6, S∈R bs×nq×|C| represents the similarity matrix, bs represents the batch size, that is, the number of images processed by the model in one forward propagation; nq represents the number of object queries, that is, the number of queries used by the model to infer the target area, D represents the number of feature channels, |C| represents the number of foreign body categories, and each element value in the similarity matrix S represents the similarity score between the target area feature 21 corresponding to the foreign body in each target area and the average feature of all foreign body category features in each foreign body category.

[0083] Step S42: using the probability conversion formula, convert the similarity matrix into a probability matrix.

[0084] It should be noted that in order to obtain the probability that the foreign objects in each target area belong to each foreign object category, it is necessary to apply the Softmax function to each row so that the similarity score is converted into a probability distribution.

[0085] In this embodiment, the Softmax function is applied to the last dimension (category dimension) of the similarity matrix S, and the matching probability between each target region feature 31 and the foreign object category features of all foreign object categories is calculated, and the similarity matrix is ​​converted into a probability matrix. The probability conversion formula corresponding to the Softmax function is:

[0086]

[0087] Among them, S i,j represents the similarity score between the i-th target region feature and the average feature of all foreign object category features in the j-th foreign object category, and the i-th target region feature corresponds to the foreign object in the i-th target region of the railway real-time image 10; exp(S i,j ) indicates S i,j The index value; |C| represents the number of foreign body categories; P(S i,j ) represents the probability that the foreign object in the i-th target area belongs to the j-th foreign object category.

[0088] Finally, after calculating the probability that all foreign objects in the target area of ​​the railway real-time image 10 belong to each foreign object category, a probability matrix R is output, whose size is R bs×nq×|C| , each element value in the probability matrix R represents the probability that each target area belongs to each foreign object category.

[0089] Step S43: Determine the foreign object category label of the foreign object in the area corresponding to the target area feature in the railway real-time image according to the probability matrix.

[0090] Among them, the most likely category label corresponding to the foreign object in each target area in the railway real-time image 10 is assigned according to each element value in the probability matrix R. For each target area, according to the corresponding probability distribution in the probability matrix R, the foreign object category corresponding to the element value with the largest probability is selected, and the category label of the foreign object category is used as the foreign object category label of the foreign object in the target area, so as to obtain the final foreign object detection result.

[0091] For example, in a railway scene, the decoding stage outputs three target area features corresponding to a real-time railway image, that is, three areas of interest (area A, area B, and area C) are detected. At the same time, the category labels corresponding to the feature sets of the four foreign object categories in the multi-image vision library are defined as follows: Category 1: stones; Category 2: branches; Category 3: plastic bags; Category 4: other debris. In order to assign the three areas to one of the four categories, for each area, the cosine similarity is used to calculate the similarity score between the target area features and the category features, and a similarity matrix can be obtained. Among them, the obtained similarity matrix is:

[0092]

[0093] In which, each row of the similarity matrix S represents region A, region B, and region C in turn, and each column represents category 1, category 2, category 3, and category 4 in turn; S i,jRepresents the similarity score between the i-th region (region A, region B and region C) and the j-th category (category 1, category 2, category 3 and category 4); i = 1, 2, 3; j = 1, 2, 3, 4, that is, each element value in the similarity matrix S represents the similarity score between each region and each category.

[0094] Taking region A as an example, the similarity scores between region A and each category are 1.5, 0.3, 0.2, and 0.1, respectively. First, the index values ​​of each similarity score are calculated as follows:

[0095] exp(1.5)=4.4817,

[0096] exp(0.3)=1.3499,

[0097] exp(0.2)=1.2214,

[0098] exp(0.1)=1.1052;

[0099] Then, the sum of all exponential values ​​is calculated as:

[0100] Sum=4.4817+1.3499+1.2214+1.1052=8.1582;

[0101] Finally, the probabilities of the foreign objects in area A output by the Softmax function to which the foreign objects belong are:

[0102]

[0103] Similarly, the probability of each foreign body category to which the foreign bodies in area B and area C belong is calculated, and the probability matrix is ​​obtained as follows:

[0104]

[0105] Among them, the element value of each row of the probability matrix is ​​the probability distribution of the foreign object category to which the foreign object belongs in the corresponding area, that is, the probability distribution of the foreign object category to which the foreign object in area A, area B and area C are [0.5492, 0.1654, 0.1497, 0.1356], [0.1720, 0.3090, 0.2085, 0.3105] and [0.1131, 0.1843, 0.4963, 0.2063] respectively.

[0106] Finally, according to the probability distribution of each foreign object category to which the foreign objects in each area belong, the category label of the foreign object category corresponding to the largest value is selected as the foreign object category label of the foreign objects in the area, and the final foreign object detection result is obtained. Among them, the probability that the foreign objects in area A belong to category 1 (stone) is the largest, with a maximum probability of 0.5492; the probability that the foreign objects in area B belong to category 4 (other sundries) is the largest, with a maximum probability of 0.3105; the probability that the foreign objects in area C belong to category 3 (plastic bags) is the largest, with a maximum probability of 0.4963. That is, the foreign object category label of the foreign objects in area A is stone, the foreign object category label of the foreign objects in area B is other sundries, and the foreign object category label of the foreign objects in area C is plastic bags.

[0107] like Figure 8 As shown in the figure, a detection result diagram detected by this method is shown. In the figure, the detection frame is used to identify the foreign body categories in the three target areas in the real-time railway image, and the category labels and corresponding probabilities of the foreign body categories to which the foreign bodies in the corresponding areas belong are marked on the detection frame. Among them, the target areas containing foreign bodies of different foreign body categories are distinguished by detection frames of different colors. In the figure, a white detection frame is used to identify that the foreign body category label of the foreign body in one of the target areas is tree, and the probability that the corresponding foreign body belongs to the tree category is 0.93; the blue detection frame is used to identify that the foreign body category labels of the foreign bodies in the other two target areas are both stone, and the probabilities that the corresponding foreign bodies belong to the stone category are 0.79 and 0.87 respectively.

[0108] In this process, in this way, the most likely category label can be assigned to each foreign object detected in the target area of ​​the real-time railway image, thereby achieving effective identification and detection of foreign objects in the railway scene. This method helps to achieve efficient detection and classification of foreign objects in complex and changeable railway environments, ensuring the safety and reliability of railway operations.

[0109] It should be noted that the above content is mainly about foreign body detection in railway scenarios, but the technical solution of the present invention is not only applicable to foreign body detection in railway scenarios, but can also be extended to other fields that require real-time detection of unknown category objects, such as unmanned driving, intelligent monitoring, etc. By adjusting the parameters and feature fusion mechanism of the multi-image vision library, it can adapt to more complex and diverse open set environments and achieve efficient target detection and classification.

[0110] Based on the same inventive concept, Fig. 9 As shown, the present invention also provides a railway foreign body detection device based on semantic fusion, the device comprising:

[0111] The image encoding unit 101 is used to perform image encoding on the acquired real-time railway image to obtain image encoding features.

[0112] The target decoding unit 102 is used to decode the image coding features according to the target query to obtain the target area features.

[0113] The visual library updating unit 103 is used to update the preset historical image visual library according to the category semantic features corresponding to the real-time railway image to obtain a multi-image visual library.

[0114] The feature matching unit 104 is used to match the target area features with the foreign object category features in the multi-image vision library to obtain foreign object detection results.

[0115] In the present application, other technical features of the above-mentioned railway foreign object detection device based on semantic fusion are the same as the features disclosed in the above-mentioned method embodiment, and will not be repeated here.

[0116] Based on the same inventive concept, the present application also provides an electronic device, which includes a processor, a memory and a communication circuit, and the processor is connected to the memory and the communication circuit respectively; wherein the communication circuit is used for communication connection, the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above method.

[0117] See also Fig.10 The electronic device described in the embodiment of the present application may specifically include a processor 210 and a memory 220. The memory 220 is coupled to the processor 210.

[0118] The processor 210 is used to control the operation of the electronic device. The processor 210 may also be referred to as a CPU (Central Processing Unit). The processor 210 may be an integrated circuit chip having the ability to process signals. The processor 210 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 210 may also be any conventional processor, etc.

[0119] The memory 220 is used to store computer programs, and may be a RAM, a ROM, or other types of storage terminals. Specifically, the memory 220 may include one or more computer-readable storage media, which may be non-transitory or transient. The memory 220 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage terminals, flash memory storage terminals. In some embodiments, the non-transitory computer-readable storage medium in the memory 220 is used to store at least one program code.

[0120] The processor 210 is used to execute the computer program stored in the memory 220 to implement the methods described in the various method embodiments of the present application.

[0121] In some embodiments, the electronic device may further include: a peripheral terminal interface 230 and at least one peripheral terminal. The processor 210, the memory 220 and the peripheral terminal interface 230 may be connected via a bus or a signal line. Each peripheral terminal may be connected to the peripheral terminal interface 230 via a bus, a signal line or a circuit board. Specifically, the peripheral terminal includes: at least one of a radio frequency circuit 240, a display screen 250, an audio circuit 260 and a power supply 270.

[0122] The peripheral terminal interface 230 may be used to connect at least one peripheral terminal related to I / O (Input / output) to the processor 210 and the memory 220. In some embodiments, the processor 210, the memory 220, and the peripheral terminal interface 230 are integrated on the same chip or circuit board; in some other implementations, any one or two of the processor 210, the memory 220, and the peripheral terminal interface 230 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0123] The radio frequency circuit 240 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 240 communicates with the communication network and other IoT devices through electromagnetic signals, and the radio frequency circuit 240 is the communication circuit of the electronic device. The radio frequency circuit 240 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 240 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, an operator identity module card, and the like. The radio frequency circuit 240 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 240 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0124] The display screen 250 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 250 is a touch display screen, the display screen 250 also has the ability to collect touch signals on the surface or above the surface of the display screen 250. The touch signal can be input to the processor 210 as a control signal for processing. At this time, the display screen 250 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 250 can be one, set on the front panel of the electronic device; in other embodiments, the display screen 250 can be at least two, respectively set on different surfaces of the electronic device or in a folding design; in other embodiments, the display screen 250 can be a flexible display screen, set on a curved surface or a folding surface of the electronic device. Even, the display screen 250 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 250 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0125] The audio circuit 260 may include a microphone and a speaker. The microphone is used to collect sound waves from the operator and the environment, and convert the sound waves into electrical signals and input them into the processor 210 for processing, or input them into the radio frequency circuit 240 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged in different parts of the electronic device. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 210 or the radio frequency circuit 240 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 260 may also include a headphone jack.

[0126] The power supply 270 is used to power various components in the electronic device. The power supply 270 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 270 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0127] For a detailed description of the functions and execution processes of each functional module or component in the electronic device embodiment of the present application, reference can be made to the description in the above-mentioned method embodiments of the present application, which will not be repeated here.

[0128] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the various embodiments of the electronic devices described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0129] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0130] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0131] Based on the same inventive concept, the present application also provides a computer-readable storage medium storing a computer program, which can be executed by a processor to implement the above method.

[0132] See also Fig.11 , if the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 300. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions / computer programs to enable an IoT device (which can be a personal computer, server, or network terminal, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical disks, and electronic terminals such as computers, mobile phones, laptops, tablet computers, cameras, etc. having the above-mentioned storage media.

[0133] The description of the execution process of the program data in the computer-readable storage medium can refer to the description in the above-mentioned method embodiments of the present application, and will not be repeated here.

[0134] It can be seen that the present invention discloses a railway foreign body detection method, device, equipment and medium based on semantic fusion, the method comprising the following steps: image encoding the acquired railway real-time image to obtain image encoding features; decoding the image encoding features according to the target query to obtain the target area features; updating the preset historical image visual library according to the category semantic features corresponding to the railway real-time image to obtain a multi-image visual library; matching the target area features with the foreign body category features in the multi-image visual library to obtain the foreign body detection results. The method of the present invention improves the ability to detect foreign bodies of unknown categories in complex scenes through semantic intersection learning of multi-image prompts, significantly reduces the computational cost while maintaining a high accuracy, and realizes real-time adaptation to the open set environment.

[0135] The above are only embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent structural transformations made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are included in the protection scope of the present invention.

Claims

1. A railway foreign body detection method based on semantic fusion, characterized in that: The method comprises the following steps: Perform image coding on the acquired railway real-time images to obtain image coding features; According to the target query, the image encoding features are decoded to obtain target area features; According to the category semantic features corresponding to the real-time railway image, a preset historical image visual library is updated to obtain a multi-image visual library; The target area features are matched with the foreign object category features in the multi-image vision library to obtain foreign object detection results.

2. The foreign body detection method according to claim 1, characterized in that: The method of encoding the acquired railway real-time image to obtain image encoding features comprises the following steps: Extracting features from the railway real-time image to obtain multi-scale features; Performing attention calculation on the multi-scale features to obtain efficient hybrid coding; The efficient hybrid coding and the multi-scale features are subjected to feature fusion to obtain the image coding features.

3. The foreign body detection method according to claim 1, characterized in that: The step of performing attention calculation on the multi-scale features to obtain efficient hybrid coding comprises the following steps: Perform deformable attention calculation on the deep features in the multi-scale features to obtain image features; The preset position code is superimposed on the image feature to obtain the efficient hybrid code.

4. The foreign body detection method according to claim 1, characterized in that: The method further comprises the steps of: The historical image visual library is constructed, and the historical image visual library stores and updates foreign body category features from different time points.

5. The foreign body detection method according to claim 1, characterized in that: The method of updating a preset historical image visual library according to the category semantic features corresponding to the railway real-time image to obtain a multi-image visual library comprises the following steps: Performing prompt coding on the railway real-time image to obtain category semantic features; According to the similarity between the category semantic features and the foreign object category features in the historical image visual library, the historical image visual library is updated to obtain a multi-image visual library.

6. The foreign body detection method according to claim 1, characterized in that: The method of matching the target area features with the foreign body category features in the multi-image visual library to obtain the foreign body detection result comprises the steps of: Calculating a similarity matrix between the target area features and the foreign object category features; Using a probability conversion formula, converting the similarity matrix into a probability matrix; According to the probability matrix, the foreign object category label of the foreign object in the area corresponding to the railway real-time image by the target area feature is determined.

7. The foreign matter detection method according to claim 6, characterized in that: The probability conversion formula is: Among them, S i,j represents the similarity score between the i-th target area feature and the average feature of all foreign object category features in the j-th foreign object category, wherein the i-th target area feature corresponds to the foreign object in the i-th target area of ​​the railway real-time image; exp(S i,j ) indicates S i,j The index value; |C| represents the number of foreign body categories; P(S i,j ) represents the probability that the foreign object in the i-th target area belongs to the j-th foreign object category.

8. A railway foreign object detection device based on semantic fusion, characterized in that: The device comprises: An image coding unit is used to perform image coding on the acquired real-time railway image to obtain image coding features; A target decoding unit, used to decode the image encoding features according to the target query to obtain target area features; A visual library updating unit, used for updating a preset historical image visual library according to the category semantic features corresponding to the railway real-time image, to obtain a multi-image visual library; The feature matching unit is used to match the target area features with the foreign object category features in the multi-image vision library to obtain foreign object detection results.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images