A mine car track foreign matter detection method and system based on a deep neural network

Abstract

The application belongs to the technical field of target detection, and provides a mine car track foreign matter detection method and system based on a deep neural network, which comprises the following steps: acquiring real-time images and text prompt words of a mine car track; extracting hierarchical multi-modal features of the acquired real-time images; obtaining image-text fusion features of the mine car track by using the hierarchical multi-modal features obtained through multi-modal deep neural network fusion and the text prompt words; detecting whether there is foreign matter in the mine car track according to the obtained image-text fusion features of the mine car track and a preset foreign matter detection model; and when there is foreign matter, identifying the size of the foreign matter and determining the position of the foreign matter, thereby completing real-time monitoring of foreign matter in the mine car track.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, specifically relating to a method and system for detecting foreign objects on mine car tracks based on deep neural networks. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Mining cars are narrow-gauge railway vehicles used in mines to transport bulk materials such as coal, ore, and waste rock. They are mainly used for rail transport in underground mine roadways, shafts, and on the surface. Foreign objects such as rolling stones intruding into mining car tracks are characterized by their suddenness and unpredictability. These foreign objects may collide with moving mining cars, leading to serious production accidents such as derailment and overturning. As the main transportation channel for ore, the inability to detect and identify foreign objects on mining car tracks in a timely and accurate manner poses a significant threat to the safety and operational efficiency of the entire mine.

[0004] Therefore, in order to ensure the safety of mine car operation and mine production, it is urgent to monitor foreign objects in the mine track area in real time. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a method and system for detecting foreign objects on mine car tracks based on deep neural networks. Based on multimodal perception and target detection, it monitors and identifies in real time whether foreign objects exist on the mine car tracks and determines the size and location of the foreign objects, thereby achieving real-time, high-precision target detection and risk warning of foreign objects on mine car tracks.

[0006] According to some embodiments, the first solution of the present invention provides a method for detecting foreign objects on mine car tracks based on deep neural networks, which adopts the following technical solution:

[0007] A method for detecting foreign objects on mine car tracks based on deep neural networks includes:

[0008] Acquire real-time images and text prompts of the mine car track;

[0009] Extract hierarchical multimodal features from the acquired real-time images;

[0010] The image-text fusion features of the mine car track are obtained by fusing hierarchical multimodal features and text prompts using multimodal deep neural networks;

[0011] Based on the obtained image and text fusion features of the mine car track and the preset foreign object detection model, detect whether there are foreign objects on the mine car track;

[0012] When foreign objects are present, the size of the foreign objects is identified and their location is determined, thus completing real-time monitoring of foreign objects on the mine car track.

[0013] As a further technical limitation, the preset foreign object detection model adopts a deep neural network detection model, which is trained using a composite loss function and gradient descent backpropagation.

[0014] Furthermore, the composite loss function for ,in, This represents the weight of each loss function, and ; Let represent different loss functions, where The text loss function is represented by the cross-entropy loss function, i.e. , This indicates the annotated text information. This represents the predicted text information from a multimodal large language model. This represents the object detection loss function, which uses the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. , Represents the object detection prediction box. Indicates a label box; This represents the first-layer image-text joint loss function. This represents the second-layer image-text joint loss function. This represents the third-layer image-text joint loss function. , and Both use a contrastive loss function. ,

[0015] ,

[0016] ;

[0017] in, It is the number of samples in a batch. It is the number of negative samples in a batch, where positive samples represent matching image-text pairs and negative samples represent non-matching image-text pairs; Indicates the first layer Image feature vectors, Represents the first positive sample in the first layer. Each text feature vector Indicates the first negative sample in the first layer One text feature vector; Indicates the second layer Image feature vectors, Indicates the first positive sample in the second layer Each text feature vector Indicates the first negative sample in the second layer One text feature vector; Indicates the third layer Image feature vectors, Indicates the third positive sample in the third layer Each text feature vector Indicates the third layer of negative samples. Each text feature vector This represents the cosine similarity measure. This represents the temperature parameter used to adjust the similarity score distribution.

[0018] Furthermore, the contrastive loss function is minimized based on gradient descent backpropagation, and the deep neural network detection model is trained through supervised learning using the loss function.

[0019] As a further technical limitation, a foreign object detection model is used to perform target detection on the obtained image-text fusion features of the mine car track. The location and size of foreign objects are marked in the real-time image of the acquired mine car track by using the target detection box, thus completing the real-time monitoring of foreign objects on the mine car track.

[0020] As a further technical limitation, hierarchical multimodal features are extracted and fused based on convolutional layers. Text prompts are mapped and matched to the same dimension by combining linear layers. The feature correlation between hierarchical multimodal features and text prompts is extracted through cross-self-attention mechanism and self-attention mechanism, and saliency features are fused. Additional concatenation operation by side path is used to collaboratively extract hierarchical multimodal features and text prompt features to construct residual neural network. The image-text fusion features of the mine car track are obtained by summing the backbone network and the side path residual network.

[0021] According to some embodiments, a second aspect of the present invention provides a foreign object detection system for mine car tracks based on a deep neural network, employing the following technical solution:

[0022] A foreign object detection system for mining truck tracks based on deep neural networks includes:

[0023] The acquisition module is configured to acquire real-time images and text prompts of the mine car track;

[0024] An extraction module is configured to extract hierarchical multimodal features from the acquired real-time images;

[0025] The fusion module is configured to use hierarchical multimodal features obtained by fusing multimodal deep neural networks and text prompts to obtain the image-text fusion features of the mine car track;

[0026] The detection module is configured to detect whether there are foreign objects on the mine car track based on the obtained image and text fusion features and the preset foreign object detection model; when foreign objects are present, the module identifies the size of the foreign objects and determines their location, thus completing the real-time monitoring of foreign objects on the mine car track.

[0027] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium, employing the following technical solution:

[0028] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the foreign object detection method for mine car tracks based on a deep neural network as described in the first aspect of the present invention.

[0029] According to some embodiments, the fourth aspect of the present invention provides an electronic device, which adopts the following technical solution:

[0030] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, it implements the steps in the foreign object detection method based on a deep neural network for mine car tracks as described in the first aspect of the present invention.

[0031] According to some embodiments, the fifth aspect of the present invention provides a computer program product, which adopts the following technical solution:

[0032] A computer program product includes software code, wherein the program in the software code performs the steps of the foreign object detection method based on deep neural networks for mine car tracks as described in the first aspect of the present invention.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] This invention is based on multimodal perception and target detection to monitor and identify in real time whether there are foreign objects on the mine car track and to determine the size and position of the foreign objects in the mine car track, thereby realizing real-time high-precision target detection and risk warning of foreign objects on the mine car track. Attached Figure Description

[0035] The accompanying drawings, which form part of this embodiment, are used to provide a further understanding of this embodiment. The illustrative embodiments and their descriptions are used to explain this embodiment and do not constitute an improper limitation of this embodiment.

[0036] Figure 1 This is a flowchart of the foreign object detection method for mine car tracks based on deep neural networks in Embodiment 1 of the present invention;

[0037] Figure 2 This is a schematic diagram of the hardware structure of the foreign object detection method based on deep neural network in Embodiment 1 of the present invention;

[0038] Figure 3 This is a schematic diagram of the foreign object detection method based on deep neural networks in Embodiment 1 of the present invention.

[0039] Figure 4 This is a flowchart of the image-text fusion process in Embodiment 1 of the present invention;

[0040] Figure 5 The diagram shows the detailed steps of the foreign object detection method for mine car tracks based on deep neural networks in Embodiment 1 of the present invention.

[0041] Figure 6 This is a structural block diagram of the foreign object detection system for mine car tracks based on deep neural networks in Embodiment 2 of the present invention;

[0042] Among them, 1. mine car track; 2. track rollers; 3. visible light camera; 4. infrared camera; 5. computer; 6. communication equipment. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0045] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0046] In this invention, terms such as "upper," "lower," "left," "right," "front," "back," "vertical," "horizontal," "side," and "bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only to facilitate the description of the structural relationships of the various components or elements of this invention and do not specifically refer to any component or element in this invention. They should not be construed as limiting the invention.

[0047] In this invention, terms such as "fixed connection," "connected," and "linked" should be interpreted broadly, indicating a fixed connection, an integral connection, or a detachable connection; a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can determine the specific meaning of these terms in this invention based on the specific circumstances, and they should not be construed as limitations on the invention.

[0048] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0049] Example 1

[0050] Embodiment 1 of this invention introduces a method for detecting foreign objects on mine car tracks based on deep neural networks.

[0051] like Figure 1 The aforementioned method for detecting foreign objects on mine car tracks based on deep neural networks includes:

[0052] Acquire real-time images and text prompts of the mine car track;

[0053] Extract hierarchical multimodal features from the acquired real-time images;

[0054] The image-text fusion features of the mine car track are obtained by fusing hierarchical multimodal features and text prompts using multimodal deep neural networks;

[0055] Based on the obtained image and text fusion features of the mine car track and the preset foreign object detection model, detect whether there are foreign objects on the mine car track;

[0056] When foreign objects are present, the size of the foreign objects is identified and their location is determined, thus completing real-time monitoring of foreign objects on the mine car track.

[0057] This embodiment uses the track boulder 2 as an example to perform foreign object detection on the mine car track 1, combined with... Figure 2 In this embodiment, visible light camera 3, infrared camera 4, computer 5, and communication equipment 6 are used to collect and detect track boulders; specifically:

[0058] Visible light camera 3, consisting of multiple wide-angle high-definition cameras (4K resolution @30fps), covers key areas of the track (slopes, curves, tunnel entrances) and acquires RGB video streams in real time; infrared light camera 4, consisting of multiple wide-angle infrared light cameras (1280 x 1024 resolution @30fps), covers key areas of the track (slopes, curves, tunnel entrances) and acquires infrared light video streams in real time; computer 5, wiredly connected to visible light camera 3 and infrared light camera 4, acquires the acquired video streams online, runs the proposed multimodal deep neural network fusion model to achieve track stone target detection, and is used for deep neural network computing framework calculation to accelerate deep neural network model calculation and meet the requirements for online track stone target detection; communication device 6, when a track stone is detected, the communication module sends the position and size information of the stone detection result to the central server through the wireless communication module to complete the track stone warning.

[0059] This embodiment uses, as follows: Figure 2 The hardware structure of the foreign object detection method for mine cart tracks based on deep neural networks is shown. It collects 500 sets of different rolling stone data samples, covering different lighting conditions, location scenarios, and characteristics such as the size, material (rock, ore), quantity, and rolling location of the rolling stones. The LabelImg tool is used for manual target annotation, forming a rolling stone detection training dataset for subsequent deep neural network model training. Simultaneously, foreign objects similar to rolling stones (such as driftwood, metal fragments, etc.) are manually labeled in the samples to improve the recognition rate of rolling stones and effectively distinguish them from other similar foreign objects.

[0060] In the process of detecting boulder spills on tracks, this embodiment employs a multimodal hierarchical deep neural network fusion model based on target detection. The model consists of text encoding, graph encoding, graph-text fusion, and target detection. The specific deep neural network model structure proposed for detection is as follows: Figure 3 As shown, its inputs are a visible light image, an infrared light image, and a text prompt, and its output is a target detection box for rolling stones on the track based on the visible light image and the infrared light image. The target detection box marks the position and size of the rolling stones appearing in the input image.

[0061] This embodiment uses, as follows: Figure 3 The deep neural network model shown includes:

[0062] (1) Text encoding module

[0063] In this embodiment, the text encoding module is based on the Qianwen multimodal large language model. The input consists of a visible light image, an infrared image, and a text prompt. The text prompt is: "The current input is a visible light image and an infrared image. The target scene is a mine cart track operation scene. Please output whether there are any rolling stones or foreign objects in the scene." By utilizing the image and text joint reasoning capabilities of the multimodal large language model, richer and more comprehensive semantic information is generated, improving the accuracy and generalization ability of target detection.

[0064] (2) Graph Coding Module

[0065] The image encoding module in this embodiment consists of convolutional layers and pooling layers, which are used to extract image features from the input visible light image and infrared light image, respectively. The convolutional features are extracted layer by layer from shallow to deep layers.

[0066] (3) Image and text fusion module

[0067] like Figure 4 As shown, this embodiment utilizes a cross-attention mechanism to fuse and extract visible light features, infrared light fusion features extracted by the graph encoding module, and text features extracted by the text encoder. The backbone first extracts visible light and infrared light fusion image features using convolutional layers, then uses linear layers to map and match text features to the same dimension. A cross-self-attention mechanism and a self-attention mechanism are employed to extract feature correlations between the two modalities, fusing saliency features. Simultaneously, an additional side-channel concatenation operation is used to collaboratively extract both features, forming a residual neural network. Finally, the backbone network and the side-channel residual network are summed to obtain the final fused features.

[0068] (4) Target detection

[0069] This embodiment performs target detection on the decoded fused features, and generates the final target detection box from the target detection head; it uses the constructed Rolling Stone detection training dataset to supervise learning through training data.

[0070] This embodiment employs a composite loss function and utilizes gradient descent backpropagation to effectively train the proposed multimodal hierarchical deep neural network fusion model. Composite Loss Function It consists of five loss functions, namely the text loss function. Object detection loss function First-layer image-text joint loss function Second-layer image-text joint loss function and the third-layer image-text joint loss function That is, the composite loss function. for ,in, This represents the weight of each loss function, and ;

[0071] Text loss function Using the cross-entropy loss function, i.e. , This indicates the annotated text information. This represents the predicted text information from a multimodal large language model.

[0072] Object detection loss function The intersection-union ratio (IUU) of the predicted bounding boxes and the ground truth bounding boxes is used. , Represents the object detection prediction box. Indicates a label box;

[0073] First-layer image-text joint loss function Second-layer image-text joint loss function and the third-layer image-text joint loss function Both use a contrastive loss function.

[0074] ,

[0075] ,

[0076] ;

[0077] in, It is the number of samples in a batch. It is the number of negative samples in a batch, where positive samples represent matching image-text pairs and negative samples represent non-matching image-text pairs; Indicates the first layer Image feature vectors, Represents the first positive sample in the first layer. Each text feature vector Indicates the first negative sample in the first layer One text feature vector; Indicates the second layer Image feature vectors, Indicates the first positive sample in the second layer Each text feature vector Indicates the first negative sample in the second layer One text feature vector; Indicates the third layer Image feature vectors, Indicates the third positive sample in the third layer Each text feature vector Indicates the third layer of negative samples. Each text feature vector This represents the cosine similarity measure. This represents the temperature parameter used to adjust the similarity score distribution.

[0078] The contrastive loss function is minimized by gradient descent backpropagation. The loss function is then used to supervise the training of the deep neural network detection model, thus completing the training of the foreign object detection model. The model can learn feature representations that result in higher similarity scores for matching image-text pairs and lower similarity scores for non-matching image-text pairs.

[0079] After training the deep neural network model in this embodiment, the model will be deployed in the track stone detection data acquisition and detection system. Utilizing online acquired visible and infrared video streams, the model will perform forward propagation calculations to generate target detection results, determine the location and size of the stones, and provide anomaly warnings. The specific process is as follows: Figure 5 As shown.

[0080] This embodiment establishes a visible light and infrared light data video stream acquisition system for track stone detection. It proposes a multimodal fusion model framework, which fully extracts potential effective features from multimodal input information through hierarchical feature fusion of text and images. These features are used for effective visual detection of target objects in multimodal images, ultimately achieving real-time, online, and high-precision target detection and risk warning for track stones. Based on multimodal perception and target detection, it monitors and identifies the presence of foreign objects on the mine car track in real time, determining the size and location of the foreign objects, thereby achieving real-time, high-precision target detection and risk warning for foreign objects on the mine car track.

[0081] Example 2

[0082] Embodiment 2 of the present invention introduces a foreign object detection system for mine car tracks based on deep neural networks.

[0083] like Figure 6 The illustrated foreign object detection system for mine car tracks based on deep neural networks includes:

[0084] The acquisition module is configured to acquire real-time images and text prompts of the mine car track;

[0085] An extraction module is configured to extract hierarchical multimodal features from the acquired real-time images;

[0086] The fusion module is configured to use hierarchical multimodal features obtained by fusing multimodal deep neural networks and text prompts to obtain the image-text fusion features of the mine car track;

[0087] The detection module is configured to detect whether there are foreign objects on the mine car track based on the obtained image and text fusion features and the preset foreign object detection model; when foreign objects are present, the module identifies the size of the foreign objects and determines their location, thus completing the real-time monitoring of foreign objects on the mine car track.

[0088] The detailed steps are the same as those of the foreign object detection method based on deep neural networks provided in Example 1, and will not be repeated here.

[0089] Example 3

[0090] Embodiment 3 of the present invention provides a computer-readable storage medium.

[0091] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the foreign object detection method for mine car tracks based on deep neural networks as described in Embodiment 1 of the present invention.

[0092] The detailed steps are the same as those of the foreign object detection method based on deep neural networks provided in Example 1, and will not be repeated here.

[0093] Example 4

[0094] Embodiment 4 of the present invention provides an electronic device.

[0095] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, it implements the steps in the foreign object detection method based on a deep neural network for mine car tracks as described in Embodiment 1 of the present invention.

[0096] The detailed steps are the same as those of the foreign object detection method based on deep neural networks provided in Example 1, and will not be repeated here.

[0097] Example 5

[0098] Embodiment 5 of the present invention provides a computer program product.

[0099] A computer program product includes software code, wherein the program in the software code performs the steps of the foreign object detection method based on deep neural network in mine car track as described in Embodiment 1 of the present invention.

[0100] The detailed steps are the same as those of the foreign object detection method based on deep neural networks provided in Example 1, and will not be repeated here.

[0101] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0102] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0106] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0107] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A method for detecting foreign objects on a mine car track based on a deep neural network, characterized by, The method comprises the following steps: acquiring real-time images of a mine car track and text prompt words; extracting hierarchical multi-modal features of the acquired real-time images; fusing the hierarchical multi-modal features and the text prompt words obtained by using a multi-modal deep neural network to obtain image-text fusion features of the mine car track; detecting whether there is a foreign object in the mine car track according to the obtained image-text fusion features of the mine car track and a preset foreign object detection model; the preset foreign object detection model adopts a deep neural network detection model, and the deep neural network detection model adopts a composite loss function for model training; The composite loss function For , wherein, represents the weight of each loss function, and ; represents different loss functions, wherein, represents a text loss function, adopts a cross-entropy loss function, that is, , represents the labeled text information, represents the predicted text information output by the core module of the open-source multi-modal large model of the text modal branch of the deep neural network detection model; The role of is to train the open-source multi-modal large model and optimize its text feature representation capability. The training process is part of the overall deep neural network detection model training process and ultimately serves to improve the detection accuracy of mine car track foreign objects; represents a target detection loss function, adopts an intersection over union of the predicted box and the real box, that is, , represents a target detection prediction box, represents a label box; represents a first layer of graph-text joint loss function, represents a second layer of graph-text joint loss function, represents a third layer of graph-text joint loss function, , and all adopt a contrast loss function, , ， ； wherein, is the number of samples in a batch, is the number of negative samples in a batch, positive samples represent matching image and text pairs, and negative samples represent non-matching image and text pairs; represents the first layer, the th image feature vector, represents the first layer, the th text feature vector in positive samples, represents the first layer, the th text feature vector in negative samples; represents the second layer, the th image feature vector, represents the second layer, the th text feature vector in positive samples, represents the second layer, the th text feature vector in negative samples; represents the third layer, the th image feature vector, represents the third layer, the th text feature vector in positive samples, represents the third layer, the th text feature vector in negative samples, represents the cosine similarity measure, represents the temperature parameter used to adjust the similarity score distribution; when there is a foreign object, the size of the foreign object is identified and the position of the foreign object is determined to complete real-time monitoring of the foreign object in the mine car track. 2.The mine car track foreign matter detection method based on a deep neural network according to claim 1, wherein, The deep neural network detection model also adopts gradient descent back propagation for model training. 3.The mine car track foreign matter detection method based on a deep neural network according to claim 1, wherein, Based on gradient descent back propagation, the contrast loss function is minimized, the deep neural network detection model is supervised by the loss function, and the training of the foreign object detection model is completed. 4.The mine car track foreign matter detection method based on a deep neural network according to claim 1, wherein, The obtained image-text fusion features of the mine car track are detected by using the foreign object detection model, the position and size of the foreign object are marked in the acquired real-time images of the mine car track by using a target detection box, and real-time monitoring of the foreign object in the mine car track is completed. 5.The mine car track foreign matter detection method based on a deep neural network according to claim 1, characterized in that, Based on convolutional layers, hierarchical multi-modal features are extracted and fused, linear layers are combined to map and match the text prompt words to the same dimension, cross self-attention mechanisms and self-attention mechanisms are used to extract feature correlations between the hierarchical multi-modal features and the text prompt words, and saliency features are fused; The features of the hierarchical multi-modal features and the text prompt words are cooperatively extracted by using a bypass additional splicing operation, which is used to form a residual neural network, and the sum of the backbone network and the bypass residual network is obtained to obtain the image-text fusion features of the mine car track.

6. A deep neural network based foreign object detection system for mine car tracks, applying a deep neural network based foreign object detection method according to any one of claims 1 to 5, characterized in that, The method comprises the following steps: an acquisition module configured to acquire real-time images of a mine car track and text prompt words; an extraction module configured to extract hierarchical multi-modal features of the acquired real-time images; a fusion module configured to fuse the hierarchical multi-modal features and the text prompt words obtained by using a multi-modal deep neural network to obtain image-text fusion features of the mine car track; a detection module configured to detect whether there is a foreign object in the mine car track according to the obtained image-text fusion features of the mine car track and a preset foreign object detection model; when there is a foreign object, the size of the foreign object is identified and the position of the foreign object is determined to complete real-time monitoring of the foreign object in the mine car track.

7. A computer readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to realize the steps of the deep neural network-based mine car track foreign object detection method in any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the program to realize the steps of the deep neural network-based mine car track foreign object detection method in any one of claims 1-5.

9. A computer program product comprising software code, characterized in that, The program in the software code executes the steps of the deep neural network-based mine car track foreign object detection method in any one of claims 1-5.

Images