A training method, device, equipment and medium for railway foreign object phrase location model

Abstract

The invention relates to the technical field of target detection, and discloses a training method, device, equipment and medium for a railway foreign object phrase positioning model. The method can cross-align an image feature vector and a text feature vector to obtain an aligned image feature vector and an aligned text feature vector. The aligned image feature vector and the aligned text feature vector are used for training, so that the model can accurately correspond the foreign object described in the text to the target area in the image, and ensure that the phrase positioning in complex scenes is accurate and correct. The method is suitable for fine-grained scene description in railway foreign object detection. The trained railway foreign object phrase positioning model can effectively execute and complete the railway foreign object phrase positioning task, and effectively realize the phrase positioning of the railway foreign object.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a training method, device, equipment and medium for a railway foreign object phrase positioning model. Background Art

[0002] Object detection refers to detecting single objects or word-level objects in an image. It is mainly used to identify and locate single objects, such as stones and branches. Phrase localization is an extension of object detection. It is necessary to locate objects that meet complex descriptions from the image. For example, when the phrase description is "stones on the rails", it is necessary to detect the object of the stone on the rails in the image. Phrase localization not only recognizes the target object, but also understands the relationship between the target and its modifiers or environmental information. For example, "stones on the rails" can be accurately identified and distinguished from "stones outside the rails" through phrase localization.

[0003] Rail foreign body detection is an important part of railway safety management. It aims to monitor the rails and their surroundings in real time and identify foreign objects that may threaten train safety, such as stones on the rails and stones outside the rails. The railway environment is complex and changeable, and the same foreign objects may have different effects in different locations. The detection system can locate the corresponding railway foreign objects in the image according to the phrase, which is of great significance to the safety of railway operation.

[0004] However, the related technology cannot effectively realize the short-term positioning of foreign objects on railways. Summary of the invention

[0005] The present invention provides a training method, device, equipment and medium for a railway foreign object phrase positioning model, which is used to solve the defect of effectively realizing the phrase positioning of railway foreign objects in the related technology. The trained railway foreign object phrase positioning model can effectively execute and complete the railway foreign object phrase positioning task, and effectively realize the phrase positioning of railway foreign objects.

[0006] In a first aspect, the present invention provides a training method for a railway foreign object phrase positioning model, wherein the railway foreign object phrase positioning model includes a recognition module, an encoding module, a feature enhancement module and a detection module; the method includes:

[0007] Obtaining a text feature vector of a railway foreign object phrase text and an image feature vector of a railway foreign object image;

[0008] Inputting the text feature vector of the railway foreign object phrase text into the recognition module, so that the recognition module performs named entity recognition on the text feature vector according to a plurality of preset phrase structure categories, and obtains a plurality of structure texts in the railway foreign object phrase text and a probability distribution of the structure category corresponding to each of the structure texts;

[0009] Determine the phrase structure category loss according to each of the structure texts and the corresponding structure category probability distribution;

[0010] Based on the encoding module and the structural category probability distribution, generating a structural text feature vector corresponding to each of the preset phrase structural categories;

[0011] Based on the feature enhancement module and the structured text feature vector, cross-feature alignment is performed on the image feature vector and the text feature vector to obtain an aligned image feature vector corresponding to the image feature vector and an aligned text feature vector corresponding to the text feature vector;

[0012] Inputting the aligned image feature vector and the aligned text feature vector into the detection module to perform target detection, obtaining corresponding detection results, and determining the detection result loss according to the detection results;

[0013] The railway foreign object phrase localization model is updated based on the detection result loss and the phrase structure category loss to obtain a trained target railway foreign object phrase localization model.

[0014] Optionally, the multiple preset phrase structure categories include modifiers, object words and relational words, and the structural category probability distribution corresponding to each of the structural texts includes the probabilities that the phrase structure categories corresponding to the structural texts are the modifiers, the object words and the relational words respectively.

[0015] Optionally, determining the phrase structure category loss according to each of the structure texts and the corresponding structure category probability distribution includes:

[0016] For any of the structured texts, according to the cross entropy loss calculation method and the structure category probability distribution corresponding to the structured text, the cross entropy loss corresponding to the structured text is calculated;

[0017] The cross entropy losses corresponding to all the structured texts are averaged to obtain the corresponding average value as the phrase structure category loss.

[0018] Optionally, generating a structural text feature vector corresponding to each of the preset phrase structural categories based on the encoding module and the structural category probability distribution includes:

[0019] For any of the structured texts, determining a maximum probability in the structure category probability distribution corresponding to the structured text, and determining the preset phrase structure category corresponding to the maximum probability as the target phrase structure category corresponding to the structured text;

[0020] When the target phrase structure category corresponding to the structure text is the modifier, encoding the structure text through the encoding module to obtain a corresponding modifier feature vector;

[0021] When the target phrase structure category corresponding to the structure text is the object word, the structure text is encoded by the encoding module to obtain a corresponding object word feature vector;

[0022] When the target phrase structure category corresponding to the structured text is the relational word, the relational word feature vector is extracted from the text feature vector by the encoding module.

[0023] Optionally, the feature enhancement module includes a modifier feature enhancement module, an object word feature enhancement module and an overall image and text feature enhancement module; the railway foreign body phrase positioning model also includes a self-attention mechanism module;

[0024] The cross-feature alignment of the image feature vector and the text feature vector based on the feature enhancement module and the structured text feature vector to obtain an aligned image feature vector corresponding to the image feature vector and an aligned text feature vector corresponding to the text feature vector includes:

[0025] Inputting the image feature vector and the modifier feature vector into the modifier feature enhancement module for cross feature alignment to obtain a first aligned image feature vector;

[0026] Inputting the first aligned image feature vector and the object word feature vector into the object word feature enhancement module for cross feature alignment to obtain a second aligned image feature vector;

[0027] Fusing the relation word feature vector with the second aligned image feature vector to obtain a fused feature vector, and performing fusion optimization on the fused feature vector through the self-attention mechanism module to obtain an optimized feature vector;

[0028] The optimized feature vector and the text feature vector are input into the overall image-text feature enhancement module for cross-feature alignment to obtain the aligned image feature vector and the aligned text feature vector.

[0029] Optionally, the detection result includes corresponding object category identification and bounding box information;

[0030] The determining the detection result loss according to the detection result includes:

[0031] Determining a category recognition loss based on a difference between the object category identifier and a true category identifier;

[0032] Determining a bounding box positioning loss based on a difference between the bounding box information and the true bounding box information;

[0033] The category recognition loss and the bounding box positioning loss are taken as the detection result loss as a whole.

[0034] Optionally, updating the railway foreign object phrase localization model based on the detection result loss and the phrase structure category loss to obtain a trained target railway foreign object phrase localization model includes:

[0035] The category recognition loss, the bounding box positioning loss and the phrase structure category loss in the detection result loss are summed to obtain a total loss;

[0036] If it is determined that the total loss is not less than the set loss, the recognition module, the feature enhancement module and / or the detection module in the railway foreign object phrase positioning model are updated to obtain an updated railway foreign object phrase positioning model, and based on the text feature vector of the new railway foreign object phrase text and the image feature vector of the new railway foreign object image, the updated railway foreign object phrase positioning model is continuously trained until a trained target phrase positioning model is obtained.

[0037] In a second aspect, the present invention provides a training device for a railway foreign object phrase positioning model, wherein the railway foreign object phrase positioning model includes a recognition module, a coding module, a feature enhancement module and a detection module; the device includes:

[0038] An acquisition unit, used for acquiring a text feature vector of a railway foreign object phrase text and an image feature vector of a railway foreign object image;

[0039] A first input unit is used to input the text feature vector of the railway foreign object phrase text into the recognition module, so that the recognition module performs named entity recognition on the text feature vector according to a plurality of preset phrase structure categories, and obtains a plurality of structure texts in the railway foreign object phrase text and a probability distribution of the structure category corresponding to each of the structure texts;

[0040] A first determining unit, configured to determine a phrase structure category loss according to each of the structure texts and the corresponding structure category probability distribution;

[0041] A generating unit, configured to generate a structural text feature vector corresponding to each of the preset phrase structural categories based on the encoding module and the structural category probability distribution;

[0042] an alignment unit, configured to perform cross-feature alignment on the image feature vector and the text feature vector based on the feature enhancement module and the structured text feature vector, so as to obtain an aligned image feature vector corresponding to the image feature vector and an aligned text feature vector corresponding to the text feature vector;

[0043] A second input unit, used for inputting the aligned image feature vector and the aligned text feature vector into the detection module to perform target detection and obtain a corresponding detection result;

[0044] A second determining unit, configured to determine a detection result loss according to the detection result;

[0045] An updating unit is used to update the railway foreign object phrase localization model based on the detection result loss and the phrase structure category loss to obtain a trained target railway foreign object phrase localization model.

[0046] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the training method of the railway foreign object phrase positioning model of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0047] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the training method for a railway foreign object phrase location model of the first aspect or any corresponding embodiment thereof.

[0048] The training method, device, equipment and medium of the railway foreign object phrase positioning model provided by the present invention can cross-align image feature vectors and text feature vectors to obtain aligned image feature vectors and aligned text feature vectors, and use the aligned image feature vectors and aligned text feature vectors for training, so that the model can accurately correspond the foreign objects described in the text to the target area in the image, ensure the accurate and correct phrase positioning in complex scenes, and is suitable for fine-grained scene description in railway foreign object detection. The trained railway foreign object phrase positioning model can effectively execute and complete the railway foreign object phrase positioning task, and effectively realize the phrase positioning of railway foreign objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A flow chart of a method for training a railway foreign object phrase location model provided by an embodiment of the present invention;

[0051] Figure 2 A flowchart of another method for training a railway foreign object phrase location model provided by an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of the structure of a training device for a railway foreign object phrase location model provided by an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Combine the following Figure 1-Figure 2 The present invention describes the training method of the railway foreign object phrase positioning model.

[0056] like Figure 1 As shown, this embodiment proposes a first method for training a railway foreign object phrase location model, wherein the railway foreign object phrase location model includes a recognition module, an encoding module, a feature enhancement module, and a detection module. The method may include the following steps:

[0057] S101, obtaining a text feature vector of a railway foreign object phrase text and an image feature vector of a railway foreign object image.

[0058] The railway foreign object phrase location model may be an open target detection model for performing phrase location tasks. It should be noted that the railway foreign object phrase location model is a model that needs to be trained to meet performance requirements. In this embodiment, after obtaining a trained railway foreign object phrase location model through machine learning, the trained railway foreign object phrase location model may be used to perform the railway foreign object phrase location task.

[0059] Among them, the railway foreign object phrase text is a phrase description text related to railway foreign objects, such as "stones on the rails" and "stones outside the rails".

[0060] The railway foreign object image is an image to be subjected to target detection based on the railway foreign object phrase text.

[0061] Among them, in this embodiment, the railway foreign object short text can be encoded to obtain a text feature vector, and the railway foreign object image can be encoded to obtain an image feature vector.

[0062] S102. Input the text feature vector of the railway foreign object short text into the recognition module, so that the recognition module performs named entity recognition on the text feature vector according to a plurality of preset phrase structure categories set, and obtains a plurality of structural texts in the railway foreign object short text and the corresponding structural category probability distribution of each structural text.

[0063] Specifically, the recognition module can be a named entity recognition module for performing named entity recognition on the short text.

[0064] Among them, the preset phrase structure category can be a phrase structure category set for the railway foreign object short text.

[0065] Optionally, the plurality of preset phrase structure categories include a modifier, an object word, and a relational word, and the corresponding structural category probability distribution of each structural text includes the probabilities that the corresponding phrase structure categories of the structural text are a modifier, an object word, and a relational word respectively.

[0066] It should be noted that the modifiers can be divided into three categories: spatial position category, environmental condition category, and state attribute category. The spatial position category modifiers are used to describe the relative position between the target and other objects or the environment, such as "on the railway track" or "beside the railway track". The environmental condition category modifiers reflect the external environmental factors where the target is located, such as light, weather, etc., for example, "in the shadow" or "the target under low light". The state attribute category modifiers describe the physical state or material change of the target itself, such as "loose stones" or "rusty fittings". Through these three categories of modifiers, the model can more accurately understand the attributes of the target and its relationship with the surrounding environment, thereby improving the accuracy and robustness of detection.

[0067] The object word is the target to be detected in the short text, such as "stone" in "stone on the railway track", and also such as "branch" in "branch on the railway track". The relational word can be defined as the remaining part of the short text except the object word and the modifier. For example, "on" and "of" in "stone on the railway track".

[0068] Specifically, the recognition module can perform named entity recognition on the text feature vector according to the set multiple preset phrase structure categories, and obtain multiple structure texts in the railway foreign object phrase text and the structure category probability distribution corresponding to each structure text. For example, when the railway foreign object phrase text is "stones on the rails", the recognition module can perform named entity recognition on the railway foreign object phrase text, identify four structure texts "railway", "on", "of" and "stone", and the probability that each structure text belongs to the preset phrase structure category. Taking "railway" as an example, the structure category probability distribution corresponding to "railway" includes the probability that "railway" belongs to a modifier, the probability that "railway" belongs to an object word, and the probability that "railway" belongs to a relational word.

[0069] S103: Determine the phrase structure category loss according to each structure text and the corresponding structure category probability distribution.

[0070] Optionally, step S103 may include:

[0071] For any structured text, the cross entropy loss corresponding to the structured text is calculated according to the cross entropy loss calculation method and the probability distribution of the structured category corresponding to the structured text;

[0072] The cross entropy losses corresponding to all structured texts are averaged to obtain the corresponding average value as the phrase structure category loss.

[0073] Specifically, this embodiment can calculate the cross entropy loss corresponding to any structure text based on the cross entropy loss calculation formula and the structure category probability distribution corresponding to the structure text. Afterwards, this embodiment can calculate the average value of the cross entropy losses corresponding to all structure texts, and use the calculated average value as the phrase structure category loss.

[0074] It should be noted that the phrase structure category loss represents the average loss of the model when classifying each word into object words, modifier words and relation words. When training the model, this embodiment can improve the model's ability to accurately recognize phrase structures by reducing this loss.

[0075] S104: Generate a structural text feature vector corresponding to each preset phrase structure category based on the encoding module and the structural category probability distribution.

[0076] Specifically, this embodiment can generate a structural text feature vector corresponding to each preset phrase structural category according to the encoding module and the structural category probability distribution corresponding to each structural text.

[0077] Optionally, for any structured text, a maximum probability is determined in a probability distribution of structure categories corresponding to the structured text, and a preset phrase structure category corresponding to the maximum probability is determined as a target phrase structure category corresponding to the structured text;

[0078] When the target phrase structure category corresponding to the structure text is a modifier, the structure text is encoded through the encoding module to obtain the corresponding modifier feature vector;

[0079] When the target phrase structure category corresponding to the structure text is an object word, the structure text is encoded by the encoding module to obtain the corresponding object word feature vector;

[0080] When the target phrase structure category corresponding to the structured text is a relational word, the relational word feature vector is extracted from the text feature vector through the encoding module.

[0081] For example, when the railway foreign object phrase text is "stones on the railroad tracks", this embodiment performs named entity recognition on the railway foreign object phrase text, identifies four structural texts "railway", "on", "of" and "stone", and the probability that each structural text belongs to a preset phrase structure category, taking "railway" as an example, if in the structural category probability distribution corresponding to "railway", the probability that "railway" belongs to a modifier is 0.7, the probability that "railway" belongs to an object word is 0.2, and the probability that "railway" belongs to a relational word is 0.1, then this embodiment can determine that the modifier is the target phrase structure category corresponding to "railway", and then this embodiment can encode "railway" through the encoding module to obtain the corresponding modifier feature vector.

[0082] Specifically, position coding in the related art is used to help the model understand the relative or absolute position of each image or sequence element in the overall structure. Drawing on the principle of position coding, the design of relation coding aims to capture the relationship between the target and the modification described in the text, and map these relationships to the corresponding image coding. Specifically, when the target phrase structure category corresponding to the structure text is a relational word, this embodiment can refer to the extraction method of position coding in the related art, and extract the relational word feature vector from the text feature vector of the railway foreign body phrase text based on the following transformation.

[0083] .

[0084] in, is the feature vector of the relation word. represents a feed-forward neural network, Text feature vector representing the railway foreign object phrase text.

[0085] S105 , based on the feature enhancement module and the structured text feature vector, cross-feature alignment is performed on the image feature vector and the text feature vector to obtain an aligned image feature vector corresponding to the image feature vector and an aligned text feature vector corresponding to the text feature vector.

[0086] Specifically, this embodiment can perform cross-feature alignment on the image feature vector of the railway foreign object image and the text feature vector of the railway foreign object phrase text according to the feature enhancement module and each structural text feature vector, thereby realizing feature alignment and feature enhancement of the image feature vector and the text feature vector, and obtaining an aligned image feature vector and an aligned text feature vector.

[0087] Optionally, in the training method of other railway foreign object phrase location models proposed in this embodiment, the feature enhancement module includes a modifier feature enhancement module, an object word feature enhancement module and an overall image and text feature enhancement module. The railway foreign object phrase location model also includes a self-attention mechanism module. At this time, step S105 may include:

[0088] Inputting the image feature vector and the modifier feature vector into the modifier feature enhancement module for cross feature alignment to obtain a first aligned image feature vector;

[0089] Inputting the first aligned image feature vector and the object word feature vector into the object word feature enhancement module for cross feature alignment to obtain a second aligned image feature vector;

[0090] The relation word feature vector is fused with the second aligned image feature vector to obtain a fused feature vector, and the fused feature vector is fused and optimized through a self-attention mechanism module to obtain an optimized feature vector;

[0091] The optimized feature vector and text feature vector are input into the overall image-text feature enhancement module for cross-feature alignment to obtain an aligned image feature vector and an aligned text feature vector.

[0092] It should be noted that the dimensions of all feature vectors in this embodiment may be the same. In this embodiment, the relationship word feature vector may be added to the second aligned image feature vector to fuse the relationship word feature vector and the second aligned image feature vector. At this time, the fused feature vector is a feature vector obtained by adding the relationship word feature vector to the second aligned image feature vector.

[0093] Specifically, this embodiment can allocate self-attention to the fused feature vector through the self-attention mechanism module, so that the image feature vector can better fuse the relationship word feature vector, so that the image feature vector can better understand the relationship words and focus on the important features in the image.

[0094] S106: Input the aligned image feature vector and the aligned text feature vector into a detection module to perform target detection and obtain corresponding detection results.

[0095] Specifically, in this embodiment, the detection module may perform target detection on the aligned image feature vector and the aligned feature vector to obtain a corresponding detection result.

[0096] Optionally, the detection result includes corresponding object category identification and bounding box information.

[0097] S107. Determine the test result loss according to the test result.

[0098] Specifically, this embodiment can calculate the corresponding detection result loss according to the detection result output by the detection module.

[0099] Optionally, in other training methods for railway foreign object phrase localization models proposed in this embodiment, when the detection result includes corresponding object category identification and bounding box information, step S107 may include:

[0100] Determine the category recognition loss based on the difference between the object category identity and the true category identity;

[0101] Determine the bounding box localization loss based on the difference between the bounding box information and the true bounding box information;

[0102] The category recognition loss and bounding box positioning loss are taken as a whole as the detection result loss.

[0103] Specifically, in order to ensure that the model effectively handles target recognition and modification information understanding in the multi-task learning process, this embodiment designs a category recognition loss to evaluate the model's prediction accuracy for the target category, using a binary cross entropy loss:

[0104] .

[0105] in, Loss for class identification. is the number of samples or the number of categories. is the true label (0 or 1), indicating Whether a category is a target. The probability of whether the category is the target predicted by the model, the value obtained after Sigmoid activation, ranges from [0,1]. Represents the loss value when the model predicts the existence of the target. Represents the loss value when the model predicts that the target does not exist.

[0106] Specifically, the bounding box positioning loss is used to evaluate the difference between the target bounding box predicted by the model and the real bounding box, and is calculated using the L1 loss. The specific formula is:

[0107] .

[0108] in, is the bounding box localization loss. is the number of samples, i.e. the number of bounding boxes. The model predicts The coordinates of a bounding box (usually including the center coordinates and width and height, or the coordinates of four vertices). is the coordinate of the true i-th bounding box. Represents the difference between the bounding box predicted by the model and the true bounding box, and the absolute value of the difference is calculated using the L1 norm.

[0109] S108. Update the railway foreign object phrase localization model based on the detection result loss and the phrase structure category loss to obtain a trained target railway foreign object phrase localization model.

[0110] Specifically, this embodiment can update the model parameters in the railway foreign object phrase positioning model according to the detection result loss and the phrase structure category loss.

[0111] Optionally, in the training method of other railway foreign object phrase location model proposed in this embodiment, step S108 may include:

[0112] The category recognition loss, bounding box positioning loss, and phrase structure category loss in the detection result loss are summed to get the total loss;

[0113] If it is determined that the total loss is not less than the set loss, the recognition module, the feature enhancement module and / or the detection module in the railway foreign object phrase localization model are updated to obtain an updated railway foreign object phrase localization model, and based on the text feature vector of the new railway foreign object phrase text and the image feature vector of the new railway foreign object image, the updated railway foreign object phrase localization model is continued to be trained until a trained target phrase localization model is obtained.

[0114] Specifically, this embodiment can assign different weights to each type of loss according to the importance of category recognition loss, bounding box positioning loss, and phrase structure category loss, and perform weighted summation according to the category recognition loss, bounding box positioning loss, and phrase structure category loss, as well as their respective weights to obtain the total loss. It should be noted that this embodiment can first initialize the weights equally, and automatically adjust the weights between the target detection loss and the modifier understanding loss by analyzing the data distribution in real time. Selecting the appropriate weight coefficient needs to be adjusted based on the performance of the model on the validation set, which involves the impact on different tasks and the trade-offs between them. The adjustment of weights is a process of experimentation and optimization, and multiple experiments are required to find the best setting.

[0115] Specifically, this embodiment can update the model parameters of the recognition module, feature enhancement module and / or detection module in the railway foreign object phrase positioning model to update the railway foreign object phrase positioning model and improve the detection accuracy of the railway foreign object phrase positioning model.

[0116] It should be noted that this embodiment can improve the accuracy and flexibility of phrase positioning. This embodiment can simultaneously handle target detection and modifier understanding tasks by introducing a multi-task learning framework including name recognition, image-text alignment, and target detection. The railway foreign object phrase positioning model can effectively handle complex phrase structures, especially when describing the relationship between the target and the environment (such as "stones on the rails" and "stones outside the rails"). This flexibility significantly improves the model's detection accuracy of railway foreign objects, and can cope with foreign object positioning problems under various language descriptions.

[0117] This embodiment can enhance the ability to adapt to environmental changes. This embodiment designs an environmental feature loss function to enable the model to adapt to common changes in lighting, weather, and viewing angles in railway scenes. Especially under extreme conditions, such as heavy rain, snow, strong winds, or low-light environments at night, the model can still maintain high stability and detection accuracy. The performance of the model under different environmental conditions ensures the reliability of railway safety.

[0118] This embodiment can achieve efficient multimodal alignment of images and texts. By optimizing the multimodal fusion strategy, this embodiment achieves efficient alignment between image and text features. The model can accurately match the foreign objects described in the text with the target area in the image, ensuring accurate positioning of phrases in complex scenes, and is suitable for fine-grained scene description in railway foreign object detection.

[0119] The training method of the railway foreign object phrase localization model proposed in this embodiment can cross-align the image feature vector and the text feature vector to obtain the aligned image feature vector and the aligned text feature vector, and use the aligned image feature vector and the aligned text feature vector for training, so that the model can accurately correspond the foreign object described in the text with the target area in the image, ensuring accurate phrase localization in complex scenes. It is suitable for fine-grained scene description in railway foreign object detection. The trained railway foreign object phrase localization model can effectively execute and complete the railway foreign object phrase localization task, and effectively realize the phrase localization of railway foreign objects.

[0120] In related technologies, open object detection has become an important research direction in the field of computer vision, aiming to handle target categories that have not been seen during training. Grounding DINO is a target detection model based on multimodal information. By combining image and text features, it aims to improve the accuracy and flexibility of target positioning. The model can effectively understand the semantics of the input text and locate the corresponding objects in the image, thus performing well in complex scenes.

[0121] This embodiment can focus on the application of open target detection technology based on the Grounding DINO model in foreign body detection on rails. By fine-tuning the model, we explore how to integrate new data and categories while retaining the original detection capabilities to improve the accuracy and generalization of the phrase localization task. This aims to provide innovative solutions for railway safety monitoring and promote the practical application of open target detection technology.

[0122] Grounding DINO includes a dual encoder-single decoder architecture, which mainly includes the following components:

[0123] Image and text backbone network: used to extract multi-scale features of images and semantic features of text. This process ensures that the model can fully understand the diversity of input data modalities.

[0124] Feature enhancer: responsible for cross-modal feature fusion, improving the alignment between image and text features, thereby enhancing the model's ability to understand the target.

[0125] Language-guided query selection module: By analyzing the input text, relevant queries are selected from image features for decoder initialization to ensure that the model can accurately focus on the target area.

[0126] Cross-modal decoder: The selected query is input into the self-attention layer and the cross-attention layer to refine the object bounding box and extract phrase information for efficient object localization and recognition.

[0127] The task of phrase localization aims to accurately locate relevant objects in an image based on the input text. Grounding DINO effectively handles this task through multimodal learning, including:

[0128] Multimodal input processing: The model can process images and text simultaneously, establish effective connections, and ensure the comprehensiveness and accuracy of object detection.

[0129] Semantic matching capability: Through advanced feature fusion technology, the model can handle different language structures and expressions to ensure the semantic consistency between the detected object and the input phrase.

[0130] Fine-grained positioning: Grounding DINO can accurately locate the area corresponding to a specific phrase and supports complex phrase combination expressions, thereby providing accurate information in special scenarios such as railway monitoring.

[0131] Despite good performance in phrase localization, the emergence of new categories in open-set scenarios still poses challenges to the adaptability and accuracy of the model. Grounding DINO performs well at the word level in open object detection tasks. By fine-tuning the common word target categories, the pre-trained model can significantly improve its accuracy when introducing new data and new categories. The model combines the original data in this fine-tuning process, which is equivalent to a new pre-training, thus maintaining good generalization ability.

[0132] However, in the phrase localization task, due to the complexity of phrases, how to effectively integrate new data with original data to ensure the accuracy and generalization ability of the model at the phrase level is still an urgent problem to be solved. This shows that in specific application scenarios (such as railway monitoring), it is crucial to further optimize the multimodal data processing solution, especially to improve the generalization ability at the phrase level.

[0133] The training method of the railway foreign object phrase location model proposed in this embodiment can be used to solve the following specific technical problems:

[0134] 1. Automatically distinguish between targets and modifiers: In railway scenarios, phrase localization tasks require the model to accurately distinguish between targets and modifiers, especially when describing the relationship between targets and the environment (such as "stones on the rails" and "stones outside the rails"). Existing models often have difficulty distinguishing between targets and modifiers when dealing with complex phrase structures, resulting in inaccurate target localization.

[0135] 2. Identification of new categories and generalization ability of models: There are many types of foreign objects in railways, and existing models perform poorly in identifying new target categories that have not been seen before, especially in open target detection scenarios, where the generalization ability of the model is relatively limited. In addition, when the model adapts to new data, it is prone to "catastrophic forgetting", that is, while learning new categories, the recognition ability of old targets decreases.

[0136] 3. Ability to adapt to environmental changes: Railway scenes are complex and changeable. Factors such as lighting, weather, and viewing angle will significantly affect the appearance of the target. Existing models are difficult to effectively adapt to these environmental changes, especially when it is necessary to understand the relationship between the target and the environment. The performance is poor, which reduces the stability and reliability of the detection system.

[0137] 4. Fusion and processing of multimodal data: The railway foreign object detection task requires the model to effectively combine image and text information. However, when dealing with complex phrases, especially phrases containing modifiers (such as "stones beside the railroad tracks"), existing multimodal models have difficulty in achieving accurate image and text alignment, resulting in unsatisfactory detection results.

[0138] This embodiment can solve the above technical problems by the following methods:

[0139] 1. Diversity data enhancement.

[0140] This embodiment can introduce diversity data enhancement technology to improve the generalization ability of the model in the task of detecting foreign objects on railway tracks. The railway environment is complex and changeable. Different weather, lighting conditions, seasonal changes, and tracks in different geographical locations may affect the appearance characteristics of foreign objects. At the same time, there are many types of foreign objects (such as stones, branches, garbage, etc.), and these factors require the model to have strong adaptability. Word-type data may cause the model to have insufficient recognition performance in practical applications, especially when encountering new scenes or unseen foreign objects, the model may lose accuracy.

[0141] To solve this problem, this embodiment can adopt a diverse data enhancement strategy to significantly improve the model's adaptability to different scenarios and types of foreign objects, thereby achieving higher detection accuracy and robustness.

[0142] Basic labeling rules: This embodiment can first formulate basic labeling rules for foreign objects on the rails. These rules not only cover the basic physical characteristics of the target, such as shape, color and size, but also include railway-specific terms and definitions of specific events. In the railway foreign object detection task, the modifiers of the target can be divided into three categories: spatial position, environmental condition and state attribute. Spatial position modifiers are used to describe the relative position between the target and other objects or the environment, such as "on the rail" or "next to the rail". Environmental condition modifiers reflect the external environmental factors of the target, such as lighting, weather, etc., such as "in the shadow" or "target under low light". State attribute modifiers describe the physical state or material changes of the target itself, such as "loose stones" or "rusted accessories". Through these three types of modifiers, the model can more accurately understand the properties of the target and its relationship with the surrounding environment, thereby improving the accuracy and robustness of detection.

[0143] Railway-specific scene annotation: In the railway foreign object detection task, the scene is not limited to the rails themselves, but also includes the environment around the rails, such as the revetments on both sides of the track, the tunnel entrances, or the space under the bridge. There may be different kinds of foreign objects in these locations, affecting the safe operation of the train. Therefore, special attention must be paid to the details in these environments during the annotation process. For example, there may be snow accumulated at the tunnel entrance, which may slide onto the rails when the train passes; fallen rocks on the revetments may roll onto the tracks during the rainy season, directly threatening driving safety. The annotation rules take this environmental information into account to provide the model with a richer contextual understanding.

[0144] Artificial intelligence generates expressions: To further expand the training data, this embodiment can use artificial intelligence to generate rich phrase expressions. This process combines annotation rules with natural language processing technology to automatically create diverse descriptions by using generative models (such as generative pre-trained transformers). These descriptions are not only based on existing annotation rules, but can also generate phrase expressions with more details or different grammatical structures to improve data diversity.

[0145] For example, for the same foreign object, artificial intelligence can generate multiple forms of expression, such as: "A loose stone was found on the railroad track, which may affect the operation of the train." "A stone is located on the railroad track, it is recommended to clear it in time." Such diverse expressions not only improve the richness of training data, but also help the model better understand different descriptions of the same scene, thereby improving the ability to understand complex scenes.

[0146] Rule-based labeling and intelligent correction: In order to ensure the accuracy and diversity of the data, this embodiment can adopt a human-machine collaborative labeling method. First, the images are manually annotated in detail according to the basic labeling rules to ensure the professionalism and consistency of the labeling. Subsequently, the artificial intelligence model generates corresponding phrase descriptions for these images. When there is an obvious inconsistency between the phrases generated by the artificial intelligence model and the semantics of the manual annotations, a manual expert can be automatically requested to review and correct them. This method not only ensures data quality, but also greatly enriches the diversity of the data set, allowing the model to be trained more efficiently.

[0147] This embodiment can construct a variety of railway foreign body phrase texts in the above manner, realize the diversity data enhancement for the railway foreign body detection task, thereby improving the generalization ability and detection accuracy of the model. The implementation of this technical solution will provide a more effective solution for railway safety monitoring.

[0148] 2. Network structure modification and multi-task learning design.

[0149] In order to solve the problem of complex phrase positioning in foreign body detection on rails, this embodiment adopts a multi-task learning training strategy. In this architecture, the model is not only responsible for the recognition of the target, but also processes the modification information related to the target (such as spatial position, environmental impact, etc.). To achieve this, the network structure of the model is modified to introduce the recognition module, encoding module and feature enhancement module.

[0150] In image annotation, we mark the precise location of the main targets, which may be objects or specific scene elements. The bounding box around each target is designed to accurately capture its spatial location in the image. We also annotate the decorative information related to the target, such as color, shape, or spatial relationship, to help the model understand how these properties affect the interpretation of the target. For phrases describing the entire scene, we annotate a bounding box that includes all relevant elements, which helps the model learn how to understand and process complex image content as a whole.

[0151] The annotation of text data involves classifying text that describes targets, modifiers, and relationships. We divide the text into three categories: text that describes the main target is labeled "one", text that describes modifier information is labeled "two", and text that describes the relationship between the target and the modifier is labeled "three". This classification method helps the model more accurately identify and classify different elements in the text when performing named entity recognition tasks.

[0152] Data processing begins with the input of images and related description text. The image is processed using Vision Transformer (ViT), which divides the image into multiple patches of equal size, and each patch is converted into a vector of fixed dimension. The processed image features have dimensions N×d, where N is the number of patches and d is the depth of the feature vector, which is generally 768. These image feature vectors can be used as image feature vectors of railway foreign body images as a whole.

[0153] The text data is converted into a vector of dimension L×d through the embedding layer, where L is the length of the text sequence. These text vectors are processed by the self-attention mechanism, and the output text feature dimension is also L×d. These text vectors can be used as the text feature vector of the railway foreign body phrase text as a whole.

[0154] Phase 1: Phrase composition identification.

[0155] The text feature vector of railway foreign object phrase text is processed using a feedforward neural network and a sigmoid module to perform a named entity recognition task to identify the target (i.e., object word), modifier, and relation word in the railway foreign object phrase text. Each type of entity (target, modifier, relation, where the relation is defined as the remaining part of the phrase excluding the target and modifier) ​​is identified separately, combined with the phrase text and corresponding encodings of the target and modifier are generated respectively. These encodings have their own length and dimension, forming a representation similar to the phrase encoding structure. This process is optimized by classification loss.

[0156] like Figure 2 As shown, in this embodiment, the text feature vector of the railway foreign body phrase text can be input into the recognition module, so that the recognition module performs named entity recognition on the text feature vector according to the set multiple preset phrase structure categories, and obtains multiple structure texts in the railway foreign body phrase text and the structure category probability distribution corresponding to each structure text. The structure category probability distribution corresponding to each structure text includes the probability that the phrase structure category corresponding to the structure text is a modifier, an object word, and a relational word. Afterwards, for any structure text, this embodiment can determine the maximum probability in the structure category probability distribution corresponding to the structure text, and determine the preset phrase structure category corresponding to the maximum probability as the target phrase structure category corresponding to the structure text; when the target phrase structure category corresponding to the structure text is a modifier, the structure text is encoded by the encoding module to obtain the corresponding modifier feature vector; when the target phrase structure category corresponding to the structure text is an object word, the structure text is encoded by the encoding module to obtain the corresponding object word feature vector; when the target phrase structure category corresponding to the structure text is a relational word, the relational word feature vector is extracted from the text feature vector by the encoding module.

[0157] It should be noted that Figure 2 The original image code is the image feature vector of the railway foreign object image, the original phrase code is the text feature vector of the railway foreign object phrase text, and the original phrase text includes multiple structural texts obtained after the text feature vector of the railway foreign object phrase text is subjected to named entity recognition by the recognition module in this embodiment. The modifier code is the modifier feature vector obtained after the structural text corresponding to the modifier is encoded in this embodiment, the target code is the object word feature vector obtained after the structural text corresponding to the object word in multiple structural texts is encoded in this embodiment, and the relationship code is the relationship word feature vector obtained by feature extraction of the original phrase code in this embodiment. The modifier feature enhancer and the target feature enhancer are the modifier feature enhancement module and the object word feature enhancement module respectively, and the overall image-text feature enhancer is the overall image-text feature enhancement module. The new image code and the new phrase code are the aligned image feature vector and the aligned text feature vector respectively.

[0158] The second stage: multiple rounds of feature alignment.

[0159] After identifying the targets, modifiers, and relations in the text, multiple rounds of feature alignment are performed:

[0160] First layer alignment (modifiers). The modifier encoding is first used in the alignment process and input into the criss-cross attention module. This step allows the model to focus on the scene or background area in the image. Through the criss-cross attention mechanism, the parts of the image features related to the modifiers receive higher attention weights. This helps the model better understand and interpret the environment and context in the image.

[0161] like Figure 2 As shown, in this embodiment, the image feature vector and the modifier feature vector can be input into the modifier feature enhancement module for cross feature alignment to obtain a first aligned image feature vector.

[0162] Second layer alignment (target word). The target word encoding is then input into the model. At this point, the model has obtained the alignment information of the modifier to the background area, and can further focus on the specific target area in the image without ignoring the influence of the modifier. This step-by-step alignment method ensures that the model can accurately identify and locate the target entity while integrating scene information. The loss function used in the alignment process includes recognition and localization losses.

[0163] As shown in the figure, in this embodiment, the first aligned image feature vector and the object word feature vector can be input into the object word feature enhancement module for cross feature alignment to obtain the second aligned image feature vector.

[0164] The third stage: relational coding generation.

[0165] In the visual model of related technologies, position encoding is used to help the model understand the relative or absolute position of each image or sequence element in the overall structure. Drawing on the principle of position encoding, the design of relation encoding aims to capture the relationship between the target and the modification described in the text and map these relationships to the corresponding image encoding.

[0166] like Figure 2 As shown, in this embodiment, the relation word feature vector can be fused with the second aligned image feature vector to obtain a fused feature vector, and further processed by the self-attention mechanism module to perform fusion optimization on the fused feature vector to obtain an optimized feature vector. The processed encoding is aligned with the overall phrase encoding to ensure that the image features fully reflect the semantic relationship described in the text.

[0167] 3. Combining continuous learning with open target detection.

[0168] In order to enhance the adaptability of the model to open target detection scenarios, this embodiment can also be combined with a continuous learning method. When introducing new categories or new data, the model is fine-tuned through progressive learning technology to ensure that the introduction of new data does not destroy the model's ability to recognize existing targets and avoid "catastrophic forgetting" of the model.

[0169] Specifically, this embodiment can gradually introduce new categories. Set to gradually introduce 5-10 new categories each time, combined with a small amount of annotated data (100-500 annotated images), to ensure that the model quickly adapts to new categories while retaining the recognition ability of old categories. Data is introduced in batches to gradually adjust the recognition ability of the model to avoid overfitting new categories.

[0170] Specifically, this embodiment can prevent forgetting based on features. Through knowledge distillation technology, feature information of old categories is retained when new categories are introduced to prevent the model from forgetting old categories. The temperature softening method is used to smooth the category distribution to ensure the balance of learning new and old categories. Through feature alignment, the feature differences between the new and old models are minimized to ensure that the introduction of new categories will not affect the recognition of old targets.

[0171] Specifically, this embodiment can perform adaptive learning adjustment. When the model introduces new categories, it will adaptively adjust the weight distribution of new and old data. The weight of new categories is higher in the early stage, and gradually balanced in the later stage. This mechanism ensures that the model can effectively identify new and old categories at the same time.

[0172] It is understandable that this embodiment can improve the efficiency and diversity of data enhancement. This embodiment can significantly enhance the diversity of data by combining intelligent generation of diverse phrase descriptions and basic annotation rules. This enhancement method ensures that the model has stronger generalization capabilities when dealing with complex and changing scenarios, and can greatly improve the accuracy and efficiency of foreign object detection, especially when facing a variety of foreign objects on the rails (such as stones, branches, tools, etc.).

[0173] This embodiment can enhance the generalization ability of open object detection. By combining continuous learning with an adaptive loss function, this embodiment can significantly improve the performance of the model in open object detection. When faced with new categories that have never been seen (such as foreign objects of different materials and shapes), the model can maintain a high level of detection accuracy while avoiding "catastrophic forgetting" caused by the introduction of new data. This makes the model more applicable in foreign object detection on rails and can effectively respond to the emergence of various new foreign objects.

[0174] This embodiment can provide a scalable open object detection solution. Through the continuous learning mechanism, it can achieve progressive updates when new data is introduced. The model can quickly adapt to new categories in the railway foreign object detection task, while maintaining high-precision recognition of existing categories. The solution is highly scalable and adapts to long-term dynamic changes in the railway safety monitoring environment, providing stable and reliable technical support for railway safety.

[0175] The railway foreign object phrase localization model of this embodiment is provided with a multi-task learning architecture dedicated to railway foreign object detection. In this embodiment, a dedicated multi-task learning architecture is designed for the railway foreign object detection task. The model can simultaneously detect foreign objects and parse the spatial relationship between the target and the rail (such as "stone on the rail"), thereby improving the accuracy of the phrase localization task. This architecture is particularly suitable for task processing under complex phrases and multiple scenarios (such as various environmental changes in the foreign object detection task on the rail), which significantly improves the generalization ability and accuracy of the model. The multi-task learning architecture is aimed at the target and modifier parsing tasks in railway foreign object detection, ensuring the efficiency and accuracy of the model in complex railway scenarios. This architecture and its optimization algorithm ensure the real-time response and efficiency of the model in complex scenarios by dynamically adjusting the task processing flow.

[0176] This embodiment implements the design of an adaptive loss function for railway scenarios. This embodiment can be based on the Grounding DINO model, and by improving the loss function, an adaptive weight adjustment mechanism is implemented, so that the model can dynamically adapt to different phrase structures (such as the relationship between the target and the modifier) ​​and complex railway environments (such as lighting, weather, etc.). Through weight adjustment, this mechanism can respond to environmental changes in real time, such as responding to extreme weather and different lighting conditions, so that the model maintains stable performance in changeable railway scenarios. The adaptive loss function can adapt the model to different phrase structures and railway environmental conditions (such as lighting, viewing angle) by dynamically adjusting the task weights. This mechanism performs particularly well in extreme environments (such as insufficient lighting and heavy rain), ensuring that the model always maintains high-precision target detection and modifier parsing capabilities.

[0177] This embodiment has a continuous learning mechanism for railway foreign object detection. This embodiment can be combined with a continuous learning mechanism so that the model can maintain the detection capability of existing target categories when introducing new foreign object categories, avoiding "catastrophic forgetting". Through batch progressive learning, a smooth transition in the process of introducing new categories is ensured, and the impact of new categories on the detection performance of old categories is avoided. By continuously optimizing the training strategy, the model can quickly adapt to new categories in the dynamic changes of railway scenes, ensuring the effect of open target detection. Through the continuous learning method, the model can adapt to new foreign object detection in railway scenes, ensuring that the introduction of new categories in open target detection tasks does not affect the detection effect of old categories. This continuous learning mechanism enables the model to maintain efficient performance in the long-term dynamic changes of railway safety monitoring, and adapt to the changing environment and newly added foreign object categories.

[0178] like Figure 3 As shown, this embodiment proposes a training device for a railway foreign object phrase location model, and the railway foreign object phrase location model includes a recognition module, a coding module, a feature enhancement module and a detection module. The device may include:

[0179] An acquisition unit 301 is used to acquire a text feature vector of a railway foreign object phrase text and an image feature vector of a railway foreign object image;

[0180] The first input unit 302 is used to input the text feature vector of the railway foreign object phrase text into the recognition module, so that the recognition module performs named entity recognition on the text feature vector according to a plurality of preset phrase structure categories, and obtains a plurality of structure texts in the railway foreign object phrase text and a structure category probability distribution corresponding to each structure text;

[0181] A first determining unit 303 is used to determine the phrase structure category loss according to each structure text and the corresponding structure category probability distribution;

[0182] A generating unit 304, configured to generate a structural text feature vector corresponding to each preset phrase structural category based on the encoding module and the structural category probability distribution;

[0183] An alignment unit 305 is used to perform cross-feature alignment on the image feature vector and the text feature vector based on the feature enhancement module and the structured text feature vector to obtain an aligned image feature vector corresponding to the image feature vector and an aligned text feature vector corresponding to the text feature vector;

[0184] The second input unit 306 is used to input the aligned image feature vector and the aligned text feature vector into the detection module to perform target detection and obtain corresponding detection results;

[0185] A second determining unit 307, configured to determine a detection result loss according to the detection result;

[0186] The updating unit 308 is used to update the railway foreign object phrase localization model based on the detection result loss and the phrase structure category loss to obtain a trained target railway foreign object phrase localization model.

[0187] It should be noted that the processing of the acquisition unit 301, the first input unit 302, the first determination unit 303, the generation unit 304, the alignment unit 305, the second input unit 306, the second determination unit 307 and the updating unit 308 and the beneficial effects thereof can be respectively referred to in Figure 1 Steps S101 to S108 in the above are not described in detail.

[0188] Optionally, multiple preset phrase structure categories include modifiers, object words and relational words, and the structural category probability distribution corresponding to each structure text includes the probabilities that the phrase structure categories corresponding to the structure text are modifiers, object words and relational words respectively.

[0189] Optionally, the first determining unit 303 is further configured to:

[0190] For any structured text, the cross entropy loss corresponding to the structured text is calculated according to the cross entropy loss calculation method and the probability distribution of the structured category corresponding to the structured text;

[0191] The cross entropy losses corresponding to all structured texts are averaged to obtain the corresponding average value as the phrase structure category loss.

[0192] Optionally, the generating unit 304 is further configured to:

[0193] For any structured text, the maximum probability is determined in the probability distribution of the structure categories corresponding to the structured text, and the preset phrase structure category corresponding to the maximum probability is determined as the target phrase structure category corresponding to the structured text;

[0194] When the target phrase structure category corresponding to the structure text is a modifier, the structure text is encoded through the encoding module to obtain the corresponding modifier feature vector;

[0195] When the target phrase structure category corresponding to the structure text is an object word, the structure text is encoded by the encoding module to obtain the corresponding object word feature vector;

[0196] When the target phrase structure category corresponding to the structured text is a relational word, the relational word feature vector is extracted from the text feature vector through the encoding module.

[0197] Optionally, the feature enhancement module includes a modifier feature enhancement module, an object word feature enhancement module and an overall image and text feature enhancement module; the railway foreign body phrase positioning model also includes a self-attention mechanism module;

[0198] The alignment unit 305 is further configured to:

[0199] Inputting the image feature vector and the modifier feature vector into the modifier feature enhancement module for cross feature alignment to obtain a first aligned image feature vector;

[0200] Inputting the first aligned image feature vector and the object word feature vector into the object word feature enhancement module for cross feature alignment to obtain a second aligned image feature vector;

[0201] The relation word feature vector is fused with the second aligned image feature vector to obtain a fused feature vector, and the fused feature vector is fused and optimized through a self-attention mechanism module to obtain an optimized feature vector;

[0202] The optimized feature vector and text feature vector are input into the overall image-text feature enhancement module for cross-feature alignment to obtain an aligned image feature vector and an aligned text feature vector.

[0203] Optionally, the detection result includes corresponding object category identification and bounding box information;

[0204] The second determining unit 307 is further configured to:

[0205] Determine the category recognition loss based on the difference between the object category identity and the true category identity;

[0206] Determine the bounding box localization loss based on the difference between the bounding box information and the true bounding box information;

[0207] The category recognition loss and bounding box positioning loss are taken as a whole as the detection result loss.

[0208] Optionally, the updating unit 308 is further configured to:

[0209] The category recognition loss, bounding box positioning loss, and phrase structure category loss in the detection result loss are summed to get the total loss;

[0210] If it is determined that the total loss is not less than the set loss, the recognition module, the feature enhancement module and / or the detection module in the railway foreign object phrase localization model are updated to obtain an updated railway foreign object phrase localization model, and based on the text feature vector of the new railway foreign object phrase text and the image feature vector of the new railway foreign object image, the updated railway foreign object phrase localization model is continued to be trained until a trained target phrase localization model is obtained.

[0211] The training device of the railway foreign object phrase localization model proposed in this embodiment can cross-align the image feature vector and the text feature vector to obtain the aligned image feature vector and the aligned text feature vector, and use the aligned image feature vector and the aligned text feature vector for training, so that the model can accurately correspond the foreign object described in the text with the target area in the image, ensuring accurate phrase localization in complex scenes. It is suitable for fine-grained scene description in railway foreign object detection. The trained railway foreign object phrase localization model can effectively execute and complete the railway foreign object phrase localization task, and effectively realize the phrase localization of railway foreign objects.

[0212] The training device of the railway foreign object phrase positioning model in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0213] The embodiment of the present invention also provides a computer device having the above Figure 3 The training device of the railway foreign object phrase positioning model is shown.

[0214] See also Figure 4 , a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0215] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0216] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0217] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function. The data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0218] The memory 20 may include a volatile memory, such as a random access memory. The memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive. The memory 20 may also include a combination of the above-mentioned types of memory.

[0219] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0220] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a railway foreign object phrase location model, characterized in that: The railway foreign body phrase positioning model includes a recognition module, a coding module, a feature enhancement module and a detection module; the method includes: Obtaining a text feature vector of a railway foreign object phrase text and an image feature vector of a railway foreign object image; Inputting the text feature vector of the railway foreign object phrase text into the recognition module, so that the recognition module performs named entity recognition on the text feature vector according to a plurality of preset phrase structure categories, and obtains a plurality of structure texts in the railway foreign object phrase text and a probability distribution of the structure category corresponding to each of the structure texts; Determine the phrase structure category loss according to each of the structure texts and the corresponding structure category probability distribution; Based on the encoding module and the structural category probability distribution, generating a structural text feature vector corresponding to each of the preset phrase structural categories; Based on the feature enhancement module and the structured text feature vector, cross-feature alignment is performed on the image feature vector and the text feature vector to obtain an aligned image feature vector corresponding to the image feature vector and an aligned text feature vector corresponding to the text feature vector; Inputting the aligned image feature vector and the aligned text feature vector into the detection module to perform target detection, obtaining corresponding detection results, and determining the detection result loss according to the detection results; The railway foreign object phrase localization model is updated based on the detection result loss and the phrase structure category loss to obtain a trained target railway foreign object phrase localization model.

2. The method according to claim 1, characterized in that The multiple preset phrase structure categories include modifiers, object words and relational words, and the structural category probability distribution corresponding to each structure text includes the probability that the phrase structure category corresponding to the structure text is the modifier, the object word and the relational word respectively.

3. The method according to claim 2, characterized in that Determining the phrase structure category loss according to each of the structure texts and the corresponding structure category probability distribution includes: For any of the structured texts, according to the cross entropy loss calculation method and the structure category probability distribution corresponding to the structured text, the cross entropy loss corresponding to the structured text is calculated; The cross entropy losses corresponding to all the structured texts are averaged to obtain the corresponding average value as the phrase structure category loss.

4. The method according to claim 2, characterized in that: The step of generating a structural text feature vector corresponding to each of the preset phrase structural categories based on the encoding module and the structural category probability distribution includes: For any of the structured texts, determining a maximum probability in the structure category probability distribution corresponding to the structured text, and determining the preset phrase structure category corresponding to the maximum probability as the target phrase structure category corresponding to the structured text; When the target phrase structure category corresponding to the structure text is the modifier, encoding the structure text through the encoding module to obtain a corresponding modifier feature vector; When the target phrase structure category corresponding to the structure text is the object word, the structure text is encoded by the encoding module to obtain a corresponding object word feature vector; When the target phrase structure category corresponding to the structured text is the relational word, the relational word feature vector is extracted from the text feature vector by the encoding module.

5. The method according to claim 4, characterized in that The feature enhancement module includes a modifier feature enhancement module, an object word feature enhancement module and an overall image and text feature enhancement module; The railway foreign object phrase positioning model also includes a self-attention mechanism module; The cross-feature alignment of the image feature vector and the text feature vector based on the feature enhancement module and the structured text feature vector to obtain an aligned image feature vector corresponding to the image feature vector and an aligned text feature vector corresponding to the text feature vector includes: Inputting the image feature vector and the modifier feature vector into the modifier feature enhancement module for cross feature alignment to obtain a first aligned image feature vector; Inputting the first aligned image feature vector and the object word feature vector into the object word feature enhancement module for cross feature alignment to obtain a second aligned image feature vector; Fusing the relation word feature vector with the second aligned image feature vector to obtain a fused feature vector, and performing fusion optimization on the fused feature vector through the self-attention mechanism module to obtain an optimized feature vector; The optimized feature vector and the text feature vector are input into the overall image-text feature enhancement module for cross-feature alignment to obtain the aligned image feature vector and the aligned text feature vector.

6. The method according to claim 1, characterized in that The detection result includes corresponding object category identification and bounding box information; The determining the detection result loss according to the detection result includes: Determining a category recognition loss based on a difference between the object category identifier and a true category identifier; Determining a bounding box positioning loss based on a difference between the bounding box information and the true bounding box information; The category recognition loss and the bounding box positioning loss are taken as the detection result loss as a whole.

7. The method according to claim 6, characterized in that The updating of the railway foreign object phrase localization model based on the detection result loss and the phrase structure category loss to obtain a trained target railway foreign object phrase localization model includes: The category recognition loss, the bounding box positioning loss and the phrase structure category loss in the detection result loss are summed to obtain a total loss; If it is determined that the total loss is not less than the set loss, the recognition module, the feature enhancement module and / or the detection module in the railway foreign object phrase positioning model are updated to obtain an updated railway foreign object phrase positioning model, and based on the text feature vector of the new railway foreign object phrase text and the image feature vector of the new railway foreign object image, the updated railway foreign object phrase positioning model is continuously trained until a trained target phrase positioning model is obtained.

8. A training device for a railway foreign object phrase location model, characterized in that: The railway foreign body phrase positioning model includes a recognition module, a coding module, a feature enhancement module and a detection module; the device includes: An acquisition unit, used for acquiring a text feature vector of a railway foreign object phrase text and an image feature vector of a railway foreign object image; A first input unit is used to input the text feature vector of the railway foreign object phrase text into the recognition module, so that the recognition module performs named entity recognition on the text feature vector according to a plurality of preset phrase structure categories, and obtains a plurality of structure texts in the railway foreign object phrase text and a probability distribution of the structure category corresponding to each of the structure texts; A first determining unit, configured to determine a phrase structure category loss according to each of the structure texts and the corresponding structure category probability distribution; A generating unit, configured to generate a structural text feature vector corresponding to each of the preset phrase structural categories based on the encoding module and the structural category probability distribution; an alignment unit, configured to perform cross-feature alignment on the image feature vector and the text feature vector based on the feature enhancement module and the structured text feature vector, so as to obtain an aligned image feature vector corresponding to the image feature vector and an aligned text feature vector corresponding to the text feature vector; A second input unit, used for inputting the aligned image feature vector and the aligned text feature vector into the detection module to perform target detection and obtain a corresponding detection result; A second determining unit, configured to determine a detection result loss according to the detection result; An updating unit is used to update the railway foreign object phrase localization model based on the detection result loss and the phrase structure category loss to obtain a trained target railway foreign object phrase localization model.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the training method for the railway foreign object phrase positioning model according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the training method for the railway foreign object phrase location model according to any one of claims 1 to 7.

Images