Railway scene understanding method and system based on panoptic segmentation and relationship detection

Abstract

The present invention provides a railway scene understanding method and system based on panoramic segmentation and relationship detection, which belongs to the technical field of railway operation safety detection based on computer vision. The panoramic segmentation model and the relationship detection model are integrated in an end-to-end model, and the model training is carried out in a multi-modal multi-category loss function superposition mode to obtain a scene understanding model, so as to realize the segmentation of each target in the railway scene and the judgment of the relationship between the targets; the scene understanding model includes a multi-dimensional feature enhancement module, a cross-coupled multi-modal encoding and decoding network and a multivariate multi-modal prediction module. The present invention performs pixel-level segmentation on each target in the environment and accurately infers the relative relationship between each target; due to the existence of the coupling structure, the information of different modes will guide each other in this process, so as to realize the mutual coordination of the panoramic segmentation task and the relationship detection task; the multi-category centralized loss function balances the convergence degree of different modal tasks and avoids the failure of a single mode.

Description

Technical Field

[0001] The present invention relates to the technical field of railway operation safety detection based on computer vision, and in particular to a railway scene understanding method and system based on panoptic segmentation and relationship detection. Background Art

[0002] As an important transportation infrastructure, railways play an increasingly important role in economic development. The operational safety of railways has attracted much attention, and it is very important to ensure the safe operation of trains. However, in the daily operation of railways, the intrusion of foreign objects in the railway limits is highly random and destructive, which is one of the main causes of railway traffic safety accidents. In addition, the abnormal state of railway inherent facilities and equipment is also an important factor affecting the safe operation of trains. Therefore, the detection of foreign object intrusion in the railway limits and the status of facilities and equipment is of vital importance for the safe operation of trains.

[0003] Foreign object detection systems based on computer vision have the advantages of large detection range, high sensitivity, and convenient deployment. They can perform real-time detection and are widely used in railway sites. At present, the detection of foreign objects in railway intrusion has covered multiple aspects such as conventional target detection, small target detection, small sample detection, and rail segmentation. Although these methods can accurately identify the category of intrusion targets or determine the target location at the pixel level, they cannot autonomously perceive the relationship between targets, and these omitted information also determines the degree of harm. Yang Liu et al. proposed the FCMNet framework, advocating to amplify the key clues in the scene of human interaction to predict the relationship between characters, so as to achieve scene understanding. They first used the backbone module to detect the bounding boxes of all people and objects and encode their representations; for each person-object bounding box candidate pair, the spatial encoder encodes the coarse-grained layout to capture the spatial configuration between them; the context encoder accumulates other auxiliary information available in other areas of the image, including local and global context; the motion encoder infers the possible motion of people and objects (i.e., which pixel areas will move with their instantaneous velocity) by predicting optical flow from static images as an approximation of possible motion. Finally, the fusion module combines all available knowledge about the candidate pairs, predicts the final interaction scores of the candidate pairs, and outputs the detected triples.

[0004] For the detection of relationships between objects, the current mainstream methods include human interaction detection methods and graph generation methods. The classic human interaction detection method has poor generalization. Since it only considers human-oriented relationship detection and ignores foreign objects that may exist in railway scenes such as fallen rocks, it is not suitable for comprehensive relationship detection in railway scenes. The general graph generation method is designed for the open world. Although it greatly expands the detection range, this generalization comes at the cost of accuracy. As the number of preset object categories and relationship categories increases, the prediction accuracy will decrease. In addition, the current relationship detection model relies on noisy target detection results to predict relationships, which will interfere with the reasoning of relationships to a certain extent. Summary of the invention

[0005] The object of the present invention is to provide a railway scene understanding method and system based on panoramic segmentation and relationship detection to solve at least one technical problem existing in the above-mentioned background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a railway scene understanding method based on panoptic segmentation and relationship detection, comprising:

[0008] Using a pre-trained scene understanding model to process the obtained railway scene graph to be identified, to obtain an identification result, wherein the identification result includes each target category in the railway scene graph and the relationship category between each target;

[0009] Among them, the scene understanding model includes: a multi-dimensional feature enhancement module, which is used to enhance features from the spatial dimension and the channel dimension to obtain enhanced features; a cross-coupled multimodal encoding and decoding network, which is used to sequentially splice the enhanced features with the target sequence and the relationship sequence between the targets to obtain a fusion feature containing multimodal information of the target category and the relationship between the targets; a multivariate multimodal prediction module, which is used to generate the segmentation results of each target and the prediction results of the subject-predicate-object triple relationship between individual targets.

[0010] Furthermore, training the scene understanding model includes: sending the railway scene image into the feature extraction network to obtain feature maps of different sizes; using a multi-dimensional feature enhancement module to enhance the features from the spatial dimension and the channel dimension; sending the enhanced features into the cross-coupled codec for decoding different modal information; embedding the decoded features into the prediction head to generate panoramic segmentation results and subject-verb-object triple prediction results; after multiple rounds of iterative optimization, the optimal model parameters are obtained based on the loss function constraints.

[0011] Furthermore, the loss function consists of two parts: a category loss function and a mask loss function. The category loss function includes a subject category loss function, a relation category loss function, and an object category loss function. The mask loss includes a subject target mask loss and an object target mask loss.

[0012] Furthermore, the multi-dimensional feature enhancement module includes a spatial feature enhancement module and a channel feature enhancement module connected in series; the spatial feature enhancement module is used to adopt an enhancement method based on cluster normalization evaluation, screen out strong feature weights through threshold gating, multiply the input features with the strong feature weights to finally obtain spatial enhancement features; the channel feature enhancement module is used to generate channel enhancement weights, multiply the channel enhancement weights with the spatial enhancement features to obtain the final enhanced features.

[0013] Furthermore, the input features are group regularized, and then general regularization is used to evaluate the importance of different feature maps. Finally, the weights generated by the evaluation are multiplied by the features after regularization transformation and normalized. The strong feature weights are screened out through threshold gating, and the input features are multiplied by the strong feature weights to finally obtain the spatial enhancement features; the spatial enhancement features are subjected to group convolution and point convolution operations respectively, where group convolution is used to extract high-dimensional information, while point convolution is used to compensate for information to avoid the loss of feature information caused by sparse convolution in the group convolution operation; then the two convolution outputs are added and average pooled to collect global information, and the pooled features are activated by softmax to generate channel enhancement weights, which are multiplied by the spatial enhancement weights to obtain the final enhanced features.

[0014] Furthermore, the cross-coupled codec includes: a multi-task shared encoder, which is used to further enhance the enhanced features and convert them into inputs acceptable to the decoder; a target decoder, whose multi-head attention mechanism input includes a query input as a target sequence processed by the self-attention mechanism, a key input as a relation sequence processed by the self-attention mechanism, and a value input as the encoder output; a relation decoder, whose multi-head attention mechanism input includes: a query input as a relation sequence processed by the self-attention mechanism, a key input as a target sequence processed by the self-attention mechanism, and a value input as the encoder output.

[0015] In a second aspect, the present invention provides a railway scene understanding system based on panoptic segmentation and relationship detection, comprising:

[0016] An acquisition module, used for acquiring a railway scene image to be identified;

[0017] A processing module processes the obtained railway scene graph to be identified using a pre-trained scene understanding model to obtain an identification result, wherein the identification result includes target categories and relationship categories between targets in the railway scene graph;

[0018] Among them, the scene understanding model includes: a multi-dimensional feature enhancement module, which is used to enhance features from the spatial dimension and the channel dimension to obtain enhanced features; a cross-coupled multimodal encoding and decoding network, which is used to sequentially splice the enhanced features with the target sequence and the relationship sequence between the targets to obtain a fusion feature containing multimodal information of the target category and the relationship between the targets; a multivariate multimodal prediction module, which is used to generate the segmentation results of each target and the prediction results of the subject-predicate-object triple relationship between individual targets.

[0019] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the railway scene understanding method based on panoramic segmentation and relationship detection as described in the first aspect is implemented.

[0020] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the railway scene understanding method based on panoramic segmentation and relationship detection as described in the first aspect.

[0021] In a fifth aspect, the present invention provides an electronic device, comprising: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the railway scene understanding method based on panoramic segmentation and relationship detection as described in the first aspect.

[0022] Terminology explanation:

[0023] Panoptic segmentation: Panoptic segmentation is a computer vision technology that can perform semantic segmentation on the input image, identify each object / region in the picture, and give corresponding semantic labels. In railway scene understanding, panoptic segmentation can be used to accurately locate key elements such as rails, intruding objects, and trains, providing a basis for subsequent scene analysis.

[0024] Relationship detection: Relationship detection refers to identifying the spatial, dynamic or semantic relationships between different objects / regions in an image or scene. In railway scenarios, relationship detection can be used to analyze the interactions and constraints between elements such as trains, tracks, and intruding objects, so as to better understand the structure and operation status of the entire scene.

[0025] Scene understanding: Scene understanding refers to the understanding and analysis of scene content in images or videos by computer vision systems. In railway scenarios, scene understanding technology can be used to identify and understand the key elements in the scene, as well as their spatial relationships and dynamic changes, to construct a complete semantic representation.

[0026] The beneficial effects of the present invention are as follows: a railway scene understanding model based on panoramic segmentation and target detection is constructed, and a multi-category centralized loss function is adopted to balance the convergence degree of different modal tasks during the model training process to avoid the failure of a single modality; the model operates in a perception and analysis manner, performs pixel-level segmentation on each target in the environment, and accurately infers the relative relationship between each target; a multi-head multimodal codec architecture is specially customized for railway scenes, and the target information and relationship information will be extracted by the target decoder and the relationship decoder respectively. Due to the existence of the coupling structure, the information of different modalities will guide each other in this process, thereby achieving mutual coordination between the panoramic segmentation task and the relationship detection task.

[0027] Additional advantages of the present invention will be more clearly given in the following description or learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0029] Figure 1 The present invention provides an overall workflow diagram of the scene understanding model described in the embodiment of the present invention.

[0030] Figure 2 This is an overall structural diagram of the scene understanding model described in an embodiment of the present invention.

[0031] Figure 3 This is a structural diagram of the multi-dimensional feature enhancement module described in an embodiment of the present invention.

[0032] Figure 4 This is a structural diagram of a cross-coupled codec according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below by the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0034] It should be understood by those skilled in the art that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0035] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with that in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0036] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or groups thereof.

[0037] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. Different embodiments or examples described in this specification and features of different embodiments or examples may be combined and combined by those skilled in the art without contradiction.

[0038] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0039] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0040] Example 1

[0041] In this embodiment 1, a railway scene understanding system based on panoramic segmentation and relationship detection is first provided, including: using an acquisition module to acquire a railway scene graph. Using a processing module to process the obtained railway scene graph to be identified using a pre-trained scene understanding model to obtain an identification result, the identification result includes each target category in the railway scene graph and the relationship category between each target; the target includes an intrusion target and railway equipment and facilities. Using an early warning module, based on the target category and the relationship category between each target, the railway scene intrusion target identification or the abnormal form early warning of railway equipment and facilities is performed. Among them, the scene understanding model includes: a multi-dimensional feature enhancement module, which is used to enhance features from the spatial dimension and the channel dimension to obtain enhanced features; a cross-coupled multimodal encoding and decoding network, which is used to sequentially splice the enhanced features with the target sequence and the relationship sequence between each target to obtain a fusion feature containing the target category and the multimodal information of the relationship between each target; a multivariate multimodal prediction module, which is used to generate the segmentation results of each target and the prediction results of the subject-predicate-object triple relationship between each target.

[0042] In this embodiment, the above-mentioned system is used to implement a railway scene understanding method based on panoptic segmentation and relationship detection, including: obtaining a railway scene graph to be identified; using a pre-trained scene understanding model to process the obtained railway scene graph to be identified to obtain a recognition result, wherein the recognition result includes target categories and relationship categories between targets in the railway scene graph; the targets include intrusion targets and railway equipment and facilities;

[0043] Based on the target categories and the relationship categories between the targets, the railway scene intrusion target identification or abnormal form warning of railway equipment and facilities is carried out;

[0044] Among them, the scene understanding model includes: a multi-dimensional feature enhancement module, which is used to enhance features from the spatial dimension and the channel dimension to obtain enhanced features; a cross-coupled multimodal encoding and decoding network, which is used to orderly splice the enhanced features with the target sequence and the relationship sequence between each target to obtain a fusion feature containing multimodal information of the target category and the relationship between each target; a multivariate multimodal prediction module, which is used to generate the segmentation results of each target and the prediction results of the subject-predicate-object triple relationship between each target.

[0045] The multi-dimensional feature enhancement module includes a spatial feature enhancement module and a channel feature enhancement module connected in series; the spatial feature enhancement module is used to adopt an enhancement method based on cluster normalization evaluation, screen out strong feature weights through threshold gating, multiply the input feature with the strong feature weight to finally obtain the spatial enhancement feature; the channel feature enhancement module is used to generate channel enhancement weights, multiply the channel enhancement weights with the spatial enhancement feature to obtain the final enhanced feature.

[0046] The input features are group regularized, and then general regularization is used to evaluate the importance of different feature maps. Finally, the weights generated by the evaluation are multiplied by the features after regularization transformation and normalized. The strong feature weights are screened out through threshold gating, and the input features are multiplied by the strong feature weights to finally obtain the spatial enhancement features; the spatial enhancement features are subjected to group convolution and point convolution operations respectively, where group convolution is used to extract high-dimensional information, while point convolution is used to compensate for information to avoid the loss of feature information caused by sparse convolution in the group convolution operation; then the two convolution outputs are added and average pooled to collect global information, and the pooled features are activated by softmax to generate channel enhancement weights, which are multiplied by the spatial enhancement weights to obtain the final enhanced features.

[0047] The cross-coupled codec includes: a multi-task shared encoder, which is used to further enhance the enhanced features and convert them into inputs acceptable to the decoder; a target decoder, whose multi-head attention mechanism input includes a query input as a target sequence processed by the self-attention mechanism, a key input as a relation sequence processed by the self-attention mechanism, and a value input as the encoder output; a relation decoder, whose multi-head attention mechanism input includes: a query input as a relation sequence processed by the self-attention mechanism, a key input as a target sequence processed by the self-attention mechanism, and a value input as the encoder output.

[0048] Example 2

[0049] In this embodiment 2, a railway scene understanding model based on panoramic segmentation and target detection is constructed. The model operates in a perceptual and analytical manner, performs pixel-level segmentation on each target in the environment, and accurately infers the relative relationship between each target. A multi-head multimodal codec architecture specially customized for railway scenes is invented. The target information and relationship information will be extracted by the target decoder and the relationship decoder respectively. Because of the existence of the coupling structure, the information of different modes will guide each other in this process, realizing the mutual coordination of the panoramic segmentation task and the relationship detection task. A multi-category centralized loss function is invented to balance the convergence degree of different modal tasks during the training process to avoid the failure of a single mode.

[0050] In this embodiment, the railway scene understanding method based on panoramic segmentation and relationship detection includes: integrating the panoramic segmentation model and the relationship detection model in an end-to-end model to realize the segmentation of each target in the railway scene and the judgment of the relationship between the targets. The scene understanding model includes three parts: a multi-dimensional feature enhancement module, a cross-coupled multimodal encoding and decoding network, and a multivariate multimodal prediction module; and the model training is performed by superimposing a multimodal multi-category loss function.

[0051] The above-mentioned scene understanding model adopts a railway scene segmentation model training method based on feature alignment and edge constraints, which includes the following steps: Step 1, first send the railway scene image into the feature extraction network to obtain feature maps of different sizes; Step 2, use a multi-dimensional feature enhancement module to enhance the features from the spatial dimension and channel dimension; Step 3, send the enhanced features into the cross-coupled codec for decoding of different modal information; Step 4, embed the decoded features into the prediction head to generate panoramic segmentation results and subject-verb-object triple prediction results; Step 5, after multiple rounds of iterative optimization, the optimal model parameters are obtained based on the loss function constraints.

[0052] The cross-coupled encoder-decoder described in step 3 consists of three parts: the first part is a multi-task shared encoder, which is used to further enhance the enhanced features and convert them into inputs that the decoder can accept. The second part is a target decoder, in which the multi-head attention mechanism input consists of three parts: the query input is the target sequence processed by the self-attention mechanism, the key input is the relation sequence processed by the self-attention mechanism, and the value input is the encoder output. The third part is a relation decoder, in which the multi-head attention mechanism input consists of three parts: the query input is the relation sequence processed by the self-attention mechanism, the key input is the target sequence processed by the self-attention mechanism, and the value input is the encoder output.

[0053] The multi-dimensional feature enhancement module described in step 2 consists of two parts: the first part is a spatial enhancement module consisting of components such as cluster normalization. The second part is a channel enhancement module consisting of components such as group convolution and point convolution. The two parts are connected in series.

[0054] The overall workflow of the scene understanding model is as follows: Figure 1 First, the acquired railway scene graph is input into the multi-dimensional feature enhancement module to obtain enhanced image features, and the enhanced image features are input into the cross-coupled encoder-decoder network. The features are fused through the target decoder and the relationship decoder between targets, and the fused features are input into the multivariate multimodal prediction module to obtain the final prediction results of the target category and the relationship category between targets.

[0055] The overall structure of the model is as follows Figure 2As shown in the figure, in the feature extraction stage, the color image I is input into the backbone network to generate a feature map f1, which contains multi-scale features. The multi-scale features are first enhanced in the spatial dimension and then in the channel dimension to obtain the final enhanced features. The enhanced features and the target position code are jointly input into the encoder (multi-task shared encoder) of the cross-coupled encoder-decoder module, and the output of the multi-task shared encoder is input into the target decoder and the relation decoder respectively. The target decoder takes the target sequence as the input of the self-attention mechanism, and the output formed is input into the multi-head attention as the query. At the same time, the multi-head attention module takes the output of the multi-task shared encoder as the value input and the relation sequence as the key input. The output of the multi-head attention module is fed into the feedforward network and the regularization layer to form the output of the target decoder. The relation decoder takes the relation sequence as the input of the self-attention mechanism, and the output formed is input into the multi-head attention module as the query. At the same time, the multi-head attention module takes the output of the encoder as the value input and the target sequence as the key input. The output of the multi-head attention module is fed into the feedforward network and the regularization layer to form the output of the relation decoder. The outputs of the relation decoder and the target decoder are summed and input into the prediction head of the multivariate multimodal prediction module to obtain the subject category, object category, subject mask, object mask and relation category.

[0056] In this embodiment, the feature extraction backbone network used is Resnet-50. However, in actual use, the backbone network is not limited to the above Resnet-50. Those skilled in the art can select a suitable feature extraction backbone network according to the specific situation.

[0057] In this embodiment, in order to further enhance the expressiveness of the features, the following Figure 3 The multi-dimensional feature enhancement module shown in the figure suppresses the redundant information extracted by the backbone network from both spatial and channel aspects. In terms of spatial feature enhancement, an enhancement method based on cluster normalization evaluation is adopted. First, the input features are processed by group regularization GN, and then the importance of different feature maps is evaluated using general regularization N. Finally, the weight W generated by the evaluation is used. α Multiply the regularized transformed features and normalize them to map them to the value range of (0-1), filter out the strong feature weight W through threshold gating Thr, multiply the input features with the strong feature weights to finally obtain the spatial enhancement feature X S ; In terms of channel feature enhancement, the spatial enhancement feature X S The group convolution operation GWC and the point convolution operation PWC are performed respectively, where the group convolution is used to extract high-dimensional information, while the point convolution is used to compensate for information to avoid the loss of feature information caused by sparse convolution in the group convolution operation. Then the feature X after adding the two convolution outputs is sumAverage pooling is performed to collect global information. The pooled features are activated by softmax to generate channel enhancement weights β, and the channel enhancement weights are multiplied by the spatial enhancement features to obtain the final enhanced features.

[0058] Although the multi-dimensional enhancement module can suppress unimportant features, the features of different modalities are still mixed, so the design Figure 4 The cross-coupled encoder and decoder shown in the figure realizes the orderly arrangement of target features and relationship features. The cross-coupled encoder and decoder includes three parts: multi-task shared encoder, target decoder and relationship decoder. The multi-task shared encoder uses the features (fine features) output by the multi-dimensional feature enhancement module as the input of the self-attention mechanism after 1×1 convolution. The output of the self-attention mechanism is summed with the features after the above 1×1 convolution and regularized before being input into the feedforward neural network FFN. The output of the feedforward neural network FFN is summed with its input again and regularized, and finally outputs the global memory V * . V * They are input into the multi-head attention mechanism of the target decoder and the multi-head attention mechanism of the relation decoder respectively. In the target decoder, the target sequence is used as the input of the self-attention mechanism. After the output of the self-attention mechanism is added and regularized with the input, the output is input into the multi-head attention module as the query. At the same time, the multi-head attention module takes the output V of the encoder as * As the value input, the relation sequence is used as the key input. After the output is added and regularized with the input, the output is sent to the feedforward network FFN and the regularization layer to form the output of the target decoder, that is, the target embedding. Similar to the target decoder, the relation decoder uses the relation sequence as the input of the self-attention mechanism. After the output of the self-attention mechanism is added and regularized with the input, the output formed is input as the query into the multi-head attention module. At the same time, the multi-head attention module takes the output V of the encoder as the input of the self-attention mechanism. * As the value input, the target sequence is used as the key input. After the output is added and regularized with the input, the output is sent to the feedforward network FFN and the regularization layer to form the output of the relation decoder, that is, the relation embedding. The target embedding and the relation embedding are jointly input to the multivariate multimodal prediction module, the mask is output by the prediction head, and the prediction result is output by the feedforward neural network.

[0059] After the cross-coupled codec stitches the multimodal features together in order, the final five-tuple instance is generated using a multivariate multimodal prediction network, which consists of two parts: a feedforward neural network and a panoramic segmentation head. The feedforward neural network predicts the subject and object bounding boxes by predicting the normalized center coordinates, height, and width relative to the input image. Then, softmax activation is used to predict the subject, object target category, and relation category; the panoramic segmentation head predicts the panoramic segmentation mask by uniformly assigning a binary mask to each subject and object object.

[0060] In the overall model training process, the overall loss function in step 5 consists of two parts: the category loss function and the mask loss function. The category loss function includes the subject category loss function, the relation category loss function, and the object category loss function. The mask loss includes the subject target mask loss and the object target mask loss. The overall loss function can be expressed as:

[0061]

[0062] in, represents the category loss of subject objects, relations, and object objects. Represents the bounding box loss of the subject and object objects, The mask loss representing the subject and object objects, g i With p i They represent the true value and the model prediction value respectively, and α is a hyperparameter representing different loss weights.

[0063] Compared with the HOI method, the application of this embodiment is more object-oriented. Thanks to the cross-coupled codec, all target features and relationship features in the current scene can be sorted out, rather than limited to the features of the subject being a person. In addition, the introduction of the multi-dimensional feature enhancement module also enables the model to obtain more robust semantic and visual features, thereby achieving more accurate segmentation and detection effects.

[0064] Example 3

[0065] This embodiment 3 provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the railway scene understanding method based on panoptic segmentation and relationship detection as described above is implemented. The method includes:

[0066] The obtained railway scene graph to be identified is processed using a pre-trained scene understanding model to obtain an identification result, wherein the identification result includes target categories and relationship categories between targets in the railway scene graph; the targets include intrusion targets and railway equipment and facilities;

[0067] Based on the target categories and the relationship categories between the targets, the railway scene intrusion target identification or abnormal form warning of railway equipment and facilities is carried out;

[0068] Among them, the scene understanding model includes: a multi-dimensional feature enhancement module, which is used to enhance features from the spatial dimension and the channel dimension to obtain enhanced features; a cross-coupled multimodal encoding and decoding network, which is used to orderly splice the enhanced features with the target sequence and the relationship sequence between each target to obtain a fusion feature containing multimodal information of the target category and the relationship between each target; a multivariate multimodal prediction module, which is used to generate the segmentation results of each target and the prediction results of the subject-predicate-object triple relationship between each target.

[0069] Example 4

[0070] This embodiment 4 provides a computer device, including a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, the processor calls the program instructions to execute the above-mentioned railway scene understanding method based on panoramic segmentation and relationship detection, the method comprising:

[0071] The obtained railway scene graph to be identified is processed using a pre-trained scene understanding model to obtain an identification result, wherein the identification result includes target categories and relationship categories between targets in the railway scene graph; the targets include intrusion targets and railway equipment and facilities;

[0072] Based on the target categories and the relationship categories between the targets, the railway scene intrusion target identification or abnormal form warning of railway equipment and facilities is carried out;

[0073] Among them, the scene understanding model includes: a multi-dimensional feature enhancement module, which is used to enhance features from the spatial dimension and the channel dimension to obtain enhanced features; a cross-coupled multimodal encoding and decoding network, which is used to orderly splice the enhanced features with the target sequence and the relationship sequence between each target to obtain a fusion feature containing multimodal information of the target category and the relationship between each target; a multivariate multimodal prediction module, which is used to generate the segmentation results of each target and the prediction results of the subject-predicate-object triple relationship between each target.

[0074] Example 5

[0075] This embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing the railway scene understanding method based on panoramic segmentation and relationship detection as described above, the method comprising:

[0076] The obtained railway scene graph to be identified is processed using a pre-trained scene understanding model to obtain an identification result, wherein the identification result includes target categories and relationship categories between targets in the railway scene graph; the targets include intrusion targets and railway equipment and facilities;

[0077] Based on the target categories and the relationship categories between the targets, the railway scene intrusion target identification or abnormal form warning of railway equipment and facilities is carried out;

[0078] Among them, the scene understanding model includes: a multi-dimensional feature enhancement module, which is used to enhance features from the spatial dimension and the channel dimension to obtain enhanced features; a cross-coupled multimodal encoding and decoding network, which is used to orderly splice the enhanced features with the target sequence and the relationship sequence between each target to obtain a fusion feature containing multimodal information of the target category and the relationship between each target; a multivariate multimodal prediction module, which is used to generate the segmentation results of each target and the prediction results of the subject-predicate-object triple relationship between each target.

[0079] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0081] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0083] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative work on the basis of the technical solution disclosed in the present invention should be included in the scope of protection of the present invention.

Claims

1. A railway scene understanding method based on panoptic segmentation and relationship detection, characterized in that: include: Using a pre-trained scene understanding model to process the obtained railway scene graph to be identified, to obtain an identification result, wherein the identification result includes each target category in the railway scene graph and the relationship category between each target; Said targets include invasion targets and railway equipment and facilities; Based on the target categories and the relationship categories between the targets, the railway scene intrusion target identification or abnormal form warning of railway equipment and facilities is carried out; The scene understanding model includes: a multi-dimensional feature enhancement module, which is used to enhance features from the spatial dimension and the channel dimension to obtain enhanced features; a cross-coupled multimodal encoding and decoding network, which is used to sequentially splice the enhanced features with the target sequence and the relationship sequence between the targets to obtain a fusion feature containing multimodal information of the target category and the relationship between the targets; a multivariate multimodal prediction module, which is used to generate the segmentation results of each target and the prediction results of the subject-verb-object triple relationship between the targets; Among them, the cross-coupled multimodal encoding and decoding network includes a multi-task shared encoder, a target decoder and a relational decoder; the multi-task shared encoder uses the features output by the multi-dimensional feature enhancement module as the input of the self-attention mechanism after 1×1 convolution, and the output of the self-attention mechanism is summed and regularized with the features after 1×1 convolution and then input into the feedforward neural network FFN. The output of the feedforward neural network FFN is summed and regularized with its input again, and finally the global memory is output; the global memory is respectively input into the multi-head attention mechanism of the target decoder and the multi-head attention mechanism of the relational decoder; in the target decoder, the target sequence is used as the input of the self-attention mechanism, and the output of the self-attention mechanism is summed and regularized with the input to form an output As a query input to the multi-head attention module, at the same time, the multi-head attention module takes the global memory as the value input and the relation sequence as the key input. After its output is summed and regularized with the input, the output is sent to the feedforward network FFN and the regularization layer to form the output of the target decoder, that is, the target embedding; the relation decoder takes the relation sequence as the input of the self-attention mechanism, and the output of the self-attention mechanism is summed and regularized with the input to form the output as the query input to the multi-head attention module. At the same time, the multi-head attention module takes the global memory as the value input and the target sequence as the key input. After its output is summed and regularized with the input, the output is sent to the feedforward network FFN and the regularization layer to form the output of the relation decoder, that is, the relation embedding.

2. The railway scene understanding method based on panoptic segmentation and relationship detection according to claim 1 is characterized in that: Training the scene understanding model includes: sending the railway scene image into the feature extraction network to obtain feature maps of different sizes; using a multi-dimensional feature enhancement module to enhance the features from the spatial dimension and the channel dimension; sending the enhanced features into the cross-coupled codec for decoding different modal information; embedding the decoded features into the prediction head to generate panoramic segmentation results and subject-verb-object triple prediction results; after multiple rounds of iterative optimization, the optimal model parameters are obtained based on the loss function constraints.

3. The railway scene understanding method based on panoptic segmentation and relationship detection according to claim 2 is characterized in that: The loss function consists of two parts: a category loss function and a mask loss function. The category loss function includes a subject category loss function, a relation category loss function, and an object category loss function. The mask loss includes a subject target mask loss and an object target mask loss.

4. The railway scene understanding method based on panoptic segmentation and relationship detection according to claim 1, characterized in that: The multi-dimensional feature enhancement module includes a spatial feature enhancement module and a channel feature enhancement module connected in series; The spatial feature enhancement module is used to adopt an enhancement method based on cluster normalization evaluation, screen out strong feature weights through threshold gating, multiply the input features with the strong feature weights to finally obtain spatial enhancement features; the channel feature enhancement module is used to generate channel enhancement weights, multiply the channel enhancement weights with the spatial enhancement features to obtain the final enhanced features.

5. The railway scene understanding method based on panoptic segmentation and relationship detection according to claim 4 is characterized in that: The input features are group regularized, and then general regularization is used to evaluate the importance of different feature maps. Finally, the weights generated by the evaluation are multiplied by the features after regularization transformation and normalized. The strong feature weights are screened out through threshold gating, and the input features are multiplied by the strong feature weights to finally obtain the spatial enhancement features. The spatial enhancement features are respectively subjected to group convolution and point convolution operations, where group convolution is used to extract high-dimensional information and point convolution is used to compensate for information to avoid feature information loss caused by sparse convolution in the group convolution operation. The two convolution outputs are then added and average pooled to collect global information. The pooled features are activated by softmax to generate channel enhancement weights, and the channel enhancement weights are multiplied by the spatial enhancement features to obtain the final enhanced features.

6. The railway scene understanding method based on panoptic segmentation and relationship detection according to claim 1, characterized in that: The cross-coupled multimodal encoding and decoding network includes: a multi-task shared encoder, which is used to further enhance the enhanced features and convert them into inputs acceptable to the decoder; a target decoder, whose multi-head attention mechanism input includes a query input as a target sequence processed by the self-attention mechanism, a key input as a relation sequence processed by the self-attention mechanism, and a value input as the encoder output; a relation decoder, whose multi-head attention mechanism input includes: a query input as a relation sequence processed by the self-attention mechanism, a key input as a target sequence processed by the self-attention mechanism, and a value input as the encoder output.

7. A railway scene understanding system based on panoptic segmentation and relationship detection, characterized in that: include: An acquisition module, used for acquiring a railway scene image to be identified; A processing module, used to process the obtained railway scene graph to be identified using a pre-trained scene understanding model to obtain an identification result, wherein the identification result includes each target category and the relationship category between each target in the railway scene graph; Said targets include invasion targets and railway equipment and facilities; The early warning module is used to identify intrusion targets in railway scenes or to warn of abnormal forms of railway equipment and facilities based on target categories and the relationship categories between targets; The scene understanding model includes: a multi-dimensional feature enhancement module, which is used to enhance features from the spatial dimension and the channel dimension to obtain enhanced features; a cross-coupled multimodal encoding and decoding network, which is used to sequentially splice the enhanced features with the target sequence and the relationship sequence between the targets to obtain a fusion feature containing multimodal information of the target category and the relationship between the targets; a multivariate multimodal prediction module, which is used to generate the segmentation results of each target and the prediction results of the subject-verb-object triple relationship between the targets; Among them, the cross-coupled multimodal encoding and decoding network includes a multi-task shared encoder, a target decoder and a relational decoder; the multi-task shared encoder uses the features output by the multi-dimensional feature enhancement module as the input of the self-attention mechanism after 1×1 convolution, and the output of the self-attention mechanism is summed and regularized with the features after 1×1 convolution and then input into the feedforward neural network FFN. The output of the feedforward neural network FFN is summed and regularized with its input again, and finally the global memory is output; the global memory is respectively input into the multi-head attention mechanism of the target decoder and the multi-head attention mechanism of the relational decoder; in the target decoder, the target sequence is used as the input of the self-attention mechanism, and the output of the self-attention mechanism is summed and regularized with the input to form an output As a query input to the multi-head attention module, at the same time, the multi-head attention module takes the global memory as the value input and the relation sequence as the key input. After its output is summed and regularized with the input, the output is sent to the feedforward network FFN and the regularization layer to form the output of the target decoder, that is, the target embedding; the relation decoder takes the relation sequence as the input of the self-attention mechanism, and the output of the self-attention mechanism is summed and regularized with the input to form the output as the query input to the multi-head attention module. At the same time, the multi-head attention module takes the global memory as the value input and the target sequence as the key input. After its output is summed and regularized with the input, the output is sent to the feedforward network FFN and the regularization layer to form the output of the relation decoder, that is, the relation embedding.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the railway scene understanding method based on panoramic segmentation and relationship detection as described in any one of claims 1-6 is implemented.

9. A computer device, characterized in that: It includes a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the railway scene understanding method based on panoramic segmentation and relationship detection as described in any one of claims 1-6.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the railway scene understanding method based on panoramic segmentation and relationship detection as described in any one of claims 1-6.

Images