An event ontology-based image semantic segmentation method and system

Abstract

The application discloses an image semantic segmentation method and system based on an event ontology, comprising: constructing a structured event ontology library; obtaining an input image to be segmented, and extracting multi-scale visual features from the input image by using a deep learning model; constructing a heterogeneous graph containing visual candidate nodes and ontology prototype nodes according to the extracted multi-scale visual features and semantic embedding vectors in the event ontology library; performing message passing and aggregation on the heterogeneous graph by using a graph neural network to obtain a potential event type corresponding to the input image and entity role instantiation configuration; generating a dynamic spatial attention mask and a logical consistency constraint according to the determined potential event type and entity role instantiation configuration, and guiding a segmentation decoder to perform weighted processing on the multi-scale visual features to generate a semantic segmentation result. The application can obtain a semantic segmentation result with good logical consistency, strong robustness and good interpretability in a complex scene.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to an image semantic segmentation method and system based on event ontology. Background Technology

[0002] Image semantic segmentation aims to assign semantic category labels to each pixel in an image. It is a core technology for achieving fine-grained scene understanding and is widely used in fields such as autonomous driving, robot navigation, and medical image analysis. Existing technologies mainly rely on deep learning models, especially fully convolutional neural networks and their derivative architectures (such as U-Net, DeepLab series, Mask2Former, etc.). These methods perform end-to-end supervised training on large-scale datasets, extract image features using backbone networks, integrate contextual information by combining feature pyramids or multi-scale fusion mechanisms, and finally generate pixel-level classification masks by a decoder. They have achieved excellent performance in standard benchmark tests.

[0003] However, existing semantic segmentation methods are essentially still at the "perception" level, mainly focusing on low-level visual features and pixel classification, lacking the ability to understand high-level semantics and dynamic relationships, and have the following shortcomings: First, they lack scene cognition at the event level; the model can identify "objects" but cannot understand "events" (such as "vehicle rear-end collisions"), and is prone to errors in scenarios requiring contextual reasoning; Second, they are difficult to explicitly model the dynamic spatiotemporal relationships between entities (such as inclusion, adjacency, occlusion, temporal sequence, etc.), and are prone to semantically inconsistent segmentation results when severe occlusion or rare combinations occur; Third, they lack robustness to long-tailed distributions and low-sample scenes, and rare events are difficult to segment accurately due to a lack of sufficient training samples; Fourth, they rely on large-scale pixel-level labeled data, which is costly and has limited generalization ability. Although some studies in recent years have attempted to introduce scene graphs or semantic constraints to assist segmentation, most are limited to static object relationship descriptions and have failed to deeply integrate structured event ontology knowledge with deep visual features to guide the segmentation process. Summary of the Invention

[0004] To address the technical problems of existing image semantic segmentation methods, such as lack of high-level scene cognition, difficulty in explicitly modeling dynamic spatiotemporal relationships between entities, insufficient robustness to long-tail and few-sample events, and poor model interpretability, this invention provides an image semantic segmentation method based on event ontology, comprising the following steps: Construct a structured event ontology library, which includes event type nodes, entity role nodes, attribute nodes, spatial topological relationship edges and time sequence relationship edges between nodes, and map each node to a semantic embedding vector; The input image to be segmented is obtained, and multi-scale visual features are extracted from the input image using a deep learning model; Based on the extracted multi-scale visual features and the semantic embedding vectors in the event ontology library, a heterogeneous graph containing visual candidate nodes and ontology prototype nodes is constructed. The heterogeneous graph is processed by a graph neural network for message passing and aggregation to obtain the potential event types and entity role instantiation configurations corresponding to the input image. Based on the identified potential event types and entity role instantiation configurations, a dynamic spatial attention mask and logical consistency constraints are generated, and the segmentation decoder is guided to perform weighted processing on the multi-scale visual features to generate semantic segmentation results.

[0005] Optionally, building a structured event ontology library specifically includes: The event body is stored using a resource description framework or property graph format; Define the metadata of the event node, which includes the event trigger words and a list of participant roles, and establish a mapping relationship between the list of participant roles and visual semantic tags; Define the constraint relationships between role nodes, including spatial location constraints and temporal state constraints; By using a contrastive learning pre-training method, the semantic embedding vectors of each node in the event ontology are initialized so that the semantic embedding vectors are close to the distance of similar visual concepts in the feature space.

[0006] Optionally, extracting multi-scale visual features specifically includes: Image features are extracted using a backbone network based on a visual transformation network or a modern convolutional network; By fusing image features from different levels through a feature pyramid network structure, multi-scale visual features are obtained. Shallow features are used as pixel-level features, mid-level features are used as object-level features, and deep features are used as scene-level features after global pooling. The pixel-level features, object-level features, and scene-level features are then mapped to the same dimensional space as the semantic embedding vectors in the event ontology library.

[0007] Optionally, constructing a heterogeneous graph and performing message passing and aggregation specifically includes: The extracted object-level features are used as query nodes, and the role nodes in the event ontology are used as prototype nodes to form a heterogeneous graph node set. An edge set is constructed based on visual spatial proximity and semantic relationships defined in the event ontology. Graph attention networks or graph transformation networks are used to perform multi-round message passing on the heterogeneous graph, enabling visual nodes to aggregate prior knowledge of the ontology, and simultaneously enabling ontology nodes to aggregate visual evidence. The confidence score for each predefined event type is calculated using a classification head. The highest score is selected as the potential event type. Based on the attention weight distribution between nodes, the instantiation location and state attributes of each entity role in the image are determined.

[0008] Optionally, generating dynamic spatial attention masks and logical consistency constraints and guiding the segmentation decoder specifically includes: Based on the determined entity role instantiation information, a spatial attention map for role perception is generated, and the spatial attention map is multiplied element-wise with the multi-scale visual features to enhance the feature response of the target region and suppress background noise. An event logic consistency loss term is introduced to penalize segmentation prediction results that violate spatial topological or temporal relationships defined in the event ontology library; The attention-modulated features are input into the mask generation network, and the network parameters are optimized by combining a total loss function including cross-entropy loss, Dess loss, and event logic consistency loss to generate the final semantic segmentation result.

[0009] On the other hand, the present invention also provides an image semantic segmentation system based on event ontology for implementing the method, comprising: The event ontology management module is used to build a structured event ontology library, which includes event type nodes, entity role nodes, attribute nodes, spatial topological relationship edges and time sequence relationship edges between nodes, and maps each node to a semantic embedding vector. A multi-scale feature extraction module is used to acquire the input image to be segmented and extract multi-scale visual features from the input image using a deep learning model; The event-visual fusion reasoning module is used to construct a heterogeneous graph containing visual candidate nodes and ontology prototype nodes based on the multi-scale visual features extracted by the multi-scale feature extraction module and the semantic embedding vector in the event ontology management module, and to use a graph neural network to perform message passing and aggregation on the heterogeneous graph to obtain the potential event type and entity role instantiation configuration corresponding to the input image. The knowledge-guided segmentation generation module is used to generate dynamic spatial attention masks and logical consistency constraints based on the determined potential event types and entity role instantiation configurations, and guide the segmentation decoder to perform weighted processing on the multi-scale visual features to generate semantic segmentation results.

[0010] Optionally, the event ontology management module includes: Ontology storage unit, used to store event ontology data using a resource description framework or property graph format; The semantic embedding generation unit, with a built-in text encoder and graph neural network submodule, is used to calculate and update the high-dimensional semantic embedding vectors of each node in the event ontology library. The ontology retrieval interface unit is used to provide ontology subgraph retrieval services based on event types.

[0011] Optionally, the multi-scale feature extraction module includes: The backbone network units, employing either visual transformation networks or modern convolutional networks, are used to extract multi-level features from the input image. The feature pyramid fusion unit is used to fuse shallow features of backbone network units as pixel-level features, mid-level features as object-level features, and deep features as scene-level features after global pooling. A cross-modal projection unit is used to map the multi-level features to the same dimensional space as the semantic embedding vectors in the event ontology library.

[0012] Optionally, the event-visual fusion reasoning module includes: The heterogeneous graph construction unit is used to dynamically construct a heterogeneous graph node set and edge set by taking the extracted object-level features as visual nodes and the role nodes in the event ontology library as ontology prototype nodes. The graph attention reasoning unit, with a built-in graph attention network or graph transformation network, is used to perform multi-round message passing on the heterogeneous graph, enabling visual nodes to aggregate prior knowledge of the ontology, and simultaneously enabling ontology nodes to aggregate visual evidence. The event discrimination and instantiation unit is used to calculate the confidence score of each predefined event type, select the highest score as the potential event type, and determine the instantiation position and state attributes of each entity role in the image based on the attention weight distribution between nodes.

[0013] Optionally, the knowledge-guided segmentation generation module includes: An attention control unit is used to generate a spatial attention map of the character perception based on the instantiation information of the entity character, and multiply the spatial attention map by the row element by element; The logical constraint calculation unit is used to calculate the event logical consistency loss during the training phase, and to penalize the segmentation prediction results that violate the spatial topological relationships or temporal relationships defined in the event ontology library. The segmentation decoding unit has a built-in mask generation network. It is used to input the features modulated by the attention control unit into the mask generation network, and optimize the network parameters by combining the total loss function to output the semantic segmentation mask.

[0014] Compared with the prior art, the present invention has the following advantages and technical effects: This invention constructs an event ontology library containing event type nodes, entity role nodes, spatial topological relationship edges, and time sequence relationship edges, and maps them to semantic embedding vectors, enabling the quantitative expression of high-level event logic and structured prior knowledge. On this basis, multi-scale visual features and ontology embedding vectors are jointly used to construct a heterogeneous graph, and graph neural networks are used for bidirectional message passing and aggregation, so as to deeply integrate visual features and ontology knowledge, thereby accurately identifying potential event types and entity role instantiation configurations in images, and achieving a cognitive improvement from object perception to event understanding.

[0015] This invention generates dynamic spatial attention masks and logical consistency constraints based on the potential event types and role instantiation configurations obtained through reasoning. These masks guide the segmentation decoder to perform weighted processing on multi-scale visual features, making the segmentation process explicitly guided by event logic and spatiotemporal relationship constraints. This maintains the logical consistency and structural integrity of the segmentation results in complex scenarios such as occlusion, lighting changes, and rare combinations. At the same time, it reduces the dependence on large-scale pixel-level labeled data and improves the generalization ability and model interpretability in long-tailed distribution and few-sample scenarios. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the "traffic accident" event structure in an embodiment of the present invention. Detailed Implementation

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0019] Example 1 like Figure 1 As shown, this embodiment provides an image semantic segmentation method based on event ontology, including the following steps: S1. Construct a structured event ontology library: Define a graph structure containing event type nodes, entity role nodes, and attribute nodes, and establish spatial topological relationship edges and time sequence relationship edges between nodes, mapping the nodes to high-dimensional semantic embedding vectors; S2. Receive and preprocess the input image: Obtain the raw image data to be segmented; S3. Multi-scale visual feature extraction and projection: A hybrid architecture deep learning model is used to extract pixel-level, object-level and scene-level multi-level visual features from the input image, and the visual features are mapped to a feature space with the same dimension as the event ontology embedding vector through a linear projection layer; S4. Visual-ontology fusion reasoning based on heterogeneous graphs: Construct a heterogeneous graph containing visual candidate nodes and ontology prototype nodes, use graph neural networks for message passing and aggregation, calculate the matching probability between the input image and each event type, and determine the optimal potential event type and the corresponding entity role instantiation configuration. S5. Semantic segmentation generation under knowledge constraints: Based on the optimal potential event type and entity role instantiation configuration, a dynamic spatial attention mask and logical consistency loss function are generated to guide the segmentation decoder to perform weighted processing on the visual features and output pixel-level semantic segmentation results that conform to the event logic.

[0020] Implementable, the construction of the structured event ontology library in step S1 specifically includes: The ontology is stored using a Resource Description Framework (RDF) or Attribute Graph format; Define the metadata of the event node, including the event trigger words, the list of participant roles and their corresponding visual semantic label mappings; Define the constraints between roles, including spatial constraints (such as containment, adjacency, occlusion) and temporal state constraints; By using a contrastive learning pre-training method, the semantic embedding vectors of each node in the ontology are initialized, making them closer to similar visual concepts in the feature space.

[0021] As a specific implementation method, the process of constructing a multi-granularity structured event ontology library and semantic embedding space includes: (1) Event ontology model definition. For example... Figure 2 As shown, in order to construct the event ontology for guiding image semantic segmentation, the structured event ontology is defined as a quintuple: ; in: (Event): A collection of event types (such as "traffic accident", "crowd gathering"); (Role): A set of participant roles (e.g., "pedestrian", "vehicle"); (Location / Scene): A collection of scene and environment types, including macro scenes (such as "highway", "indoor corridor", "operating room") and micro environmental elements (such as "slippery road surface", "low light area", "narrow passage"). (Attribute): A collection of attributes (description) Specific characteristics, such as speed, color, and material); (Relations): A multidimensional set of relations that includes not only relations between entities, but also explicit binding relations (semantic constraints) between events / roles and the environment, such as spatial topological relations (next_to, inside, occluding) and time series between events (before, during, after), and semantic constraints (e.g., "driver" must be inside "vehicle").

[0022] (2) Event ontology storage. The event ontology is stored in the format of Resource Description Framework (RDF) or Property Graph. It can be constructed manually or through a semi-automatic ontology tool.

[0023] (3) Semantic vectorization of the event ontology. Using pre-trained large-scale language models (such as BERT or CLIP's text encoder), the semantic vectorization of each event type node in the ontology is performed. and character nodes The text description is encoded to generate an initial semantic embedding vector. .

[0024] (4) Event Knowledge Graph Enhancement: Graph Convolutional Network (GCN) is introduced to propagate information on the ontology graph, update node embeddings to integrate structured prior knowledge of the neighborhood, and obtain the final ontology prototype vector library. .

[0025] (5) Pre-trained knowledge base: Extract subgraphs using existing knowledge graphs (such as ConceptNet, VisualGenome), or generate event templates for specific domains with the assistance of large language models (LLM). Supports online learning, and automatically suggests adding new event types when unknown high-frequency co-occurrence patterns are detected.

[0026] The multi-scale visual feature extraction and projection in step S3, which is feasible, specifically includes: The backbone network is based on visual Transformers (such as Swing Transformer) or modern convolutional networks (such as ConvNeXt). Features from different levels are fused using a Feature Pyramid Network (FPN) structure. Shallow features retain edge details as pixel-level features, mid-level features combine region proposals as object-level features, and deep features are global pooled as scene-level features. Design a learnable linear projection module to uniformly map the aforementioned multi-scale visual features to a preset common embedding dimension, thereby achieving cross-modal alignment with the event ontology vector.

[0027] As a specific implementation method, the input image preprocessing and multi-scale visual feature extraction process includes: (1) Data augmentation and standardization: for input images The process involves random flipping, color jittering, and normalization.

[0028] (2) Hybrid backbone network extraction: It is recommended to use Swin Transformer V2 or ConvNeXt-V2 as the backbone network. Compared to traditional CNNs, the Transformer architecture is better at capturing long-range dependencies and is suitable for understanding the global scene context; while ConvNeXt provides efficient convolutional inductive bias. Features are extracted through the backbone network, and a multi-scale feature pyramid (FPN) is constructed: Pixel-level features (F_{pix}): taken from the shallow layer (Stride 4) of the backbone network, preserving high-resolution edge information for fine boundary segmentation.

[0029] Object-level features (F_{obj}): Taken from the middle layer (Stride 8 / 16), combined with the Region Proposal Network (RPN) or the anchorless detector head, to extract the feature vector of potential object instances.

[0030] Scene-level features (F_{scene}): taken from deep layers (Stride 32) and subjected to global average pooling (GAP) to encode the overall scene context (such as "rainy day" or "night").

[0031] (3) Cross-modal projection alignment: Design a learnable cross-modal projection head to map multi-level visual features to the same dimensional space as the ontology embedding (e.g., 512-dimensional or 768-dimensional) to obtain a visual feature sequence. Each of them A potential object region in the corresponding image (obtained through a sliding window or a pre-generated candidate box).

[0032] The feasible step S4, the visual-ontology fusion reasoning step based on heterogeneous graphs, specifically includes: S41. Heterogeneous graph construction: The object-level visual features extracted in step S3 are used as query nodes, and the event ontology role nodes in step S1 are used as prototype nodes to jointly form a heterogeneous graph node set; the edge set is constructed based on visual spatial proximity and semantic relationships defined by the ontology. S42. Graph Attention Reasoning: Utilize Graph Attention Network (GAT) or Graph Transformer module to perform multi-round message passing on heterogeneous graphs, enabling visual nodes to aggregate prior knowledge of the ontology, while enabling ontology nodes to aggregate visual evidence. S43. Event Discrimination and Instantiation: Calculate the confidence score of each predefined event type using the classification head, select the highest score as the potential event type, and determine the specific instantiation location and state attributes of each entity role in the image based on the attention weight distribution between nodes.

[0033] As a specific implementation method, the process of constructing a visual-ontology heterogeneous graph and performing bidirectional reasoning includes: (1) Visual-Ontology Alignment: Cross-modal mapping: using contrastive learning loss to map visual features Embedded with role nodes in the event body Alignment is performed. The goal is to ensure that the feature vectors of objects in the image are spatially close to the vectors of entities in the ontology.

[0034] The formula for the loss function of contrastive learning is: ; (2) Dynamic construction of heterogeneous graphs. Construct a heterogeneous graph. .in This represents the set of candidate object nodes detected in the image and the set of event / role prototype nodes in the ontology. This represents a set containing visual spatial proximity relationships and semantic relationships defined by the ontology, and also includes edges between nodes in the image and the ontology (where the cosine similarity between two nodes exceeds a specified threshold). (Then a connection is established).

[0035] (3) Graph Neural Network Message Passing: Using a Hetero-Graph Attention Network (Hetero-GAT) Round message passing.

[0036] Knowledge injection phase: The ontology node aggregates its inherent logical constraints (such as "rider" must be adjacent to "motorcycle") and passes them to the connected visual nodes, modifying the representation of the visual nodes so that they contain not only visual appearance information, but also semantic role information.

[0037] Evidence feedback phase: The visual node sends its instantiated feature evidence back to the ontology node to dynamically update the activation scores of each event type in the current image.

[0038] (4) Event recognition and role instantiation: First, the updated ontology event node scores are read, processed by a fully connected layer, and the matching probability of each predefined event type is calculated using the Softmax function. Based on this, the target event type with the highest confidence level is selected. Subsequently, according to The associated ontology character nodes are used to select the corresponding set of character instances from the visual node set. This establishes a mapping relationship between "image regions" and "event roles". .

[0039] The semantic segmentation generation under knowledge constraints in step S5, which is feasible, specifically includes: S51. Dynamic attention guidance: Based on the entity role instantiation information determined in step S4, generate a spatial attention map for role perception, and multiply the attention map element-wise with the visual features in step S3 to enhance the feature response of the target area and suppress background noise. S52. Logical consistency loss constraint: During model training or inference optimization, an event logical consistency loss term is introduced. This loss term is used to penalize segmentation prediction results that violate the spatial topological relationships or temporal relationships defined in the event ontology. S53. Decoding Output: Input the attention-modulated features into the mask generation network (such as Mask2Former or UperNet decoder), and optimize the network parameters by combining the total loss function (including cross-entropy loss, Dice loss and the aforementioned logical consistency loss) to generate the final semantic segmentation mask.

[0040] As a specific implementation method, the process of dynamic segmentation decoding and joint optimization under knowledge constraints includes: (1) Dynamic spatial attention generation: based on the mapping relationship obtained in step three Multimodal features Embedded with physical characters Combine, using the formula: ; Generate dynamic spatial attention mask This mask can dynamically adjust feature weights according to specific semantic scenarios: assigning high weights to role regions that conform to the event logic (such as "goal" and "football" in the "football shot" event), while suppressing interference from irrelevant backgrounds (such as the candidate area where "rider" is away from "motorcycle" or the stands), thereby achieving role-aware attention.

[0041] (2) Guided feature decoding: Multi-scale visual features are input into a segmentation decoder (such as an improved Mask2Former, DeepLab, or UPerNet). The decoder receives these event-knowledge-modulated features and, combined with dynamic attention masks, performs refined pixel classification predictions on the multi-scale visual features, ultimately generating a preliminary segmentation result. That is, a pixel-level semantic segmentation mask.

[0042] (3) Construct a composite loss function for end-to-end training: To ensure the accuracy of the segmentation results at the pixel, semantic, and logical levels, the total loss function is defined as: ; in, The standard segmentation loss is adopted, using cross-entropy loss ( ) and Dice loss ( The weighted sum of ) is used to constrain the basic pixel-level classification accuracy; For cross-modal alignment loss, the consistency between visual features and semantic concepts is ensured by minimizing the distance between visual region features and character ontology prototype vectors (usually using contrastive loss). The core logic consistency regularization loss aims to explicitly penalize predictions that violate the ontology definition (e.g., the ontology definition "role A must be above role B" but the prediction result is the opposite). By calculating the distance penalty for pixel pairs that violate spatial topological constraints, the model is forced to learn a logically consistent topology.

[0043] In addition, to further improve edge quality, the framework also utilizes event-triggered edge priors and applies weighted loss at object boundaries to specifically improve the segmentation accuracy of small targets and complex boundaries.

[0044] Furthermore, the process of outputting the final segmentation result includes: The trained model is used to infer the test image and output a pixel-level semantic segmentation mask. It can also optionally output the event type label corresponding to the image and the confidence distribution of each role, thus achieving a dual output of "segmentation + understanding".

[0045] On the other hand, this embodiment also provides an image semantic segmentation system based on event ontology, used to implement the method, including: The event ontology management module is used to build a structured event ontology library, which includes event type nodes, entity role nodes, attribute nodes, spatial topological relationship edges and time sequence relationship edges between nodes, and maps each node to a semantic embedding vector. A multi-scale feature extraction module is used to acquire the input image to be segmented and extract multi-scale visual features from the input image using a deep learning model; The event-visual fusion reasoning module is used to construct a heterogeneous graph containing visual candidate nodes and ontology prototype nodes based on the multi-scale visual features extracted by the multi-scale feature extraction module and the semantic embedding vector in the event ontology management module, and to use a graph neural network to perform message passing and aggregation on the heterogeneous graph to obtain the potential event type and entity role instantiation configuration corresponding to the input image. The knowledge-guided segmentation generation module is used to generate dynamic spatial attention masks and logical consistency constraints based on the determined potential event types and entity role instantiation configurations, and guide the segmentation decoder to perform weighted processing on the multi-scale visual features to generate semantic segmentation results.

[0046] Implementable, the event ontology management module includes: Ontology storage unit, used to store event ontology data using a resource description framework or property graph format; The semantic embedding generation unit, with a built-in text encoder and graph neural network submodule, is used to calculate and update the high-dimensional semantic embedding vectors of each node in the event ontology library. The ontology retrieval interface unit is used to provide ontology subgraph retrieval services based on event types.

[0047] Implementable, the multi-scale feature extraction module includes: The backbone network units, employing either visual transformation networks or modern convolutional networks, are used to extract multi-level features from the input image. The feature pyramid fusion unit is used to fuse shallow features of backbone network units as pixel-level features, mid-level features as object-level features, and deep features as scene-level features after global pooling. A cross-modal projection unit is used to map the multi-level features to the same dimensional space as the semantic embedding vectors in the event ontology library.

[0048] Implementable, the event-visual fusion reasoning module includes: The heterogeneous graph construction unit is used to dynamically construct a heterogeneous graph node set and edge set by taking the extracted object-level features as visual nodes and the role nodes in the event ontology library as ontology prototype nodes. The graph attention reasoning unit, with a built-in graph attention network or graph transformation network, is used to perform multi-round message passing on the heterogeneous graph, enabling visual nodes to aggregate prior knowledge of the ontology, and simultaneously enabling ontology nodes to aggregate visual evidence. The event discrimination and instantiation unit is used to calculate the confidence score of each predefined event type, select the highest score as the potential event type, and determine the instantiation position and state attributes of each entity role in the image based on the attention weight distribution between nodes.

[0049] Implementable, the knowledge-guided segmentation generation module includes: An attention control unit is used to generate a spatial attention map of the character perception based on the instantiation information of the entity character, and multiply the spatial attention map by the row element by element; The logical constraint calculation unit is used to calculate the event logical consistency loss during the training phase, and to penalize the segmentation prediction results that violate the spatial topological relationships or temporal relationships defined in the event ontology library. The segmentation decoding unit has a built-in mask generation network. It is used to input the features modulated by the attention control unit into the mask generation network, and optimize the network parameters by combining the total loss function to output the semantic segmentation mask.

[0050] Compared to existing technologies: This embodiment introduces a structured event ontology library, integrating high-level event logic, role constraints, and prior knowledge of spatiotemporal relationships into the visual segmentation process. Specifically, based on the deep fusion of event ontology and heterogeneous graphs, the model is no longer limited to pixel-level classification but can identify the types of events and entity roles occurring in an image, achieving a cognitive upgrade from "perceiving objects" to "understanding scenes," significantly improving semantic understanding and reasoning accuracy in complex scenarios.

[0051] Meanwhile, this embodiment utilizes heterogeneous graph neural networks to explicitly model the spatial topological and temporal relationships between entities, and injects these relationships as strong prior knowledge into the segmentation process. This enables the model to make accurate inferences based on event logic even when faced with interference such as severe occlusion, drastic lighting changes, or cluttered backgrounds, resulting in a segmentation mask with higher integrity and logical consistency. Furthermore, the event ontology, as a structured knowledge base, does not rely on a large number of labeled samples. When facing long-tailed distributions or rare events, the model can leverage the logical rules within the ontology for reasoning transfer, maintaining good segmentation performance even in scenarios with few or even zero samples, effectively alleviating the dependence on large-scale pixel-level labeled data.

[0052] The reasoning process in this embodiment is transparent and traceable. By visualizing attention weights and event matching paths, users can clearly understand the basis for the model's specific segmentation decisions. Furthermore, because the event ontology is decoupled from the segmentation network, the segmentation strategy can be flexibly adjusted by modifying or expanding the ontology library without retraining the model, greatly improving the system's maintainability and adaptability. Finally, by designing a differentiable logical consistency loss function and a dynamic attention mechanism, this embodiment integrates symbolic knowledge reasoning with deep neural networks, achieving end-to-end joint optimization from ontology construction to segmentation generation. This ensures high performance while enhancing the model's generalization ability and interpretability.

[0053] Example 2 This embodiment provides a specific implementation scheme for an image semantic segmentation system based on event ontology. The system aims to solve the semantic segmentation problem in complex traffic scenarios, especially for the understanding and segmentation of specific events such as "vehicle rear-end collision" and "pedestrian crossing".

[0054] Step 1: Construction and initialization of the event ontology knowledge base: This step aims to build a structured traffic event ontology and transform it into a computer-computable vector form.

[0055] Ontology definition and entry: Construct a transportation domain ontology using ontology editing tools (such as Protégé), defining core concepts (event types and role types, etc.) and relationships: (1) Event type ( ): Defined as "Normal driving", "Rear-end collision", "Pedestrian crossing the road" .

[0056] (2) Role type ): Defined as "Vehicles", "pedestrians", "drivers", "roads" .

[0057] (3) Attributes ): Defined as Speed, color, material .

[0058] Spatiotemporal relationship ( ): Define logical constraints, for example: Constraint 1: If the event is a "vehicle rear-end collision", there must be two "vehicle" entities, and their spatial relationship must be "in contact" or "the distance between them is less than 1 meter".

[0059] Constraint 2: If the event is "pedestrian crossing", then the "pedestrian" must be on the "road" and the angle between the direction of movement and the direction of the road must be greater than 45 degrees.

[0060] Ontology vectorization and storage: (1) Construct a symbolic event ontology library: Store the defined event ontology triples into the Neo4j graph database, establish a symbolic storage structure for entities and relations, and form a persistent event knowledge base.

[0061] (2) Constructing a vectorized semantic space: The text descriptions of the ontology nodes are encoded using the BERT-base model to generate 768-dimensional initial semantic embedding vectors; these vectors are loaded into memory to construct the event ontology vector space. And establish a mapping relationship between vector indexes and database node IDs.

[0062] Step 2, Image Preprocessing and Feature Extraction: This step processes the input traffic monitoring images to extract multi-scale visual features.

[0063] Step 2.1 Data Input and Augmentation: Get resolution Traffic surveillance images .

[0064] Image preprocessing: random horizontal flipping (probability 0.5), color dithering (brightness and contrast variation range) ), and normalize the pixel values ​​to Interval.

[0065] Step 2.2 Feature extraction of hybrid backbone network: The Swin Transformer V2-Tiny was used as the backbone network. Multi-scale feature pyramids were extracted. Pixel-level features Features derived from the Patch Embedding layer, with a stride of 4 and a size of [size missing]. Used to preserve edge details.

[0066] Object-level features Output from Stage 2, step size 16, size is , used to capture potential object instances.

[0067] Scene-level features : Global Average Pooling (GAP) output from Stage 4, size is , used to encode the overall environment (such as "rainy day", "night").

[0068] Step 2.3 Cross-modal projection: Design a projection head consisting of two fully connected layers. Each feature vector in the image is mapped to a 768-dimensional space to obtain a set of visual candidate nodes. This aligns it with the ontology embedding space generated by BERT.

[0069] Step 3, Visual-Ontology Heterogeneous Graph Reasoning: This step is the core reasoning process, which deeply integrates visual features with ontology knowledge by constructing a graph network.

[0070] Step 3.1 Heterogeneous graph construction: Building heterogeneous graphs in graph computing engines The set of nodes Includes visual candidate nodes (from) ) and ontology prototype nodes (from Neo4j query results). Edge set Includes all edges (calculated by the cosine similarity between visual nodes and ontology nodes; if the similarity is greater than a threshold). Then establish a connecting edge.

[0071] Step 3.2 Message Passing and Updates: Three rounds were performed using the Hetero-Graph Attention Network (Hetero-GAT). Message transmission, the transmission process includes knowledge injection and evidence feedback, taking a "rear-end collision" event as an example: Knowledge injection process: The event node "rear-end collision" in the ontology aggregates the logical constraint "must contain two vehicles" onto the connected visual nodes.

[0072] Evidence feedback process: The visual node sends back the features actually detected in the image (such as two cars actually being close together in a certain area) to the "rear-end collision" ontology node to increase its activation score.

[0073] Step 3.3 Event Recognition: After three rounds of iteration, the activation values ​​of the event nodes are read. Assuming the two cars in the image are extremely close (0.5 meters), the "rear-end collision" node scores the highest (e.g., The system determines the current scene as a "rear-end collision". Based on the determination result, visual nodes associated with "rear-end collision" are filtered out, thus identifying the role instances of the "front vehicle" and the "rear vehicle".

[0074] Step 4: Dynamic segmentation, decoding, and optimization: This step uses the event information obtained through inference to dynamically adjust the attention of the segmentation model and generate the final mask.

[0075] Step 4.1 Dynamic attention mask generation: Embed the characters obtained through reasoning With multi-scale visual features Concatenation. A dynamic attention mask is generated using an MLP layer and a sigmoid activation function. For example, in a rear-end collision, the mask automatically increases the weight of the contact area between the two vehicles while suppressing the weight of irrelevant background elements such as roadside trees.

[0076] Step 4.2 Segmentation Decoding and Loss Calculation: The weighted features are input into the Mask2Former decoder to generate a pixel-level segmentation mask. Then calculate the joint loss: set the weighting coefficients. Segmentation loss Using cross-entropy loss ( ) and Dice loss ( The weighted sum of () solves the class imbalance problem. Logical consistency loss. Check if the prediction results violate ontology constraints. For example, if the prediction is that the "pedestrian" is inside the "vehicle" (which violates common sense physics), calculate the penalty term and add it to the total loss.

[0077] Step 4.3 Model Training: (1) Train the Mask2Former model using the AdamW optimizer, with an initial learning rate set to The batch size is 16, which balances training stability and efficiency.

[0078] (2) Training was based on a traffic dataset containing 5,000 labeled images, with a total of 100 rounds. Validation was performed every 10 epochs to monitor performance. At the same time, the CosineAnnealingLR learning rate scheduler and early stopping mechanism (patience value 10~15, monitoring val mIoU) were adopted to alleviate overfitting, improve the model's generalization ability, and ensure that the training specifications were reproducible.

[0079] Step 5, Output the results: After training is complete, input the image to be tested into the system: (1) Visual output: Output a segmentation mask with the same resolution as the original image, clearly marking the pixel areas such as "vehicles", "pedestrians", and "roads".

[0080] (2) Semantic output: Display the text at the top of the interface: "Event detected: vehicle rear-end collision (confidence 92%)".

[0081] (3) Explainable output: The reasoning path is highlighted in the sidebar: “According to the rule: the distance between the two vehicles is <1 meter and the relative speed is >0.”

[0082] As an additional implementation method, the hardware environment for model training and running is as follows: The above embodiments are run on a server equipped with an NVIDIA A100 GPU (40GB VRAM), an Intel Xeon Gold 6330 processor, and 128GB of memory.

[0083] System software dependencies: Python 3.8, PyTorch 1.12, PyTorch Geometric (for graph neural networks), Transformers (for BERT).

[0084] Through the above specific implementation methods, the present invention successfully injects high-level event logic knowledge into the low-level pixel segmentation process, achieving more accurate and logical image understanding in complex scenarios.

[0085] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image semantic segmentation method based on event ontology, characterized in that, Includes the following steps: Construct a structured event ontology library, which includes event type nodes, entity role nodes, attribute nodes, spatial topological relationship edges and time sequence relationship edges between nodes, and map each node to a semantic embedding vector; The input image to be segmented is obtained, and multi-scale visual features are extracted from the input image using a deep learning model; Based on the extracted multi-scale visual features and the semantic embedding vectors in the event ontology library, a heterogeneous graph containing visual candidate nodes and ontology prototype nodes is constructed. The heterogeneous graph is processed by a graph neural network for message passing and aggregation to obtain the potential event types and entity role instantiation configurations corresponding to the input image. Based on the identified potential event types and entity role instantiation configurations, a dynamic spatial attention mask and logical consistency constraints are generated, and the segmentation decoder is guided to perform weighted processing on the multi-scale visual features to generate semantic segmentation results.

2. The method according to claim 1, specifically comprising constructing a structured event ontology library: The event body is stored using a resource description framework or property graph format; Define the metadata of the event node, which includes the event trigger words and a list of participant roles, and establish a mapping relationship between the list of participant roles and visual semantic tags; Define the constraint relationships between role nodes, including spatial location constraints and temporal state constraints; By using a contrastive learning pre-training method, the semantic embedding vectors of each node in the event ontology are initialized so that the semantic embedding vectors are close to the distance of similar visual concepts in the feature space.

3. The method according to claim 1, specifically comprising: extracting multi-scale visual features including: Image features are extracted using a backbone network based on a visual transformation network or a modern convolutional network; By fusing image features from different levels through a feature pyramid network structure, multi-scale visual features are obtained. Shallow features are used as pixel-level features, mid-level features are used as object-level features, and deep features are used as scene-level features after global pooling. The pixel-level features, object-level features, and scene-level features are then mapped to the same dimensional space as the semantic embedding vectors in the event ontology library.

4. The method according to claim 1, specifically comprising constructing a heterogeneous graph and performing message passing and aggregation, includes: The extracted object-level features are used as query nodes, and the role nodes in the event ontology are used as prototype nodes to form a heterogeneous graph node set. An edge set is constructed based on visual spatial proximity and semantic relationships defined in the event ontology. Graph attention networks or graph transformation networks are used to perform multi-round message passing on the heterogeneous graph, enabling visual nodes to aggregate prior knowledge of the ontology, and simultaneously enabling ontology nodes to aggregate visual evidence. The confidence score for each predefined event type is calculated using a classification head. The highest score is selected as the potential event type. Based on the attention weight distribution between nodes, the instantiation location and state attributes of each entity role in the image are determined.

5. The method according to claim 1, specifically comprising generating a dynamic spatial attention mask and logical consistency constraints and guiding the segmentation decoder, includes: Based on the determined entity role instantiation information, a spatial attention map for role perception is generated, and the spatial attention map is multiplied element-wise with the multi-scale visual features to enhance the feature response of the target region and suppress background noise. An event logic consistency loss term is introduced to penalize segmentation prediction results that violate spatial topological or temporal relationships defined in the event ontology library; The attention-modulated features are input into the mask generation network, and the network parameters are optimized by combining a total loss function including cross-entropy loss, Dess loss, and event logic consistency loss to generate the final semantic segmentation result.

6. An image semantic segmentation system based on event ontology, characterized in that, For implementing the method according to any one of claims 1-5, comprising: The event ontology management module is used to build a structured event ontology library, which includes event type nodes, entity role nodes, attribute nodes, spatial topological relationship edges and time sequence relationship edges between nodes, and maps each node to a semantic embedding vector. A multi-scale feature extraction module is used to acquire the input image to be segmented and extract multi-scale visual features from the input image using a deep learning model; The event-visual fusion reasoning module is used to construct a heterogeneous graph containing visual candidate nodes and ontology prototype nodes based on the multi-scale visual features extracted by the multi-scale feature extraction module and the semantic embedding vector in the event ontology management module, and to use a graph neural network to perform message passing and aggregation on the heterogeneous graph to obtain the potential event type and entity role instantiation configuration corresponding to the input image. The knowledge-guided segmentation generation module is used to generate dynamic spatial attention masks and logical consistency constraints based on the determined potential event types and entity role instantiation configurations, and guide the segmentation decoder to perform weighted processing on the multi-scale visual features to generate semantic segmentation results.

7. The system according to claim 6, wherein the event ontology management module comprises: Ontology storage unit, used to store event ontology data using a resource description framework or property graph format; The semantic embedding generation unit, with a built-in text encoder and graph neural network submodule, is used to calculate and update the high-dimensional semantic embedding vectors of each node in the event ontology library. The ontology retrieval interface unit is used to provide ontology subgraph retrieval services based on event types.

8. The system according to claim 6, wherein the multi-scale feature extraction module comprises: The backbone network units, employing either visual transformation networks or modern convolutional networks, are used to extract multi-level features from the input image. The feature pyramid fusion unit is used to fuse shallow features of backbone network units as pixel-level features, mid-level features as object-level features, and deep features as scene-level features after global pooling. A cross-modal projection unit is used to map the multi-level features to the same dimensional space as the semantic embedding vectors in the event ontology library.

9. The system according to claim 6, wherein the event-visual fusion reasoning module comprises: The heterogeneous graph construction unit is used to dynamically construct a heterogeneous graph node set and edge set by taking the extracted object-level features as visual nodes and the role nodes in the event ontology library as ontology prototype nodes. The graph attention reasoning unit, with a built-in graph attention network or graph transformation network, is used to perform multi-round message passing on the heterogeneous graph, enabling visual nodes to aggregate prior knowledge of the ontology, and simultaneously enabling ontology nodes to aggregate visual evidence. The event discrimination and instantiation unit is used to calculate the confidence score of each predefined event type, select the highest score as the potential event type, and determine the instantiation position and state attributes of each entity role in the image based on the attention weight distribution between nodes.

10. The system according to claim 6, wherein the knowledge-guided segmentation generation module comprises: An attention control unit is used to generate a spatial attention map of the character perception based on the instantiation information of the entity character, and multiply the spatial attention map by the row element by element; The logical constraint calculation unit is used to calculate the event logical consistency loss during the training phase, and to penalize the segmentation prediction results that violate the spatial topological relationships or temporal relationships defined in the event ontology library. The segmentation decoding unit has a built-in mask generation network. It is used to input the features modulated by the attention control unit into the mask generation network, and optimize the network parameters by combining the total loss function to output the semantic segmentation mask.

Images