Scene sketch semantic segmentation method and system based on language guidance and graph transformer

Abstract

The application provides a scene sketch semantic segmentation method and system based on language guidance and graph Transformer, and relates to the technical field of computer vision. The method comprises the following steps: obtaining stroke sequence data of a scene sketch; performing multi-modal feature extraction on the stroke sequence data to obtain texture features, semantic features and geometric features of each stroke respectively; fusing the three features through a cross-modal attention mechanism to obtain a fused feature representation of each stroke; constructing a graph structure based on the spatial or temporal adjacency relationship between strokes, taking strokes as nodes and adjacency relationships as edges, and taking the fused feature representation as the initial feature of the node; inputting the graph structure into a graph Transformer encoder for message passing and node feature updating to obtain updated node features; and predicting the semantic category of each stroke based on the updated node features to complete the semantic segmentation of the scene sketch.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a scene sketch semantic segmentation method and system based on language guidance and graph Transformer. BACKGROUND

[0002] The statements in this section merely provide background technology related to the present disclosure and do not necessarily constitute prior art.

[0003] As a natural and efficient visual expression method, hand-drawn sketches have important application value in many fields such as architectural design, game animation prototype design, and human-computer interaction. Compared with traditional pixel-based image processing methods, stroke-based recognition methods can better fit the cognitive process of human drawing. Stroke data directly records the original trajectory information of the drawing, including the coordinate sequence of the brush stroke, the time sequence, etc., and its time sequence can reflect the drawing intention. The topological relationship between strokes (such as intersection and inclusion) can also be directly used to infer object association. Existing research shows that in a single object classification task, stroke-based methods based on recurrent neural networks (RNN) or time Transformer can improve the accuracy by 10-15% compared with image classification models.

[0004] However, there are several technical problems in applying stroke information to scene sketch semantic segmentation:

[0005] (1) The strokes of a scene sketch may belong to different objects alternately, and the strokes of different objects are mixed. Existing methods are difficult to effectively separate the mixed strokes and establish the correspondence between objects and strokes.

[0006] (2) Traditional stroke segmentation methods mainly focus on the geometric features of single strokes, but lack modeling of spatial relationships. Scene understanding needs to model high-level semantics such as occlusion and relative position between objects.

[0007] (3) The stroke styles in different fields differ significantly, and it is difficult to capture features of small objects. Existing methods perform poorly in related data sets.

[0008] Therefore, there is an urgent need for a semantic segmentation method that can combine stroke temporal features and multi-modal fusion of semantic understanding, solve the problems of multi-object stroke separation, spatial relationship reasoning, and small sample learning robustness, and realize high-precision scene sketch segmentation. SUMMARY

[0009] To solve the above problems, the present application provides a scene sketch semantic segmentation method and system based on language guidance and graph Transformer, which is used to realize fine semantic segmentation of sketch strokes.

[0010] The first aspect of the present application provides a scene sketch semantic segmentation method based on language guidance and graph Transformer, comprising:

[0011] Obtaining stroke sequence data of a scene sketch;

[0012] Performing multi-modal feature extraction on the stroke sequence data to obtain texture features , semantic features and geometric features of each stroke respectively;

[0013] Fusing the texture features, semantic features and geometric features through a cross-modal attention mechanism to obtain a fused feature representation of each stroke;

[0014] Constructing a graph structure based on the spatial or temporal adjacency relationship between strokes, wherein the nodes are strokes and the edges are adjacency relationships, and taking the fused feature representation as the initial feature of the nodes;

[0015] Inputting the graph structure into a graph Transformer encoder for message passing and node feature updating to obtain updated node features;

[0016] Based on the updated node features, predicting the semantic category of each stroke to complete the semantic segmentation of the scene sketch.

[0017] Further, the semantic features are generated by a text encoder of a pre-trained language-visual model ;

[0018] In the training stage, real category label texts are input into the text encoder to generate semantic features: ;

[0019] In the inference stage, the mean of the embedding vectors of all N preset category label texts is used as a proxy semantic feature: .

[0020] Further, the execution steps of the cross-modal attention mechanism include:

[0021] The texture features and semantic features are respectively transformed and added after linear projection weight matrix and transformation to generate query vectors ;

[0022] The geometric features are respectively transformed and added after linear projection weight matrix and transform, to generate a key vector and a value vector wherein, ;

[0023] calculate a query vector Q and a scaled dot product similarity with the key vector K , normalized by a softmax function to obtain an attention weight;

[0024] perform weighted summation on the value vector V using the attention weight to obtain a fused feature output : wherein is the dimension of the key vector.

[0025] Further, the output of the cross-modal attention mechanism is processed via a residual connection and layer normalization to finally output a feature H represented as: H= LayerNorm( + ), wherein is the original input feature of the module.

[0026] Further, the geometric feature includes at least one of the following: length, direction, minimum bounding box information, and drawing timing information of the stroke.

[0027] Further, the graph Transformer encoder updates the node-level feature through a graph attention layer, and the update formula of the node in the first l layer is: wherein is the node feature in the first l layer, and GAT(·) is a graph attention operation.

[0028] Further, a joint loss function combining cross-entropy loss and geometric consistency loss is used for model training, and the total loss is represented as: wherein, is the node-level cross-entropy loss, is a geometric loss calculated based on the predicted segmentation and the true value bounding box intersection over union IoU, and λ is a hyperparameter for balancing the two loss terms.

[0029] The second aspect of the present application provides a scene sketch semantic segmentation system based on language guidance and graph Transformer, comprising:

[0030] a data acquisition unit configured to acquire stroke sequence data of a scene sketch;

[0031] A multi-modal feature extraction unit is configured to perform multi-modal feature extraction on the stroke sequence data to obtain texture features, semantic features and geometric features of each stroke respectively.

[0032] A cross-modal feature fusion unit is configured to fuse the texture features, semantic features and geometric features through a cross-modal attention mechanism to obtain a fused feature representation of each stroke.

[0033] A graph structure construction unit is configured to construct a graph structure based on spatial or temporal adjacency relationships between strokes, wherein nodes are strokes and edges are adjacency relationships, and the fused feature representation is taken as an initial feature of the nodes.

[0034] A graph encoding and node updating unit is configured to input the graph structure into a graph Transformer encoder to perform message passing and node feature updating, thereby obtaining updated node features.

[0035] A semantic prediction and output unit is configured to perform semantic category prediction on each stroke based on the updated node features, thereby completing semantic segmentation of the scene sketch.

[0036] A third aspect of the present application provides a device for semantic segmentation of a scene sketch based on language guidance and graph Transformer, which comprises a memory and a processor; the memory is configured to store a computer program; the processor is configured to implement the above-mentioned method for semantic segmentation of a scene sketch based on language guidance and graph Transformer when the computer program is executed.

[0037] A fourth aspect of the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for semantic segmentation of a scene sketch based on language guidance and graph Transformer is implemented.

[0038] Compared with the prior art, the method and system for semantic segmentation of a scene sketch based on language guidance and graph Transformer provided by the present application have the following advantages:

[0039] (1) For the problem of mixed strokes of multiple objects, the present application is based on multi-modal feature extraction and cross-modal attention fusion technology features, and through the collaborative processing of stroke texture, semantic and geometric features, accurate separation and semantic aggregation of mixed strokes are realized, and the segmentation accuracy in complex scenes is improved. The semantic feature text embedding is explicitly injected at the node level, and the attention is recalibrated with geometric information as a spatial anchor point, effectively eliminating the semantic ambiguity and positioning deviation of similar strokes.

[0040] ​​​(2) In view of the problem of insufficient spatial relationship modeling, the application is based on graph structure construction and graph Transformer encoder, and the spatial and temporal adjacency relationship between strokes is modeled explicitly, and a hierarchical message passing mechanism is combined to enhance the reasoning ability of high-level semantics such as occlusion and relative position between objects.

[0041] (3) To solve the problem of small sample generalization, the application is based on conditional semantic embedding mechanism (CSE), and the real class semantics is injected in the training stage, and the class mean agent is used in the reasoning stage, which effectively avoids label leakage and improves the generalization ability of the model to unknown classes and cross-domain data.

[0042] (4) The application also improves the fine-grained segmentation quality, structural consistency and training stability through the asymmetric cross-modal attention mechanism (CAM) and the joint loss function, and the three types of modalities are aligned in a unified dimension, which can be seamlessly integrated with multi-head attention, residual / norm / regularization modules, and has good engineering scalability and deployment feasibility. BRIEF DESCRIPTION OF DRAWINGS

[0043] The drawings accompanying the specification of this disclosure serve to provide further understanding of the disclosure, and the illustrative embodiments of the disclosure and their descriptions are used to explain the disclosure, and do not constitute an improper limitation on the disclosure.

[0044] Figure 1 is a flowchart of the scene sketch semantic segmentation method based on language guidance and graph Transformer provided by the embodiment one of the application;

[0045] Figure 2 is a whole architecture diagram of the sketch semantic segmentation method based on multi-modal graph Transformer provided by the embodiment one of the application;

[0046] Figure 3 is a cross-modal attention mechanism (CAM) structure schematic diagram provided by the embodiment one of the application;

[0047] Figure 4 is a sketch semantic segmentation flowchart based on multi-modal graph Transformer provided by the embodiment one of the application;

[0048] Figure 5 is a schematic diagram of the scene sketch semantic segmentation system based on language guidance and graph Transformer provided by the embodiment two of the application. DETAILED DESCRIPTION

[0049] It should be pointed out that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise specified, all technical and scientific terms used in the application have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0050] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0051] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0052] All data acquisition in this embodiment is based on legal regulations and user consent, and the legal application of data.

[0053] Term explanation:

[0054] Multimodal Fusion: refers to the integration of features from different sources or modalities (such as texture, semantics, geometry) to enhance the expressive power and robustness of the model.

[0055] Graph Transformer: a graph neural network based on the Transformer architecture, used to process graph-structured data, modeling and updating features between nodes through self-attention mechanisms.

[0056] Conditional Semantic Embedding (CSE): a method of dynamically injecting semantic information during training and inference to enhance the model's ability to distinguish between classes and generalize across classes.

[0057] Cross-modal Attention Mechanism (CAM): an asymmetric attention mechanism that aligns and fuses features from different modalities (such as semantics, texture, geometry) to avoid feature flooding and improve structural perception.

[0058] Geometric Features: refers to features related to the spatial properties and morphology of strokes, such as length, direction, minimum bounding box (Bounding Box), drawing timing, etc.

[0059] Node-level update: In a graph neural network, each node updates its features based on information from its neighboring nodes.

[0060] Example 1

[0061] Please see Figure 1 This invention provides a scene sketch semantic segmentation method based on language guidance and graph Transformer, comprising:

[0062] Obtain the stroke sequence data of the scene sketch;

[0063] Multimodal feature extraction is performed on the stroke sequence data to obtain the texture features of each stroke. semantic features and geometric features ;

[0064] The texture features, semantic features, and geometric features are fused using a cross-modal attention mechanism to obtain the fused feature representation of each stroke;

[0065] A graph structure is constructed based on the spatial or temporal adjacency relationship between strokes, where nodes are strokes and edges are adjacency relationships, and the fused feature representation is used as the initial feature of the node.

[0066] The graph structure is input into a graph Transformer encoder for message passing and node feature updating to obtain the updated node features.

[0067] Based on the updated node features, semantic category prediction is performed for each stroke to complete the semantic segmentation of the scene sketch.

[0068] The overall process of this method first obtains local texture features, semantic features, and geometric features of strokes through parallel feature extraction layers. These features are then input into a cross-modal fusion module, which utilizes an asymmetric attention mechanism to achieve directional alignment and deep fusion of semantic, texture, and geometric information. The fused node features serve as the initial state of a graph Transformer, undergoing message passing and hierarchical structure modeling through a multi-layer graph attention encoder, ultimately generating a semantic segmentation prediction for each stroke node. During training, the model enhances its class discrimination ability through conditional semantic embedding, and during inference, it ensures cross-class generalization through zero-shot proxies. Simultaneously, the multi-modal fusion strategy ensures the comprehensive expression of node-level features across semantic, texture, and spatial dimensions, thereby achieving fine-grained, robust, and high-precision sketch segmentation.

[0069] Specifically, the semantic features Text encoder using a pre-trained language-visual model generate;

[0070] In the training phase, real class label texts are used The text encoder is input to generate semantic features: ;

[0071] In the inference phase, the mean of the embedding vectors of all N preset class label texts is used as a proxy semantic feature: . .

[0072] Specifically, the execution steps of the cross-modal attention mechanism include:

[0073] The texture features and semantic features are respectively transformed by linear projection weight matrices and and then added to generate query vectors ;

[0074] The geometry features are respectively transformed by linear projection weight matrices and to generate key vectors and value vectors , where ;

[0075] The scaled dot product similarity of the query vectors Q and the key vectors K is calculated, and the attention weights are obtained after normalization by the softmax function;

[0076] The value vectors V are weighted and summed using the attention weights to obtain the fused feature output : , where is the dimension of the key vector.

[0077] Specifically, the output of the cross-modal attention mechanism is processed via residual connection and layer normalization, and the final output feature H is represented as: H= LayerNorm( + ), where is the original input feature of the module.

[0078] Specifically, the geometry features include at least one of the following: length, direction, minimum bounding box information, drawing timing information.

[0079] Specifically, the graph Transformer encoder updates the node-level features by a graph attention layer, and the first l The update formula of the layer node is: wherein is the node feature of the first l layer, and GAT(·) is a graph attention operation.

[0080] Specifically, a joint loss function combining cross-entropy loss and geometric consistency loss is used for model training, and the total loss is expressed as: wherein, is the node-level cross-entropy loss, is the geometric loss calculated based on the predicted segmentation and the true value bounding box intersection over union (IoU), and λ is a hyperparameter for balancing the two loss terms.

[0081] In one specific embodiment, as Figure 2 illustrated, the complete processing flow of the method of the present application is shown. The flow starts with inputting the sketch stroke data, and three types of heterogeneous features are extracted in parallel from the texture feature source, the semantic feature source (CLIP) and the geometric feature source (spatial / temporal). Subsequently, through the conditional semantic embedding (CSE) module, the embedding of the true class label is injected in the training stage to strengthen the supervision, and in the inference stage, the mean of all class embeddings is used as a proxy to ensure the zero-shot generalization capability. In the feature fusion stage, the core asymmetric attention mechanism (CAM) is used, which jointly uses texture and semantic features as queries (Q) and geometric features as keys (K) and values (V), and through weighted aggregation and residual normalization, the directional alignment and deep fusion of multi-modal information are realized. The fused features are used to construct a graph structure and input into the graph encoding module, which performs message passing and node updating through edge feature enhancement and multi-layer graph attention network, during which the gating coupling can be introduced to balance the weights of different feature sources. Finally, the network performs node-level semantic segmentation prediction, outputs the classification results of each stroke, and performs end-to-end optimization through the loss function combining cross-entropy and geometric consistency constraints. This flowchart systematically demonstrates the whole process from multi-modal feature extraction, adaptive semantic embedding, cross-modal fusion to structured prediction, which embodies the integrity and innovation of the method of the present application.

[0082] The application adopts a three-way parallel feature extraction architecture to capture key information of sketch strokes from different modalities: first, the texture features of strokes are extracted by a convolutional neural network (CNN) to accurately depict the curvature changes, endpoint morphology and local details of lines; second, a semantic feature text encoder is used to generate high-level semantic embeddings, and category prior knowledge is introduced to enhance semantic discrimination ability; finally, the geometric attributes are encoded to extract spatial features including stroke length, drawing timing information and minimum bounding box position, providing a basis for subsequent fusion and structure modeling.

[0083] The conditional semantic embedding mechanism (CSE) is used to eliminate the semantic distribution difference between the training and inference stages. In the training stage, the model uses the real class label to retrieve the precomputed text embedding and injects it into the stroke node feature to enhance the class discrimination ability:

[0084]

[0085] In the inference stage, to achieve zero-shot generalization, the model uses the mean of all class text embeddings as a proxy:

[0086]

[0087] Through the stage adaptive strategy, CSE can make full use of the supervision signal in the training stage and maintain stable generalization ability for unknown classes in the inference stage, effectively bridging the training-inference gap.

[0088] As Figure 3 , the calculation process of the core cross-modal attention mechanism (CAM) in the application is detailed. This module is responsible for fusing texture, semantic and geometric heterogeneous features. Specifically, the input texture features and semantic features are jointly generated after linear projection (Linear) to generate query vectors (Q), while the geometric features are separately projected to generate key vectors (K) and value vectors (V), forming an asymmetric attention architecture. After that, the query vector and the key vector perform matrix multiplication (Matmul) to calculate the similarity, and the result is scaled and input into the Softmax function to obtain the attention weight. Finally, the value vector (V) is weighted and summed by the weight to realize the re-alignment and calibration of semantic and texture information with geometric features as spatial anchors, and the fused features are output. This mechanism effectively solves the feature alignment problem in heterogeneous modal fusion and is the key to improving stroke positioning and segmentation accuracy.

[0089] In the cross-modal attention mechanism (CAM), in order to effectively integrate texture, semantics and geometry three kinds of features, the application designs a heterogeneous query-key value attention architecture. In the specific implementation, the texture and semantic features jointly constitute the query vector Q, and the geometry feature occupies the key value space K / V, forming an asymmetric interaction path:

[0090]

[0091]

[0092] The model dynamically calculates the correlation between texture and semantic information and geometry features through scaled dot product attention, and performs weighted aggregation according to the attention weight:

[0093]

[0094] At the output end, residual connection and layer normalization are adopted, and a light gating unit can be set to balance the original features and geometry guided correction. This design effectively avoids the feature flooding problem in the fusion of heterogeneous modalities in the traditional symmetric attention, and forces the model to learn the directional mapping of semantics / texture to geometry, thereby improving the stroke positioning accuracy and structure perception ability.

[0095] Figure 4 The core architecture of the multi-modal graph Transformer (M3F-GT) network proposed by the application is shown. The architecture takes the stroke sequence of the sketch as input, first obtains the texture features, semantic features and geometry features of the strokes through parallel multi-modal feature extraction branches. These features are then sent to the cross-modal fusion module (M3F), and the directional alignment and deep fusion of semantic and texture information in the geometry space are realized through the asymmetric attention mechanism (CAM). The fused features are constructed into a graph structure, where the nodes represent the strokes and the edges represent the spatio-temporal adjacency relationship between the strokes. This graph structure is passed through a graph Transformer (GT) encoder for multi-layer message passing and node feature updating to explicitly model the semantic relationship and hierarchical structure between the strokes. Finally, the semantic categories of each stroke are output through a softmax classifier, realizing pixel-level sketch segmentation. The graph clearly shows the technical route of the application for solving the stroke separation and semantic segmentation problem by fusing multi-modal information and graph structure reasoning.

[0096] The fused node embedding is used as the initial state of the graph Transformer, and message passing and hierarchical feature aggregation are performed through multiple layers of graph attention networks. Each layer of graph attention updates the features of the nodes through the adjacency matrix, and combines residual connection and layer normalization to keep the information flow stable:

[0097]

[0098] The construction of the graph structure is based on stroke adjacency relationship, including spatial distance and drawing timing information, ensuring that each node can maintain structural consistency when receiving neighborhood messages. The gating coupling module can dynamically adjust the weights of semantic / textural features and geometric features in the node-level update process to improve the balance of local and global features.

[0099] The loss function design and training strategy combine the node-level cross-entropy loss and geometric consistency constraint in the training objective function, where the geometric 0 consistency is measured by the IoU of the predicted segmentation result and the stroke bounding box, thereby enhancing the structural constraint:

[0100]

[0101] Embodiment Two

[0102] As shown in Figure 5 , the embodiment provides a scene sketch semantic segmentation system based on language guidance and graph Transformer, which includes:

[0103] A data acquisition unit is configured to acquire stroke sequence data of a scene sketch.

[0104] A multi-modal feature extraction unit is configured to perform multi-modal feature extraction on the stroke sequence data to obtain texture features , semantic features and geometric features of each stroke, respectively.

[0105] A cross-modal feature fusion unit is configured to fuse the texture features, semantic features and geometric features through a cross-modal attention mechanism to obtain a fused feature representation of each stroke.

[0106] A graph structure construction unit is configured to construct a graph structure based on spatial or temporal adjacency relationship between strokes, wherein the nodes are strokes and the edges are adjacency relationship, and the fused feature representation is used as the initial feature of the node.

[0107] A graph encoding and node updating unit is configured to input the graph structure into a graph Transformer encoder for message passing and node feature updating to obtain updated node features.

[0108] A semantic prediction and output unit is configured to predict the semantic class of each stroke based on the updated node features to complete the semantic segmentation of the scene sketch.

[0109] Embodiment Three

[0110] The embodiment provides a scene sketch semantic segmentation device based on language guidance and a graph Transformer, the device comprises a memory and a processor; the memory is used for storing a computer program; the processor is used for realizing the scene sketch semantic segmentation method based on language guidance and the graph Transformer when the computer program is executed.

[0111] The processor is connected with the memory, the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method in the embodiment one.

[0112] It should be understood that, in the embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSPs, application-specific integrated circuits ASICs, ready-to-program gate arrays FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0113] The memory can include read-only memory and random access memory, and provide instructions and data for the processor, and a part of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0114] In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software.

[0115] The method in the embodiment one can be directly embodied as execution completion of a hardware processor, or execution completion of a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads information in the memory and combines hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0116] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiment can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software mode depends on the specific application and design constraints of the technical solution. The professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0117] Embodiment four

[0118] A computer readable storage medium storing a computer program is provided in still another embodiment of the present application. When the computer program is executed by a processor, a language guidance and graph Transformer based scene sketch semantic segmentation method as described above is implemented.

[0119] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiments can be included. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc. In the present application, the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application. In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0120] Although the present application is disclosed as above, the protection scope of the present application is not limited to this. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, and these changes and modifications will fall within the protection scope of the present application.

Claims

1. A method for scene sketch semantic segmentation based on language guidance and graph Transformer, characterized in that, The method comprises: obtaining stroke sequence data of a scene sketch; Multi-modal feature extraction is performed on the stroke sequence data to obtain texture features, semantic features and geometric features of each stroke respectively The semantic features are generated by a text encoder of a pre-trained language-visual model In the training stage, real class label texts are input into the text encoder to generate semantic features In the inference stage, the mean of the embedding vectors of all N preset class label texts is used as a proxy semantic feature ​​​ fusing the texture feature, the semantic feature and the geometric feature through a cross-modal attention mechanism to obtain a fused feature representation of each stroke; the execution steps of the cross-modal attention mechanism comprise: The texture features and semantic features are respectively transformed by linear projection weight matrices and and then added to generate a query vector ; transforming the geometric features respectively by a linear projection weight matrix and transforming, generating a key vector and a value vector wherein ; Computing query vectors Q Similarity with key vectors K The scaled dot product is normalized by a softmax function to obtain the attention weight. using the attention weights on the value vectors V performing a weighted sum to obtain a fused feature output : wherein is the dimension of the key vector; constructing a graph structure based on spatial or temporal adjacency relationships between strokes, wherein nodes are strokes and edges are adjacency relationships, and taking the fused feature representation as an initial feature of the node; inputting the graph structure into a graph Transformer encoder for message passing and node feature updating to obtain an updated node feature; based on the updated node feature, predicting a semantic category for each stroke to complete semantic segmentation of the scene sketch.

2. The method of claim 1, wherein, The output of the cross-modal attention mechanism is processed via a residual connection and layer normalization, and the final output feature H is represented as: H= LayerNorm( + ), where is the original input feature of the module.

3. The method of claim 1, wherein, The geometric feature comprises at least one of the following: length, direction, minimum bounding box information and drawing timing information of the stroke.

4. The method of claim 1, wherein, The graph Transformer encoder performs node-level feature updates through a graph attention layer. l The update formula for layer nodes is: ,in For the first l The node features of the layer, and GAT(·) is the graph attention operation.

5. The method of claim 1, wherein, The model is trained by using a joint loss function combining a cross-entropy loss and a geometric consistency loss, and the total loss is represented as: wherein, is a node-level cross-entropy loss, is a geometric loss calculated based on an intersection over union IoU between a predicted segmentation and a ground truth bounding box, and λ is a hyperparameter balancing the two loss terms.

6. A language-guided and graph Transformer-based scene sketch semantic segmentation system, characterized in that, The method comprises: a data acquisition unit configured to acquire stroke sequence data of a scene sketch; The multimodal feature extraction unit is used to perform multimodal feature extraction on the stroke sequence data to obtain the texture features of each stroke. semantic features and geometric features The semantic features Text encoder using a pre-trained language-visual model Generate; during the training phase, use real category-labeled text. Input the text encoder to generate semantic features: During the reasoning phase, all N pre-defined category label texts are used. The mean of the embedding vectors is used as the proxy semantic feature: ; a cross-modal feature fusion unit configured to fuse the texture feature, the semantic feature and the geometric feature through a cross-modal attention mechanism to obtain a fused feature representation of each stroke; the execution steps of the cross-modal attention mechanism comprise: The texture features and semantic features are respectively transformed by linear projection weight matrices and and then added to generate a query vector ; transforming the geometric features respectively by a linear projection weight matrix and transforming, generating a key vector and a value vector wherein ; Computing query vectors Q Similarity with key vectors K The scaled dot product is normalized by a softmax function to obtain the attention weight. using the attention weights on the value vectors V performing a weighted sum to obtain a fused feature output : wherein is the dimension of the key vector; a graph structure construction unit configured to construct a graph structure based on spatial or temporal adjacency relationships between strokes, wherein nodes are strokes and edges are adjacency relationships, and taking the fused feature representation as an initial feature of the node; a graph encoding and node updating unit configured to input the graph structure into a graph Transformer encoder for message passing and node feature updating to obtain an updated node feature; a semantic prediction and output unit configured to predict a semantic category for each stroke based on the updated node feature to complete semantic segmentation of the scene sketch.

7. A language guidance and graph Transformer based scene sketch semantic segmentation device, characterized in that, The device comprises a memory and a processor; the memory is configured to store a computer program; the processor is configured to implement the scene sketch semantic segmentation method based on language guidance and graph Transformer according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the method for semantic segmentation of a scene sketch based on language guidance and graph Transformer according to any one of claims 1 to 5 is implemented.

Images