Cross-modal plant station main wiring blueprint primitive feature extraction and representation design method, system and equipment based on natural language description and medium

Abstract

The invention discloses a cross-modal plant station main wiring blueprint primitive feature extraction and representation design method, system and device based on natural language description and a medium, and belongs to the technical field of electrical engineering digitization and power system automation, and the method comprises the steps: respectively extracting multi-scale visual features of a blueprint image and semantic features of text prompt; cross-modal feature interaction fusion is realized through a feature enhancement layer, and language-guided target query is generated based on feature similarity; visual positioning and semantic information are fused in query through iterative optimization of a cross-modal decoder; and outputting a primitive bounding box and category prediction which are completely matched with the text description. The end-to-end open vocabulary detection method is constructed by successfully adapting a target detection model DINO detection framework based on Transform to the field of a plant station electrical main wiring blueprint and introducing a multi-level cross-modal fusion mechanism and text representation.

Description

Technical Field

[0001] This invention relates to the field of electrical engineering digitization and power system automation technology, specifically to a method, system, equipment, and medium for extracting and representing the feature elements of cross-modal power plant main wiring blueprints based on natural language description. Background Technology

[0002] Existing methods for identifying main wiring diagrams in power plants primarily rely on convolutional neural networks (CNNs) for object detection and image processing. They extract global and local visual features of elements and combine this with structured representations and rule matching to achieve component identification. However, these methods are essentially closed-set detection paradigms, with their recognition categories strictly limited to a predefined training set. They cannot respond to unforeseen or newly added element categories in the blueprint design, resulting in insufficient generalization ability. While emerging open-set detection models (such as DINO) have shown zero-shot detection potential, their structures are simplistic and lack effective mechanisms for incorporating and integrating natural language instructions. Furthermore, some language-guided models (such as GLIP) often struggle to handle complex technical descriptions due to insufficient flexibility. Therefore, existing technologies cannot achieve the goal of flexibly and accurately querying and detecting arbitrary main wiring diagram elements through free text. Summary of the Invention

[0003] In view of the above-mentioned problems, the present invention provides a method, system, equipment and medium for extracting and characterizing the basic features of cross-modal power plant main wiring blueprints based on natural language description.

[0004] Therefore, the technical problem solved by this invention is: how to overcome the limitations of existing plant main wiring blueprint recognition methods, which rely on closed category sets and cannot respond to flexible text commands, so as to achieve end-to-end, open-vocabulary arbitrary blueprint element detection. Specifically, it is necessary to solve the problem of effectively adapting the Transformer-based object detection model DINO to the professional blueprint domain and breaking through the single visual modality to achieve deep fusion of text and image.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a method for feature extraction and characterization of cross-modal power plant main wiring blueprint elements based on natural language description, comprising, The process involves acquiring the main wiring blueprint image of the plant and the text prompt describing the target to be detected; extracting multi-scale visual features from the blueprint image using an image encoder and extracting text features from the text prompt using a language encoder; performing cross-modal interaction and fusion of multi-scale visual features and text features through a feature enhancement layer to generate cross-modal image features; and performing language-guided query initialization based on the similarity between cross-modal image features and text features, dynamically selecting image region features related to text semantics to generate the target query. The target query, cross-modal image features, and text features are input into the cross-modal decoder. In the cross-modal decoder, the target query is progressively optimized by iteratively performing cross-modal attention interactions to model the relationships between queries and focus on relevant visual context. Based on the optimized target query, the bounding box and category prediction of the target described in the text prompt are output.

[0006] As a preferred embodiment of the cross-modal power plant main wiring blueprint primitive feature extraction and representation design method based on natural language description described in this invention, wherein: the acquisition of the power plant main wiring blueprint image and the text prompt describing the target to be detected include, Obtain the blueprint image of the main electrical wiring of the plant to be analyzed, along with text prompts describing specific graphic elements to be detected using open-ended terms.

[0007] By performing semantic parsing and feature extraction on the text prompts, a machine-understandable text representation is formed.

[0008] The blueprint image and text representation are used as input to a cross-modal open target detection system to initiate an end-to-end primitive detection process.

[0009] As a preferred embodiment of the cross-modal power plant main wiring blueprint primitive feature extraction and representation design method based on natural language description described in this invention, the step of extracting multi-scale visual features of the blueprint image using an image encoder and extracting text features of text prompts using a language encoder includes, A visual encoder is used to extract multi-scale visual features with different resolutions from the blueprint image.

[0010] The text prompts are subjected to deep semantic understanding using a language encoder to generate corresponding text feature representations.

[0011] The text prompt is divided and represented by structured semantic units.

[0012] As a preferred embodiment of the cross-modal power plant main wiring blueprint primitive feature extraction and representation design method based on natural language description described in this invention, the step of generating cross-modal image features by performing cross-modal interaction and fusion of multi-scale visual features and text features through a feature enhancement layer includes: In the feature enhancement layer, the multi-scale visual features and text features are guided to perform bidirectional cross-modal attention interaction.

[0013] The cross-modal attention interaction injects textual semantic information into visual features.

[0014] By leveraging visual contextual information, text features are optimized, and the resulting image features are enhanced through cross-modal alignment.

[0015] As a preferred embodiment of the cross-modal power plant main wiring blueprint primitive feature extraction and representation design method based on natural language description described in this invention, wherein: the execution language guides query initialization, dynamically selecting image region features related to text semantics to generate target queries, including, The correlation between the cross-modal image features and text features is calculated to evaluate the degree of association between each image region and the text semantics.

[0016] Based on the aforementioned correlation, key region features most relevant to the text prompt are dynamically selected from all image regions.

[0017] The selected key region features are used as a set of target queries for initialization.

[0018] The beneficial effects of this preferred technical solution are that by calculating the cross-modal feature correlation and dynamically selecting key region features based on this, the linguistic information can accurately guide the detection process; specifically, semantic correlation replaces the traditional random initialization or fixed anchor box mechanism, so that the target query focuses on the image region that is highly related to the text description from the beginning of decoding, thereby improving the model convergence speed and initial positioning accuracy.

[0019] As a preferred embodiment of the cross-modal power plant main wiring blueprint primitive feature extraction and representation design method based on natural language description described in this invention, the step of iteratively optimizing the target query through cross-modal attention interaction includes: In the cross-modal decoder, a self-attention mechanism is used to model the global relationships between target queries; By employing an image cross-attention mechanism, each target query can focus on extracting the most relevant visual contextual information from cross-modal image features.

[0020] By employing a text cross-attention mechanism, semantic features of the text are injected into the target query.

[0021] The beneficial effect of this preferred technical solution is that, through the collaborative optimization mechanism composed of self-attention, image cross-attention and text cross-attention, the target query can synchronously model the dependencies between queries, refine visual positioning information and enhance semantic alignment capabilities during the decoding process.

[0022] As a preferred embodiment of the cross-modal power plant main wiring blueprint primitive feature extraction and representation design method based on natural language description described in this invention, wherein: the step of outputting the bounding box and category prediction of the target described in the text prompt based on the optimized target query includes, The final target query, optimized by the cross-modal decoder, is fed into the prediction head network.

[0023] The prediction head decodes each target query into corresponding bounding box coordinate parameters, thus completing the localization of text targets.

[0024] Based on the semantic information fused in the query, a category label prediction corresponding to the text prompt is generated.

[0025] Output primitive detection results that match the input text description. Each result includes the bounding box of the text's location in the blueprint and the identified category.

[0026] The beneficial effects of this preferred technical solution are that by directly mapping the optimized query that deeply integrates visual and semantic information to the detection result, an end-to-end conversion from cross-modal understanding to structured output is achieved. This not only ensures that the final output bounding box has accurate spatial positioning, but also ensures that its category label is highly consistent with the semantics of the text prompt, thereby directly outputting complete, accurate primitive detection results that are completely matched with natural language instructions.

[0027] This invention provides a cross-modal power plant main wiring blueprint feature extraction and characterization design system based on natural language description.

[0028] To address the aforementioned technical problems, this invention provides the following technical solution: a cross-modal power plant main wiring blueprint feature extraction and characterization design system based on natural language description, comprising: a data input and preprocessing module, a dual-modal feature encoding module, a cross-modal fusion and enhancement module, a collaborative decoding and prediction module, and a result output and visualization module.

[0029] The data input and preprocessing module acquires the main wiring blueprint image of the plant and the text prompts describing the target to be detected.

[0030] The dual-modal feature encoding module uses an image encoder to extract multi-scale visual features of the blueprint image and a language encoder to extract text features of the text prompt.

[0031] Cross-modal image features are generated by performing cross-modal interaction and fusion of multi-scale visual and textual features through a feature enhancement layer.

[0032] The cross-modal fusion and enhancement module performs language-guided query initialization based on the similarity between cross-modal image features and text features, and dynamically selects image region features related to text semantics to generate target queries.

[0033] The collaborative decoding and prediction module inputs the target query, cross-modal image features, and text features into the cross-modal decoder.

[0034] In the cross-modal decoder, the target query is progressively optimized by iteratively executing cross-modal attention interactions to model the relationships between queries and focus on relevant visual context.

[0035] The result output and visualization module outputs the bounding box and category prediction of the target described in the text prompt based on the optimized target query.

[0036] The present invention provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the cross-modal power plant main wiring blueprint feature extraction and characterization design method based on natural language description.

[0037] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the cross-modal power plant main wiring blueprint feature extraction and characterization design method based on natural language description.

[0038] The beneficial effects of this invention are as follows: By successfully adapting the Transformer-based target detection model DINO detection framework to the field of power plant electrical main wiring blueprints, and introducing a multi-level cross-modal fusion mechanism and text representation, an end-to-end open vocabulary detection method is constructed. It can accurately locate and identify any specified graphic element in the blueprint directly based on natural language instructions, thus improving the flexibility of blueprint recognition technology. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the overall process of a cross-modal power plant main wiring blueprint feature extraction and characterization design method based on natural language description, provided as an embodiment of the present invention.

[0041] Figure 2 This is an overall framework diagram of a cross-modal power plant main wiring blueprint feature extraction and characterization design system based on natural language description, provided as an embodiment of the present invention. Detailed Implementation

[0042] To make the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0043] Example 1, referring to Figure 1 This is one embodiment of the present invention, which provides a method for feature extraction and characterization design of cross-modal power plant main wiring blueprint primitives based on natural language description, including: S1. Obtain the blueprint image of the main wiring diagram of the plant and the text prompt describing the target to be inspected.

[0044] S2. Use an image encoder to extract multi-scale visual features of the blueprint image and a language encoder to extract text features of the text prompt.

[0045] S3. Cross-modal interaction and fusion of multi-scale visual features and text features are performed through the feature enhancement layer to generate cross-modal image features.

[0046] S4. Based on the similarity between cross-modal image features and text features, perform language-guided query initialization and dynamically select image region features related to text semantics to generate target queries.

[0047] S5. Input the target query, cross-modal image features, and text features into the cross-modal decoder.

[0048] S6. In the cross-modal decoder, the target query is gradually optimized by iteratively executing cross-modal attention interactions to model the relationship between queries and focus on relevant visual context.

[0049] S7. Based on the optimized target query, output the bounding box and category prediction of the target described in the text prompt.

[0050] This invention achieves open-vocabulary detection of any element in the main wiring diagram of a plant by constructing an end-to-end cross-modal detection framework. Specifically, it involves injecting deep textual semantics into the entire process of visual feature extraction, query initialization, and decoding optimization through a multi-level cross-modal fusion mechanism, enabling accurate understanding of the semantic connotation of natural language instructions; utilizing language-guided query initialization and a triple attention collaborative optimization mechanism to improve the model's localization accuracy and robustness in identifying text-described targets in complex blueprint scenarios; and finally, through semantically aligned prediction output, it effectively overcomes the limitations of traditional methods that rely on closed category sets.

[0051] Example 2, an embodiment of the present invention, provides a method for feature extraction and characterization design of cross-modal power plant main wiring blueprint elements based on natural language description, based on the previous embodiment, including: The steps A1-A3 are as follows: Obtaining the main wiring blueprint image of the plant and the text prompts describing the target to be inspected in S1. A1. Obtain the blueprint image of the main electrical wiring of the plant to be analyzed, as well as text prompts describing the specific graphic elements to be detected in open-ended terms.

[0052] A2. By performing semantic analysis and feature extraction on text prompts, a machine-understandable text representation is formed.

[0053] A3. Use blueprint images and text representations as inputs to a cross-modal open target detection system to initiate an end-to-end primitive detection process.

[0054] In this embodiment, the open-ended vocabulary in A1, i.e., natural language form, is specifically achieved through multi-level semantic parsing and structuring of the input text prompts: First, the system uses a pre-trained semantic understanding module to perform word segmentation, part-of-speech tagging, and dependency parsing on the original text to identify core graph entities and their modification relationships; then, based on professional dictionaries and grammatical rules in the field of electrical engineering, the identified entities are subjected to semantic disambiguation and concept normalization to ensure the unified representation of synonyms such as "PT" and "voltage transformer"; finally, through a structured encoder based on an attention mechanism, the processed semantic units are constructed into a graph structure or sequential representation with hierarchical relationships, thereby transforming unstructured natural language input into machine-understandable, structured query conditions that retain complete semantic information, providing accurate language guidance for subsequent cross-modal alignment.

[0055] In one alternative implementation, the open-ended vocabulary can be a complete natural language sentence. This is achieved by directly receiving a complete descriptive statement (e.g., "all outgoing circuit breakers and voltage transformers in the location diagram") from the user input, and then using a language encoder to globally encode the entire sentence, capturing its overall semantic intent to guide the model in joint detection.

[0056] In another alternative implementation, open-ended vocabulary can also be composed of combined phrases consisting of independent semantic units. This is achieved as follows: First, the complex long sentence or phrase input by the user (e.g., “XX change #1 main change and 110kV I mother”) is broken down into multiple independent semantic clauses or keywords (e.g., broken down into “XX change”, “#1 main change”, and “110kVI mother”); then, each independent semantic unit is encoded separately or isolated through a masked attention mechanism to form a clause-level text representation; finally, based on these discrete semantic units, the model detects all corresponding primitive targets in the graph in parallel or sequentially.

[0057] In this embodiment, the language encoder in S2 is a text encoder. Specifically, the encoder first performs word segmentation and embedding processing on the input text, transforming each word into a dense vector; then, it calculates the global dependencies between words through a multi-layer self-attention mechanism, dynamically adjusting the contextual representation of each word; next, it performs non-linear transformation through a feedforward neural network, gradually refining feature representations with multi-layer semantic abstraction; finally, the output text feature vector retains the integrity of the original semantics and includes grammatical structure and logical relationship knowledge obtained through large-scale pre-training.

[0058] In one alternative implementation, the language encoder can be a pre-trained model based on the BERT architecture. Specifically, this is achieved by segmenting the input text (e.g., "#1 main variable") into words and adding special tags (e.g., [CLS] and [SEP]), then inputting it into a BERT model composed of multiple Transformer encoders; dynamically aggregating the semantic information of each word and its context through its self-attention mechanism, and finally taking the output vector corresponding to the [CLS] tag as the global semantic representation of the entire sentence, or taking the sequence of all output word vectors as the fine-grained semantic representation.

[0059] In another alternative implementation, the language encoder can also be a variant model based on clause-level encoding. Specifically, this is achieved as follows: First, the complex query text (e.g., “#1 main transformer and 110kV I bus in the XX transformer main wiring diagram”) is broken down into multiple independent semantic clauses (e.g., “XX transformer”, “#1 main transformer”, “110kV I bus”). Then, each clause is independently encoded by inputting it into a BERT base encoder with shared parameters, generating a set of independent clause-level feature vectors. Finally, this set of vectors together constitutes a distributed representation of the entire text prompt, thereby avoiding long-distance dependency interference and achieving accurate, decoupled semantic understanding of multi-target descriptions.

[0060] Furthermore, in S2, the extraction of multi-scale visual features from the blueprint image using an image encoder and the extraction of text features from the text prompt using a language encoder include steps B1-B3: B1. Use a visual encoder to extract multi-scale visual features with different resolutions from blueprint images.

[0061] Specifically, the image encoder (Swin Transformer) is used to extract multi-scale visual features from the main wiring blueprint of the plant.

[0062] By outputting electrical primitive feature maps at different resolutions through multi-scale features, it is used to detect primitive targets of different sizes.

[0063] Vanilla Image Features: The original image feature representation is designed and constructed as... .

[0064] in, For the number of image tokens, =256 is the feature dimension.

[0065] B2. Utilize a language encoder to perform deep semantic understanding of text prompts and generate corresponding text feature representations; Specifically, text features are extracted using a text encoder (BERT).

[0066] Text Image Features: The original text feature representation is designed and constructed as... .

[0067] in, The number of text tokens (maximum length is designed to be 256).

[0068] Clause-level text representation: The text is split into independent clauses (the clause “#1 main transformer 110kV I bus and disconnecting switch with grounding switch in the main wiring diagram of XX transformer” is split into “XX transformer”, “#1 main transformer”, “110kV I bus”, and “disconnecting switch (including grounding switch)”) to avoid cross-clause attention interference.

[0069] B3. Perform structured semantic unit division and representation of text prompts.

[0070] In the embodiments of this application, the structured semantic unit division and representation in B3 involves splitting the text into independent semantic clauses and encoding them. First, a sentence segmentation model based on grammar rules or deep learning is used to identify key boundaries representing semantic transformations in the text (such as specific punctuation marks, conjunctions, and phrase structures), dividing the coherent query statement into several clause units carrying independent semantics. Then, an isolation encoding strategy is adopted, using physically separated independent encoders or logical isolation methods using attention masks to ensure that each clause obtains a self-contained semantic representation in the feature space, while blocking semantic interference between different clauses. Finally, a generation is generated that maintains the semantic integrity within the clause.

[0071] In one optional implementation, the structured semantic unit partitioning and representation can be achieved through physical splitting and independent encoding using punctuation and conjunctions. Specifically, this is implemented as follows: First, the system physically splits the input long sentence (such as "#1 main transformer, 110kV I bus and disconnector with grounding switch in the XX transformer main wiring diagram") into multiple clause units ("XX transformer main wiring diagram", "#1 main transformer", "110kV I bus", "disconnector with grounding switch") at the string level based on commas, pauses, and conjunctions such as "and", "and"; then, each clause is treated as an independent text input and forward-propagated through a language encoder with shared weights to generate a set of clause-level feature vectors that are decoupled in semantic space.

[0072] In another alternative implementation, the structured semantic unit partitioning and representation can also be achieved through logical splitting and joint encoding using an attention masking mechanism. Specifically, this is achieved by maintaining the integrity of the input text sequence, but introducing a logical mask matrix in the self-attention computation of the language encoder. This mask matrix allows full interaction between words within a clause, but prohibits cross-clause attention computation between words in different clauses. Thus, in a single forward propagation process, it utilizes the complete context for word meaning disambiguation and achieves isolation of different semantic units at the representation level, ultimately outputting a set of semantically pure clause-level features.

[0073] Furthermore, step C1-C3 in S3 involves cross-modal interaction and fusion of multi-scale visual and textual features through a feature enhancement layer to generate cross-modal image features: C1. In the feature enhancement layer, guide multi-scale visual features and text features to perform bidirectional cross-modal attention interaction.

[0074] C2. Inject textual semantic information into visual features through cross-modal attention interaction.

[0075] C3. Optimize text features using visual context information and output image features that have been cross-modal aligned and enhanced.

[0076] Furthermore, the execution language in S4 guides query initialization, dynamically selecting image region features relevant to text semantics to generate the target query, including steps D1-D3: D1. Calculate the correlation between cross-modal image features and text features to evaluate the degree of association between each image region and text semantics.

[0077] Specifically, the first step is to calculate the features of the blueprint image. Text features The similarity matrix (via dot product) ).

[0078] Step 2: Take the maximum value for each row (corresponding to the blueprint image token) to obtain the overall relevance of each token to the text.

[0079] Step 3: Select the top results with the highest relevance. A blueprint image token is used as the initial query (e.g.) =900).

[0080] in, This represents a two-dimensional matrix, indicating the original correlation score between blueprint image features and text features. Represents the feature matrix of the blueprint image. This represents the transpose of the text feature matrix. This indicates that the maximum value is obtained along the last dimension of the two-dimensional matrix. Indicates the output result. This represents selecting the k highest-scoring values ​​and their corresponding position indices.

[0081] D2. Based on relevance, dynamically select the top K key region features most relevant to the text prompt from all image regions.

[0082] D3. Initialize the selected key region features as a set of target queries.

[0083] In this embodiment, the cross-modal attention interaction in S5, namely self-attention, image cross-attention, and text cross-attention, firstly establishes a global dependency between query vectors through the self-attention layer, enabling each query to perceive the spatial distribution and semantic association of other detected targets; then, through the image cross-attention layer, the query vector is used as the retrieval key to perform deformable sampling of cross-modal image features, focusing on the most relevant local visual context to refine spatial coordinates; finally, through the text cross-attention layer, the visually enhanced query is used as the retrieval key again to extract the most matching semantic description from the text features to enhance the category discrimination ability.

[0084] In one alternative implementation, cross-modal attention interaction can be a standardized three-layer attention structure executed sequentially. Specifically, this is achieved as follows: In each decoder layer, self-attention is first performed on the target query set, modeling the spatial and semantic dependencies among all detected targets by calculating the association weights between queries; then, image cross-attention is performed, using the query output from self-attention as the Query and the cross-modal image features as the Key and Value, dynamically sampling relevant image regions through a deformable attention mechanism to refine the visual localization information for each query; finally, text cross-attention is performed, again using the query output from image cross-attention as the Query and the text features as the Key and Value, injecting the most relevant text semantics into the query through global attention computation, thereby consolidating and enhancing its category attributes.

[0085] In another alternative implementation, cross-modal attention interaction can also be a structure that fuses image and text cross-attention after parallel computation. Specifically, this is achieved as follows: for the query output by the self-attention module, image cross-attention and text cross-attention operations are performed in parallel to obtain two sets of query vectors optimized by visual and semantic information, respectively; subsequently, through a learnable gating fusion module or a simple weighted summation method, these two sets of query vectors are integrated into an optimized query that simultaneously contains strong visual localization information and strong semantic discriminative information, and then fed into a feedforward network for further processing.

[0086] Furthermore, in S5, the target query is progressively optimized through iterative execution of cross-modal attention interactions, including steps E1-E3: E1. In the cross-modal decoder, the global relationship between each target query is modeled through a self-attention mechanism; Specifically, the feature enhancement layer includes Deformable Self-Attention and cross-modal attention (Image-to-Text / Text-to-Image).

[0087] Feature alignment is optimized by using contrastive loss and localization loss.

[0088] Deformable Self-Attention is used to enhance local details of blueprint image features (such as primitive edges).

[0089] Text-to-Image Cross-Attention is used to inject text features into image features, highlighting text-related regions (such as visual features corresponding to "circuit breaker").

[0090] Image-to-Text Cross-Attention is used to inversely optimize text features and enhance multimodal consistency.

[0091] Loss function: Feature alignment is optimized through contrastive loss and localization loss.

[0092] E2. Through the image cross-attention mechanism, each target query focuses on extracting the most relevant visual context information from cross-modal image features.

[0093] Specifically, Structure 1: Self-Attention: Modeling global relationships between queries (such as the positional association between "#1 main variable" and "110kV I mother").

[0094] Structure 2: Image Cross-Attention: Extracts visual information relevant to the query from image features.

[0095] Structure 3: Text Cross-Attention: Injects text features into the query to enhance semantic alignment (e.g., strengthens visual features related to "red").

[0096] Structure 4: Feedforward Network (FFN): Nonlinear transformation and feature enhancement.

[0097] The final output consists of a bounding box and a category prediction (e.g., "circuit breaker" or "voltage transformer") for each query. E3. By using a text cross-attention mechanism, textual features and semantics are injected into the target query.

[0098] Furthermore, in S6, based on the optimized target query, the output text prompt describing the target's bounding box and category prediction includes steps F1-F4: F1 feeds the final target query, optimized by the cross-modal decoder, into the prediction head network.

[0099] F2. Using the prediction head, each target query is decoded into the corresponding bounding box coordinate parameters to complete the localization of the text target.

[0100] F3. Based on the semantic information fused in the query, generate category label predictions corresponding to the text prompts.

[0101] F4. Output the primitive detection results that match the input text description. Each result includes the bounding box of the text's location in the blueprint and the identified category.

[0102] Example 3, referring to Figure 2 This embodiment of the present invention provides a cross-modal power plant main wiring blueprint feature extraction and characterization design system based on natural language description, including: a data input and preprocessing module, a dual-modal feature encoding module, a cross-modal fusion and enhancement module, a collaborative decoding and prediction module, and a result output and visualization module.

[0103] The data input and preprocessing module acquires the main wiring blueprint image of the plant and the text prompts describing the target to be inspected.

[0104] The dual-modal feature encoding module uses an image encoder to extract multi-scale visual features from the blueprint image and a language encoder to extract text features from the text prompts.

[0105] Cross-modal image features are generated by performing cross-modal interaction and fusion of multi-scale visual and textual features through a feature enhancement layer.

[0106] The cross-modal fusion and enhancement module performs language-guided query initialization based on the similarity between cross-modal image features and text features, and dynamically selects image region features that are related to text semantics to generate the target query.

[0107] The collaborative decoding and prediction module inputs the target query, cross-modal image features, and text features into the cross-modal decoder.

[0108] In the cross-modal decoder, the target query is progressively optimized by iteratively executing cross-modal attention interactions to model the relationships between queries and focus on relevant visual context.

[0109] The results output and visualization module outputs the bounding box and category prediction of the target described in the text prompt, based on the optimized target query.

[0110] This embodiment also provides an electronic device applicable to the method of extracting and representing the features of cross-modal plant main wiring blueprint elements based on natural language description, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the method of extracting and representing the features of cross-modal plant main wiring blueprint elements based on natural language description proposed in the above embodiment.

[0111] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the cross-modal power plant main wiring blueprint feature extraction and characterization design method based on natural language description proposed in the above embodiments.

[0112] The storage medium proposed in this embodiment and the method for extracting and characterizing cross-modal power plant main wiring blueprint features based on natural language description proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0113] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0114] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for feature extraction and representation design of cross-modal power plant main wiring blueprint primitives based on natural language description, characterized by: include, Obtain the main wiring blueprint image of the plant and the text prompts describing the target to be inspected; Multi-scale visual features of blueprint images are extracted using an image encoder, and text features of text prompts are extracted using a language encoder. Cross-modal interaction and fusion of multi-scale visual and textual features are performed through a feature enhancement layer to generate cross-modal image features; Based on the similarity between cross-modal image features and text features, language-guided query initialization is performed, and image region features related to text semantics are dynamically selected to generate the target query; Input the target query, cross-modal image features, and text features into the cross-modal decoder; In the cross-modal decoder, the target query is progressively optimized by iteratively executing cross-modal attention interactions to model the relationships between queries and focus on relevant visual context; Based on the optimized target query, the output text prompt describes the bounding box and category prediction of the target.

2. The method for feature extraction and representation design of cross-modal power plant main wiring blueprint primitives based on natural language description as described in claim 1, characterized in that: The acquisition of the main wiring blueprint image of the plant and the text prompts describing the target to be detected include, Obtain the blueprint image of the main electrical wiring of the plant to be analyzed, as well as text prompts describing specific graphic elements to be detected in open-ended terms; By performing semantic parsing and feature extraction on the text prompts, a machine-understandable text representation is formed; The blueprint image and text representation are used as input to a cross-modal open target detection system to initiate an end-to-end primitive detection process.

3. The method for feature extraction and representation design of cross-modal power plant main wiring blueprint primitives based on natural language description as described in claim 2, characterized in that: The process of extracting multi-scale visual features from the blueprint image using an image encoder and extracting text features from the text prompt using a language encoder includes: A visual encoder is used to extract multi-scale visual features with different resolutions from the blueprint image; The text prompt is subjected to deep semantic understanding using a language encoder to generate corresponding text feature representations. The text prompt is divided and represented by structured semantic units.

4. The method for feature extraction and representation design of cross-modal power plant main wiring blueprint primitives based on natural language description as described in claim 3, characterized in that: The method of performing cross-modal interaction and fusion of multi-scale visual and textual features through a feature enhancement layer to generate cross-modal image features includes: In the feature enhancement layer, the multi-scale visual features and text features are guided to perform bidirectional cross-modal attention interaction; Through the aforementioned cross-modal attention interaction, textual semantic information is injected into visual features; By leveraging visual contextual information, text features are optimized, and the resulting image features are enhanced through cross-modal alignment.

5. The method for feature extraction and representation design of cross-modal power plant main wiring blueprint primitives based on natural language description as described in claim 4, characterized in that: The execution language guides query initialization, dynamically selecting image region features related to text semantics to generate the target query, including: The correlation between the cross-modal image features and text features is calculated to evaluate the degree of association between each image region and the text semantics; Based on the aforementioned correlation, key region features most relevant to the text prompt are dynamically selected from all image regions; The selected key region features are used as a set of target queries for initialization.

6. The method for feature extraction and representation design of cross-modal power plant main wiring blueprint primitives based on natural language description as described in claim 4, characterized in that: The stepwise optimization of the target query through iterative execution of cross-modal attention interactions includes: In the cross-modal decoder, a self-attention mechanism is used to model the global relationships between target queries; Through the image cross-attention mechanism, each target query focuses on extracting the most relevant visual context information from cross-modal image features; By employing a text cross-attention mechanism, semantic features of the text are injected into the target query.

7. The method for feature extraction and representation design of cross-modal power plant main wiring blueprint primitives based on natural language description as described in claim 4, characterized in that: The step of outputting the bounding box and category prediction of the target in the text prompt based on the optimized target query includes, The final target query, optimized by the cross-modal decoder, is fed into the prediction head network; The prediction head decodes each target query into corresponding bounding box coordinate parameters, thus completing the localization of text targets; Based on the semantic information fused in the query, generate category label predictions corresponding to the text prompts; Output primitive detection results that match the input text description. Each result includes the bounding box of the text's location in the blueprint and the identified category.

8. A cross-modal power plant main wiring blueprint feature extraction and representation design system based on natural language description, employing the cross-modal power plant main wiring blueprint feature extraction and representation design method based on natural language description as described in any one of claims 1 to 7, characterized in that, include: The system includes a data input and preprocessing module, a bimodal feature encoding module, a cross-modal fusion and enhancement module, a collaborative decoding and prediction module, and a result output and visualization module. The data input and preprocessing module acquires the main wiring blueprint image of the plant and text prompts describing the target to be detected; The dual-modal feature encoding module uses an image encoder to extract multi-scale visual features of the blueprint image and a language encoder to extract text features of the text prompt. Cross-modal interaction and fusion of multi-scale visual and textual features are performed through a feature enhancement layer to generate cross-modal image features; The cross-modal fusion and enhancement module performs language-guided query initialization based on the similarity between cross-modal image features and text features, and dynamically selects image region features related to text semantics to generate target queries; The collaborative decoding and prediction module inputs the target query, cross-modal image features, and text features into the cross-modal decoder; In the cross-modal decoder, the target query is progressively optimized by iteratively executing cross-modal attention interactions to model the relationships between queries and focus on relevant visual context; The result output and visualization module outputs the bounding box and category prediction of the target described in the text prompt based on the optimized target query.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the cross-modal power plant main wiring blueprint feature extraction and characterization design method based on natural language description as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the cross-modal power plant main wiring blueprint feature extraction and characterization design method based on natural language description as described in any one of claims 1 to 7.

Images