Construction method and equipment of smart city knowledge graph, and medium

Abstract

The invention discloses a smart city knowledge graph construction method and device and a medium, and relates to the technical field of knowledge graphs. The method comprises the following steps: acquiring multi-modal data in a smart city; performing feature extraction on the image by using a visual encoder to generate an image feature sequence, and performing feature extraction on the text by using a language encoder to generate a text feature sequence; inputting to a feature fusion layer, and performing bidirectional interaction through a cross-modal attention mechanism to generate joint representation; obtaining a preset natural language instruction template, combining the joint representation and the natural language instruction template into a prompt, and inputting the prompt into the multi-modal large model after instruction fine tuning to generate structured entity relationship knowledge; and analyzing the structured entity relationship knowledge, converting the structured entity relationship knowledge into a knowledge triple, and performing entity linking, conflict detection and resolution with the knowledge graph to obtain a new knowledge graph. According to the method, accurate, efficient and continuous dynamic evolution of the knowledge graph is realized.

Description

Technical Field

[0001] This application relates to the field of knowledge graph technology, and in particular to a method, device and medium for constructing a smart city knowledge graph. Background Technology

[0002] In constructing knowledge graphs for smart cities, traditional methods typically employ a pipeline approach, processing multimodal data such as images and text independently before merging them at the knowledge level. This method separates visual feature extraction from textual semantic analysis, making it difficult to establish fine-grained connections between pixel-level image information and word-level textual semantics. Consequently, spatial relationships in images and event descriptions in text are often processed within their respective modalities, failing to effectively capture deep semantic correspondences across modalities.

[0003] Furthermore, this phased processing approach is prone to error accumulation across multiple independent subtasks and heavily relies on manually designed rules and feature engineering. As urban data continues to expand and dynamically change, traditional methods face significant bottlenecks in terms of accuracy, automation, and adaptability, making it difficult to support efficient and accurate knowledge extraction and graph evolution from multi-source heterogeneous data.

[0004] Based on the above analysis, the problems and shortcomings of the existing technology are as follows: Traditional knowledge graph construction methods in the present technology suffer from semantic loss, error accumulation, and low automation due to the isolated processing of multimodal data. Summary of the Invention

[0005] This application provides a method, device, and medium for constructing a smart city knowledge graph, which can solve the problems of semantic association loss, error accumulation, and low automation caused by the isolated processing of multimodal data in traditional knowledge graph construction methods in the prior art.

[0006] In a first aspect, embodiments of this application provide a method for constructing a smart city knowledge graph. The method includes: acquiring multimodal data from a smart city, the multimodal data including images and text; using a visual encoder to extract features from the images to generate image feature sequences, and using a language encoder to extract features from the text to generate text feature sequences; inputting the image feature sequences and text feature sequences into a feature fusion layer, and performing bidirectional interaction through a cross-modal attention mechanism to generate a joint representation; acquiring a preset natural language instruction template, combining the joint representation with the natural language instruction template to form a prompt, and inputting it into a finely tuned multimodal large model to generate structured entity relationship knowledge; parsing the structured entity relationship knowledge, converting it into knowledge triples, and performing entity linking, conflict detection, and resolution with the knowledge graph to obtain a new knowledge graph.

[0007] In one implementation of this application, a visual encoder is used to extract features from image data to generate an image feature sequence, and a language encoder is used to extract features from text data to generate a text feature sequence. Specifically, this includes: using a visual encoder to segment the image into image blocks and convert them into a feature vector sequence, adding positional encoding to obtain an image feature sequence; using a language encoder to perform word segmentation on the text to obtain a text feature sequence; and processing the image feature sequence and text feature sequence through a multi-layer Transformer encoder to capture long-range dependencies and contextual semantic information.

[0008] In one implementation of this application, image feature sequences and text feature sequences are input into a feature fusion layer, and bidirectional interaction is performed through a cross-modal attention mechanism to generate a joint representation. Specifically, this includes: concatenating the image feature sequences and text feature sequences along the sequence dimension to form a multimodal feature sequence; enabling bidirectional interaction between the image feature sequences and text feature sequences through multi-head cross-attention calculation; introducing a contrastive learning strategy to construct positive and negative sample pairs, and optimizing the contrastive loss function to make positive samples closer in the joint representation space and negative samples farther apart in the joint representation space.

[0009] In one implementation of this application, before obtaining a preset natural language instruction template, combining the joint representation with the instruction template into a prompt input, and inputting it into the multimodal large model after instruction fine-tuning to generate structured entity relationship knowledge, the method further includes: constructing a training dataset, which includes multimodal input, natural language instructions, and structured output; and using the training dataset to perform supervised fine-tuning of the multimodal large model.

[0010] In one implementation of this application, structured entity relationship knowledge is parsed, converted into knowledge triples, and then linked with a knowledge graph for entity linking, conflict detection, and resolution to obtain a new knowledge graph. Specifically, this includes: parsing the structured entity relationship knowledge into a JSON format string; extracting an entity list and a relationship list from the JSON format string; and organizing the entity list and relationship list into knowledge triples, where each knowledge triple includes a subject, a relationship, and an object.

[0011] In one implementation of this application, the method further includes: calculating the similarity between the newly extracted entity and existing entities in the knowledge graph in terms of name string, attribute features, and contextual dimensions; determining whether the newly extracted entity is an existing entity or a new entity that needs to be created based on a preset threshold; establishing association links for existing entities, and creating unique identifiers and incorporating new entities into the knowledge graph.

[0012] In one implementation of this application, the method further includes: detecting attribute value contradictions, relational logic conflicts, and spatiotemporal consistency conflicts in knowledge triples; resolving conflicts using a priority-based multi-strategy fusion mechanism; determining the final knowledge graph based on the resolution results; and ensuring data integrity through transactional operations.

[0013] In one implementation of this application, after acquiring multimodal data from a smart city, which includes images and text, the method further includes: unifying the resolution, adjusting the size, and converting the format of the images to generate standardized images; and removing noise, filtering irrelevant symbols, and standardizing the encoding of the text to generate standardized text.

[0014] Secondly, embodiments of this application also provide a device for constructing a smart city knowledge graph. The device includes at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to: acquire multimodal data in a smart city, including images and text; extract features from images using a visual encoder to generate image feature sequences, and extract features from text using a language encoder to generate text feature sequences; input the image feature sequences and text feature sequences into a feature fusion layer, and perform bidirectional interaction through a cross-modal attention mechanism to generate a joint representation; acquire a preset natural language instruction template, combine the joint representation with the natural language instruction template to form a prompt, and input it into a fine-tuned multimodal large model to generate structured entity relationship knowledge; parse the structured entity relationship knowledge, convert it into knowledge triples, and perform entity linking, conflict detection, and resolution with the knowledge graph to obtain a new knowledge graph.

[0015] Thirdly, embodiments of this application also provide a non-volatile computer storage medium for constructing a smart city knowledge graph, storing computer-executable instructions. These computer-executable instructions are configured to: acquire multimodal data from the smart city, including images and text; extract features from the images using a visual encoder to generate image feature sequences, and extract features from the text using a language encoder to generate text feature sequences; input the image feature sequences and text feature sequences into a feature fusion layer, and perform bidirectional interaction through a cross-modal attention mechanism to generate a joint representation; acquire a preset natural language instruction template, combine the joint representation with the natural language instruction template to form a prompt, and input it into the fine-tuned multimodal large model to generate structured entity relationship knowledge; parse the structured entity relationship knowledge, convert it into knowledge triples, and perform entity linking, conflict detection, and resolution with the knowledge graph to obtain a new knowledge graph.

[0016] This application provides a method, device, and medium for constructing a smart city knowledge graph. By introducing a cross-modal attention mechanism in the multimodal representation layer for early deep fusion, fine-grained semantic alignment between image pixels and text words is achieved, avoiding information loss during later fusion from the source. Instruction fine-tuning technology is used to adapt the large multimodal model into an end-to-end structured knowledge generator, replacing multiple extraction models sequentially in the traditional process, significantly improving the automation level and accuracy of the construction process. Furthermore, a priority-based multi-strategy conflict resolution mechanism ensures the quality and consistency of the fusion of new knowledge with the existing knowledge graph, thereby achieving accurate, efficient, and continuously dynamic evolution of the knowledge graph, providing deep cognitive support for smart city applications. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a method for constructing a smart city knowledge graph, as provided in an embodiment of this application; Figure 2 This is a schematic diagram of the internal structure of a smart city knowledge graph provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] This application provides a method, device, and medium for constructing a smart city knowledge graph, which solves the problems of semantic association loss, error accumulation, and low automation caused by the isolated processing of multimodal data in traditional knowledge graph construction methods in the prior art.

[0020] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0021] Figure 1 This is a flowchart illustrating a method for constructing a smart city knowledge graph, as provided in an embodiment of this application. Figure 1 As shown in the figure, the method for constructing a smart city knowledge graph provided in this application embodiment specifically includes the following steps: Step 10: Acquire multimodal data from the smart city, including images and text; First, it is understandable that the input layer is the data entry point for the entire system, responsible for receiving and processing diverse raw multimodal data from the smart city environment. The main sources include: visual data: real-time video streams from traffic cameras, satellite remote sensing images, drone aerial photos, etc., which mainly reflect the physical state and spatial information of urban operation; and text data: government documents and reports, public opinions and event descriptions on social media, weather forecasts issued by meteorological departments, etc., which mainly carry semantic information such as event descriptions, policies and regulations, and public perception.

[0022] As an optional embodiment, after acquiring multimodal data from a smart city, which includes images and text, the method may further include: step 101: performing resolution unification, size adjustment, and format conversion on the images to generate standardized images; step 102: performing noise removal, irrelevant symbol filtering, and encoding standardization on the text to generate standardized text.

[0023] In this step, to facilitate subsequent model training, the input layer needs to perform preliminary processing and standardization of the raw data. This includes unifying the resolution, adjusting the size, and converting the format of image data, and removing noise, filtering irrelevant symbols, and standardizing the encoding of text data. Simultaneously, this layer is also responsible for extracting and recording key metadata, such as data source, acquisition time, and spatial location, to support subsequent correlation and alignment between multimodal data. Through data preprocessing, the input layer transforms the raw heterogeneous data into standardized data with a unified format and controllable quality, providing standardized data input for subsequent feature encoding and cross-modal fusion modules.

[0024] Step 20: Use a visual encoder to extract features from the image and generate an image feature sequence; use a language encoder to extract features from the text and generate a text feature sequence. As an optional embodiment, a visual encoder is used to extract features from image data to generate an image feature sequence, and a language encoder is used to extract features from text data to generate a text feature sequence. Specifically, this may include: Step 201: Using a visual encoder, the image is segmented into image blocks and converted into a feature vector sequence, and positional encoding is added to obtain an image feature sequence; Step 202: Using a language encoder, the text is segmented to obtain a text feature sequence; Step 203: The image feature sequence and the text feature sequence are processed by a multi-layer Transformer encoder to capture long-range dependencies and contextual semantic information.

[0025] In this step, the feature encoding layer is responsible for converting the preprocessed, standardized data from the input layer into a deep feature representation rich in semantic information. This layer employs a parallel and independent dual-channel architecture to efficiently extract features for both visual and textual modalities. For image data, the system uses a Transformer-based visual encoder. The input image is first segmented into fixed-size image patches. Each patch is linearly projected and converted into a feature vector, with learnable positional encoding added to preserve spatial information. These processed image patch sequences are then input into a multi-layer Transformer encoder, which uses a self-attention mechanism to capture long-range dependencies between image patches, ultimately outputting a high-dimensional image feature sequence containing global contextual information.

[0026] For text data, the system uses the language model BERT (Bidirectional Encoder Representations from Transformers) for encoding. The input text is first processed by word segmentation, which is converted into a sequence of lexical units and special CLS (Classification Symbol) and SEP (Separator) tags are added. These tagged sequences are then input into BERT's multi-layer Transformer encoder, which deeply mines the semantic relationships and contextual information between words through a bidirectional attention mechanism, and finally generates a text feature sequence rich in semantic information.

[0027] The output of this layer is an image feature sequence that retains rich visual spatial information and a text feature sequence that contains deep semantic information. While maintaining their respective modal characteristics, these feature sequences provide a good feature foundation for subsequent cross-modal fusion.

[0028] Step 30: Input the image feature sequence and text feature sequence into the feature fusion layer, and generate a joint representation through bidirectional interaction via a cross-modal attention mechanism; As an optional embodiment, image feature sequences and text feature sequences are input into a feature fusion layer, and bidirectional interaction is performed through a cross-modal attention mechanism to generate a joint representation. Specifically, this may include: Step 301: concatenating the image feature sequence and the text feature sequence in the sequence dimension to form a multimodal feature sequence; Step 302: enabling bidirectional interaction between the image feature sequence and the text feature sequence through multi-head cross-attention calculation; Step 303: introducing a contrastive learning strategy to construct positive and negative sample pairs, and optimizing the contrastive loss function to make positive samples closer in the joint representation space and negative samples farther apart in the joint representation space.

[0029] In this step, the design goal of the feature fusion layer is to break down the semantic barriers between visual and textual modalities, achieving an organic unification of information from both modalities at the deep representation level. This layer receives image feature sequences and text feature sequences from the feature encoding layer as input, and outputs a deeply fused joint representation through a fusion mechanism based on the Transformer cross-modal attention mechanism. Specifically, the system first concatenates the image feature sequences and text feature sequences along the sequence dimension to form a multimodal feature sequence; then, it inputs this sequence into a cross-modal Transformer encoder for deep interaction. Within this encoder, multi-head cross-attention computation achieves bidirectional interaction between text features and image features, as well as between image features and text features. In text-to-image attention, each text word calculates its association weight with all image regions, thus learning to focus on the most relevant visual regions. Similarly, in image-to-text attention, each image region searches for the most matching text descriptive words. This refined cross-attention mechanism can establish fine-grained semantic correspondences between pixel-level and word-level semantics.

[0030] To further enhance the fusion effect, this layer introduces a contrastive learning strategy. The system constructs positive and negative sample pairs: positive sample pairs consist of semantically matched images and text, while negative sample pairs consist of randomly combined unrelated images and text. By optimizing the InfoNCE loss function, the distance between positive sample pairs in the joint representation space becomes closer, while the distance between negative sample pairs is pushed further apart. This process not only enhances the model's discriminative ability, but more importantly, drives the entire network to learn a semantically highly aligned cross-modal representation space.

[0031] After multiple layers of cross-attention computation and contrastive learning optimization, this layer finally outputs a unified joint representation. This joint representation not only retains the feature information of the original modality, but more importantly, it contains rich cross-modal interaction context, providing a solid and rich feature foundation for downstream knowledge extraction tasks. This early deep fusion method avoids the information loss problem of traditional late fusion and realizes multimodal semantic understanding.

[0032] Step 40: Obtain the preset natural language instruction template, combine the joint representation with the natural language instruction template into a prompt, and input it into the multimodal large model after instruction fine-tuning to generate structured entity relationship knowledge; In this step, the task of the knowledge generation layer is to directly generate structured knowledge from the fused joint representation. This layer abandons the complex pipeline of traditionally serializing multiple discrete models and adopts an end-to-end generative paradigm. Specifically, LLaVA is selected as the core generation engine, which is a multimodal large model capable of simultaneously understanding visual and textual information and responding through natural language. The implementation of this layer includes two key stages: instruction fine-tuning in the training stage and structured generation in the inference stage.

[0033] As an optional embodiment, before obtaining a preset natural language instruction template, combining the joint representation with the instruction template into a prompt input, and inputting it into the multimodal large model after instruction fine-tuning to generate structured entity relationship knowledge, the method may further include: Step 01: Constructing a training dataset, which includes multimodal input, natural language instructions, and structured output; Step 02: Using the training dataset to perform supervised fine-tuning of the multimodal large model.

[0034] In this step, during the instruction fine-tuning phase, specific training data is constructed to optimize the LLaVA model. The training samples are in a triplet format: Input: Joint representations from the fusion layer; Instructions: Natural language prompts, such as "Please analyze the following city multimodal data, identify all entities and their relationships, and output in strict JSON format"; Output: A structured JSON string conforming to the instructions, containing a list of entities and a list of relationships. Through supervised fine-tuning, LLaVA learns how to understand and execute complex instructions for structured knowledge extraction based on multimodal inputs.

[0035] During the inference and generation phase, the system combines the fused joint representation with a pre-set instruction template to form a complete prompt input into the fine-tuned LLaVA model. Based on its powerful multimodal understanding and generation capabilities, the model directly outputs a well-formatted JSON string, which fully contains the identified entity information and the relationship information between entities.

[0036] Finally, using a lightweight JSON parser, the system parses the JSON string output by the model into standard knowledge triples: subject, relation, and object, completing the end-to-end transformation from raw multimodal data to structured knowledge, achieving higher accuracy and better generalization performance.

[0037] Step 50: Parse the structured entity relationship knowledge, convert it into knowledge triples, and link them with the knowledge graph through entity linking, conflict detection and resolution to obtain a new knowledge graph.

[0038] In this step, the knowledge fusion layer is a crucial link in ensuring the continuous evolution of the knowledge graph. This layer employs a multi-step processing flow to guarantee the quality and reliability of knowledge fusion. In the entity linking stage, the system uses a deep matching algorithm based on similarity calculation. For each newly extracted entity, the system calculates its similarity to existing entities in the knowledge graph across multiple dimensions: semantic similarity of name strings, matching degree of attribute features, and relevance to the context. By setting adaptive thresholds, the system can accurately determine whether a new entity needs a unique identifier or an existing entity needs to establish a link. In the conflict detection and resolution stage, the system performs in-depth knowledge consistency verification. Conflict detection includes not only direct contradictions in attribute values ​​but also logical conflicts in relationships and spatiotemporal consistency conflicts. The system employs a multi-strategy fusion resolution mechanism: first, it considers the authority weight of the data source; second, it compares the newness of the timestamps; and finally, it refers to the confidence score output by the model. In the graph update stage, the system uses transactional operations to ensure data integrity. All verified new knowledge triples are batch-processed and optimized before being written to the knowledge base through an efficient graph database operation interface.

[0039] As an optional implementation, the structured entity relationship knowledge is parsed, converted into knowledge triples, and then linked with the knowledge graph for entity linking, conflict detection, and resolution to obtain a new knowledge graph. Specifically, this may include: Step 501: Parsing the structured entity relationship knowledge into a JSON format string; Step 502: Extracting the entity list and relationship list from the JSON format string; Step 503: Organizing the entity list and relationship list into knowledge triples, where each knowledge triple includes a subject, a relationship, and an object.

[0040] In this step, the structured entity relationship knowledge output by the multimodal large model is parsed into a JSON format string. Each element in the entity list contains a unique entity ID, entity name, entity type, and attribute key-value pairs. Each element in the relationship list contains a relationship ID, subject entity ID, object entity ID, relationship type, and related attributes. Triples can include attribute information such as timestamps and confidence levels.

[0041] As an optional embodiment, the method may further include: Step 504: Calculate the similarity between the newly extracted entity and the existing entities in the knowledge graph in terms of name string, attribute features, and contextual dimensions; Step 505: Determine whether the newly extracted entity is an existing entity or a new entity that needs to be created based on a preset threshold; Step 506: For existing entities, establish association links; for new entities, create a unique identifier and incorporate it into the knowledge graph.

[0042] As an optional embodiment, the method may further include: step 507: detecting attribute value contradictions, relational logic conflicts, and spatiotemporal consistency conflicts in knowledge triples; step 508: resolving conflicts using a priority-based multi-strategy fusion mechanism; step 509: determining the final knowledge graph based on the resolution results, and ensuring data integrity through transactional operations.

[0043] In this step, conflicts arise due to several factors: attribute value contradictions (different values ​​for the same attribute within the same entity); relational logic conflicts (logical contradictions between relationships between entities and existing relationships); and spatiotemporal consistency conflicts (inconsistencies between the temporal and spatial information of entities or relationships and existing records). A priority-based multi-strategy fusion mechanism is employed to resolve these conflicts. Priority rules may include: data source authority weight (official data takes precedence over ordinary sensor data); timestamp freshness (newer data takes precedence over older data); and model output confidence. Based on the conflict resolution results, the knowledge graph is updated, and transactional operations are used to ensure data integrity. This means that updates to all relevant triples either succeed entirely or fail entirely and are rolled back, preventing data inconsistencies.

[0044] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a device for constructing a smart city knowledge graph, the structure of which is as follows: Figure 2 As shown.

[0045] Figure 2 This is a schematic diagram of the internal structure of a device for constructing a smart city knowledge graph, provided as an embodiment of this application. Figure 2 As shown, the device includes: At least one processor 201; And a memory 202 that is communicatively connected to at least one processor; The memory 202 stores instructions executable by at least one processor 201, which enables the processor 201 to: acquire multimodal data from a smart city, including images and text; extract features from the images using a visual encoder to generate image feature sequences, and extract features from the text using a language encoder to generate text feature sequences; input the image feature sequences and text feature sequences into a feature fusion layer, and generate a joint representation through bidirectional interaction via a cross-modal attention mechanism; acquire a preset natural language instruction template, combine the joint representation with the natural language instruction template to form a prompt, and input it into the multimodal large model after instruction fine-tuning to generate structured entity relationship knowledge; parse the structured entity relationship knowledge, convert it into knowledge triples, and perform entity linking, conflict detection, and resolution with the knowledge graph to obtain a new knowledge graph.

[0046] Some embodiments of this application provide corresponding to Figure 1A non-volatile computer storage medium for constructing a smart city knowledge graph stores computer-executable instructions. These instructions are configured to: acquire multimodal data from the smart city, including images and text; extract features from the images using a visual encoder to generate image feature sequences, and extract features from the text using a language encoder to generate text feature sequences; input the image and text feature sequences into a feature fusion layer, and generate a joint representation through bidirectional interaction via a cross-modal attention mechanism; acquire a preset natural language instruction template, combine the joint representation with the natural language instruction template to form a prompt, and input it into a fine-tuned multimodal large model to generate structured entity relationship knowledge; parse the structured entity relationship knowledge, convert it into knowledge triples, and perform entity linking, conflict detection, and resolution with the knowledge graph to obtain a new knowledge graph.

[0047] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0048] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0049] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0050] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0051] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0052] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0053] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0054] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0055] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0056] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0057] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for constructing a smart city knowledge graph, characterized in that, The method includes: Acquire multimodal data from smart cities, including images and text; The image is used to extract features using a visual encoder to generate an image feature sequence, and the text is used to extract features using a language encoder to generate a text feature sequence. The image feature sequence and text feature sequence are input into the feature fusion layer, and bidirectional interaction is performed through a cross-modal attention mechanism to generate a joint representation; Obtain a preset natural language instruction template, combine the joint representation with the natural language instruction template to form a prompt, and input it into the multimodal large model after instruction fine-tuning to generate structured entity relationship knowledge; The structured entity relationship knowledge is parsed, converted into knowledge triples, and then linked with the knowledge graph for entity linking, conflict detection, and resolution to obtain a new knowledge graph.

2. The method for constructing a smart city knowledge graph according to claim 1, characterized in that, The step of using a visual encoder to extract features from the image data and generate an image feature sequence, and using a language encoder to extract features from the text data and generate a text feature sequence, specifically includes: The image is segmented into image patches using a visual encoder and converted into a feature vector sequence. Position encoding is then added to obtain the image feature sequence. The text is segmented using a language encoder to obtain the text feature sequence; The image feature sequence and text feature sequence are processed by a multi-layer Transformer encoder to capture long-range dependencies and contextual semantic information.

3. The method for constructing a smart city knowledge graph according to claim 1, characterized in that, The image feature sequence and text feature sequence are input into the feature fusion layer, and bidirectional interaction is performed through a cross-modal attention mechanism to generate a joint representation, specifically including: The image feature sequence and the text feature sequence are concatenated along the sequence dimension to form a multimodal feature sequence; Multi-head cross-attention computation enables bidirectional interaction between image feature sequences and text feature sequences; A contrastive learning strategy is introduced to construct positive and negative sample pairs. By optimizing the contrastive loss function, positive samples are made closer together in the space of the joint representation, while negative samples are made farther apart in the space of the joint representation.

4. The method for constructing a smart city knowledge graph according to claim 1, characterized in that, Before obtaining a preset natural language instruction template, combining the joint representation with the instruction template to form a prompt input, and inputting it into the fine-tuned multimodal large model to generate structured entity relationship knowledge, the method further includes: Construct a training dataset, which includes multimodal inputs, natural language instructions, and structured outputs; The multimodal large model is then fine-tuned using the training dataset.

5. The method for constructing a smart city knowledge graph according to claim 1, characterized in that, The process of parsing the structured entity relationship knowledge, converting it into knowledge triples, and then linking it with the knowledge graph through entity linking, conflict detection, and resolution to obtain a new knowledge graph specifically includes: The structured entity relationship knowledge is parsed into a JSON format string; Extract the entity list and relationship list from the JSON format string; The entity list and relation list are organized into knowledge triples, which include a subject, a relation, and an object.

6. The method for constructing a smart city knowledge graph according to claim 5, characterized in that, The method further includes: Calculate the similarity between the newly extracted entities and existing entities in the knowledge graph in terms of name strings, attribute features, and contextual dimensions; Based on a preset threshold, determine whether the newly extracted entity is an existing entity or a new entity that needs to be created; For the existing entities, establish associated links; for the new entities, create unique identifiers and incorporate them into the knowledge graph.

7. The method for constructing a smart city knowledge graph according to claim 1, characterized in that, The method further includes: Detect attribute value contradictions, relational logic conflicts, and spatiotemporal consistency conflicts in the knowledge triples; A priority-based multi-strategy fusion mechanism is used for conflict resolution. The final knowledge graph is determined based on the resolution results, and data integrity is ensured through transactional operations.

8. The method for constructing a smart city knowledge graph according to claim 1, characterized in that, After acquiring multimodal data from a smart city, including images and text, the method further includes: The image is then subjected to resolution unification, size adjustment, and format conversion to generate a standardized image; The text is subjected to noise removal, irrelevant symbol filtering, and encoding standardization to generate standardized text.

9. A device for constructing a smart city knowledge graph, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Acquire multimodal data from smart cities, including images and text; The image is used to extract features using a visual encoder to generate an image feature sequence, and the text is used to extract features using a language encoder to generate a text feature sequence. The image feature sequence and text feature sequence are input into the feature fusion layer, and bidirectional interaction is performed through a cross-modal attention mechanism to generate a joint representation; Obtain a preset natural language instruction template, combine the joint representation with the natural language instruction template to form a prompt, and input it into the multimodal large model after instruction fine-tuning to generate structured entity relationship knowledge; The structured entity relationship knowledge is parsed, converted into knowledge triples, and then linked with the knowledge graph for entity linking, conflict detection, and resolution to obtain a new knowledge graph.

10. A non-volatile computer storage medium for constructing a smart city knowledge graph, storing computer-executable instructions, characterized in that, The computer-executable instructions are set as follows: Acquire multimodal data from smart cities, including images and text; The image is used to extract features using a visual encoder to generate an image feature sequence, and the text is used to extract features using a language encoder to generate a text feature sequence. The image feature sequence and text feature sequence are input into the feature fusion layer, and bidirectional interaction is performed through a cross-modal attention mechanism to generate a joint representation; Obtain a preset natural language instruction template, combine the joint representation with the natural language instruction template to form a prompt, and input it into the multimodal large model after instruction fine-tuning to generate structured entity relationship knowledge; The structured entity relationship knowledge is parsed, converted into knowledge triples, and then linked with the knowledge graph for entity linking, conflict detection, and resolution to obtain a new knowledge graph.

Images