Large model fine-tuning method, system and smart city anomaly detection method

Abstract

The application relates to the technical field of artificial intelligence, in particular to a large model fine-tuning method and system and a smart city anomaly detection method, which comprises the following steps: constructing a domain knowledge graph based on domain standard knowledge data of a target city, performing knowledge representation learning and vector coding processing on the domain knowledge graph, and constructing a domain knowledge vector space; performing feature coding on pre-collected city multi-modal data, obtaining corresponding city multi-modal features, and performing linear projection on the city multi-modal features to map the city multi-modal features to the domain knowledge vector space, thereby obtaining multi-modal semantic fusion features that realize the same-space semantic alignment expression of multi-modal data and domain knowledge; and inputting the multi-modal semantic fusion features into an encoder of a pre-trained general large model to perform parameter fine-tuning and domain knowledge embedding on the general large model, thereby obtaining a city large model applied to a smart city. The application improves the application capability of the model to domain knowledge, thereby improving the decision reliability.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to large model fine-tuning methods, systems, and anomaly detection methods for smart cities. Background Technology

[0002] While current high-efficiency parameter fine-tuning techniques can reduce the training cost of adapting general large-scale models to specific domains, traditional fine-tuning methods heavily rely on scenario-based data-driven learning. They lack in-depth guidance from structured domain knowledge specific to urban infrastructure, making it difficult to understand professional content such as urban equipment mechanisms, facility association rules, and operation and maintenance business logic. When faced with complex and heterogeneous urban multimodal data, their cross-modal semantic fusion capabilities are weak, easily leading to problems such as insufficient generalization, uninterpretable inference results, and limited anomaly identification accuracy. This makes them unsuitable for the high-precision, high-reliability management and control application requirements of smart cities. In contrast, domain knowledge graphs, as a means to systematically carry urban entity elements, relationships, and industry standard rules, can provide structured prior support for adapting large-scale models to vertical scenarios, effectively compensating for the insufficient logical reasoning capabilities of purely data-driven models. For example, patent CN120317343A (classification number G06N) provides a method for constructing a knowledge graph in the field of science and technology innovation based on a large model; patent CN120745771A (classification number G06N) provides a method for processing professional knowledge domain data based on a large language model; and patent CN120338067A (classification number G06N) provides a method for optimizing a knowledge graph based on the fusion of a large model and multimodal data.

[0003] However, most current methods of integrating knowledge graphs with large models are rather superficial, generally adopting a plug-in approach of text splicing and external retrieval calls. They only supplement knowledge information during the model input or inference stages, failing to perform unified vectorized encoding and representation learning of knowledge graphs, and making it difficult to construct a standardized, unified-dimensional domain knowledge vector space. Summary of the Invention

[0004] (a) Technical problems to be solved In view of the above-mentioned shortcomings and deficiencies of the existing technology, this application provides a large model fine-tuning method, system and smart city anomaly detection method. It solves the problem that the knowledge graph and the large model are combined in an external way, and knowledge information is only supplemented in the model input or inference stage. This makes it impossible to perform unified vectorization encoding and representation learning of the knowledge graph, and it is difficult to build a standardized and unified domain knowledge vector space. As a result, domain knowledge cannot be deeply integrated into the optimization process of feature encoding and hidden layer representation of the large model. Consequently, the large model has weak cross-modal understanding ability, uninterpretable inference results and limited anomaly recognition accuracy when facing the complex scenario of smart cities.

[0005] (II) Technical Solution To achieve the above objectives, the main technical solutions adopted in this application include: In a first aspect, embodiments of this application provide a method for fine-tuning a large model, the method being applied in the field of smart cities, the method comprising: S1. Based on the domain standard knowledge data of the target city input by the user, construct a domain knowledge graph, and input the domain knowledge graph into a pre-set TransE model to perform knowledge representation learning and vector encoding processing on the domain knowledge graph to construct a domain knowledge vector space; wherein, the domain knowledge vector space is a low-dimensional dense feature space composed of entity vectors and relation vectors of the same dimension; S2. Feature encoding is performed on the pre-collected urban multimodal data to obtain the urban multimodal features corresponding to the urban multimodal data. Linear projection is then performed on the urban multimodal features to map them to the domain knowledge vector space, thereby obtaining multimodal semantic fusion features that achieve co-spatial semantic alignment between multimodal data and domain knowledge. Here, the urban multimodal data consists of multi-type heterogeneous data sources collected in urban scenarios. S3. Input the multimodal semantic fusion features into the encoder of the pre-trained general large model to fine-tune the parameters and embed domain knowledge into the general large model to obtain a city large model applicable to smart cities.

[0006] Optionally, based on the domain standard knowledge data of the target city input by the user, a domain knowledge graph is constructed, including: S11. Preprocess the domain standard knowledge data input by the user, the preprocessing including data cleaning, data standardization and redundant information removal; S12. Based on the pre-set word segmentation model, extract terms from the pre-processed domain standard knowledge data to obtain the corresponding domain entities and the attribute information corresponding to each domain entity. S13. Based on the attribute information corresponding to each domain entity and the preprocessed domain standard knowledge data, obtain the relationship types between domain entities; S14. Construct a domain knowledge graph based on domain entities, the attribute information corresponding to each domain entity, and the relationship types between domain entities.

[0007] Optionally, the domain knowledge graph is input into a pre-set TransE model to perform knowledge representation learning and vector encoding processing on the domain knowledge graph, constructing a domain knowledge vector space, including: S15. Extract all triples in the domain knowledge graph, and based on the pre-set vector dimension, perform uniform distribution initialization on the head entity, entity relation and tail entity in each triple to obtain the triple feature vector corresponding to each triple; wherein, each triple includes a head entity, entity relation and tail entity, and each triple feature includes a head entity vector, an entity relation vector and a tail entity vector. S16. Based on the attribute information corresponding to each domain entity in the domain knowledge graph, label the head entity vector and tail entity vector in each triplet feature vector with head entity label and tail entity label, and based on the relationship type between each domain entity in the domain knowledge graph, label the entity relationship vector in each triplet feature vector with entity relationship label. S17. Construct a triplet training sample set based on the triplet feature vector corresponding to each triplet, and the head entity vector label, entity relation vector label and tail entity vector label corresponding to each triplet feature vector. S18. Input the triplet training sample set into the pre-set TransE model to construct the domain knowledge vector space.

[0008] Optionally, a triplet training sample set is constructed based on the triplet feature vector corresponding to each triplet, and the head entity vector label, entity relation vector label, and tail entity vector label corresponding to each triplet feature vector, including: S171. Take each triplet feature vector as a positive training sample, and based on the head entity vector label, entity relation vector label and tail entity vector label corresponding to each triplet feature vector, randomly replace the head entity vector or tail entity vector in each positive training sample to generate the negative training sample corresponding to each positive training sample. S172. Based on all positive and negative training samples, construct a triplet training sample set.

[0009] Optionally, the triple training sample set is input into a pre-set TransE model to construct a domain knowledge vector space, including: S181. Input the triplet training sample set into the pre-set TransE model to obtain the corresponding sample loss; and based on the corresponding sample loss, iteratively update all head entity vectors, entity relation vectors and tail entity vectors in the triplet training sample set through backpropagation gradient. S182. Repeat step S181 until the pre-set training objective is achieved. Integrate all updated head entity vectors, entity relation vectors, and tail entity vectors into the same continuous feature space to construct the domain knowledge vector space.

[0010] Optionally, the triplet training sample set is input into a pre-set TransE model to obtain the corresponding sample loss, including: S1811. Based on each positive training sample in the triplet training sample set, construct the L2 regularization term corresponding to each positive training sample; the L2 regularization term is: ; Where h is the head entity vector in the positive training sample, r is the entity relation vector in the positive training sample, t is the tail entity vector in the positive training sample, C is the L2 regularization term corresponding to the positive training sample, i is the index of the positive training sample in the triplet training sample set, and ||·|| is the L2 norm. S1812. Based on each positive training sample in the triplet training sample set, and the corresponding negative training sample and L2 regularization term for each positive training sample, obtain the corresponding sample loss.

[0011] Optionally, S1812 includes: Based on each positive training sample in the triplet training sample set, and the corresponding negative training sample and L2 regularization term for each positive training sample, the corresponding sample loss is obtained; the sample loss is: ; Where L is the corresponding sample loss, λ e For the pre-set entity label constraint weights, λ reg The pre-defined relation label constraint weights, and γ is a pre-defined fixed interval hyperparameter. The head entity vector in the negative training samples. The entity relation vector in the negative training samples. This represents the tail entity vector in the negative training samples.

[0012] Optionally, S3 includes: S31. Embed the multimodal semantic fusion features into the 6th to 10th intermediate feature layers of the general large model Transformer encoder to perform weighted fusion and residual connection between the multimodal semantic fusion features and the original hidden layer features of the model to obtain an updated large model. S32. Based on the triplet feature vector corresponding to each triplet, and the head entity vector label, entity relation vector label and tail entity vector label corresponding to each triplet feature vector, fine-tune the parameters of the updated large model to obtain a city large model applicable to smart cities.

[0013] Secondly, embodiments of this application provide a large model fine-tuning system, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described large model fine-tuning method.

[0014] Thirdly, embodiments of this application provide a smart city anomaly detection method, including: Real-time multimodal data within the target city is acquired and input into a large city model to obtain structured inference results. The large city model is the one described above. The structured inference results include disease type, risk level, and treatment recommendations.

[0015] (III) Beneficial Effects This application presents a large model fine-tuning method that embeds knowledge vectors from the target domain into a general large model. This allows knowledge to participate in semantic understanding, feature encoding, and decision reasoning throughout the process. This enhances the model's ability to absorb and utilize domain knowledge, while also improving the analytical accuracy and decision reliability when facing the complex scenario of smart cities. Attached Figure Description

[0016] Figure 1 A flowchart of the large model fine-tuning method provided in the embodiments of this application; Figure 2 A flowchart for constructing a domain knowledge graph is provided for embodiments of this application; Figure 3 A flowchart for constructing the domain knowledge vector space provided in the embodiments of this application; Figure 4 A flowchart for fine-tuning a general large model provided for embodiments of this application. Detailed Implementation

[0017] To better explain and facilitate understanding of this application, the following detailed description of the application is provided in conjunction with the accompanying drawings and specific embodiments.

[0018] While current high-efficiency parameter fine-tuning techniques reduce domain adaptation costs, traditional fine-tuning relies heavily on data-driven approaches and lacks guidance from structured knowledge specific to urban infrastructure. This makes it difficult to understand specialized content such as equipment mechanisms and operation and maintenance rules, and its cross-modal semantic fusion capabilities are weak, leading to problems such as insufficient generalization and uninterpretable reasoning, thus failing to meet the high-precision management and control requirements of smart cities. Domain knowledge graphs, on the other hand, can systematically carry urban entities, relationships, and industry rules, providing structured support for large-scale model adaptation and compensating for the reasoning deficiencies of purely data-driven models. However, current methods of integrating knowledge graphs with large models are relatively superficial, often employing add-on approaches such as text concatenation and external retrieval, supplementing knowledge only during the input or reasoning stages. This fails to enable unified vectorized encoding and representation learning of knowledge graphs, making it difficult to construct a standardized domain knowledge vector space.

[0019] To this end, the large model fine-tuning method provided in this application embeds domain knowledge vectors into a general large model, allowing knowledge to participate in semantic understanding, feature encoding, and decision reasoning throughout the process. This not only improves the model's ability to absorb and utilize domain knowledge, but also significantly enhances its analytical accuracy and decision reliability in smart city scenarios.

[0020] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application can be understood more clearly and thoroughly, and that the scope of this application can be fully conveyed to those skilled in the art.

[0021] This application provides a method for fine-tuning a large model, which is applied in the field of smart cities, such as... Figure 1 As shown, the method includes: S1. Based on the domain standard knowledge data of the target city input by the user, construct a domain knowledge graph, and input the domain knowledge graph into a pre-set TransE model to perform knowledge representation learning and vector encoding processing on the domain knowledge graph to construct a domain knowledge vector space; wherein, the domain knowledge vector space is a low-dimensional dense feature space composed of entity vectors and relation vectors of the same dimension; S2. Feature encoding is performed on the pre-collected urban multimodal data to obtain the urban multimodal features corresponding to the urban multimodal data. Linear projection is then performed on the urban multimodal features to map them to the domain knowledge vector space, thereby obtaining multimodal semantic fusion features that achieve co-spatial semantic alignment between multimodal data and domain knowledge. Here, the urban multimodal data consists of multi-type heterogeneous data sources collected in urban scenarios. S3. Input the multimodal semantic fusion features into the encoder of the pre-trained general large model to fine-tune the parameters and embed domain knowledge into the general large model, and obtain a city large model applicable to smart cities.

[0022] This embodiment provides a large model fine-tuning method that embeds knowledge vectors from the target domain into a general large model. This allows knowledge to participate in semantic understanding, feature encoding, and decision reasoning throughout the process, thereby improving the model's ability to absorb and utilize domain knowledge and enhancing the analytical accuracy and decision reliability when dealing with smart cities.

[0023] Optionally, in one specific embodiment, a domain knowledge graph is constructed based on the domain standard knowledge data of the target city input by the user, such as... Figure 2 As shown, it includes: S11. Preprocess the domain standard knowledge data input by the user. Preprocessing includes data cleaning, data standardization, and removal of redundant information. S12. Based on the pre-set word segmentation model, extract terms from the pre-processed domain standard knowledge data to obtain the corresponding domain entities and the attribute information corresponding to each domain entity. S13. Based on the attribute information corresponding to each domain entity and the preprocessed domain standard knowledge data, obtain the relationship types between domain entities; S14. Construct a domain knowledge graph based on domain entities, the attribute information corresponding to each domain entity, and the relationship types between domain entities.

[0024] Specifically, this embodiment integrates multi-source domain information such as domain texts, equipment manuals, operation and maintenance rules, fault cases, and historical data from both edge and cloud platforms to construct an entity system covering equipment, faults, locations, operating conditions, and handling measures, as well as a relationship system of belonging, triggering, located, needing, and associating. This serves as the domain standard knowledge data for constructing a domain knowledge graph. The construction process of the domain knowledge graph includes the following steps: First, acquire the smart city domain standard knowledge data input by the user. Data sources include unstructured text materials such as urban infrastructure operation and maintenance specifications, bridge and utility tunnel technical standards, facility operation and maintenance manuals, defect treatment procedures, and equipment technical documents. Preprocess the raw input domain standard knowledge data, namely: Data cleaning removes invalid content such as garbled characters, invalid symbols, duplicate fragments, and incomplete sentences; data standardization unifies professional terminology, standardizes sentence formats, and unifies units of measurement and facility naming rules; redundant information removal deletes duplicate descriptions, irrelevant popular science content, and general irrelevant language, retaining core business knowledge related to urban facilities, diseases, monitoring, operation and maintenance, and equipment, resulting in well-organized, concise, and effective standardized domain knowledge text.

[0025] For example, the original data contained colloquial expressions, repetitive content, and invalid symbols. For instance, "Bridge cracks and crack defects can cause structural hazards, and bridge cracks are very dangerous #@". After preprocessing, the data was standardized and simplified into the standard text: Bridge cracks can easily lead to structural safety hazards in bridges.

[0026] Secondly, a pre-trained domain-specific word segmentation model is invoked. This model is adapted to a specialized lexicon for urban infrastructure and smart city management, and performs accurate word segmentation and key term extraction on the text in the pre-processed domain standard knowledge data.

[0027] Based on entity recognition rules and a domain dictionary, core domain entities are automatically filtered and extracted from text. Simultaneously, combined with the semantic context of the text, attribute information corresponding to each domain entity is mined and bound, completing entity element extraction and attribute annotation. Specifically, a word segmentation tool is used to perform full word segmentation of the text, combined with a specialized dictionary for urban infrastructure (bridges, tunnels, pipe corridors, roads, structures, sensors, defects, treatment) for forced word segmentation matching, ensuring that domain terms are not segmented. High-frequency words, professional phrases, and fixed collocations are automatically identified through word frequency statistics, mutual information, and left and right information entropy to filter out high-confidence candidate terms. Stop words, function words, and generic words are filtered out, retaining only noun terms to form a candidate entity set.

[0028] Pre-trained models such as BERT and RoBeRTa are used to semantically vectorize candidate terms. Terms with similar meanings are automatically clustered through semantic similarity calculation. For example, "crack," "transverse crack," and "vertical crack" are clustered into the "crack" category; "settlement," "uneven settlement," and "excessive settlement" are clustered into the "settlement" category. The system automatically determines the type based on industry rules found in domain standard knowledge data. For instance, "bridge," "tunnel," "pipe gallery," "pier," "wall," and "pipeline" are identified as facility / structure entities; "crack," "leakage," "damage," "deformation," and "detachment" are identified as defect entities; "stress," "displacement," "settlement value," "temperature and humidity," and "concentration" are identified as monitoring indicator entities; and "inspection," "reinforcement," "repair," "sealing," and "replacement" are identified as treatment measure entities. The clustering results are automatically categorized to output domain entities and their corresponding attribute information.

[0029] For example, the text to be processed reads: "Bridge cracks and abutment seepage are both structural defects of bridges. Cracks are characterized by expansion and concealment, while seepage is characterized by corrosiveness and long-term nature." Domain entities extracted: bridge cracks, abutment seepage, and bridge structural defects. Matching entity attributes: bridge cracks, with attributes of expansion and concealment; abutment seepage, with attributes of corrosiveness and long-term nature; bridge structural defects, with attributes of infrastructure defects requiring regular monitoring.

[0030] Furthermore, based on all extracted domain entities and entity attributes, and combined with preprocessed complete domain standard knowledge text, semantic relationships between different entities are mined through semantic rule matching and domain relation dictionary constraints. Standardized relationship types are defined and categorized; subordinate, causal, action, attribution, harm, and disposal relationships are distinguished to clarify the logical connections between pairs of entities, forming a standardized set of entity relationship relationships. That is: For example, by combining the semantics of business text, we can mine the relationships between entities: Bridge cracks are a type of bridge structural defect; bridge abutment seepage is a type of bridge structural defect; bridge cracks cause structural safety hazards; bridge abutment seepage exacerbates steel reinforcement corrosion. This establishes standardized entity relationship types such as "belongs to," "causes," "exacerbates," and "acts upon."

[0031] Finally, by integrating all the elements obtained in the preceding steps, the domain entities are used as knowledge graph nodes, entity attributes are used as feature description information of the nodes, and the relationship types between entities are used as connection edges between nodes. In accordance with the standard architecture of knowledge graphs, node input, attribute mounting, and relationship link binding are completed, and the entities, attributes, and relational logic of the urban infrastructure domain are organized in a structured manner. In the end, a complete, logically closed-loop domain knowledge graph that fits the business of smart cities is built.

[0032] The graph is constructed by using domain entities as nodes and the relationship types between domain entities as connecting lines.

[0033] This embodiment collects standard knowledge data in the smart city domain through multi-source fusion, and sequentially performs data cleaning, standardization, and redundancy removal preprocessing. Combined with a dedicated word segmentation model, it accurately extracts domain entities and their corresponding attributes. Relying on semantic rules and domain dictionaries, it mines standardized relationships between entities. Finally, with entities as nodes, attributes as supplements, and relationships as connecting edges, it completes the standardized construction of a domain knowledge graph. This effectively unifies the expression of professional terminology in urban infrastructure, organizes heterogeneous knowledge content from multiple sources, and accurately precipitates core domain elements and logical relationships such as equipment, defects, operation and maintenance, and disposal. It avoids the problems of scattered and messy original knowledge data, inconsistent expression, and unclear relationships, ensuring the standardization, completeness, and logical rigor of the constructed knowledge graph. This provides high-quality, structured, and highly adaptable underlying knowledge support for subsequent TransE model knowledge representation learning, domain knowledge vector space construction, and deep embedding and precise fine-tuning of domain knowledge in large models.

[0034] Optionally, in a specific embodiment, the domain knowledge graph is input into a pre-set TransE model to perform knowledge representation learning and vector encoding processing on the domain knowledge graph, thereby constructing a domain knowledge vector space, such as... Figure 3 As shown, it includes: S15. Extract all triples in the domain knowledge graph, and based on the pre-set vector dimension, initialize the head entity, entity relation, and tail entity in each triple in a uniform distribution to obtain the triple feature vector corresponding to each triple; wherein, each triple includes a head entity, entity relation, and tail entity, and each triple feature includes a head entity vector, an entity relation vector, and a tail entity vector. S16. Based on the attribute information corresponding to each domain entity in the domain knowledge graph, label the head entity vector and tail entity vector in each triplet feature vector with head entity label and tail entity label, and based on the relationship type between each domain entity in the domain knowledge graph, label the entity relationship vector in each triplet feature vector with entity relationship label. S17. Construct a triplet training sample set based on the triplet feature vector corresponding to each triplet, and the head entity vector label, entity relation vector label and tail entity vector label corresponding to each triplet feature vector. S18. Input the triplet training sample set into the pre-set TransE model to construct the domain knowledge vector space.

[0035] Specifically, a triplet training sample set is constructed based on the triplet feature vector corresponding to each triplet, and the head entity vector label, entity relation vector label, and tail entity vector label corresponding to each triplet feature vector, including: S171. Take each triplet feature vector as a positive training sample, and based on the head entity vector label, entity relation vector label and tail entity vector label corresponding to each triplet feature vector, randomly replace the head entity vector or tail entity vector in each positive training sample to generate the negative training sample corresponding to each positive training sample. S172. Based on all positive and negative training samples, construct a triplet training sample set.

[0036] Furthermore, the triplet training sample set is input into the pre-set TransE model to construct a domain knowledge vector space, including: S181. Input the triplet training sample set into the pre-set TransE model to obtain the corresponding sample loss; and based on the corresponding sample loss, iteratively update all head entity vectors, entity relation vectors and tail entity vectors in the triplet training sample set through backpropagation gradient. S182. Repeat step S181 until the pre-set training objective is achieved. Integrate all updated head entity vectors, entity relation vectors, and tail entity vectors into the same continuous feature space to construct the domain knowledge vector space.

[0037] Specifically, the TransE model is used to learn representations of the knowledge graph, generating standardized entity vectors and relation vectors to provide a unified low-dimensional semantic representation for subsequent knowledge fusion. This includes: Based on the ontology structure of the domain knowledge graph constructed above, all (structured) triples are extracted from the domain knowledge graph. The triples have a unified format: (head entity h, relation r, tail entity t). For example: (pier, contain, support), (crack, trigger, structural risk), (settlement, exceed, threshold), (leakage, treatment, sealing). All domain entities and the types of relations between domain entities are uniquely numbered to form an entity index table and a relation index table.

[0038] Secondly, a uniform vector dimension d is set (e.g., 128 / 256 / 512 dimensions, based on the general large model to be embedded). A d-dimensional vector is randomly initialized for each domain entity (head entity and tail entity) to obtain the head entity vector and tail entity vector. A d-dimensional vector is randomly initialized for each entity relationship to obtain the entity relationship vector. All vectors are initialized with a uniform distribution and their values ​​are limited to a fixed range to ensure training stability. At this time, the vectors have no actual semantics and are only initial parameters.

[0039] Furthermore, based on the attribute information of entities in each domain in the above steps, each head entity vector and tail entity vector is labeled with a category label, such as facility, component, disease, indicator, and measure; each entity relationship vector is labeled with an entity relationship label, such as subordinate, composition, location, trigger, symptom-cause, measure-cause, judgment, and exceeding limit; the category information is added to TransE training as a priori constraints, so that entity vectors of the same type are closer in space and the directions of different relationship vectors are more clearly distinguished.

[0040] Subsequently, the core optimization objective of TransE is defined as follows: Head entity vector + Entity relation vector ≈ Tail entity vector. For correct triples (positive training samples), the distance between the head entity vector + entity relation vector and the tail entity vector is minimized. For incorrect triples (negative training samples, randomly generated negative samples), the distance between the head entity vector + entity relation vector and the tail entity vector is maximized. Positive training samples are derived from triple feature vectors in the knowledge graph, while negative training samples are generated by randomly replacing the head or tail entity, ensuring they do not exist in the knowledge graph.

[0041] Finally, the vectors are updated using the Stochastic Gradient Descent (SGD) / Adam optimizer. In each iteration, a batch of triplet feature vectors is randomly selected (labeled positive and negative samples are uniformly summarized, shuffled, and divided into batches to form a standardized triplet training sample set that can be directly input into the model for iterative training, achieving a balanced configuration of positive and negative samples to meet the requirements of TransE model loss calculation and parameter update), such as combinations related to tunnels, pipe corridors, defects, and monitoring indicators. The positive and negative sample losses are calculated to automatically correct the values ​​of the head entity vector, tail entity vector, and entity relationship vector through backpropagation gradient. Semantic aggregation is generated during the iterative update process: for similar entities, the vector spatial distance of bridge piers, abutments, and foundations continuously approaches; for similar relationships, the direction of entity relationship vectors such as triggering, causing, and inducing tends to be consistent; and for business association binding, defect entities such as leakage, damage, and corrosion form a fixed semantic correspondence with entities such as sealing, anti-corrosion, and maintenance.

[0042] When the sample loss stabilizes and no longer decreases significantly, the clustering of entities such as facilities, diseases, and monitoring indicators is clear, and the semantic similarity of similar infrastructure entities reaches the set threshold, the model is considered to have converged, and training is stopped.

[0043] The trained head entity vectors, tail entity vectors, and entity relationship vectors are integrated to build a dedicated domain knowledge vector space, namely the domain knowledge vector space.

[0044] This embodiment extracts standardized triples from the domain knowledge graph and initializes them with unified dimensional vectors. Multi-dimensional labeling is then performed by combining entity attributes and relation types. A balanced and complete training sample set is constructed using native triples of positive samples and randomly replaced entities to generate negative samples. Iterative training and gradient backpropagation updates are performed using the TransE model, with the spatial mapping relationship between head entity vectors, relation vectors, and tail entity vectors as the optimization objective. By leveraging prior constraints of category and relation labels, the semantic vector space of similar entities is clustered, and the representation of similar relations tends to be unified, effectively strengthening the semantic relevance and distinguishability of domain knowledge and continuously optimizing the process. The dense semantic representation of entity and relation vectors avoids the problems of scattered vector representation and semantic ambiguity. After the model loss converges and the feature distribution stabilizes, all optimized low-dimensional dense vectors are integrated to construct a standardized, dimensionally unified, and semantically ordered domain knowledge vector space. This enables unified vectorized encoding and deep representation learning of structured knowledge in the urban infrastructure field. It provides standardized and highly semantically consistent underlying knowledge representation support for subsequent linear projection alignment of urban multimodal features, cross-modal semantic fusion, and deep embedding and precise parameter fine-tuning of domain knowledge in large models. This significantly improves the understanding and reasoning accuracy of large models in the field of smart cities.

[0045] Optionally, in a specific embodiment, the triplet training sample set is input into a pre-set TransE model to obtain the corresponding sample loss, including: S1811. Based on each positive training sample in the triplet training sample set, construct the L2 regularization term corresponding to each positive training sample; the L2 regularization term is: ; Where h is the head entity vector in the positive training sample, r is the entity relation vector in the positive training sample, t is the tail entity vector in the positive training sample, C is the L2 regularization term corresponding to the positive training sample, and i is the index of the positive training sample in the triplet training sample set. S1812. Based on each positive training sample in the triplet training sample set, and the corresponding negative training sample and L2 regularization term for each positive training sample, obtain the corresponding sample loss.

[0046] Furthermore, based on each positive training sample in the triplet training sample set, and the corresponding negative training sample and L2 regularization term for each positive training sample, the corresponding sample loss is obtained; the sample loss is: ; Where L is the corresponding sample loss, λ e For the pre-set entity label constraint weights, λ reg The pre-defined relation label constraint weights, and γ is a pre-defined fixed interval hyperparameter. The head entity vector in the negative training samples. The entity relation vector in the negative training samples. This represents the tail entity vector in the negative training samples.

[0047] Specifically, for each positive training sample in the triplet training sample set, perform the following operations: Take the head entity vector, entity relation vector, and tail entity vector of the positive training sample, generate the corresponding negative training sample, and calculate the L2 regularization term C for the positive sample. i ; ; Where h is the head entity vector in the positive training sample, r is the entity relation vector in the positive training sample, t is the tail entity vector in the positive training sample, C is the L2 regularization term corresponding to the positive training sample, i is the index of the positive training sample in the triplet training sample set, and ||·|| is the L2 norm.

[0048] Based on each positive training sample in the triplet training sample set, and the corresponding negative training sample and L2 regularization term for each positive training sample, the corresponding sample loss is obtained; the sample loss is: ; Where L is the corresponding sample loss, λ e The pre-set entity label constraint weights, the larger the weights, the stronger the energy score constraint, λ. reg The pre-defined relation label constraint weights represent the model's strengths. A larger weight results in stronger regularization constraints and a lower risk of overfitting. γ is a pre-defined fixed interval hyperparameter; a larger parameter indicates a more stringent requirement for the model to distinguish between positive and negative samples. The head entity vector in the negative training samples. The entity relation vector in the negative training samples. This represents the tail entity vector in the negative training samples.

[0049] The goal of model training is to minimize the total loss so that the distance between positive training samples becomes smaller and the distance between negative samples becomes larger; at the same time, the size of all vectors is kept within a reasonable range.

[0050] This embodiment effectively suppresses model overfitting and improves the stability of the knowledge vector space by adding an L2 regularization term to constrain the magnitude of entity and relation vectors. Combined with an improved margin loss function with labeled constraint weights, the representation distance between positive and negative samples can be precisely increased, strengthening the semantic distinction between domain entities and relations. By continuously optimizing vector representations by minimizing the overall sample loss, the structured semantics of the knowledge graph are accurately encoded, providing highly robust standardized knowledge vector support for subsequent large-scale model knowledge embedding and domain adaptation.

[0051] Optionally, in a specific embodiment, S2, the pre-collected urban multimodal data is feature-encoded to obtain the corresponding urban multimodal features, and the urban multimodal features are linearly projected to map the urban multimodal features to the domain knowledge vector space, thereby obtaining multimodal semantic fusion features that achieve co-spatial semantic alignment between multimodal data and domain knowledge; wherein, the urban multimodal data is multi-type heterogeneous data source data collected in urban scenarios. Specifically, this includes: firstly, encoding the corresponding features of multimodal data such as images / videos, sensor time series, and text commands respectively; mapping each modal feature to the knowledge graph vector space through a linear projection layer to achieve co-spatial representation; then performing three layers of semantic binding: entity alignment, relation alignment, and scene alignment, and minimizing the distance between data features and knowledge vectors by constraining alignment loss to ensure accurate binding between multi-source data and domain knowledge. That is: Perform the following steps on the images, videos, sensor data, and text data collected on-site in the target city: For images / videos of bridge cracks, tunnel leaks, pipe gallery damage, and structural deformation, visual feature extraction and target detection algorithms are used to extract visual features such as defects, structures, and locations. For time-series monitoring data such as settlement, displacement, vibration, stress, temperature and humidity, water pressure, and gas concentration, time-series pattern matching and statistical analysis are used to extract time-series feature vectors such as exceeding limits, sudden changes, and abnormal trends. For texts such as inspection records, alarm texts, maintenance work orders, and structural descriptions, jieba word segmentation, BERT encoding, and word2vec vectorization are used to extract phrase and sentence vectors.

[0052] Visual, temporal, and textual features are uniformly mapped to the urban infrastructure knowledge graph vector space through a linear projection layer, completing the alignment of facility entities, disease relationships, and working conditions, and realizing the semantic expression of multimodal data in the same space.

[0053] This embodiment achieves co-space mapping between multimodal features and domain knowledge vector space through linear projection. Combined with three-layer semantic alignment and alignment loss constraints, it ensures accurate binding between multimodal data and domain knowledge, breaks down the semantic barriers of modality, and provides semantically consistent and highly adaptable multimodal fusion feature support for subsequent large model knowledge embedding and parameter fine-tuning.

[0054] Optionally, in one specific embodiment, S3, as Figure 4 As shown, it includes: S31. Embed the multimodal semantic fusion features into the 6th to 10th intermediate feature layers of the general large model Transformer encoder to perform weighted fusion and residual connection between the multimodal semantic fusion features and the original hidden layer features of the model to obtain an updated large model. S32. Based on the triplet feature vector corresponding to each triplet, and the head entity vector label, entity relation vector label and tail entity vector label corresponding to each triplet feature vector, fine-tune the parameters of the updated large model to obtain a city large model applicable to smart cities.

[0055] Specifically, the aligned knowledge vectors are embedded in the middle of layers 6-10 of the large model's Transformer encoder, and weighted fusion and residual connection are performed with the model's hidden layer features. A knowledge gating unit is introduced to dynamically adjust the knowledge injection intensity to protect the pre-trained basic semantics from changing. Knowledge participates in the entire process of forward propagation and gradient backpropagation, and three types of joint training—entity matching, relation reasoning, and scene discrimination—are carried out simultaneously. Multi-task weighted loss and alignment loss are jointly optimized to achieve deep collaboration between data and knowledge.

[0056] This embodiment embeds multimodal fusion features into the intermediate layer of the Transformer encoder of a large model. By combining weighted fusion, residual connections, and knowledge gating unit adjustments, it not only preserves the original basic semantic capabilities of the large model but also achieves deep fusion of multimodal features and knowledge vectors. At the same time, through multi-task joint training and loss optimization, it effectively avoids semantic bias and overfitting problems, allowing knowledge to be deeply integrated into the model representation process. This enhances the model's understanding of semantics and business logic related to smart cities, ensuring that the large model can accurately meet the needs of smart city scenarios, balancing model stability and adaptability, and providing reliable support for subsequent accurate inference and efficient response.

[0057] Building upon this, this embodiment stores the domain knowledge graph and the city big model in the cloud, enabling the cloud to undertake global training and knowledge update tasks; the edge loads the knowledge subgraph and lightweight model, responsible for local real-time inference; the edge and cloud share the same knowledge encoding table, and the edge only uploads gradients and knowledge increments to complete lightweight synchronization; an edge-cloud consistency loss constraint is constructed to ensure that edge and cloud features are distributed in the same way, so as to achieve the same knowledge source, the same feature distribution, and the same decision result between the edge and cloud, which significantly reduces response latency.

[0058] In addition, embodiments of this application provide a large model fine-tuning system, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to perform the large model fine-tuning method described above.

[0059] Furthermore, embodiments of this application provide a smart city anomaly detection method, including: Real-time multimodal data within the target city is acquired and input into a large city model to obtain structured inference results. The large city model is the one described above. The structured inference results include disease type, risk level, and treatment recommendations.

[0060] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0061] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0062] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first and second features are in direct contact, or that they are in indirect contact through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0063] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0064] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for fine-tuning a large model, characterized in that, The method is applied in the field of smart cities, and the method includes: S1. Based on the domain standard knowledge data of the target city input by the user, construct a domain knowledge graph, and input the domain knowledge graph into a pre-set TransE model to perform knowledge representation learning and vector encoding processing on the domain knowledge graph to construct a domain knowledge vector space; wherein, the domain knowledge vector space is a low-dimensional dense feature space composed of entity vectors and relation vectors of the same dimension; S2. Feature encoding is performed on the pre-collected urban multimodal data to obtain the urban multimodal features corresponding to the urban multimodal data. Linear projection is then performed on the urban multimodal features to map them to the domain knowledge vector space, thereby obtaining multimodal semantic fusion features that achieve co-spatial semantic alignment between multimodal data and domain knowledge. Here, the urban multimodal data consists of multi-type heterogeneous data sources collected in urban scenarios. S3. Input the multimodal semantic fusion features into the encoder of the pre-trained general large model to fine-tune the parameters and embed domain knowledge into the general large model to obtain a city large model applicable to smart cities.

2. The large model fine-tuning method according to claim 1, characterized in that, Based on the domain standard knowledge data of the target city input by the user, a domain knowledge graph is constructed, including: S11. Preprocess the domain standard knowledge data input by the user, the preprocessing including data cleaning, data standardization and redundant information removal; S12. Based on the pre-set word segmentation model, extract terms from the pre-processed domain standard knowledge data to obtain the corresponding domain entities and the attribute information corresponding to each domain entity. S13. Based on the attribute information corresponding to each domain entity and the preprocessed domain standard knowledge data, obtain the relationship types between domain entities; S14. Construct a domain knowledge graph based on domain entities, the attribute information corresponding to each domain entity, and the relationship types between domain entities.

3. The large model fine-tuning method according to claim 1, characterized in that, The domain knowledge graph is input into a pre-configured TransE model to perform knowledge representation learning and vector encoding processing on the domain knowledge graph, constructing a domain knowledge vector space, including: S15. Extract all triples in the domain knowledge graph, and based on the pre-set vector dimension, perform uniform distribution initialization on the head entity, entity relation and tail entity in each triple to obtain the triple feature vector corresponding to each triple; wherein, each triple includes a head entity, entity relation and tail entity, and each triple feature includes a head entity vector, an entity relation vector and a tail entity vector. S16. Based on the attribute information corresponding to each domain entity in the domain knowledge graph, label the head entity vector and tail entity vector in each triplet feature vector with head entity label and tail entity label, and based on the relationship type between each domain entity in the domain knowledge graph, label the entity relationship vector in each triplet feature vector with entity relationship label. S17. Construct a triplet training sample set based on the triplet feature vector corresponding to each triplet, and the head entity vector label, entity relation vector label and tail entity vector label corresponding to each triplet feature vector. S18. Input the triplet training sample set into the pre-set TransE model to construct the domain knowledge vector space.

4. The large model fine-tuning method according to claim 3, characterized in that, Based on the triplet feature vector corresponding to each triplet, and the head entity vector label, entity relation vector label, and tail entity vector label corresponding to each triplet feature vector, a triplet training sample set is constructed, including: S171. Take each triplet feature vector as a positive training sample, and based on the head entity vector label, entity relation vector label and tail entity vector label corresponding to each triplet feature vector, randomly replace the head entity vector or tail entity vector in each positive training sample to generate the negative training sample corresponding to each positive training sample. S172. Based on all positive and negative training samples, construct a triplet training sample set.

5. The large model fine-tuning method according to claim 4, characterized in that, Input the triplet training sample set into the pre-set TransE model to construct the domain knowledge vector space, including: S181. Input the triplet training sample set into the pre-set TransE model to obtain the corresponding sample loss; and based on the corresponding sample loss, iteratively update all head entity vectors, entity relation vectors and tail entity vectors in the triplet training sample set through backpropagation gradient. S182. Repeat step S181 until the pre-set training objective is achieved. Integrate all updated head entity vectors, entity relation vectors, and tail entity vectors into the same continuous feature space to construct the domain knowledge vector space.

6. The large model fine-tuning method according to claim 5, characterized in that, Input the triplet training sample set into the pre-set TransE model to obtain the corresponding sample loss, including: S1811. Based on each positive training sample in the triplet training sample set, construct the L2 regularization term corresponding to each positive training sample; the L2 regularization term is: ； Where h is the head entity vector in the positive training sample, r is the entity relation vector in the positive training sample, t is the tail entity vector in the positive training sample, C is the L2 regularization term corresponding to the positive training sample, i is the index of the positive training sample in the triplet training sample set, and ||·|| is the L2 norm. S1812. Based on each positive training sample in the triplet training sample set, and the corresponding negative training sample and L2 regularization term for each positive training sample, obtain the corresponding sample loss.

7. The large model fine-tuning method according to claim 6, characterized in that, S1812 includes: Based on each positive training sample in the triplet training sample set, and the corresponding negative training sample and L2 regularization term for each positive training sample, the corresponding sample loss is obtained; the sample loss is: ； Where L is the corresponding sample loss, λ e For the pre-set entity label constraint weights, λ reg The pre-defined relation label constraint weights, and γ is a pre-defined fixed interval hyperparameter. The head entity vector in the negative training samples. The entity relation vector in the negative training samples. This represents the tail entity vector in the negative training samples.

8. The large model fine-tuning method according to claim 3, characterized in that, S3 includes: S31. Embed the multimodal semantic fusion features into the 6th to 10th intermediate feature layers of the general large model Transformer encoder to perform weighted fusion and residual connection between the multimodal semantic fusion features and the original hidden layer features of the model to obtain an updated large model. S32. Based on the triplet feature vector corresponding to each triplet, and the head entity vector label, entity relation vector label and tail entity vector label corresponding to each triplet feature vector, fine-tune the parameters of the updated large model to obtain a city large model applicable to smart cities.

9. A large model fine-tuning system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the large model fine-tuning method according to any one of claims 1-8.

10. A method for anomaly detection in smart cities, characterized in that, include: Real-time multimodal data within the target city is acquired and input into a large city model to obtain structured inference results of the multimodal real-time data. The large-scale urban model is the large-scale urban model described in any one of claims 1-8; The structured reasoning results include disease type, risk level, and treatment recommendations.

Images