A multi-modal large model hidden danger identification method and system for the urban energy industry

Abstract

This invention discloses a multimodal large-scale model hazard identification method and system for the urban energy industry, belonging to the field of automatic hazard identification technology in the energy industry. It includes S1, data preprocessing; S2, multi-dimensional low-rank parameter fine-tuning: constructing a ROI-oriented multi-branch LoRA fine-tuning framework to perform local visual feature fine-tuning on the image ROI feature set, generating global image fusion features; constructing a field-level differentiated LoRA fine-tuning module group to perform field-level independent semantic fine-tuning on the structured text field set, generating exclusive text features for each field; S3, cross-modal feature fusion and decision output. This invention's multimodal large-scale model hazard identification method and system for the urban energy industry constructs a full-link structure of pre-calibration locking of the area of ​​interest – dual-mode initial screening – refined ROI extraction – calibration and adaptation fine-tuning, deeply integrating calibration information into the entire process of data preprocessing and multi-dimensional low-rank parameter fine-tuning, improving image processing efficiency and hazard identification capabilities.

Description

Technical Field

[0001] This invention specifically relates to a multimodal large-scale model hazard identification method and system for the urban energy industry, belonging to the field of automatic hazard identification technology in the energy industry. Background Technology

[0002] As the core lifeline of urban operations, the urban energy industry faces complex operating scenarios, diverse types of hazards, and stringent safety requirements, placing extremely high demands on the accuracy, standardization, and closed-loop management capabilities of hazard identification. In recent years, intelligent inspection systems based on image analysis have been gradually replacing traditional manual inspection methods. The emergence of large-scale models and multimodal model technologies has transformed hazard identification from "single-target detection" to "semantic understanding and description." For example, the multimodal hazard identification, monitoring, and early warning method and system disclosed in Chinese Patent Publication No. CN120804618B can generate traceable causal chains when hazards are still in their initial stages, significantly improving the accuracy and advance warning capabilities. However, this structure cannot be applied to professional scenarios in the urban energy industry, and existing multimodal large-scale models still have the following shortcomings: 1. Non-standardized identification output and poor resolvability: Existing multimodal large-scale models typically use free text in the hazard identification process. 1. The output of recognition results in a single format lacks unified structural constraints, resulting in problems such as arbitrary expression, missing elements, and semantic inconsistencies in the output content; 2. High false positive and false negative rates in complex scenarios: Hazard images in the urban energy industry are often collected in complex on-site environments with many background interference factors, and some hazard areas are small in size and have indistinct features. Existing technologies mostly use fine-tuning methods of global feature learning, which makes it difficult to focus on key hazard areas and effectively filter background noise, easily leading to false positives and false negatives; 3. Information interference between fields and insufficient professional adaptability: Existing multimodal models cannot split text fields with different functions such as hazard name, hazard description, and rectification suggestions, resulting in information interference between fields. Problems such as inaccurate hazard name recognition, missing descriptive elements, and rectification suggestions deviating from the core issues occur frequently, and the recognition accuracy in professional scenarios is much lower than in general scenarios; 4. Rectification suggestions lack standardized basis and cannot achieve closed-loop management. Summary of the Invention

[0003] To address the aforementioned issues, this invention proposes a multimodal large-scale model-based hazard identification method and system for the urban energy industry. It constructs a full-link structure encompassing pre-calibration to lock the region of interest, dual-mode initial screening, refined ROI extraction, and calibration adaptation fine-tuning. The calibration information is deeply integrated into the entire process of data preprocessing and multi-dimensional low-rank parameter fine-tuning, thoroughly resolving the problems of low efficiency in full-image processing, redundant background interference, and poor targeting of indiscriminate feature learning, thereby significantly improving image processing efficiency and hazard identification capabilities.

[0004] The present invention provides a multimodal large-scale model hazard identification method for the urban energy industry, comprising: S1. Data Preprocessing: Preprocess the image data and text data of potential hazards in the urban energy industry to obtain the ROI feature set of the hazard images and the structured text field set. S2. Multi-dimensional low-rank parameter fine-tuning: Construct a ROI-oriented multi-branch LoRA fine-tuning framework to perform local visual feature fine-tuning on the image ROI feature set and generate global image fusion features; construct a field-level differentiated LoRA fine-tuning module group to perform field-level independent semantic fine-tuning on the structured text field set and generate exclusive text features for each field; the multi-dimensional low-rank parameter fine-tuning supports incremental fine-tuning mode: when a new hazard type is added, only the corresponding ROI LoRA branch and the corresponding field fine-tuning module need to be added, without retraining the backbone network and existing branches, and the incremental fine-tuning time is ≤4 hours; S3. Cross-modal feature fusion and decision output: A dual-path cross-attention mechanism is used to perform cross-modal deep fusion of the global image fusion features and the specific text features of each field to obtain fused multimodal features; based on the fused multimodal features, a structured hazard identification result is output, and the fused multimodal features are matched with the preset urban energy industry hazard knowledge base to generate compliant and standardized rectification suggestions, completing the closed-loop output of the entire hazard identification process.

[0005] Furthermore, the preprocessing of the hazard image data is as follows: S11. Construct an image dataset of potential hazards in the urban energy industry. The dataset covers gas, electricity, heat, and water supply and drainage scenarios. Label the category, bounding box, risk level, and semantic label of each hazard sample. S12. Based on the dataset, complete the pre-training of the YOLOv8 object detection model. The pre-training uses the CIoU loss function. The input image size is uniformly set to 640×640, the confidence threshold is set to 0.5, and the non-maximum suppression threshold is set to 0.45. S13. The input potential hazard image is processed by the pre-trained YOLOv8 model to generate candidate ROIs. Each ROI contains location information, confidence score and preliminary category information. Valid ROIs are retained by confidence score screening. S14. Crop the effective ROIs and uniformly scale them to 224×224 size. Perform data augmentation by random flipping, brightness adjustment, and Gaussian noise addition with an enhancement ratio of 1:4. Finally, generate an image ROI feature set that is adapted and finely adjusted.

[0006] Furthermore, the preprocessing of the hazard text data is as follows: S15. Construct a professional terminology dictionary and anchor keyword library for potential hazards in the urban energy industry. The anchor keyword library includes three categories: anchor keywords for hazard names, anchor keywords for hazard descriptions, and anchor keywords for rectification suggestions. S16. The rule engine is built using the Aho-Corasick multi-pattern matching algorithm and combined with the anchor keyword matching mechanism to accurately split the original hidden danger text into three core fields: hidden danger name, hidden danger description, and rectification suggestions. S17. Clean the text of each field after splitting, remove meaningless characters and stop words, complete word segmentation based on the terminology dictionary, use the BERT-wwm pre-trained model for semantic encoding, generate 768-dimensional standardized semantic vectors, and form a structured text field set.

[0007] Furthermore, the process of fine-tuning the local visual features of the image ROI feature set is as follows: S21. Configure an independent LoRA fine-tuning branch for each valid ROI. The branch adopts a two-level architecture of feature enhancement and low-rank adaptation. All branches share the multimodal large model visual encoder backbone network. Freeze all parameters of the backbone network. S22. The feature enhancement submodule of each LoRA branch adopts the convolutional block attention module CBAM, which sequentially enhances the fine-grained texture features and high-level semantic features within the ROI through channel attention and spatial attention, and suppresses background noise. S23, the low-rank adaptor submodule of each LoRA branch, rank Set to 8, scaling factor Set to 16, only for the attention layer of the visual encoder. The matrix is ​​decomposed into low-rank components and its parameters are fine-tuned to extract local features of a single ROI. S24. Use an attention-weighted fusion strategy to aggregate all ROI features and calculate the initial attention weight for each ROI feature. The calculation formula is as follows: ; in, For the first Initial attention weights for each ROI feature. Let this be the confidence level of the ROI. The cosine similarity between the ROI and the preset hazard category is used. For the characteristic significance of this ROI, Let be the weighting coefficient, satisfying The preferred value is ; S25. The initial attention weights are normalized using the Softmax function. Based on the normalized weights, all ROI features are weighted and summed to generate global image fusion features.

[0008] Furthermore, the field-level differentiated LoRA fine-tuning module group is constructed as follows: The field-level differentiated LoRA fine-tuning module group includes a hazard name fine-tuning module, a hazard description fine-tuning module, and a rectification suggestion fine-tuning module. These three modules are independent of each other and all freeze the multimodal large model language encoder backbone network, fine-tuning only the low-rank parameters of the attention layer and feedforward network layer. Specifically, this includes: S26. The hazard name fine-tuning module adopts a dual-path structure of keyword enhancement and semantic alignment, LoRA rank. Set the learning rate to 4. The iteration step number is set to 1000; a 4-head keyword attention submodule is added before the low-rank adaptation layer to strengthen the extraction of core keyword features. At the same time, a semantic vector library of urban energy industry hidden danger standard categories is introduced, and the semantic alignment between hidden danger names and standard categories is achieved by comparing loss constraints. S27. The hazard description fine-tuning module adopts a context modeling + feature selection architecture, LoRA rank Set the learning rate to 8. The number of iterations was set to 2000. The semantic associations of the text context were captured by the Bi-LSTM network, the core semantic features were optimized by the low-rank adaptation layer, and redundant semantic information was removed by the gating screening mechanism to focus on the core descriptive features of the hidden dangers. S28. The rectification suggestion fine-tuning module adopts a logical modeling + standard constraint structure, LoRA rank. Set the learning rate to 16. The number of iterations was set to 3000. During the low-rank fine-tuning process, an industry-standard semantic vector library was introduced, and a joint loss function of cross-entropy loss and standard constraint loss was adopted. The generated content was constrained to conform to industry standards, while the semantic expression of core elements such as rectification measures, responsible parties, and completion deadlines was strengthened.

[0009] Furthermore, the specific process of cross-modal feature fusion is as follows: S31. Unify the dimensions of the global image fusion features and the specific text features of each field by mapping the image features to 768 dimensions through a 1×1 convolutional layer, so as to keep the dimensions of the text features consistent. S32. Layer normalization is used to calibrate the distribution of the two modal features. Set to 1e-5 to eliminate the feature distribution differences between modalities and output the standardized image and text features; S33. The standardized image features and text features are concatenated to form a multimodal feature sequence, which is then input into a cross-modal fusion layer consisting of three Transformer encoders. S34. The cross-modal fusion layer mines the element correlation within the feature sequence through a multi-head self-attention module, and introduces a dual-path cross-attention mechanism to calculate the attention weights of image features to text features and text features to image features respectively, so as to realize bidirectional feature interaction and enhancement, and finally output fused multimodal features. The specific process for decision output is as follows: S35. The multimodal features are integrated into the classification submodule and the localization submodule. The classification submodule outputs the predicted results of the hazard category and risk level through a fully connected layer and a Softmax function. The localization submodule outputs the accurate location coordinates of the hazard through a regression network, forming a structured recognition result that includes the hazard name, location, category, risk level, and confidence level. S36. Construct a knowledge base for potential hazards in the urban energy industry, using a hybrid storage architecture of MySQL and Redis to classify and store standard definitions, characteristic parameters, compliance rectification specifications, industry standard clauses, and historical handling cases of potential hazards. S37. Using the cosine similarity algorithm, calculate the matching degree between the fused multimodal features and the standard hidden danger feature vectors in the knowledge base, with a preset matching degree threshold of 0.85. S38. When the matching degree is ≥0.85, the standardized rectification suggestion corresponding to the standard hidden danger item with the highest matching degree is called and output together with the identification result; when the matching degree is <0.85, the knowledge base update reminder is automatically triggered, the predicted features and identification results are pushed to the professional review end, and after the review is approved, they are added to the knowledge base. At the same time, a temporary rectification suggestion is generated and output based on the closest standard clause.

[0010] Furthermore, during data preprocessing, the inspection points are pre-calibrated and a calibration library is constructed. Based on the calibration information in the calibration library, the image pre-interest region is locked, and a hazard image ROI feature set is generated by combining the target detection algorithm. During the multi-dimensional low-rank parameter fine-tuning, the hazard image ROI feature set is locally visually fine-tuned, and the feature attention weight is optimized by combining the calibration point information to generate global image fusion features.

[0011] Furthermore, the calibration library is constructed and its application is specifically as follows: Calibration library construction: For fixed inspection scenarios in the urban energy industry, including fixed monitoring points, preset docking points of inspection robots, and fixed shooting points of manual inspection, a calibration library is constructed. Each collection point is assigned a unique ID, a focus mode, and a predefined focus area. The predefined focus area is a pre-ROI, which is the area of ​​equipment and components that need to be monitored at that point. The library also includes a baseline image set of the point under normal conditions, a standard hazard type library corresponding to the point, and compliance regulations. Real-time matching of acquired images and calibration information: When an image of a potential hazard is acquired, the ID of the acquisition point corresponding to the image is obtained simultaneously, and all calibration information corresponding to that point is retrieved from the calibration library. Pre-focused region focus mode locking: The focus mode is adaptively switched according to the ID to complete the core region extraction and remove irrelevant background regions; the focus mode includes a fixed region mode and an anomaly comparison mode; the fixed region mode directly crops the acquired image based on the mapped predefined focus region, retaining only the core monitoring region as the image to be processed; the anomaly comparison mode extracts the global scene feature fingerprint of the acquired image through the DINO-V2 model, performs cosine similarity matching with the manually calibrated calibration library, automatically matches the corresponding manually calibrated point, obtains the matching ID, and then performs pre-ROI acquisition according to the focus mode.

[0012] Furthermore, the acquired image undergoes viewpoint normalization processing. Based on the unified coordinate system after viewpoint normalization, the manually calibrated predefined reference coordinates of the region of interest are directly mapped to the normalized image. The viewpoint normalization processing is as follows: Basic attribute labeling of inspection points: Assign a unique ID to each inspection point, label the point name, the scene to which it belongs and the area of ​​responsibility, and complete the manual labeling of the point location; Baseline image acquisition: Manually take one baseline main image of the location under normal conditions, and no less than nine auxiliary baseline images under different lighting conditions, angles, and equipment operating conditions; Pre-monitoring area labeling: On the baseline main map, mark the baseline coordinates of the equipment or component areas that need to be monitored by selecting them with a box.

[0013] A multimodal large-scale model hazard identification system for the urban energy industry is provided, which implements a multimodal large-scale model hazard identification method for the urban energy industry. The system uses Qwen-VL or LLaVA-1.5 open-source multimodal large-scale model as the base model. The system includes: The multimodal data acquisition module is used to collect multi-source data such as hidden danger image data, equipment operation logs, and on-site text feedback at urban energy operation sites, and supports access from various devices such as cameras, inspection robots, and mobile terminals. The data preprocessing module communicates with the multimodal data acquisition module to complete image ROI extraction, filtering, cropping and enhancement, as well as text field splitting, cleaning and semantic encoding, and output standardized ROI feature sets and structured text field sets; The multi-dimensional fine-tuning module communicates with the data preprocessing module and has a built-in ROI-oriented multi-branch LoRA fine-tuning framework and field-level differentiated LoRA fine-tuning module group. It is used to complete the precise fine-tuning of local visual features and field-level semantic features, and output global image fusion features and field-specific text features. The cross-modal fusion and decision module communicates with the multi-dimensional fine-tuning module to complete the cross-modal deep fusion of image and text features, output structured hazard identification results, and at the same time link with the hazard knowledge base to complete feature matching and generate standardized rectification suggestions. The hazard knowledge base module communicates with the cross-modal fusion and decision-making module to store standard data, normative clauses and handling cases of hazards in the urban energy industry, and supports dynamic updates and rapid feature matching. The results output and closed-loop management module communicates and connects with the cross-modal fusion and decision-making module to visualize the identification results and rectification suggestions. It supports data export, hazard warning, rectification progress tracking and archiving, and realizes closed-loop management of the entire process of hazard disposal.

[0014] Furthermore, the system also includes an image acquisition location calibration unit, which is used to pre-calibrate the inspection points, construct and maintain the calibration library, and perform real-time matching of the acquired images and calibration information to output the pre-interest area; the image acquisition location calibration unit is connected to the data preprocessing module; the multi-dimensional fine-tuning module has a built-in multi-branch LoRA fine-tuning framework for ROI guidance and calibration point adaptation.

[0015] Compared with existing technologies, the multimodal large-scale model hazard identification method and system of the present invention for the urban energy industry has the following advantages: 1. Improve image processing efficiency and hazard identification capability: By locking the pre-marked pre-interest area, the image processing range is narrowed from the entire image to the core monitoring area, greatly reducing the computational load of image preprocessing and the processing latency of a single image. The target detection speed of YOLOv8 is increased many times, and it can simultaneously support real-time hazard identification of thousands of fixed points, fully meeting the processing needs of large-scale inspection scenarios in the urban energy industry.

[0016] 2. Significantly improved hazard identification accuracy and anti-interference capability: Through the two-level ROI mechanism of pre-calibrated pre-ROI and YOLOv8 refined ROI, background interference in non-interested areas is completely eliminated. Combined with feature enhancement and weight optimization of calibration and adaptation, the accuracy of identifying small-sized hazards is further improved, reducing false detection rate and false negative rate, and the identification stability under harsh scenarios such as complex lighting and occlusion is significantly improved.

[0017] 3. Comprehensive optimization of model fine-tuning efficiency and scenario adaptability: Based on the LoRA branch grouping management of calibration points, feature sharing within the same scenario is realized, further reducing the number of training parameters to be fine-tuned and shortening the fine-tuning time; when adding new inspection points, only the point calibration and incremental fine-tuning of the corresponding LoRA branch need to be completed, without adjusting the backbone network and existing branches, improving the efficiency of new point adaptation and enabling rapid coverage of various fixed inspection scenarios in the urban energy industry.

[0018] 4. Pre-screening of invalid data to reduce system redundancy load: Through the pre-screening mechanism of benchmark image comparison, invalid images with viewpoint shift, image occlusion, and no abnormalities can be filtered in advance, reducing the amount of invalid calculations and preventing invalid images from entering the subsequent recognition process, thereby reducing the false alarm rate of the system and improving the overall operational stability.

[0019] 5. The relevance and feasibility of rectification suggestions have been further enhanced: Through the linkage of standardized information between the calibration library and the knowledge base, the rectification suggestions not only comply with general industry standards, but also fit the specific equipment types, management requirements and site conditions of the location, maintaining the standard compliance rate and improving the first-time pass rate of on-site rectification, truly achieving efficient control of the entire process from hazard identification to rectification closure.

[0020] 6. By using ROI-guided multi-branch LoRA fine-tuning to replace the traditional global fine-tuning mode, it accurately focuses on the core area of ​​the hidden danger and effectively filters out background interference from complex sites. Compared with global LoRA fine-tuning, it improves the accuracy of identifying small-sized hidden dangers, reduces the false detection rate and false negative rate, and significantly improves the reliability of the identification results. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the multimodal large model hazard identification process in Embodiment 1 of the present invention.

[0022] Figure 2 This is a schematic diagram of the preprocessing flow for hidden danger image data according to the present invention.

[0023] Figure 3 This is a schematic diagram of the preprocessing flow for potentially hazardous text data in this invention.

[0024] Figure 4 This is a schematic diagram illustrating the process of fine-tuning local visual features for the image ROI feature set of this invention.

[0025] Figure 5 This is a schematic diagram illustrating the specific process of cross-modal feature fusion according to the present invention.

[0026] Figure 6 This is a schematic diagram illustrating the specific process of decision output in this invention.

[0027] Figure 7This is a schematic diagram of the multimodal large model hazard identification process in Embodiment 2 of the present invention.

[0028] Figure 8 This is a schematic diagram of the real-time matching process between acquired images and calibration information in Embodiment 2 of the present invention.

[0029] Figure 9 This is a schematic diagram of the structure of a multimodal large-scale hazard identification system for the urban energy industry according to Embodiment 1 of the present invention.

[0030] Figure 10 This is a schematic diagram of the structure of a multimodal large-scale hazard identification system for the urban energy industry according to Embodiment 2 of the present invention. Detailed Implementation

[0031] Example 1: like Figures 1 to 6 The multimodal large-scale model hazard identification method for the urban energy industry, as shown, includes: S1. Data Preprocessing: Preprocess the image data and text data of potential hazards in the urban energy industry to obtain the ROI feature set of the hazard images and the structured text field set. S2. Multi-dimensional low-rank parameter fine-tuning: Construct a ROI-oriented multi-branch LoRA fine-tuning framework to perform local visual feature fine-tuning on the image ROI feature set and generate global image fusion features; construct a field-level differentiated LoRA fine-tuning module group to perform field-level independent semantic fine-tuning on the structured text field set and generate exclusive text features for each field; the multi-dimensional low-rank parameter fine-tuning supports incremental fine-tuning mode: when a new hazard type is added, only the corresponding ROI LoRA branch and the corresponding field fine-tuning module need to be added, without retraining the backbone network and existing branches, and the incremental fine-tuning time is ≤4 hours; S3. Cross-modal feature fusion and decision output: A dual-path cross-attention mechanism is used to perform cross-modal deep fusion of the global image fusion features and the specific text features of each field to obtain fused multimodal features; based on the fused multimodal features, a structured hazard identification result is output, and the fused multimodal features are matched with the preset urban energy industry hazard knowledge base to generate compliant and standardized rectification suggestions, completing the closed-loop output of the entire hazard identification process.

[0032] The image data of the potential hazard is preprocessed as follows: S11. Construct an image dataset of potential hazards in the urban energy industry. The dataset covers gas, electricity, heat, and water supply and drainage scenarios. Label the category, bounding box, risk level, and semantic label of each hazard sample. S12. Based on the dataset, complete the pre-training of the YOLOv8 object detection model. The pre-training uses the CIoU loss function. The input image size is uniformly set to 640×640, the confidence threshold is set to 0.5, and the non-maximum suppression threshold is set to 0.45. S13. The input potential hazard image is processed by the pre-trained YOLOv8 model to generate candidate ROIs. Each ROI contains location information, confidence score and preliminary category information. Valid ROIs are retained by confidence score screening. S14. Crop the effective ROIs and uniformly scale them to 224×224 size. Perform data augmentation by random flipping, brightness adjustment, and Gaussian noise addition with an enhancement ratio of 1:4. Finally, generate an image ROI feature set that is adapted and finely adjusted.

[0033] The preprocessing of the potential hazard text data is as follows: S15. Construct a professional terminology dictionary and anchor keyword library for potential hazards in the urban energy industry. The anchor keyword library includes three categories: anchor keywords for hazard names, anchor keywords for hazard descriptions, and anchor keywords for rectification suggestions. S16. The rule engine is built using the Aho-Corasick multi-pattern matching algorithm and combined with the anchor keyword matching mechanism to accurately split the original hidden danger text into three core fields: hidden danger name, hidden danger description, and rectification suggestions. S17. Clean the text of each field after splitting, remove meaningless characters and stop words, complete word segmentation based on the terminology dictionary, use the BERT-wwm pre-trained model for semantic encoding, generate 768-dimensional standardized semantic vectors, and form a structured text field set.

[0034] The process of fine-tuning local visual features of the image ROI feature set is as follows: S21. Configure an independent LoRA fine-tuning branch for each valid ROI. The branch adopts a two-level architecture of feature enhancement and low-rank adaptation. All branches share the multimodal large model visual encoder backbone network. Freeze all parameters of the backbone network. S22. The feature enhancement submodule of each LoRA branch adopts the convolutional block attention module CBAM, which sequentially enhances the fine-grained texture features and high-level semantic features within the ROI through channel attention and spatial attention, and suppresses background noise. S23, the low-rank adaptor submodule of each LoRA branch, rank Set to 8, scaling factor Set to 16, only for the attention layer of the visual encoder. The matrix is ​​decomposed into low-rank components and its parameters are fine-tuned to extract local features of a single ROI. S24. Use an attention-weighted fusion strategy to aggregate all ROI features and calculate the initial attention weight for each ROI feature. The calculation formula is as follows: ; in, For the first Initial attention weights for each ROI feature. Let this be the confidence level of the ROI. The cosine similarity between the ROI and the preset hazard category is used. For the characteristic significance of this ROI, Let be the weighting coefficient, satisfying The preferred value is ; S25. The initial attention weights are normalized using the Softmax function. Based on the normalized weights, all ROI features are weighted and summed to generate global image fusion features.

[0035] The field-level differentiated LoRA fine-tuning module group is constructed as follows: The field-level differentiated LoRA fine-tuning module group includes a hazard name fine-tuning module, a hazard description fine-tuning module, and a rectification suggestion fine-tuning module. These three modules are independent of each other and all freeze the multimodal large model language encoder backbone network, fine-tuning only the low-rank parameters of the attention layer and feedforward network layer. Specifically, this includes: S26. The hazard name fine-tuning module adopts a dual-path structure of keyword enhancement and semantic alignment, LoRA rank. Set the learning rate to 4. The iteration step number is set to 1000; a 4-head keyword attention submodule is added before the low-rank adaptation layer to strengthen the extraction of core keyword features. At the same time, a semantic vector library of urban energy industry hidden danger standard categories is introduced, and the semantic alignment between hidden danger names and standard categories is achieved by comparing loss constraints. S27. The hazard description fine-tuning module adopts a context modeling + feature selection architecture, LoRA rank Set the learning rate to 8. The number of iterations was set to 2000. The semantic associations of the text context were captured by the Bi-LSTM network, the core semantic features were optimized by the low-rank adaptation layer, and redundant semantic information was removed by the gating screening mechanism to focus on the core descriptive features of the hidden dangers. S28. The rectification suggestion fine-tuning module adopts a logical modeling + standard constraint structure, LoRA rank. Set the learning rate to 16. The number of iterations was set to 3000. During the low-rank fine-tuning process, an industry-standard semantic vector library was introduced, and a joint loss function of cross-entropy loss and standard constraint loss was adopted. The generated content was constrained to conform to industry standards, while the semantic expression of core elements such as rectification measures, responsible parties, and completion deadlines was strengthened.

[0036] The specific process of cross-modal feature fusion is as follows: S31. Unify the dimensions of the global image fusion features and the specific text features of each field by mapping the image features to 768 dimensions through a 1×1 convolutional layer, so as to keep the dimensions of the text features consistent. S32. Layer normalization is used to calibrate the distribution of the two modal features. Set to 1e-5 to eliminate the feature distribution differences between modalities and output the standardized image and text features; S33. The standardized image features and text features are concatenated to form a multimodal feature sequence, which is then input into a cross-modal fusion layer consisting of three Transformer encoders. S34. The cross-modal fusion layer mines the element correlation within the feature sequence through a multi-head self-attention module, and introduces a dual-path cross-attention mechanism to calculate the attention weights of image features to text features and text features to image features respectively, so as to realize bidirectional feature interaction and enhancement, and finally output fused multimodal features. The specific process for decision output is as follows: S35. The multimodal features are integrated into the classification submodule and the localization submodule. The classification submodule outputs the predicted results of the hazard category and risk level through a fully connected layer and a Softmax function. The localization submodule outputs the accurate location coordinates of the hazard through a regression network, forming a structured recognition result that includes the hazard name, location, category, risk level, and confidence level. S36. Construct a knowledge base for potential hazards in the urban energy industry, using a hybrid storage architecture of MySQL and Redis to classify and store standard definitions, characteristic parameters, compliance rectification specifications, industry standard clauses, and historical handling cases of potential hazards. S37. Using the cosine similarity algorithm, calculate the matching degree between the fused multimodal features and the standard hidden danger feature vectors in the knowledge base, with a preset matching degree threshold of 0.85. S38. When the matching degree is ≥0.85, the standardized rectification suggestion corresponding to the standard hidden danger item with the highest matching degree is called and output together with the identification result; when the matching degree is <0.85, the knowledge base update reminder is automatically triggered, the predicted features and identification results are pushed to the professional review end, and after the review is approved, they are added to the knowledge base. At the same time, a temporary rectification suggestion is generated and output based on the closest standard clause.

[0037] Example 2: like Figure 7 and Figure 8 The multimodal large-scale model hazard identification method for the urban energy industry, as shown, involves the following steps during data preprocessing: First, the inspection points are pre-calibrated, and a calibration library is constructed. Based on the calibration information in the library, the image's region of interest (ROI) is locked, and a hazard image ROI feature set is generated using a target detection algorithm. Second, during the multi-dimensional low-rank parameter fine-tuning, the hazard image ROI feature set undergoes local visual feature fine-tuning, and the feature attention weights are optimized using the calibration point information to generate global image fusion features.

[0038] The calibration library is constructed and its application is as follows: Calibration library construction: For fixed inspection scenarios in the urban energy industry, including fixed monitoring points, preset docking points of inspection robots, and fixed shooting points of manual inspection, a calibration library is constructed. Each collection point is assigned a unique ID, a focus mode, and a predefined focus area. The predefined focus area is a pre-ROI, which is the area of ​​equipment and components that need to be monitored at that point. The library also includes a baseline image set of the point under normal conditions, a standard hazard type library corresponding to the point, and compliance regulations. Real-time matching of acquired images and calibration information: When an image of a potential hazard is acquired, the ID of the acquisition point corresponding to the image is obtained simultaneously, and all calibration information corresponding to that point is retrieved from the calibration library. Pre-focused region focus mode locking: The focus mode is adaptively switched according to the ID to complete the core region extraction and remove irrelevant background regions; the focus mode includes a fixed region mode and an anomaly comparison mode; the fixed region mode directly crops the acquired image based on the mapped predefined focus region, retaining only the core monitoring region as the image to be processed; the anomaly comparison mode extracts the global scene feature fingerprint of the acquired image through the DINO-V2 model, performs cosine similarity matching with the manually calibrated calibration library, automatically matches the corresponding manually calibrated point, obtains the matching ID, and then performs pre-ROI acquisition according to the focus mode.

[0039] The acquired images undergo viewpoint normalization processing. Based on the unified coordinate system after viewpoint normalization, the manually calibrated predefined reference coordinates of the region of interest are directly mapped to the normalized image. The viewpoint normalization processing is as follows: Basic attribute labeling of inspection points: Assign a unique ID to each inspection point, label the point name, the scene to which it belongs and the area of ​​responsibility, and complete the manual labeling of the point location; Baseline image acquisition: Manually take one baseline main image of the location under normal conditions, and no less than nine auxiliary baseline images under different lighting conditions, angles, and equipment operating conditions; Pre-monitoring area labeling: On the baseline main map, mark the baseline coordinates of the equipment or component areas that need to be monitored by selecting them with a box.

[0040] like Figure 9 As shown, a multimodal large-scale model hazard identification system for the urban energy industry is provided to implement a multimodal large-scale model hazard identification method for the urban energy industry. The system uses Qwen-VL or LLaVA-1.5 open-source multimodal large-scale models as the base model. The system includes: The multimodal data acquisition module is used to collect multi-source data such as hidden danger image data, equipment operation logs, and on-site text feedback at urban energy operation sites, and supports access from various devices such as cameras, inspection robots, and mobile terminals. The data preprocessing module communicates with the multimodal data acquisition module to complete image ROI extraction, filtering, cropping and enhancement, as well as text field splitting, cleaning and semantic encoding, and output standardized ROI feature sets and structured text field sets; The multi-dimensional fine-tuning module communicates with the data preprocessing module and has a built-in ROI-oriented multi-branch LoRA fine-tuning framework and field-level differentiated LoRA fine-tuning module group. It is used to complete the precise fine-tuning of local visual features and field-level semantic features, and output global image fusion features and field-specific text features. The cross-modal fusion and decision module communicates with the multi-dimensional fine-tuning module to complete the cross-modal deep fusion of image and text features, output structured hazard identification results, and at the same time link with the hazard knowledge base to complete feature matching and generate standardized rectification suggestions. The hazard knowledge base module communicates with the cross-modal fusion and decision-making module to store standard data, normative clauses and handling cases of hazards in the urban energy industry, and supports dynamic updates and rapid feature matching. The results output and closed-loop management module communicates and connects with the cross-modal fusion and decision-making module to visualize the identification results and rectification suggestions. It supports data export, hazard warning, rectification progress tracking and archiving, and realizes closed-loop management of the entire process of hazard disposal.

[0041] like Figure 10 As shown, the system also includes an image acquisition location calibration unit, which is used to complete the pre-calibration of inspection points, the construction and maintenance of the calibration library, and the real-time matching of acquired images and calibration information, and output the pre-interest area; the image acquisition location calibration unit is connected to the data preprocessing module; the multi-dimensional fine-tuning module has a built-in multi-branch LoRA fine-tuning framework for ROI guidance and calibration point adaptation.

[0042] Based on the ROI's ID, LoRA branches are grouped for management. Points within the same scene type and the same hazard type library share underlying visual feature weights, significantly reducing redundant training parameters and improving fine-tuning efficiency. Local feature enhancement for calibration adaptation: Each LoRA branch's feature enhancement submodule, combined with the hazard type library calibrated by the corresponding point ID, uses a CBAM convolutional block attention module to specifically enhance the fine-grained texture features and high-level semantic features corresponding to key monitored hazards at that point. For example, for points calibrated as "pressure gauge over-range," the feature weights of the dial scale and pointer area are enhanced; for points "pipeline interface leakage," the texture features of the interface sealing area are enhanced, while suppressing background noise in non-interested areas, improving the targeting of feature learning. Adaptive low-rank adaptation: Each LoRA branch's low-rank adaptation submodule adaptively adjusts the rank according to the complexity of the hazard calibrated at the point. (4-16 range), simple scenarios (such as instrument readings) Set to 4 for complex scenarios (such as pipe corrosion and cable aging). Set to 16, scaling factor The value is uniformly set to 16, applicable only to the attention layer of the visual encoder. The matrix is ​​decomposed into low rank and its parameters are fine-tuned to minimize training cost while maintaining feature representation capability.

[0043] Application Example 1: This application example uses the scenario of inspecting an urban underground gas pipeline network. The basic multimodal large model adopts the Qwen-VL-7B open-source model. The specific implementation steps are as follows: S1. Dataset Construction and Pre-training: A dedicated dataset for urban gas pipeline network hazards was constructed. The image dataset contains 12,000 samples, covering 6 major categories and 28 subcategories of common gas pipeline network hazards, including pipeline corrosion, weld cracking, valve leakage, interface loosening, pipeline occupancy, and anti-corrosion layer damage. Each sample is labeled with its bounding box, hazard category, risk level, and semantic label. The text dataset contains 8,000 sets of corresponding hazard text data, covering all elements of hazard name, hazard description, and rectification suggestions, matching the requirements of the "Safety Technical Regulations for Operation, Maintenance and Emergency Repair of Urban Gas Facilities" CJJ51-2016. Based on the image dataset, the YOLOv8s model was pre-trained. The pre-training batch size was set to 16, the optimizer was AdamW, the initial learning rate was set to 1e-3, the training epochs were set to 100, and the CIoU loss function was used. The final model achieved an mAP@0.5 of 96.3% on the test set.

[0044] S2. Data Preprocessing: For the on-site images of the gas pipeline network collected during inspections, the images are processed using a pre-trained YOLOv8s model to generate candidate ROIs. A confidence threshold of 0.5 is set to filter out valid ROIs. The valid ROIs are then cropped and uniformly scaled to 224×224 pixels. Data augmentation is performed through random horizontal flipping, ±15% brightness adjustment, and Gaussian noise addition. Four augmented samples are generated for each original ROI, forming an ROI feature set. For the text data on potential hazards in the gas pipeline network, a gas industry dictionary containing 1200 professional terms is constructed, along with three anchor keyword libraries for hazard names, descriptions, and rectification suggestions. An Aho-Corasick algorithm is used to build a rule engine, accurately splitting the original text into three core fields. Jieba word segmentation is used to complete word segmentation, removing stop words and meaningless characters. A BERT-wwm pre-trained model is used for semantic encoding, generating a 768-dimensional standardized semantic vector to form a structured text field set.

[0045] S3. Multi-dimensional Low-Rank Parameter Fine-tuning: Construct an ROI-oriented multi-branch LoRA fine-tuning framework, configuring an independent LoRA branch for each valid ROI. The branches adopt a two-level architecture of CBAM feature enhancement + low-rank adaptation, and LoRA rank... =8, scaling factor =16, dropout=0.1, freeze the Qwen-VL visual encoder backbone network, and fine-tune only the Q and V matrices of the attention layer; calculate the attention weights for each ROI feature, and set the weight coefficients to 16. =0.5、 =0.35、 =0.15, weight normalization is performed through Softmax, and the weighted summation of all ROI features is performed to generate global image fusion features.

[0046] Construct a group of field-level differentiated LoRA fine-tuning modules: Hazard Name Fine-tuning Module: Adopts a 4-head keyword attention + semantic alignment structure, LoRA rank r=4, learning rate 1e-4, iteration steps 1000, introduces the standard category semantic vector library of gas pipeline network hazards, and uses contrastive loss to constrain semantic alignment; Hazard description fine-tuning module: Employs a Bi-LSTM context modeling + gated feature selection architecture, LoRA rank =8, learning rate 2e-4, number of iterations 2000; The rectification suggestion fine-tuning module adopts logical modeling + standardized constraint structure, LoRA rank. =16, learning rate 3e-4, number of iterations 3000, introduce the CJJ51-2016 specification semantic vector library, and use joint loss function to constrain specification compliance.

[0047] All three modules freeze the Qwen-VL language encoder backbone network, only fine-tuning the low-rank parameters of the attention layer and feedforward network layer, and output the exclusive text features of each field after completion.

[0048] S4. Cross-modal feature fusion and decision output: The global image fusion features are mapped to 768 dimensions through 1×1 convolution, which is consistent with the dimension of the text features. LayerNorm is used to complete the distribution calibration. The image features and text features are sequentially concatenated and input into a cross-modal fusion layer composed of 3 Transformer encoders. The features are enhanced by bidirectional interaction through 8-head self-attention and dual-path cross attention, and the fused multimodal features are output.

[0049] The multimodal feature input classification and localization submodule is integrated to output a structured recognition result including hazard name, location, category, risk level, and confidence level. At the same time, the cosine similarity algorithm is used to match the integrated features with the gas pipeline network hazard knowledge base, with the matching degree threshold set to 0.85.

[0050] S5. System Deployment and Closed-Loop Management: This invention will be deployed on the intelligent inspection platform of the city gas company, connecting with inspection robots, mobile inspection terminals, and pipeline monitoring cameras to realize real-time collection, identification, early warning, and issuance of rectification instructions for on-site hidden danger data, track the rectification progress and review results, and achieve closed-loop management of the entire process of hidden danger control.

[0051] In this application example 1, regarding the potential hazard of loose valve interfaces in an underground pipeline network, the model recognition confidence level reached 98.7%, the structured output elements were complete, and the matching degree with the knowledge base reached 0.92. The corresponding standard clauses were directly invoked to generate standardized rectification suggestions: "1. Immediately close the upstream and downstream control valves of the valve and depressurize the interface; 2. Disassemble the interface, replace the sealing gasket, and clean and protect the threads from corrosion; 3. Retighten the interface according to the standard torque and conduct an airtightness test. The test pressure should meet the requirements of CJJ51-2016 standard; 4. Review the rectification within 3 days after completion to confirm that there is no potential leakage hazard."

[0052] Application Example 2: Taking the fixed inspection scenario of a 10kV substation in a city as an example, the basic multimodal large model adopts the Qwen-VL-7B open source model, and the specific steps are as follows: Pre-calibration of monitoring points and construction of calibration library: For the eight fixed monitoring points in the target power distribution room (low-voltage switchgear incoming line switch, capacitor bank, transformer, busbar joint, grounding switch, cable terminal head, temperature and humidity instrument, and fire-fighting equipment), each point is assigned a unique ID to complete full-dimensional calibration: the installation position, shooting angle, and field of view of the camera at each point are calibrated to ensure the consistency of the collected image perspective; the predefined areas of interest at each point are accurately marked, for example, the low-voltage switchgear points are marked with three pre-ROIs: switch contacts, wiring terminals, and instrument display areas, with a calibration accuracy of ±3 pixels; 30 normal reference images of each point at different times, under different lighting conditions, and under different load conditions are collected to build a reference image set; the corresponding hidden danger type library for each point (such as switch overheating, loose wiring, instrument over-range, etc.) and the corresponding standard clauses of "Design Code for 10kV and Below Substations" GB50053 and "Electric Power Safety Work Regulations" are bound to complete the construction of the calibration library; Data preprocessing: Fixed cameras in the power distribution room collect images at a frequency of 5 minutes / time, and upload the location IDs synchronously. The system automatically matches the calibration library information, and uses a fixed region mode to crop out predefined regions of interest. The size of the images to be processed is reduced from the original 1920×1080 to an average of 300×300, and more than 90% of irrelevant backgrounds are removed. The cropped images to be processed are input into a pre-trained YOLOv8s model to generate candidate ROIs. The confidence threshold is set to 0.5. After filtering the effective ROIs, standardization and data augmentation are performed to generate ROI feature sets. At the same time, the accompanying inspection text data is structurally decomposed and semantically encoded to generate a structured text field set. Multi-dimensional low-rank parameter fine-tuning: An independent LoRA branch is configured for each effective ROI, grouped by location scenario, with branches within the same power distribution room scenario sharing underlying features; combined with the hazard type library calibrated for each location, the corresponding hazard features are enhanced through the CBAM module, adaptively adjusting the LoRA rank. Instrument-type points =4, Equipment connector type points =8; The optimized weight calculation formula is used to complete the ROI feature fusion and generate global image fusion features; At the same time, the three text fields are independently fine-tuned, and the rectification suggestion module optimizes the constraints in combination with the power specification clauses bound to the point and generates field-specific text features; Cross-modal fusion and decision output: Complete the cross-modal fusion of image and text features, output structured hazard identification results, and simultaneously match them with the hazard knowledge base to generate rectification suggestions that meet the requirements of the site specifications.

[0053] According to on-site testing, in this embodiment, the processing latency of a single image was reduced from 73ms to 31ms, greatly improving the processing efficiency; the accuracy rate of identifying small target hazards such as loose wiring and instruments exceeding their range reached 98.6%, and the false detection rate was reduced to 1.1%, fully meeting the stringent requirements of real-time inspection of power distribution rooms.

[0054] Application Example 2 presents a multimodal large-scale model hazard identification method and system for the urban energy industry, which has the following characteristics: 1. By using ROI-guided multi-branch LoRA fine-tuning to replace the traditional global fine-tuning mode, it accurately focuses on the core area of ​​the hidden danger and effectively filters out background interference from complex sites. Compared with global LoRA fine-tuning, it improves the accuracy of identifying small-sized hidden dangers, reduces the false detection rate and false negative rate, and significantly improves the reliability of the identification results.

[0055] 2. Achieve standardized and structured output of identification results: This invention achieves independent and accurate training of hazard names, descriptions, and rectification suggestions through field-level differentiated LoRA fine-tuning, completely avoiding information interference between fields, ensuring the completeness and standardization of structured text elements in the output, and can be directly parsed and reused by urban energy industry business management systems.

[0056] 3. Ensuring the compliance and feasibility of rectification suggestions: This invention, through a knowledge base-linked rectification suggestion generation mechanism, deeply binds the identification results with national, industry, and enterprise-level safety standards. The generated rectification suggestions achieve a 100% compliance rate, completely solving the problems of generalized and unfounded rectification suggestions in existing technologies, and providing authoritative and implementable guidance for on-site rectification.

[0057] 4. Extremely high fine-tuning efficiency and industry adaptability: The fine-tuning scheme of this invention only fine-tunes the low-rank parameters of the LoRA branch, freezes the backbone network, and the number of training parameters is only 0.5% of that of global fine-tuning, reducing training time by more than 85%. At the same time, it supports incremental fine-tuning. When adding new types of hazards, there is no need to retrain the entire model. The incremental fine-tuning time is ≤4 hours, which can quickly adapt to new hazard scenarios and business needs in the urban energy industry and has extremely strong industry adaptability.

[0058] 5. Achieve closed-loop management of the entire process of hazard control: This invention effectively connects the hazard identification and rectification process through structured output and knowledge base linkage. Combined with the result output and closed-loop management module, it realizes closed-loop management of the entire process of identification-assessment-rectification-review-archiving, and fully supports the implementation of the dual control system for safe production in the urban energy industry.

[0059] The above embodiments are merely preferred embodiments of the present invention. Therefore, all equivalent changes or modifications made to the structure, features and principles described in the claims of the present invention are included within the scope of the present invention.

Claims

1. A method for identifying potential hazards in a multimodal large-scale model for the urban energy industry, characterized by: include: S1. Data Preprocessing: Preprocess the image data and text data of potential hazards in the urban energy industry to obtain the ROI feature set of the hazard images and the structured text field set. S2. Multi-dimensional low-rank parameter fine-tuning: Construct a multi-branch LoRA fine-tuning framework based on ROI guidance, perform local visual feature fine-tuning on the image ROI feature set, and generate global image fusion features; construct a field-level differentiated LoRA fine-tuning module group, perform field-level independent semantic fine-tuning on the structured text field set, and generate exclusive text features for each field. S3. Cross-modal feature fusion and decision output: A dual-path cross-attention mechanism is used to perform cross-modal deep fusion of the global image fusion features and the specific text features of each field to obtain fused multimodal features; based on the fused multimodal features, a structured hazard identification result is output, and the fused multimodal features are matched with the preset urban energy industry hazard knowledge base to generate compliant and standardized rectification suggestions, completing the closed-loop output of the entire hazard identification process.

2. The multimodal large-scale model hazard identification method for the urban energy industry according to claim 1, characterized in that: During data preprocessing, the inspection points are pre-calibrated and a calibration library is constructed. Based on the calibration information in the calibration library, the image pre-interest region is locked, and a hazard image ROI feature set is generated by combining the target detection algorithm. During the multi-dimensional low-rank parameter fine-tuning, the hazard image ROI feature set is locally visually fine-tuned, and the feature attention weight is optimized by combining the calibration point information to generate global image fusion features.

3. The multimodal large-scale model hazard identification method for the urban energy industry according to claim 2, characterized in that: The calibration library is constructed and its application is detailed as follows: Calibration library construction: For fixed inspection scenarios in the urban energy industry, including fixed monitoring points, preset docking points of inspection robots, and fixed shooting points of manual inspection, a calibration library is constructed. Each collection point is assigned a unique ID, a focus mode, and a predefined focus area. The predefined focus area is a pre-ROI, which is the area of ​​equipment and components that need to be monitored at that point. The library also includes a baseline image set of the point under normal conditions, a standard hazard type library corresponding to the point, and compliance regulations. Real-time matching of acquired images and calibration information: When an image of a potential hazard is acquired, the ID of the acquisition point corresponding to the image is obtained simultaneously, and all calibration information corresponding to that point is retrieved from the calibration library. Pre-focused region focus mode locking: The focus mode is adaptively switched according to the ID to complete the core region extraction and remove irrelevant background regions; the focus mode includes a fixed region mode and an anomaly comparison mode; the fixed region mode directly crops the acquired image based on the mapped predefined focus region, retaining only the core monitoring region as the image to be processed; the anomaly comparison mode extracts the global scene feature fingerprint of the acquired image through the DINO-V2 model, performs cosine similarity matching with the manually calibrated calibration library, automatically matches the corresponding manually calibrated point, obtains the matching ID, and then performs pre-ROI acquisition according to the focus mode.

4. The multimodal large-scale model hazard identification method for the urban energy industry according to claim 3, characterized in that: The acquired images undergo viewpoint normalization processing. Based on the unified coordinate system after viewpoint normalization, the manually calibrated predefined reference coordinates of the area of ​​interest are directly mapped to the normalized image. The viewpoint normalization processing includes the calibration of basic point attributes, acquisition of reference images, and annotation of the pre-interest area.

5. The multimodal large-scale model hazard identification method for the urban energy industry according to claim 1, characterized in that: The image data of the potential hazard is preprocessed as follows: S11. Construct an image dataset of potential hazards in the urban energy industry, and annotate the category, bounding box, risk level, and semantic label of each hazard sample in the dataset; S12. Based on the dataset, complete the pre-training of the YOLOv8 object detection model. The pre-training uses the CIoU loss function. S13. The input potential hazard image is processed by the pre-trained YOLOv8 model to generate candidate ROIs. Each ROI contains location information, confidence score and preliminary category information. Valid ROIs are retained by confidence score screening. S14. After standardizing the effective ROIs, the final image ROI feature set with fine-tuning is generated. The preprocessing of the potential hazard text data is as follows: S15. Construct a professional terminology dictionary and anchor keyword library for potential hazards in the urban energy industry. The anchor keyword library includes three categories: anchor keywords for hazard names, anchor keywords for hazard descriptions, and anchor keywords for rectification suggestions. S16. The rule engine is built using the Aho-Corasick multi-pattern matching algorithm and combined with the anchor keyword matching mechanism to accurately split the original hidden danger text into three core fields: hidden danger name, hidden danger description, and rectification suggestions. S17. Clean and segment the text of each field after splitting, use the BERT-wwm pre-trained model for semantic encoding, generate semantic vectors, and form a structured text field set.

6. The multimodal large-scale model hazard identification method for the urban energy industry according to claim 1, characterized in that: The process of fine-tuning local visual features of the image ROI feature set is as follows: S21. Configure an independent LoRA fine-tuning branch for each valid ROI. The branch adopts a two-level architecture of feature enhancement and low-rank adaptation. All branches share the multimodal large model visual encoder backbone network. Freeze all parameters of the backbone network. S22. The feature enhancement submodule of each LoRA branch adopts the convolutional block attention module CBAM, which sequentially enhances the fine-grained texture features and high-level semantic features within the ROI through channel attention and spatial attention, and suppresses background noise. S23, the low-rank adaptor submodule of each LoRA branch, rank Set to 8, scaling factor Set to 16, only for the attention layer of the visual encoder. The matrix is ​​decomposed into low-rank components and its parameters are fine-tuned to extract local features of a single ROI. S24. Use an attention-weighted fusion strategy to aggregate all ROI features and calculate the initial attention weight for each ROI feature. S25. The initial attention weights are normalized using the Softmax function. Based on the normalized weights, all ROI features are weighted and summed to generate global image fusion features.

7. The multimodal large-scale model hazard identification method for the urban energy industry according to claim 1, characterized in that: The field-level differentiated LoRA fine-tuning module group is constructed as follows: The field-level differentiated LoRA fine-tuning module group includes a hazard name fine-tuning module, a hazard description fine-tuning module, and a rectification suggestion fine-tuning module. The three modules are independent of each other, and all of them freeze the multimodal large model language encoder backbone network, and only fine-tune the low-rank parameters of the attention layer and the feedforward network layer. The hazard name fine-tuning module adopts a keyword enhancement + semantic alignment dual-path structure, the hazard description fine-tuning module adopts a context modeling + feature selection architecture, and the rectification suggestion fine-tuning module adopts a logical modeling + normative constraint structure.

8. The multimodal large-scale model hazard identification method for the urban energy industry according to claim 1, characterized in that: The specific process of cross-modal feature fusion is as follows: the dimensions of the global image fusion features and the specific text features of each field are unified, and layer normalization is used to calibrate the distribution of the two modal features, outputting standardized image features and text features; the standardized image features and text features are concatenated to form a multimodal feature sequence, which is input into a cross-modal fusion layer composed of 3 Transformer encoders; the cross-modal fusion layer mines the element correlation within the feature sequence through a multi-head self-attention module, and introduces a dual-path cross-attention mechanism to calculate the attention weights of image features to text features and text features to image features, respectively, to achieve bidirectional feature interaction and enhancement, and finally output fused multimodal features.

9. A multimodal large-scale model hazard identification system for the urban energy industry, used to implement the multimodal large-scale model hazard identification method for the urban energy industry as described in any one of claims 1, characterized in that: The system includes a multimodal data acquisition module, a data preprocessing module, a multidimensional fine-tuning module, and a cross-modal fusion and decision-making module that are connected in sequence; the cross-modal fusion and decision-making module is connected in sequence to a hidden danger knowledge base module and a result output and closed-loop management module.

10. The multimodal large-scale hazard identification system for the urban energy industry according to claim 9, characterized in that: The system also includes an image acquisition location calibration unit, which is connected to the data preprocessing module; the multi-dimensional fine-tuning module has a built-in multi-branch LoRA fine-tuning framework that adapts to ROI guidance and calibration points.

Images