Multi-modal data integration and intelligent analysis system and device for power infrastructure platform

Abstract

The application discloses a kind of multi-modal data integration and intelligent analysis system and equipment of electric power infrastructure platform, and unifiedly builds the hierarchical architecture containing data acquisition layer, data processing layer, model training layer, intelligent analysis layer and application layer, is equipped with cross-layer two-way data backflow link, with offline training and online analysis double passage.This application covers structured, semi-structured, unstructured multi-source data of electric power infrastructure, completes data standardization management by multistage cleaning and exclusive preprocessing algorithm, realizes model efficient iteration relying on offline and incremental double passage, combines cross-modal multi-head attention mechanism to fuse image and text features, and realizes progress, quality, safety key information intelligent extraction and reasoning by matching custom BIO labeling system.The application solves the industry pain points of traditional electric power infrastructure data fragmentation, mode fragmentation, analysis lag and low efficiency of artificial research and judgment, can realize real-time monitoring and intelligent decision of construction site, and effectively improves infrastructure management and control precision and efficiency.

Description

Technical Field

[0001] This invention relates to the field of power infrastructure data processing technology, specifically to a multimodal data integration and intelligent analysis system and equipment. Background Technology

[0002] Against the backdrop of rapid development in the energy industry, power infrastructure projects are large-scale and highly complex, involving multiple stages such as design, construction, supervision, and operation and maintenance. Traditional infrastructure management methods rely heavily on manual records and paper documents, resulting in information silos, low data utilization efficiency, and insufficient decision support. With the advancement of information and intelligent technologies, the integration and intelligent analysis of multi-source heterogeneous data throughout the entire lifecycle of infrastructure projects has become crucial for improving project management efficiency and quality.

[0003] While existing infrastructure management platforms have incorporated some information technology, they still face the following challenges when processing multimodal data such as text, images, and videos: Data integration difficulties: Data from different sources and in different formats is difficult to manage and analyze in a unified manner.

[0004] Low information extraction efficiency: It relies on manual extraction of key information from a large number of documents, which is time-consuming, labor-intensive and prone to errors.

[0005] Insufficient decision support: The lack of in-depth analysis and intelligent recommendation functions based on multimodal data makes it difficult to support scientific decision-making.

[0006] Therefore, there is an urgent need for an infrastructure management and control platform that can efficiently integrate and analyze multimodal data to improve project management efficiency and decision-making level. Summary of the Invention

[0007] The multimodal data integration and intelligent analysis system and method proposed in this invention can at least solve one of the technical problems in the background art.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal data integration and intelligent analysis system for a power infrastructure platform includes: a data acquisition layer, a data processing layer, a model training layer, an intelligent analysis layer, and an application layer connected in sequence, and is equipped with a bidirectional data return link, enabling cross-layer interaction and offline / online dual-pathway operation; The data acquisition layer transmits the collected raw multimodal data stream to the data processing layer. The data processing layer outputs the standardized multimodal dataset, which has been cleaned, aligned, labeled, and structured indexed, to the model training layer. Based on the standardized multimodal dataset, the model training layer completes the fusion of the multimodal dataset and distributes it to the intelligent analysis layer. The intelligent analysis layer delivers the structured analysis results, feature vectors, alarm information, cross-modal retrieval results, and prediction indicators to the application layer through RESTful-API, WebSocket real-time push, and RPC interface.

[0009] Furthermore, the present invention also includes: the pathway between the data processing layer and the model training layer is divided into an offline pathway and an incremental pathway; The offline pathway is responsible for packaging the dataset and pushing it to the training dataset repository, while the incremental pathway is responsible for continuously synchronizing newly added labeled samples and supporting incremental fine-tuning of the model; it includes metadata such as modality type, timestamp, spatial coordinates, label, and data quality metrics. The data processing layer and the intelligent analysis layer establish a data transmission path, so that the pre-processed streaming multimodal data in real-time business scenarios can be directly sent to the intelligent analysis layer without passing through the training layer.

[0010] Furthermore, the original multimodal data of this invention includes: Structured data includes construction progress data, quality inspection data, and cost data; Semi-structured data: including construction logs, supervision reports, and design change orders; Unstructured data includes design drawings, on-site images, video surveillance streams, and voice recordings.

[0011] Furthermore, the standardized multimodal dataset acquisition method of the present invention includes: Let the original dataset be... The cleaning function clean() satisfies: In the formula, valid(d) i To verify data integrity, To test data repeatability; Let the image In the formula, H is the image height, W is the image width, and C is the number of image channels; Preprocessing is performed, and the specific process is as follows: Grayscale conversion:

[0012] Binarization: Where θ is the adaptive threshold; Gaussian noise reduction: ,in, The kernel is a Gaussian kernel, and σ is the smoothing parameter. This is a two-dimensional convolution operation; For text sequences Perform word segmentation Further marking of parts of speech Named entity recognition ; in, This refers to the m-th unit of the original text, where m is the length of the original sequence. This refers to the k-th word element after word segmentation, where k is the length of the word element sequence after segmentation. Let $j$ be the $i$-th word element obtained from word segmentation, $POS()$ be the part-of-speech tagging function, and $j$ be the total number of entities identified in the text. Let j be the entity type of the j-th entity. The text value corresponding to the j-th entity is selected.

[0013] Furthermore, the multimodal data fusion method of the present invention includes: Let the multimodal labeled dataset be... ,in, For the image data of the i-th sample; For the i-th sample text data, For word embedding dimension. Label the samples, where Y is the set of labels and N is the total number of samples in the dataset; Given multimodal fusion conditions , () is a multimodal fusion function, the target output is a text sequence. T is the length of the target sequence token.

[0014] Define the loss function for conditional language modeling as:

[0015] The total training loss is obtained by averaging all samples and introducing a regularization term:

[0016] Let CLM loss be the value of the i-th sample. For regularization terms, These are the regularization weight coefficients; The AdamW optimizer is used to refine parameters, and further deep fusion and intelligent analysis of multimodal data are performed. The process is as follows: Let the image features be Text features ; Transmembrane attention computation: ; ; ;

[0017] In the formula, , , For learnable projection matrices, For single-head attention dimension, For the number of images, For the number of texts; Multi-head attention fusion output:

[0018] Where h is the number of attention heads, To output the projection matrix, ; The fused multimodal is as follows: .

[0019] Furthermore, the method for obtaining predictive indicators by the intelligent analysis layer of the present invention includes: The input multimodal data is: ; Key information is extracted using the Extracct() function, which is as follows: In the formula For the names of the information fields to be extracted, To extract values, Score the confidence level; Information extraction is performed using sequence labeling. The BIO (Begin-Inside-Outside) labeling scheme is applied to the text sequence to define a tag set. ; Among them, B- For the starting token of the corresponding business field entity, I- O is the internal continuation token of the corresponding business field entity, and O is an irrelevant token that does not belong to any target extracted field; Let the hidden state representation of the t-th text token obtained by the multimodal encoder be: Conditional probability of sequence labeling:

[0020] In the formula, The label projection weight matrix, For bias vectors, The latent features of the text token after fusing image information.

[0021] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0022] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0023] As can be seen from the above technical solution, the present invention has the following beneficial effects: Improve management efficiency: By using automation and intelligent methods, manual operation and intervention are reduced, thereby improving management efficiency.

[0024] Reduced management costs: Reduced reliance on manual recording and organization, thus reducing labor and time costs.

[0025] Enhancing the scientific nature of decision-making: Deep analysis and intelligent recommendations based on multimodal data provide managers with more comprehensive and accurate information support, thereby enhancing the scientific nature and rationality of decision-making.

[0026] Improve safety risk management: Enhance infrastructure safety management by utilizing system-wide hazard identification data, risk warning information, and AI video alarm information. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the multimodal data integration and intelligent analysis system of the power infrastructure platform of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0029] like Figure 1 As shown, the multimodal data integration and intelligent analysis system of the power infrastructure platform described in this embodiment includes a data acquisition layer, a data processing layer, a model training layer, an intelligent analysis layer and an application layer connected in sequence. It is also equipped with a bidirectional data return link, which can realize cross-layer interaction and offline / online dual-path. The data acquisition layer transmits the collected raw multimodal data stream to the data processing layer. The data processing layer outputs the standardized multimodal dataset, which has been cleaned, aligned, labeled, and structured indexed, to the model training layer. Based on the standardized multimodal dataset, the model training layer completes the fusion of the multimodal dataset and distributes it to the intelligent analysis layer. The intelligent analysis layer delivers the structured analysis results, feature vectors, alarm information, cross-modal retrieval results, and prediction indicators to the application layer through RESTful-API, WebSocket real-time push, and RPC interface.

[0030] The pathway between the data processing layer and the model training layer is divided into offline pathway and incremental pathway; The offline pathway is responsible for packaging the dataset and pushing it to the training dataset repository, while the incremental pathway is responsible for continuously synchronizing newly added labeled samples and supporting incremental fine-tuning of the model; it includes metadata such as modality type, timestamp, spatial coordinates, label, and data quality metrics.

[0031] The data processing layer and the intelligent analysis layer establish a data transmission path, allowing pre-processed streaming multimodal data in real-time business scenarios to be directly sent to the intelligent analysis layer for inference without passing through the training layer, thus supporting online real-time analysis.

[0032] The data acquisition layer collects multimodal data, including: Structured data includes construction progress data (Gantt chart data, milestone data), quality inspection data (concrete strength test values, weld inspection pass rate, etc.), and cost data (budget execution rate, change fee statistics). Semi-structured data includes construction logs (including fields such as date, personnel, procedures, and equipment usage), supervision reports (including inspection items, rating levels, and rectification opinions), and design change orders (including reasons for changes, scope of impact, and approval status). Unstructured data includes design drawings (CAD files, PDF drawings), on-site images (construction photos, hidden works record photos), video surveillance streams (real-time video of the construction site, playback recordings), and audio recordings (construction meeting recordings, on-site intercom records).

[0033] The data processing layer cleans and formally defines the raw multimodal data uploaded by the data acquisition layer. Let the original dataset be... The cleaning function clean() satisfies: In the formula, valid(d) i To verify data integrity, To test data repeatability.

[0034] Let the image In the formula, H is the image height, W is the image width, and C is the number of image channels; Preprocessing is performed, and the specific process is as follows: Grayscale conversion:

[0035] Binarization: Where θ is the adaptive threshold; Gaussian noise reduction: ,in, The kernel is a Gaussian kernel, and σ is the smoothing parameter. This is a two-dimensional convolution operation.

[0036] For text sequences Perform word segmentation Further marking of parts of speech Named entity recognition ,in, This refers to the m-th unit of the original text, where m is the length of the original sequence. This refers to the k-th word element after word segmentation, where k is the length of the word element sequence after segmentation. Let $j$ be the $i$-th word element obtained from word segmentation, $POS()$ be the part-of-speech tagging function, and $j$ be the total number of entities identified in the text. Let j be the entity type of the j-th entity. The text value corresponding to the j-th entity is selected.

[0037] Multimodal feature extraction is performed using mathematical models, and features are extracted from images using CNNs. In the formula It is a convolutional neural network. This represents all trainable weights of the CNN; Image feature dimension Use bidirectional LSTM to extract features from the text:

[0038]

[0039] in, Let be the word embedding vector of the t-th word. This is the hidden state from the previous moment. All trainable parameters of the bidirectional LSTM This represents the hidden state at the last moment of the forward LSTM. This represents the hidden state at the last moment of the inverse LSTM. This represents the feature dimensions of the concatenated text.

[0040] Each layer interacts efficiently through data interfaces and APIs, forming a complete closed loop for data processing and analysis.

[0041] The model training layer is based on a large multimodal model, which trains and analyzes multimodal data. The process is as follows: Let the multimodal labeled dataset be... ,in, For the image data of the i-th sample; For the i-th sample text data, For word embedding dimension. Label the samples, where Y is the set of labels and N is the total number of samples in the dataset.

[0042] The system automatically pairs design drawings and construction logs for the same project and time period, using timestamps and project IDs for matching; it also uses existing project report conclusions (such as quality assessment levels) as weak monitoring signals to automatically label multimodal data for the corresponding time period.

[0043] Given multimodal fusion conditions , () is a multimodal fusion function, the target output is a text sequence. T is the length of the target sequence token.

[0044] Define the loss function for conditional language modeling as:

[0045] The total training loss is obtained by averaging all samples and introducing a regularization term:

[0046] Let CLM loss be the value of the i-th sample. For regularization terms, This is the regularization weight coefficient.

[0047] The AdamW optimizer is used to refine parameters, and further deep fusion and intelligent analysis of multimodal data are performed. The process is as follows: Let the image features be Text features ; Transmembrane attention computation: ; ; ;

[0048] In the formula, , , For learnable projection matrices, For single-head attention dimension, For the number of images, The number of texts.

[0049] Multi-head attention fusion output:

[0050] Where h is the number of attention heads, To output the projection matrix, .

[0051] The fused multimodal is as follows:

[0052] The analysis process is as follows: Modal encoding: Deep features are extracted from image and text data using their respective encoders; Cross-modal alignment: Semantic alignment between image regions and text fragments is established through an attention mechanism; Fusion inference: The fused multimodal features are input into the downstream task network for inference; Result output: Structured analysis results are generated, including classification labels, key information extraction results, anomaly scores, etc.

[0053] The intelligent analysis layer extracts key information such as project progress, quality, and safety from multimodal data, and combines it with multimodal models to conduct in-depth analysis of several projects, providing intelligent decision support. The process is as follows: The input multimodal data is: ; Key information is extracted using the Extracct() function, which is as follows: In the formula For the names of the information fields to be extracted, To extract values, Calculate the confidence level.

[0054] Information extraction is performed using sequence labeling. The BIO (Begin-Inside-Outside) labeling scheme is applied to the text sequence to define a tag set. ; Among them, B- For the starting token of the corresponding business field entity, I- O is the internal continuation token of the corresponding business field entity, and O is an irrelevant token that does not belong to any target extracted field.

[0055] Let the hidden state representation of the t-th text token obtained by the multimodal encoder be: Conditional probability of sequence labeling:

[0056] In the formula, The label projection weight matrix, For bias vectors, The latent features of the text token after fusing image information.

[0057] Taking a certain construction log segment and its corresponding on-site photos as an example, the extraction results are shown in Table 1 below: Table 1. Feature extraction results of a certain construction log segment and corresponding on-site photos

[0058] By comparing and analyzing historical and real-time data, anomalies can be identified promptly during project execution. Based on the current project status and historical data, scientific and reasonable decision-making suggestions can be provided to managers.

[0059] Here is an example of a decision recommendation: Scenario 1: Schedule Deviation Warning; Current status: The installation progress of the main plant steel structure is deviating by 12% (lagging behind); Historical data: Installation efficiency has been declining for the past two weeks; Recommended Decision: a) Add an installation team (the estimated recovery time is 5 days, and the additional cost is about 80,000 yuan). b) Adjust the flow rhythm of subsequent processes and compress non-critical paths (it is estimated that the recovery process will take 7 days with no additional cost). c) Maintain the status quo and apply for an extension of the construction period (the total construction period is expected to be extended by 10 days). System recommendation: Option b, with the best overall cost-effectiveness (based on the evaluation of reinforcement learning strategies in similar historical scenarios).

[0060] Scenario 2: Handling quality anomalies; Current status: The strength test value of the concrete test block for the foundation bolts of Tower No. 3 is 8% lower than the design value; Historical data: The same batch of concrete passed tests at other tower foundations; Recommended Decision: a) Conduct a comprehensive non-destructive test on the tower base to assess whether rework is necessary; b) Expanded sampling and testing will be conducted only on this batch of concrete; c) Record the deviation without further processing; System recommendation: Option a, based on the risk assessment model (full detection is triggered when the safety factor is <1.2).

[0061] Scenario 3: Security Risk Warning; Current status: AI video detected 3 instances of workers not wearing safety helmets at the construction site, and a weather warning indicates thunderstorms are expected within the next 2 hours; Historical data: One minor safety incident occurred in this work area last month; Recommended Decision: a) Immediately suspend all outdoor operations and organize a short safety training session; b) Notify the safety officer to increase patrols and not suspend operations; c) Warning notifications are only sent via system push notifications; System recommendation: Option a, security risk score 85 / 100 (exceeding the threshold of 70), immediate intervention is recommended.

[0062] To verify the effectiveness of this system, a test dataset was constructed based on historical data from a large-scale power infrastructure project, and simulation experiments were conducted. Data source: Full-cycle data of a thermal power infrastructure project from 2019 to 2025 Data scope: 12,846 design drawings, 35,680 construction logs, 8,942 supervision reports, and approximately 1,200 hours of video surveillance footage. Labeling results: Labeling was performed by 5 experts with experience in infrastructure management, and the labeling consistency (Fleiss' Kappa) reached 0.82. Training / validation / test set split ratio: 70% / 15% / 15% Parameter settings: Hardware environment: 4×NVIDIA A100 (80GB) GPU cluster; Software framework: PyTorch 2.0, based on the LLaMA-2 architecture for multimodal large models; Training hyperparameters: learning rate η=3e-4, batch size=128, training epochs=50; Baselines for comparison: Text-CNN (pure text), ResNet-152 (pure image), and LateFusion (simple concatenation and fusion). Experimental results are shown in Table 2; Table 2. Historical Data Test Results of a Large-Scale Power Infrastructure Project

[0063] Experimental results show that the method of the present invention significantly outperforms the baseline method in terms of accuracy, recall, and F1 score: compared with the LateFusion method, the F1 score is improved by 8.2 percentage points; compared with the pure text method (Text-CNN), the F1 score is improved by 14.0 percentage points; and compared with the pure image method (ResNet-152), the F1 score is improved by 19.2 percentage points.

[0064] In summary, this system specifically covers three core data categories—structured, semi-structured, and unstructured—in the power infrastructure scenario. It comprehensively encompasses all business data, including construction progress, quality and cost, supervision logs, design documents, on-site images, monitoring videos, and conference audio, completely breaking down the barriers of traditional power infrastructure systems characterized by single data sources, isolated modalities, and incomplete data coverage. Simultaneously, through integrity verification and duplicate removal rules, combined with formal preprocessing algorithms such as image grayscale conversion, binarization, Gaussian denoising, and text segmentation, part-of-speech tagging, and named entity recognition, the system standardizes and reconstructs the original, chaotic, multimodal data in a structured manner. On the one hand, it effectively removes invalid and noisy data and corrects data defects, significantly improving the overall quality and usability of original power infrastructure data. On the other hand, it achieves unified indexing and standardized output of data in different formats, dimensions, and scenarios, providing a high-quality, highly consistent data foundation for subsequent model training and intelligent analysis, thus solving the industry pain point of the difficulty in uniformly utilizing multi-source data in power infrastructure.

[0065] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0066] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0067] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the multimodal data integration and intelligent analysis system and method for any of the power infrastructure platforms described in the above embodiments.

[0068] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0069] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0070] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0071] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0072] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal data integration and intelligent analysis system for power infrastructure platform, characterized in that, Comprise: Comprise data acquisition layer, data processing layer, model training layer, intelligent analysis layer and application layer connected in turn, with data backflow bidirectional link, can realize cross-layer interaction, offline / online double channel; Data acquisition layer transmits the collected original multi-modal data stream to data processing layer, data processing layer outputs the standardized multi-modal data set after cleaning, alignment, labeling and structured indexing to model training layer; Model training layer completes multi-modal data set fusion based on standardized multi-modal data set and issues to intelligent analysis layer; Intelligent analysis layer pushes the structured analysis results, feature vectors, alarm information, cross-modal search results and prediction indexes to application layer through RESTful-API, WebSocket real-time push and RPC interface.

2. The multi-modal data integration and intelligent analysis system of power infrastructure platform of claim 1, wherein, Also comprise: The channel between data processing layer and model training layer is divided into offline channel and incremental channel; Offline channel is responsible for packaging data set and pushing to training data set warehouse, incremental channel is responsible for continuously synchronizing new labeled samples and supporting model incremental fine-tuning; accompanying metadata: modal type, timestamp, spatial coordinate, labeling label, data quality index; Data processing layer and intelligent analysis layer establish data transmission channel, and send preprocessed stream multi-modal data in real-time business scene to intelligent analysis layer without training layer.

3. The multi-modal data integration and intelligent analysis system of power infrastructure platform of claim 1, wherein, Raw multi-modal data includes: Structured data: including construction progress data, quality detection data, cost data; Semi-structured data: including construction log, supervision report, design change order; Unstructured data: including design drawings, field images, video monitoring stream, voice recording.

4. The multi-modal data integration and intelligent analysis system of power infrastructure platform of claim 1, wherein, Standardized multi-modal data set acquisition method includes: Let the original dataset be The cleaning function clean() satisfies: where valid(d i ) is a test for data integrity, is a test for data redundancy; Let image where H is the image height, W is the image width, and C is the number of image channels. Preprocessing, the specific process is as follows: grayscale: Binarization: where θ is an adaptive threshold value; Gaussian denoising: where, is a Gaussian kernel, and σ is a smoothing parameter, is a two-dimensional convolution operation; For text sequences Perform word segmentation , mark part of speech Named entity recognition ; wherein, is the mth unit of the original text, m is the length of the original sequence, is the kth word unit after segmentation, k is the length of the word unit sequence after segmentation, is the i th word unit obtained by segmentation, POS() is a part-of-speech tagging function, j is the total number of entities recognized by the text, is the entity type of the jth entity, is the value of the text corresponding to the jth entity.

5. The multi-modal data integration and intelligent analysis system of power infrastructure platform of claim 1, wherein, Multi-modal data fusion method includes: Let the multi-modal annotated dataset be wherein, is the i-th sample image data; is the i-th sample text data, is the word embedding dimension, is the sample annotation label, Y is the label set, and N is the total number of samples in the dataset. Given the multi-modal fusion condition , () is a multi-modal fusion function, the target output text sequence T is the target sequence token length; The conditional language modeling loss function is defined as: Take the average of all samples and introduce a regularization term to get the total training loss: CLM loss for the i-th sample, regularization term, regularization weight coefficient; AdamW optimizer is used to execute parameter change, perform multi-modal data deep fusion and intelligent analysis, the process is as follows: Let the image feature be , and the text feature ; Transmembrane state attention computation: ; ; ; wherein, , , is a learnable projection matrix, is an attention head dimension, is a number of images, is a number of texts; Multi-head attention fusion output: wherein h is the number of attention heads, is the output projection matrix, ; The fused multi-modal is: 。 6. The multi-modal data integration and intelligent analysis system of power infrastructure platform of claim 1, wherein: Intelligent analysis layer obtains prediction index method includes: The input multimodal data is: ; The key information is extracted by a function Extract(), wherein the Extract() function is: , wherein is a field name of the extracted information, is an extracted value, is a confidence score; Information extraction is performed by sequence labeling, using the BIO tagging method for text sequences, defining a label set ; where B- is the start token for the corresponding business field entity, I- is the internal continuation token for the corresponding business field entity, and O is an irrelevant token that does not belong to any target extraction field.​​ Let the hidden state representation of the t-th text token obtained by the multi-modal encoder be denoted as The label conditional probability of sequence labeling is: In the formula, is a label projection weight matrix, is a bias vector, is a text token hidden feature after fusing image information. 7.A computer device, comprising a memory and a processor, and characterized in that, The memory stores a computer program, and the computer program is executed by the processor to make the processor execute the method in any one of claims 1-6.

Images