Multi-source data fusion method and device applied to power grid production field, equipment and medium

Abstract

The invention discloses a multi-source data fusion method, device, equipment and medium applied to a power grid production field, applied to the technical field of data fusion, and the method comprises the steps: obtaining real-time data of a target power grid production field, carrying out the feature extraction of the real-time video data, and obtaining video space features; performing feature extraction on the real-time audio data to obtain audio acoustic features; performing feature extraction on the real-time environment data to obtain environment time sequence features; constructing a cross-modal relation graph based on the video spatial features, the audio acoustic features and the environment time sequence features; inputting the cross-modal relation graph into a constructed cross-modal fusion model for iterative transmission processing to obtain cross-modal space-time fusion data; and performing output processing on the cross-modal space-time fusion data to obtain a fusion output result. According to the method provided by the invention, the technical problems of insufficient complex anomaly perception capability, low recognition precision and high false alarm rate caused by lack of deep feature interaction in the prior art can be solved.

Description

Technical Field

[0001] This invention relates to the field of data fusion technology, and in particular to a method, apparatus, equipment and medium for multi-source data fusion applied in power grid production sites. Background Technology

[0002] With the rapid development of smart grid and Internet of Things technologies, higher requirements are being placed on the processing of multi-source data in power grid production sites, material management, asset inventory and other aspects.

[0003] In existing technologies, multi-source data fusion methods are loosely coupled and post-fusion, meaning that data of different modalities, such as video, audio, and environmental data, are usually collected, transmitted, and processed independently, and then simply correlated or spliced ​​at the decision-making level. This lacks deep, feature-level interaction, which leads to low accuracy and high false alarm rate in the identification of complex anomalies. Summary of the Invention

[0004] This invention provides a multi-source data fusion method, device, equipment, and medium for use in power grid production sites. It addresses the technical problems in existing technologies, such as insufficient ability to perceive complex anomalies, low identification accuracy, and high false alarm rate due to the lack of deep feature interaction. This provides key technical support for achieving real-time, accurate, and intelligent safety monitoring and operation and maintenance management in power grid production sites.

[0005] To address the aforementioned technical problems, this invention provides a multi-source data fusion method applied to power grid production sites, the method comprising: Acquire real-time data from the target power grid production site, wherein the real-time data includes at least real-time video data, real-time audio data, and real-time environmental data; Feature extraction is performed on the real-time video data to obtain video spatial features; feature extraction is performed on the real-time audio data to obtain audio acoustic features; feature extraction is performed on the real-time environmental data to obtain environmental temporal features; Based on the video spatial features, the audio acoustic features, and the environmental temporal features, a cross-modal relationship diagram of the target power grid production site is constructed; The cross-modal relationship graph is input into the constructed cross-modal fusion model for iterative transmission processing to obtain cross-modal spatiotemporal fusion data; The cross-modal spatiotemporal fusion data is processed to obtain the fusion output result.

[0006] As one preferred embodiment, the step of extracting features from the real-time video data to obtain video spatial features includes: The real-time video data is subjected to keyframe extraction and image enhancement processing in sequence to obtain a video frame sequence; The video frame sequence is used to extract features to obtain the video spatial features.

[0007] As one preferred embodiment, the step of extracting features from the real-time audio data to obtain audio acoustic features includes: The real-time audio data is subjected to frame-by-frame windowing processing to obtain an audio frame sequence; The audio frame sequence is transformed using the Fast Fourier Transform technique, and the resulting Mel spectrogram is used to extract features to obtain the audio acoustic features.

[0008] As one preferred embodiment, the step of extracting features from the real-time environmental data to obtain environmental temporal features includes: The real-time environmental data is preprocessed to obtain environmental time-series data; Long Short-Term Memory (LSTM) network technology is used to extract features from the environmental time-series data to obtain the environmental time-series features.

[0009] As one preferred embodiment, the step of constructing a cross-modal relationship diagram of the target power grid production site based on the video spatial features, the audio acoustic features, and the environmental temporal features includes: The video spatial features, the audio acoustic features, and the environmental temporal features are used as feature nodes to construct an initial node set; Spatiotemporal correlation analysis and semantic correlation analysis are performed sequentially on the initial node set to obtain the cross-modal relationship diagram of the target power grid production site.

[0010] As one preferred embodiment, the step of inputting the cross-modal relationship graph into the constructed cross-modal fusion model for iterative transfer processing to obtain cross-modal spatiotemporal fusion data includes: The cross-modal fusion model is constructed using a graph attention network; The cross-modal relationship graph is input into the cross-modal fusion model for iterative transfer processing to obtain cross-modal spatiotemporal fusion data.

[0011] As one preferred embodiment, the step of outputting the cross-modal spatiotemporal fusion data to obtain the fusion output result includes: The equipment status information of the target power grid production site is embedded into the cross-modal spatiotemporal fusion data to generate the final display image; The final displayed image is rendered in real time to generate the fused output result.

[0012] The present invention also provides a multi-source data fusion device for use in power grid production sites, the device comprising: The acquisition module is used to acquire real-time data from the target power grid production site, wherein the real-time data includes at least real-time video data, real-time audio data, and real-time environmental data; The extraction module is used to extract features from the real-time video data to obtain video spatial features; extract features from the real-time audio data to obtain audio acoustic features; and extract features from the real-time environmental data to obtain environmental temporal features. A construction module is used to construct a cross-modal relationship diagram of the target power grid production site based on the video spatial features, the audio acoustic features, and the environmental temporal features; The iteration module is used to input the cross-modal relationship graph into the constructed cross-modal fusion model for iterative transmission processing to obtain cross-modal spatiotemporal fusion data; The fusion module is used to process the cross-modal spatiotemporal fusion data to obtain the fusion output result.

[0013] The present invention also provides a multi-source data fusion device for use in power grid production sites, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the multi-source data fusion method for use in power grid production sites as described above.

[0014] The present invention further provides a computer-readable storage medium storing a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the multi-source data fusion method applied to power grid production sites as described above.

[0015] Compared with the prior art, the beneficial effects of the present invention are at least one of the following: This invention acquires real-time data from a target power grid production site, including at least real-time video data, real-time audio data, and real-time environmental data. Features are extracted from the real-time video data to obtain video spatial features; features are extracted from the real-time audio data to obtain audio acoustic features; features are extracted from the real-time environmental data to obtain environmental temporal features; a cross-modal relationship diagram of the target power grid production site is constructed based on the video spatial features, the audio acoustic features, and the environmental temporal features; the cross-modal relationship diagram is input into a constructed cross-modal fusion model for iterative transmission processing to obtain cross-modal spatiotemporal fused data; and the cross-modal spatiotemporal fused data is output to obtain a fused output result.

[0016] Compared with existing technologies, this invention constructs a cross-modal relationship graph, transforms the features of video, audio, and environmental data into graph nodes and explicitly models their relationships, and then uses a graph neural network for iterative message passing, enabling full interaction and enhancement of features at the deep semantic level. Furthermore, spatiotemporal alignment constraints are used to ensure fusion accuracy, thereby solving the technical problems of insufficient perception of complex anomalies, low recognition accuracy, and high false alarm rate caused by the lack of deep feature interaction. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a multi-source data fusion method applied to power grid production sites in one embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a multi-source data fusion device applied to a power grid production site in one embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a multi-source data fusion device applied to a power grid production site in one embodiment of the present invention; Figure label: Among them, 11. Acquisition module; 12. Extraction module; 13. Construction module; 14. Iteration module; 15. Fusion module; 21. Processor; 22. Memory. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] In the description of this invention, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0020] One embodiment of the present invention provides a multi-source data fusion method applied to power grid production sites. For details, please refer to [link to documentation]. Figure 1 , Figure 1 The diagram shown is a flowchart illustrating a multi-source data fusion method applied to power grid production sites according to one embodiment of the present invention. The method includes: S1: Acquire real-time data from the target power grid production site, wherein the real-time data includes at least real-time video data, real-time audio data, and real-time environmental data; S2: Extract features from the real-time video data to obtain video spatial features; extract features from the real-time audio data to obtain audio acoustic features; extract features from the real-time environmental data to obtain environmental temporal features; S3: Based on the video spatial features, the audio acoustic features, and the environmental temporal features, construct a cross-modal relationship diagram of the target power grid production site; S4: Input the cross-modal relationship graph into the constructed cross-modal fusion model for iterative transfer processing to obtain cross-modal spatiotemporal fusion data; S5: Perform output processing on the cross-modal spatiotemporal fusion data to obtain the fusion output result.

[0021] Specifically, real-time data from the target power grid production site is acquired, wherein the real-time data includes at least real-time video data, real-time audio data, and real-time environmental data.

[0022] Preferably, a 5-megapixel CMOS sensor supporting H.265 encoding and a maximum frame rate of 60fps is used to acquire real-time video data; a 4-microphone array with a sampling rate of 48kHz and a dynamic range of ≥110dB is used to acquire real-time audio data; and integrated temperature, humidity, illuminance, and triaxial acceleration sensors are used to acquire real-time environmental data.

[0023] Furthermore, feature extraction is performed on the real-time video data to obtain video spatial features, including: sequentially performing keyframe extraction and image enhancement processing on the real-time video data to obtain a video frame sequence; and using lightweight convolutional neural network technology to extract features from the video frame sequence to obtain the video spatial features.

[0024] Specifically, keyframe extraction processing is performed on real-time video data. By calculating the differences between consecutive frames or setting time intervals, representative frames with significant content changes are selected from the lengthy video stream.

[0025] To improve image quality by addressing issues such as blurriness and low contrast caused by changes in lighting, weather conditions, or camera shake, and to highlight the detailed features of power grid equipment, image enhancement processing is performed on the extracted keyframes. Techniques such as histogram equalization, contrast stretching, or filter denoising are employed to optimize image quality.

[0026] Lightweight convolutional neural network techniques, such as MobileNet or ShuffleNet, are used to extract deep features from the enhanced video frame sequence. Through its multi-layer convolutional structure, the network automatically learns hierarchical features from low to high levels from pixels, transforming the original pixel data into deep learning features that can effectively characterize the device's appearance, structural integrity, and state information such as hot spots, i.e., video spatial features.

[0027] Furthermore, feature extraction is performed on the real-time audio data to obtain audio acoustic features, including: performing frame-by-frame windowing processing on the real-time audio data to obtain an audio frame sequence; performing fast Fourier transform processing on the audio frame sequence; and extracting features from the obtained Mel spectrogram to obtain the audio acoustic features.

[0028] Specifically, the real-time audio data is first processed by framing and windowing. The continuous audio signal is divided into short frames with a fixed duration, such as 20 to 40 milliseconds. Each frame of data is multiplied by a window function such as a Hanning window or a Hamming window. The purpose of this step is to assume that the short audio signal is stationary and to reduce spectral leakage caused by signal truncation by windowing, so as to prepare a stable and reliable signal segment for subsequent frequency domain analysis.

[0029] Next, the Fast Fourier Transform technique is used to convert each windowed audio frame from a time-domain signal to a frequency-domain representation, calculate its spectrum, then map the linear frequency scale onto the Mel scale that conforms to the characteristics of human hearing, and calculate the energy within each Mel frequency band, finally generating a visualized Mel spectrogram, with the vertical axis representing Mel frequency, the horizontal axis representing time, and the color depth representing energy intensity, thus more intuitively capturing the spectral structure of the sound and its evolution over time.

[0030] The generated Mel spectrogram is used as input, and a convolutional neural network is used for feature extraction. The convolutional neural network automatically learns local features and global patterns in the spectrogram through its convolutional and pooling layers, and finally outputs a highly abstract feature vector. By automatically identifying and extracting acoustic fingerprint information closely related to the operating status of power grid equipment, such as unique spectral patterns of events like normal humming, abnormal discharge, or mechanical vibration, acoustic features characterizing the audio are obtained.

[0031] Furthermore, feature extraction is performed on the real-time environmental data to obtain environmental time-series features, including: preprocessing the real-time environmental data to obtain environmental time-series data; and using long short-term memory network technology to extract features from the environmental time-series data to obtain the environmental time-series features.

[0032] First, real-time environmental data is preprocessed, specifically including handling missing and outlier values. For example, data gaps are filled by interpolation or statistical methods are used to identify and correct erroneous data that deviates significantly from the normal range. Then, data standardization is performed to uniformly convert environmental parameters with different dimensions and numerical ranges, such as temperature, humidity, and air pressure, to a similar scale.

[0033] Long Short-Term Memory (LSTM) network technology is used to extract features from preprocessed environmental time-series data. Through its unique internal gating mechanism, namely the input gate, forget gate, and output gate, the LSTM network selectively remembers, forgets, and outputs information, thereby capturing the dynamic evolution patterns and dependencies of environmental parameters over a long period of time. By mining the temporal patterns in the environmental data, such as the trend of increasing temperature, the periodic fluctuation of humidity, or the cooperative changes between multiple parameters, the information of the entire time-series segment is finally reduced to a long-term contextual semantic feature vector, namely environmental time-series features.

[0034] Furthermore, based on the video spatial features, the audio acoustic features, and the environmental temporal features, a cross-modal relationship diagram of the target power grid production site is constructed, including: using the video spatial features, the audio acoustic features, and the environmental temporal features as feature nodes to construct an initial node set; performing spatiotemporal correlation analysis and semantic correlation analysis on the initial node set in sequence to obtain the cross-modal relationship diagram of the target power grid production site.

[0035] Specifically, video spatial features, audio acoustic features, and environmental temporal features are defined as three independent nodes in the graph structure, and heterogeneous multimodal data are uniformly transformed into basic elements in graph theory to form an initial node set.

[0036] Spatiotemporal correlation analysis is performed on the initial set of nodes. By calculating the timestamp differences and physical spatial distances of the data corresponding to the features of each node, their synchronicity and proximity in the spatiotemporal dimension are quantified. That is, based on the hard correlation of the objective physical world, it is ensured that event features that coexist in time and space can be preferentially correlated to form a spatiotemporal framework.

[0037] Then, semantic association analysis is performed, using methods such as cosine similarity or neural networks to calculate the similarity of different modal feature vectors at the high-level semantic level, for example, to identify the inherent causal relationship between transformer hot spot image features and abnormal sound features at a specific frequency.

[0038] Based on the analysis results of spatiotemporal and semantic correlations, edge weights are assigned between nodes, integrating the diverse information from the power grid site into a unified model containing rich spatiotemporal and semantic correlations, and constructing a complete weighted cross-modal relationship graph.

[0039] Furthermore, the cross-modal relationship graph is input into the constructed cross-modal fusion model for iterative transmission processing to obtain cross-modal spatiotemporal fusion data, including: constructing the cross-modal fusion model using a graph attention network; and inputting the cross-modal relationship graph into the cross-modal fusion model for iterative transmission processing to obtain cross-modal spatiotemporal fusion data.

[0040] A cross-modal fusion model is constructed using a graph attention network. The attention mechanism can dynamically calculate a weight coefficient for the connection between each node in the graph and its neighboring nodes. The cross-modal relationship graph is input into the model for iterative transmission. In each iteration, each feature node aggregates the feature information of all its neighboring nodes after the attention weights are applied, and then merges it with its own features to update its own feature representation. After multiple iterations, the information can be fully propagated in the entire graph structure, realizing deep cross-modal feature interaction, and allowing video, audio and environmental features to complement and enhance each other.

[0041] After several rounds of iteration, the features of each node are deeply integrated with global multimodal information. Finally, the graph pooling operation aggregates all node features into a unified feature vector. At this point, the feature vector is a joint feature representation rich in spatiotemporal context and cross-modal semantics, i.e., cross-modal spatiotemporal fusion data.

[0042] The cross-modal spatiotemporal fusion data is processed to obtain a fusion output result, including: embedding the equipment status information of the target power grid production site into the cross-modal spatiotemporal fusion data to generate a final display image; and performing real-time rendering on the final display image to generate the fusion output result.

[0043] Specifically, the equipment status information of the target power grid production site is first embedded into the cross-modal spatiotemporal fusion data. This involves associating the fusion data with equipment ledgers, real-time operating parameters, and historical status data. Through information overlay or tag mapping, the abstract fusion features are transformed into specific equipment status descriptions.

[0044] The generated final display image is rendered in real time. Using computer graphics technology, elements such as equipment status information, multimodal fusion results, and alarm levels are overlaid on the real scene or two-dimensional plan view of the power grid site in a visual form such as highlighting, heat map, and data panel.

[0045] After obtaining the fusion output result, the fusion output result is transmitted to the cloud. At the same time, when the system detects network interruption, buffer overflow or mode loss, it automatically executes task restart or missing data reconstruction; missing modes are filled in by KNN interpolation or time series prediction; when key events such as human heat source or abnormal sound waves are identified, local alarm is triggered and uploaded to the cloud in real time.

[0046] Specifically, in each iteration, each feature node receives information from its connected neighbor nodes (i.e., features from other modalities). For example, a video node receives feature information from audio nodes and environment nodes.

[0047] Each node utilizes a learnable aggregation function, such as attention-based weighted summation, to fuse contextual information from different modalities with its own features, thereby updating and enriching its feature representation. After multiple iterations, the features of each node are deeply integrated with the global multimodal context.

[0048] Therefore, by constructing a cross-modal relationship graph, the features of video, audio, and environmental data are transformed into graph nodes and their relationships are explicitly modeled. Then, graph neural networks are used for iterative message passing, which enables the features of each modality to fully interact and enhance at the deep semantic level. Spatiotemporal alignment constraints are added to ensure fusion accuracy, thereby solving the technical problems of insufficient perception of complex anomalies, low recognition accuracy, and high false alarm rate caused by the lack of deep feature interaction.

[0049] Another embodiment of the present invention provides a multi-source data fusion device applied to power grid production sites. For details, please refer to [link to relevant documentation]. Figure 2 , Figure 2 The diagram shown illustrates the structure of a multi-source data fusion device applied in a power grid production site according to one embodiment of the present invention. The device includes: The acquisition module 11 is used to acquire real-time data from the target power grid production site, wherein the real-time data includes at least real-time video data, real-time audio data, and real-time environmental data; Extraction module 12 is used to extract features from the real-time video data to obtain video spatial features; extract features from the real-time audio data to obtain audio acoustic features; and extract features from the real-time environmental data to obtain environmental temporal features. Construction module 13 is used to construct a cross-modal relationship diagram of the target power grid production site based on the video spatial features, the audio acoustic features, and the environmental temporal features; Iteration module 14 is used to input the cross-modal relationship graph into the constructed cross-modal fusion model for iterative transmission processing to obtain cross-modal spatiotemporal fusion data; The fusion module 15 is used to process the cross-modal spatiotemporal fusion data to obtain the fusion output result.

[0050] See Figure 3This is a schematic diagram of the structure of a multi-source data fusion device applied to a power grid production site according to an embodiment of the present invention. The multi-source data fusion device for power grid production sites provided in this embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and configured to be executed by the processor 21. When the processor 21 executes the computer program, it implements the steps described in the above embodiment of the multi-source data fusion method for power grid production sites, for example... Figure 1 The steps S1 to S5 described above; or, when the processor 21 executes the computer program, it implements the functions of each module in the above-described device embodiments, such as the acquisition module 11.

[0051] For example, the computer program can be divided into one or more modules, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the multi-source data fusion device applied to the power grid production site. For example, the computer program can be divided into an acquisition module 11, an extraction module 12, a construction module 13, etc., with the specific functions of each module as follows: The acquisition module 11 is used to acquire real-time data from the target power grid production site, wherein the real-time data includes at least real-time video data, real-time audio data, and real-time environmental data; Extraction module 12 is used to extract features from the real-time video data to obtain video spatial features; extract features from the real-time audio data to obtain audio acoustic features; and extract features from the real-time environmental data to obtain environmental temporal features. Construction module 13 is used to construct a cross-modal relationship diagram of the target power grid production site based on the video spatial features, the audio acoustic features, and the environmental temporal features; Iteration module 14 is used to input the cross-modal relationship graph into the constructed cross-modal fusion model for iterative transmission processing to obtain cross-modal spatiotemporal fusion data; The fusion module 15 is used to process the cross-modal spatiotemporal fusion data to obtain the fusion output result.

[0052] The multi-source data fusion device applied to power grid production sites may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of a multi-source data fusion device applied to power grid production sites and does not constitute a limitation on such a device. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the multi-source data fusion device applied to power grid production sites may also include input / output devices, network access devices, buses, etc.

[0053] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the multi-source data fusion equipment applied at the power grid production site, connecting various parts of the equipment via various interfaces and lines.

[0054] The memory 22 can be used to store the computer program and / or modules. The processor 21 implements various functions of the multi-source data fusion equipment applied to the power grid production site by running or executing the computer program and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0055] The modules integrated into the multi-source data fusion equipment applied to power grid production sites, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0056] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0057] Accordingly, embodiments of the present invention provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform steps in the multi-source data fusion method applied to power grid production sites as described in the above embodiments, for example... Figure 1 Steps S1 to S5 as described above.

[0058] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A multi-source data fusion method applied to a power grid production site, characterized in that, The method comprises the following steps: acquiring real-time data of a target power grid production site, wherein the real-time data at least comprises real-time video data, real-time audio data and real-time environmental data; extracting features from the real-time video data to obtain video spatial features, extracting features from the real-time audio data to obtain audio acoustic features, and extracting features from the real-time environmental data to obtain environmental time sequence features; constructing a cross-modal relationship graph of the target power grid production site based on the video spatial features, the audio acoustic features and the environmental time sequence features; inputting the cross-modal relationship graph into a constructed cross-modal fusion model for iterative transmission processing to obtain cross-modal spatio-temporal fusion data; output processing the cross-modal spatio-temporal fusion data to obtain a fusion output result.

2. The multi-source data fusion method applied to the power grid production site according to claim 1, characterized in that, The method comprises the following steps: extracting features from the real-time video data to obtain video spatial features, comprising: performing key frame extraction processing and image enhancement processing on the real-time video data in sequence to obtain a video frame sequence; 3. The multi-source data fusion method applied to the power grid production site according to claim 1, characterized in that, extracting features from the video frame sequence by using a lightweight convolutional neural network technology to obtain the video spatial features. The method comprises the following steps: extracting features from the real-time audio data to obtain audio acoustic features, comprising:

4. The multi-source data fusion method applied to the power grid production site according to claim 1, characterized in that, performing frame windowing processing on the real-time audio data to obtain an audio frame sequence; performing conversion processing on the audio frame sequence by using a fast Fourier transform technology, and extracting features from the obtained mel spectrum graph to obtain the audio acoustic features. The method comprises the following steps:

5. The multi-source data fusion method applied to the power grid production site according to claim 1, characterized in that, extracting features from the real-time environmental data to obtain environmental time sequence features, comprising: performing data preprocessing on the real-time environmental data to obtain environmental time sequence data; extracting features from the environmental time sequence data by using a long short-term memory network technology to obtain the environmental time sequence features.

6. The multi-source data fusion method applied to the power grid production site according to claim 1, characterized in that, The method comprises the following steps: constructing an initial node set by taking the video spatial features, the audio acoustic features and the environmental time sequence features as feature nodes; performing spatio-temporal correlation analysis and semantic correlation analysis on the initial node set in sequence to obtain the cross-modal relationship graph of the target power grid production site.

7. The multi-source data fusion method applied to the power grid production site according to claim 1, characterized in that, The method comprises the following steps: constructing the cross-modal fusion model by using a graph attention network; inputting the cross-modal relationship graph into the cross-modal fusion model for iterative transmission processing to obtain cross-modal spatio-temporal fusion data.

8. A multi-source data fusion device applied to a power grid production site, characterized in that, The method comprises the following steps: embedding device state information of the target power grid production site in the cross-modal spatio-temporal fusion data to generate a final display image; performing real-time rendering on the final display image to generate the fusion output result. The method comprises the following steps: an acquisition module configured to acquire real-time data of a target power grid production site, wherein the real-time data at least comprises real-time video data, real-time audio data and real-time environmental data; extracting a video spatial feature from the real-time video data, extracting an audio acoustic feature from the real-time audio data, and extracting an environment time-series feature from the real-time environment data; constructing a cross-modal relationship graph of the target power grid production site based on the video spatial feature, the audio acoustic feature, and the environment time-series feature; inputting the cross-modal relationship graph into a constructed cross-modal fusion model for iterative transmission processing to obtain cross-modal spatio-temporal fusion data; outputting the cross-modal spatio-temporal fusion data to obtain a fusion output result.

9. A multi-source data fusion device applied to a power grid production site, characterized in that, The computer readable storage medium stores a computer program, and when a device in which the computer readable storage medium is located executes the computer program, the application of the multi-source data fusion method for the power grid production site is implemented.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when a device in which the computer readable storage medium is located executes the computer program, the application of the multi-source data fusion method for the power grid production site is implemented.

Images