New energy multi-modal data-oriented labeling method and system

Abstract

The invention provides an annotation method and system for new energy multi-modal data, and relates to the technical field of data annotation, and the method comprises the following steps: obtaining multi-modal data of new energy equipment, preprocessing the multi-modal data, mapping the preprocessed multi-modal data to the same feature space, and obtaining mapped multi-modal data; based on the type of the multi-modal data, performing multi-modal spatial statistical analysis on the feature vector of the mapped multi-modal data by using a semi-variance function, and generating a cross-modal spatial constraint rule in combination with a cross-modal hash algorithm; and based on a cross-modal spatial constraint rule, labeling and classifying the multi-modal data of the new energy equipment, and generating a new energy multi-modal labeling result. According to the method, the accuracy and engineering applicability of new energy multi-modal data labeling are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of data annotation technology, and more specifically, to an annotation method and system for multimodal data of new energy. Background Technology

[0002] With the rapid development of the new energy industry (such as photovoltaics, wind power, and energy storage), the demand for intelligent operation and maintenance of new energy equipment is becoming increasingly prominent. During operation, new energy equipment generates massive amounts of multimodal data, including visual data, text data, audio data, and time-series data. This multimodal data contains crucial information such as equipment status and fault characteristics, serving as the core basis for achieving equipment health monitoring, fault diagnosis, and predictive maintenance. An increasing number of application scenarios require the annotation of large-scale multimodal data, such as text, images, audio, and video.

[0003] Currently, when labeling data, traditional methods only focus on single-modal semantic matching and do not quantify the spatial autocorrelation characteristics of multimodal data, resulting in the neglect of the spatial correlation of cross-modal features. At the same time, the physical characteristics and real-time operation data of new energy equipment are not included in the labeling rules, causing the labeling results to be out of touch with the actual operating status of the equipment. Ultimately, this affects the accuracy of decision-making in scenarios such as intelligent operation and maintenance and fault detection.

[0004] No effective solutions have yet been proposed to address the problems in the relevant technologies. Summary of the Invention

[0005] In view of this, the present invention provides a labeling method and system for multimodal data of new energy sources to solve the problems mentioned above.

[0006] To solve the above problems, the specific technical solution adopted by the present invention is as follows:

[0007] According to a first aspect of the present invention, a method for labeling multimodal data of new energy sources is provided, comprising the following steps:

[0008] S1. Acquire multimodal data of new energy equipment, preprocess the multimodal data, and map the preprocessed multimodal data to the same feature space to obtain the mapped multimodal data;

[0009] S2. Based on the type of multimodal data, the semivariance function is used to perform multimodal spatial statistical analysis on the feature vectors of the mapped multimodal data, and cross-modal hashing algorithm is combined to generate cross-modal spatial constraint rules.

[0010] S3. Based on cross-modal space constraint rules, the multimodal data of new energy equipment is labeled and classified to generate the labeling results of new energy multimodal data.

[0011] Preferably, the step of performing multimodal spatial statistical analysis on the feature vectors of the mapped multimodal data using a semivariance function based on the type of multimodal data, and generating cross-modal spatial constraint rules by combining a cross-modal hashing algorithm, includes the following steps:

[0012] S21. Based on the type of multimodal data and combined with the preset feature extraction method library, feature vectors are extracted for each type of mapped multimodal data, and the feature vectors of each type of data are associated with spatial coordinates to construct a multimodal cross semivariance function.

[0013] S22. Generate cross-modal hash codes based on the multimodal cross semivariance function and the cross-modal hashing algorithm;

[0014] S23. Based on the correlation between the multimodal cross-semivariance function and the cross-modal hash code, generate cross-modal space constraint rules.

[0015] Preferably, the step of extracting feature vectors for each type of mapped multimodal data based on the type of multimodal data and in conjunction with a preset feature extraction method library, and associating the feature vectors of each modality data with spatial coordinates to construct a multimodal cross-semivariance function includes the following steps:

[0016] S211. Based on the type of multimodal data, extract the corresponding feature extraction method from the preset feature extraction method library, and perform feature extraction on each type of mapped modal data to obtain the feature vector of each modality.

[0017] S212. Based on the spatial attributes of each modality data, configure a spatial coordinate system for each modality data, and bind the feature vector of each modality to the corresponding spatial coordinate system to form a structured association pair;

[0018] S213. Based on the structured association pairs, construct the initial multimodal cross semivariogram function, and optimize the initial multimodal cross semivariogram function by extracting spatial statistical parameters to obtain the final multimodal cross semivariogram function.

[0019] Preferably, the step of constructing an initial multimodal cross-semivariance function based on structured association pairs, and optimizing the initial multimodal cross-semivariance function by extracting spatial statistical parameters to obtain the final multimodal cross-semivariance function includes the following steps:

[0020] S2131. Based on the spatial coordinate system of structured association pairs, calculate the cross semivariance value between feature vectors of different modes with a preset spatial distance step size as the interval.

[0021] S2132. Use a spherical model to fit the cross semivariance values ​​and generate the initial multimodal cross semivariance function;

[0022] S2133. Extract spatial statistical parameters from the initial semivariance function, and adjust and optimize the spatial statistical parameters based on the physical characteristics of the new energy equipment to obtain optimized spatial statistical parameters; the spatial statistical parameters include nugget constant, sill value and range.

[0023] S2134. Substitute the optimized spatial statistical parameters into the initial multimodal cross semivariogram function to generate the final multimodal cross semivariogram function.

[0024] Preferably, the step of generating cross-modal hash codes based on the multimodal cross-semivariance function and combined with the cross-modal hashing algorithm includes the following steps:

[0025] S221. Map the parameters of the multimodal cross semivariogram function to the input layer of the cross-modal hashing algorithm to impose space constraints on the cross-modal hashing algorithm;

[0026] S222. Based on the feature vector of each modality data, combined with spatial constraints, feature standardization and hash encoding are performed to generate an initial hash code;

[0027] S223. A cross-modal comparison learning strategy based on the FLIP model is used to optimize the initial hash code to obtain an optimized hash code.

[0028] S224. Based on the preset spatial parameter threshold of the multimodal cross semivariogram function, verify the spatial constraint satisfaction of the optimized hash code, and output the final cross-modal hash code after the verification is passed.

[0029] Preferably, the cross-modal comparison learning strategy based on the FLIP model optimizes the initial hash code to obtain the optimized hash code, including the following steps:

[0030] S2231. Based on the nugget constant and range in the multimodal cross-variance function, adjust the Hamming distance distribution of the initial hash code to obtain the adjusted hash code;

[0031] S2232. Using the cross-modal comparative learning strategy of the FLIP model, semantic alignment is performed on the feature vectors of each modality to generate semantically aligned feature vectors.

[0032] S2233. The hash bit length is dynamically adjusted based on the base value, and the hash code is regenerated by combining the semantically aligned feature vector to obtain the optimized hash code.

[0033] Preferably, the step of generating cross-modal space constraint rules based on the correlation between the multimodal cross-semivariance function and the cross-modal hash code includes the following steps:

[0034] S231. Map the spatial statistical parameters of the multimodal cross semivariogram function to the spatial constraint dimension of the cross-modal hash code to obtain the parameter mapping result;

[0035] S232. Based on the parameter mapping results and combined with the Hamming distance distribution of cross-modal hash codes, preliminary spatial constraint rules are generated.

[0036] S233. Based on the physical characteristics and real-time operating data of new energy equipment, the preliminary spatial constraint rules are dynamically optimized to obtain the optimized spatial constraint rules.

[0037] S234. Verify the optimized spatial constraint rules, and output the final cross-modal spatial constraint rules after the verification is passed.

[0038] According to a second aspect of the present invention, a labeling system for multimodal data of new energy sources is provided, the system comprising:

[0039] The data acquisition and processing module is used to acquire multimodal data of new energy equipment, preprocess the multimodal data, and map the preprocessed multimodal data to the same feature space to obtain the mapped multimodal data.

[0040] The constraint rule generation module is used to perform multimodal spatial statistical analysis on the feature vectors of the mapped multimodal data based on the type of multimodal data, using the semivariance function, and to generate cross-modal spatial constraint rules in combination with the cross-modal hashing algorithm.

[0041] The data annotation module is used to annotate and classify multimodal data of new energy equipment based on cross-modal space constraint rules, and generate annotation results for new energy multimodal data.

[0042] According to a third aspect of the present invention, an electronic device is provided, the electronic device comprising: one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the programs to implement the steps of the above-described method.

[0043] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored therein, wherein the steps of the above-described method are implemented when the computer program controls the device in which the computer-readable storage medium is located to execute during runtime.

[0044] The beneficial effects of this invention are as follows:

[0045] 1. This invention achieves efficient annotation of multimodal data of new energy equipment by preprocessing multimodal data and unifying feature space, performing semivariance function space statistical analysis and generating cross-modal hash constraint rules. It integrates spatial distribution characteristics and semantic similarity, and fits the physical characteristics of the equipment and real-time operation data. Finally, it outputs accurate annotation results, supports efficient decision-making in scenarios such as intelligent operation and maintenance and fault detection, and significantly improves the accuracy and engineering applicability of new energy multimodal data annotation.

[0046] 2. This invention quantifies spatial parameters such as nugget constant and range by using a multimodal cross-semivariance function, which can accurately capture the spatial correlation patterns of different modal data. Combined with the semantic alignment of cross-modal hashing algorithm and FLIP model, the semantic consistency of cross-modal data is strengthened on the basis of spatial constraints. By dynamically optimizing parameters and rules through equipment physical characteristics and real-time data, the constraint rules are adapted to the special characteristics of new energy scenarios, providing high-quality multimodal data label support for new energy equipment status monitoring and fault diagnosis. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0048] Figure 1 This is a flowchart of a method for labeling multimodal data of new energy sources according to an embodiment of the present invention;

[0049] Figure 2 This is a principle block diagram of a labeling system for multimodal data of new energy according to an embodiment of the present invention;

[0050] Figure 3 This is a flowchart illustrating the generation of constraint rules in a labeling method for new energy multimodal data according to an embodiment of the present invention.

[0051] Figure 4 This is a flowchart illustrating the generation of the final multimodal cross semivariogram function in a labeling method for new energy multimodal data according to an embodiment of the present invention;

[0052] Figure 5 This is a block diagram of the hardware structure of the host device in a method for labeling multimodal data of new energy according to an embodiment of the present invention.

[0053] In the picture:

[0054] 1. Data acquisition and processing module; 2. Constraint rule generation module; 3. Data annotation module. Detailed Implementation

[0055] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0056] The methods and embodiments provided in this application can be executed on a host device or a similar computing device. Taking running on a host device as an example, such as... Figure 5 As shown, the host device may include one or more ( Figure 5 Only one is shown in the diagram. The processor (which may include, but is not limited to, a microprocessor (MCU) or programmable logic device (FPGA), etc.) and storage for storing data are also shown. The host device may further include transmission devices for communication functions and input / output devices. Those skilled in the art will understand that... Figure 5 The structure shown is for illustrative purposes only and does not limit the structure of the host device described above. For example, the host device may also include components that are larger than... Figure 5 The more or fewer components shown, or having the same Figure 5 The different configurations shown.

[0057] The memory can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the exception handling method in this embodiment. The processor executes various functional applications and data processing by running the computer program stored in the memory, thus implementing the above-described method. The memory may include high-speed random access memory (RAM) and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the host device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0058] Transmission devices are used to receive or send data over a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the host device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0059] According to an embodiment of the present invention, a method and system for labeling multimodal data of new energy is provided.

[0060] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 1 and Figure 3-4 As shown, according to a first embodiment of the present invention, a method for labeling multimodal data of new energy sources is provided, comprising the following steps:

[0061] S1. Acquire multimodal data of new energy equipment, preprocess the multimodal data, and map the preprocessed multimodal data to the same feature space to obtain the mapped multimodal data;

[0062] Specifically, for new energy scenarios such as photovoltaic power stations, wind farms, and energy storage power stations, the core equipment includes photovoltaic modules, wind turbines, inverters, and battery clusters. Multi-source heterogeneous data is collected, and typical modalities include: visual data, text data, audio data, and time-series data.

[0063] The visual data includes infrared thermal images, visible light images, and drone inspection video frames; the text data includes operation and maintenance logs, fault reports, and equipment parameter tables; the audio data includes vibration sound waves and ambient audio during equipment operation; and the time-series data includes real-time monitoring data from sensors.

[0064] Targeted preprocessing is performed to address the noise characteristics and format differences of data from different modalities, with the aim of noise reduction, format standardization, and preservation of core information; specifically:

[0065] The visual data preprocessing involves using Gaussian filtering to remove thermal noise from infrared images, median filtering to eliminate salt-and-pepper noise from visible light images, cropping the effective area containing the target device, and aligning multiple time-segment images of the same device using image registration technology; pixel values ​​are normalized to the range of [0,1] to unify image size.

[0066] Text data preprocessing involves removing meaningless characters, correcting typos, extracting key information using regular expressions, and filtering stop words using a new energy-related dictionary to retain core semantic words.

[0067] Audio data preprocessing involves using wavelet threshold denoising to eliminate environmental noise, preserving the characteristic frequencies of device operation, and segmenting long audio files according to the device's operating cycle to convert them into Mel spectrograms (converting time-domain signals into frequency-time domain two-dimensional features) to facilitate subsequent feature extraction.

[0068] For time-series data preprocessing, linear interpolation or LSTM prediction is used to supplement missing data caused by sensor disconnection; high-frequency noise is filtered out using moving average to retain trend characteristics; and time-series data from different sensors are aligned by timestamp to ensure that data in the same time period can be correlated.

[0069] The original feature dimensions and distribution patterns of data from different modalities differ significantly, making direct cross-modal association analysis impossible. Cross-modal embedding models are required, which are pre-trained on large-scale new energy multimodal data to learn semantic associations between modalities. For scenarios without pre-trained models, adversarial learning (such as cross-modal adversarial networks) is used. The generator maps modal features to a shared space, and the discriminator distinguishes the source of features, ultimately ensuring that semantically similar features from different modalities are distributed consistently in the shared space.

[0070] S2. Based on the type of multimodal data, the semivariance function is used to perform multimodal spatial statistical analysis on the feature vectors of the mapped multimodal data, and cross-modal hashing algorithm is combined to generate cross-modal spatial constraint rules.

[0071] As a preferred embodiment, the step of performing multimodal spatial statistical analysis on the feature vectors of the mapped multimodal data using a semivariance function based on the type of multimodal data, and generating cross-modal spatial constraint rules by combining a cross-modal hashing algorithm, includes the following steps:

[0072] S21. Based on the type of multimodal data and combined with the preset feature extraction method library, feature vectors are extracted for each type of mapped multimodal data, and the feature vectors of each type of data are associated with spatial coordinates to construct a multimodal cross semivariance function.

[0073] In a preferred embodiment, the step of extracting feature vectors for each type of mapped multimodal data based on the type of multimodal data and in conjunction with a preset feature extraction method library, and associating the feature vectors of each modality data with spatial coordinates to construct a multimodal cross-semivariance function includes the following steps:

[0074] S211. Based on the type of multimodal data, extract the corresponding feature extraction method from the preset feature extraction method library, and perform feature extraction on each type of mapped modal data to obtain the feature vector of each modality.

[0075] Within the same feature space, for each type of modality data, a fixed-dimensional feature vector is extracted using a pre-defined feature extraction method library. For example, for visual modality, a CNN model, such as ResNet50, is called from the library to extract deep features from the preprocessed image, such as the temperature gradient and texture complexity of hotspot regions, outputting a 256-dimensional feature vector. For text modality, a BERT model fine-tuned from new energy domain corpus is called to extract semantic features from the preprocessed text, outputting a 256-dimensional feature vector. For audio modality, a Mel-spectrum CNN model is called to extract spectral features from audio segments, outputting a 256-dimensional feature vector. For temporal modality, a Transformer model is called to extract trend features from aligned temporal data, outputting a 256-dimensional feature vector.

[0076] S212. Based on the spatial attributes of each modality data, configure a spatial coordinate system for each modality data, and bind the feature vector of each modality to the corresponding spatial coordinate system to form a structured association pair;

[0077] It should be noted that the spatial properties of different modalities of data differ significantly, specifically:

[0078] The spatial attributes of visual modalities are explicitly correlated with physical space. For example, the infrared thermal image of a photovoltaic module and the visible light photograph of a wind turbine blade directly correspond to the physical location of the equipment. For instance, the installation area of ​​a photovoltaic string is located at 116.2°E, 39.7°N, and the pixel position within the image can be mapped to a physical area on the surface of the equipment, such as 30cm from the upper left corner of the module.

[0079] The spatial attributes of text modalities are implicit semantic spatial associations. For example, if a hot spot appears on a photovoltaic string in the operation and maintenance log, resulting in a 10% power reduction, its spatial information is implicit in the text entity and needs to be mapped to specific physical coordinates through the equipment ledger; at the same time, the semantics of the text itself constitute semantic spatial associations.

[0080] The spatial attributes of audio modalities are associated with both physical space and time. For example, the vibration audio of a wind turbine gearbox has a fixed sampling point next to the equipment (the physical coordinates are known), and the audio signal changes with time (such as the abnormal noise characteristics from 10:00 to 10:05). It is necessary to associate both the spatial location and the timestamp.

[0081] The spatial attributes of time-series modes are strongly correlated with sensor location and time. For example, the voltage and current time-series data of photovoltaic modules are directly bound to the sensor installation location and acquisition time, with both spatial and temporal attributes being explicit.

[0082] Based on the above spatial attributes, a dedicated coordinate system is configured for each mode to ensure that the coordinates accurately reflect its spatial correlation characteristics. Specifically:

[0083] The visual modality uses a dual coordinate system of physical coordinates and pixel coordinates. The physical coordinates are based on the geographical latitude and longitude of the device installation, while the pixel coordinates are two-dimensional pixel coordinates with the upper left corner of the image as the origin, and the unit is pixels. The dual coordinate systems are linked by an image calibration matrix (e.g., 1 pixel corresponds to 0.5 cm physical length) to achieve accurate mapping from pixel position to device physical position.

[0084] The text modality employs a hybrid coordinate system combining physical device coordinates and semantic space coordinates. Physical device coordinates are obtained by querying the equipment ledger from the device entities mentioned in the text to obtain their installation latitude and longitude. Semantic space coordinates are achieved by reducing the dimensionality of the text feature vectors (256 dimensions) to two dimensions using the t-SNE algorithm, reflecting the semantic correlation between texts. The hybrid coordinate system preserves both physical location correlations and semantic similarities, adapting to the characteristic of text semantics driving spatial correlation.

[0085] The audio modality uses a three-dimensional spatiotemporal coordinate system (lat, lng, t), where lat and lng are the installation latitude and longitude of the audio acquisition device (such as a microphone), with an accuracy of 0.5 meters, and t is the timestamp of the audio acquisition, with an accuracy of milliseconds.

[0086] The temporal modality is the same as the audio modality, and it also uses a three-dimensional spatiotemporal coordinate system.

[0087] S213. Based on the structured association pairs, construct the initial multimodal cross semivariogram function, and optimize the initial multimodal cross semivariogram function by extracting spatial statistical parameters to obtain the final multimodal cross semivariogram function.

[0088] In a preferred embodiment, the step of constructing an initial multimodal cross-semivariance function based on structured association pairs, and optimizing the initial multimodal cross-semivariance function by extracting spatial statistical parameters to obtain the final multimodal cross-semivariance function includes the following steps:

[0089] S2131. Based on the spatial coordinate system of structured association pairs, calculate the cross semivariance value between feature vectors of different modes with a preset spatial distance step size as the interval.

[0090] It should be noted that the spatial distance step size is the benchmark for dividing spatial distance intervals, and its setting value is based on the physical scale of new energy equipment. Specifically:

[0091] For photovoltaic scenarios, the spacing between photovoltaic strings is about 3-5 meters, and the size of a single module is about 1.6×1 meter, with a step size of 1 meter; for wind power scenarios, the spacing between wind turbines is about 500-1000 meters, with a step size of 50 meters.

[0092] The cross-semivariance (CSA) measures the average difference between feature vectors of different modalities at a given spatial distance. It requires extracting the spatial coordinates of all mapped cross-modal data pairs from the structured association pairs, calculating the spatial distance for each pair, dividing the spatial distance into different intervals according to a step size, and calculating the mean of the squared differences of the feature vectors for each interval to obtain the CSA. The specific calculation formula is as follows:

[0093] ;

[0094] In the formula, γ ij (h) represents the cross-sigma of mode i and mode j at a spatial distance h, where h represents the interval corresponding to the preset spatial distance step size, and N(h) represents the number of cross-modal feature vector pairs whose spatial distances belong to the interval h. i,k and γ j,k Let represent the feature vectors of mode i and mode j in the k-th pair of mapped cross-modal data, respectively.

[0095] S2132. Use a spherical model to fit the cross semivariance values ​​and generate the initial multimodal cross semivariance function;

[0096] It should be noted that the spherical model is a classic model in geostatistics for fitting spatial variability. When applied to new energy scenarios, it accurately describes the pattern where, when spatial distances are small, feature differences increase rapidly with increasing distance; beyond a certain distance, the differences tend to stabilize. This aligns with the spatial correlation characteristics of equipment data, where data within a single string are strongly correlated, and the correlation weakens beyond the string's range. The model parameters (nuclear constant, sill value, range) have clear physical meanings, facilitating subsequent optimization based on equipment characteristics. The initial expression for the multimodal cross-semivariance function is:

[0097] ;

[0098] In the formula, c0 represents the nugget constant, the semivariance value when h=0, which reflects small-scale random variation (such as feature fluctuations caused by device sensor noise and image shooting angle errors), c represents the partial sill value, c0+c is the sill value, which reflects the total spatial variation of multimodal features. The larger the value, the more significant the differences in cross-modal features. a represents the range, which is the distance when the semivariance value reaches the sill value, reflecting the maximum range of spatial correlation of multimodal data.

[0099] S2133. Extract spatial statistical parameters from the initial semivariance function, and adjust and optimize the spatial statistical parameters based on the physical characteristics of the new energy equipment to obtain optimized spatial statistical parameters; the spatial statistical parameters include nugget constant, sill value and range.

[0100] Specifically, spatial statistical parameters are extracted from the fitted initial semivariogram function and then optimized by combining them with the physical characteristics of new energy equipment (such as the vibration attenuation law of wind turbine blades and the hot spot propagation characteristics of photovoltaic panels). For example:

[0101] If the range of the wind turbine vibration sound pattern does not match the actual effective monitoring range of the equipment (e.g., 500m), the range needs to be adjusted to match the physical characteristics. If the photovoltaic hot spot is too large (e.g., >20%), it indicates strong local noise interference, and feature extraction needs to be optimized through noise reduction algorithms (e.g., wavelet denoising). The base station needs to be consistent with the overall variation of the equipment. For example, the overall hot spot variation intensity of the photovoltaic power station needs to be positively correlated with the base station.

[0102] S2134. Substitute the optimized spatial statistical parameters into the initial multimodal cross semivariogram function to generate the final multimodal cross semivariogram function.

[0103] S22. Generate cross-modal hash codes based on the multimodal cross semivariance function and the cross-modal hashing algorithm;

[0104] In a preferred embodiment, the step of generating cross-modal hash codes based on the multimodal cross-semivariance function and combined with a cross-modal hashing algorithm includes the following steps:

[0105] S221. Map the parameters of the multimodal cross semivariogram function to the input layer of the cross-modal hashing algorithm to impose space constraints on the cross-modal hashing algorithm;

[0106] It should be noted that this step maps the nugget constant, range, and sill value of the multimodal cross-variance function as spatial constraint parameters to the input layer of the cross-modal hashing algorithm (e.g., adjusting the weight of the regularization term in the loss function and the spatial attention module parameters in the encoder structure). This enables the hashing algorithm to actively adapt to the spatial distribution characteristics of new energy equipment when generating hash codes. For example, the nugget constant suppresses the interference of local noise on cross-modal matching, the range constrains the spatial autocorrelation range that the hash code needs to retain, and the sill balances information retention and computational efficiency. Ultimately, this ensures that the generated hash code has both semantic discriminative power and conforms to spatial statistical laws, providing accurate quantitative support for the subsequent generation of cross-modal spatial constraint rules.

[0107] S222. Based on the feature vector of each modality data, combined with spatial constraints, feature standardization and hash encoding are performed to generate an initial hash code;

[0108] It should be noted that Z-score normalization or Min-Max normalization is performed on the feature vectors of each modality to eliminate dimensional differences. For example, after normalization, the spectral features of the acoustic modality fall within the same numerical range as the image features of the visual modality, ensuring the fairness of the hash coding. Combined with spatial constraint parameters (such as a and c), Locality Sensitive Hashing (LSH) or deep hashing networks are used to generate initial hash codes. For example, by adjusting the kernel width parameter of the hash function, the Hamming distance distribution is matched to the spatial autocorrelation range (range a), ensuring that samples with similar spatial distances have smaller Hamming distances.

[0109] S223. A cross-modal comparison learning strategy based on the FLIP model is used to optimize the initial hash code to obtain an optimized hash code.

[0110] In a preferred embodiment, the cross-modal comparison learning strategy based on the FLIP model optimizes the initial hash code to obtain the optimized hash code, including the following steps:

[0111] S2231. Based on the nugget constant and range in the multimodal cross-variance function, adjust the Hamming distance distribution of the initial hash code to obtain the adjusted hash code;

[0112] It should be noted that the dynamic adjustment of the Hamming distance distribution of the initial hash code based on the nugget constant and range of the multimodal cross-variance function specifically includes:

[0113] When the nugget constant is high (e.g., >0.15), it indicates that local noise (e.g., dust on the surface of photovoltaic panels, turbulence of wind turbines) strongly interferes with the data. It is necessary to increase the Hamming distance threshold δ (e.g., adjust it from 0.2 to 0.3) to suppress cross-modal mismatches caused by noise. Conversely, when the nugget constant is low, δ should be reduced to improve the matching accuracy of sensitive modes (e.g., hot spot edge features).

[0114] The effective range of spatial autocorrelation is defined by the range, and the Hamming distance distribution needs to be forcibly associated with the range. For example, when the range a = 300m (typical monitoring range of photovoltaic power station), the Hamming distance range needs to be extended to [0, 0.5] to ensure that the cross-modal characteristics of samples within 300m (such as hot spot images and infrared thermal images) are preferentially matched, and to avoid interference from samples with too far spatial distance.

[0115] For example, in a wind power scenario, if the nugget constant of the wind turbine vibration soundprint is 0.18 (strong local turbulence interference), then the Hamming distance threshold is adjusted to δ=0.25 to avoid the soundprint noise causing mis-association with maintenance text (such as bearing failure); if the range is 500m (effective monitoring radius of the wind turbine), then the Hamming distance distribution is extended to [0,0.6] to ensure cross-modal matching between the soundprint and the vibration image within 500m.

[0116] S2232. Using the cross-modal comparative learning strategy of the FLIP model, semantic alignment is performed on the feature vectors of each modality to generate semantically aligned feature vectors.

[0117] It should be noted that by utilizing the cross-modal comparative learning capability of the FLIP (FastLanguage-ImagePre-training) model, feature vectors from different modalities are mapped to a unified semantic space, thereby enhancing the semantic consistency of cross-modal data. The FLIP model has a dual encoder structure, which includes a text encoder and a multimodal encoder, respectively mapping feature vectors from modalities such as text, image, and audio to a semantic space of the same dimension.

[0118] The contrastive learning mechanism employs the InfoNCE loss function to shorten the hash code distance between positive sample pairs (spatial distance h < a and semantic matching, such as hot spot image - infrared thermography - maintenance text) and widen the distance between negative sample pairs (h ≥ a or semantic mismatch, such as normal area image - fault text). For example, the feature vectors of the photovoltaic panel hot spot image and the corresponding infrared thermography image are input into a dual encoder. Through contrastive learning, their hash codes are aligned in the semantic space, ensuring a strong semantic association between the hot spot and the high-temperature anomaly.

[0119] S2233. The hash bit length is dynamically adjusted based on the base value, and the hash code is regenerated by combining the semantically aligned feature vector to obtain the optimized hash code.

[0120] It should be noted that when dynamically adjusting the hash bit length based on the sill value (reflecting the intensity of total spatial variation) and regenerating the hash code by combining it with the semantically aligned feature vector, the logic for adjusting the bit length of the sill value is as follows: a high sill value indicates a large total spatial variation (such as a complex overall hotspot distribution in a photovoltaic power station), requiring an increase in the hash bit length (e.g., from 64 bits to 128 bits) to retain more spatial variation information and avoid the loss of key features; a low sill value reduces the bit length (e.g., to 32 bits) to improve computational efficiency and avoid interference from redundant information. Therefore, the semantically aligned feature vector is input into the optimized hash encoder (such as a deep hash network), and the hash code is regenerated by combining it with the adjusted hash bit length. For example, in a photovoltaic scenario, if the sill value = 0.85 (strong total hotspot variation), the hash bit length is adjusted to 128 bits. The hash code is then regenerated by combining the semantically aligned hotspot image with infrared thermal imaging features, ensuring that the hash code reflects both the semantic similarity between the hotspot and high temperature and the spatial distribution characteristics (such as the hotspot diffusion range).

[0121] S224. Based on the preset spatial parameter threshold of the multimodal cross semivariogram function, verify the spatial constraint satisfaction of the optimized hash code, and output the final cross-modal hash code after the verification is passed.

[0122] Specifically, the hash code conforms to the spatial correlation pattern of multimodal data by using preset thresholds for the optimized spatial parameters in the multimodal cross-variance function. Specifically, the preset thresholds include: a spatial correlation distance threshold based on range; a "small-scale variation tolerance threshold" based on the nugget constant; and an overall variation adaptation threshold based on the sill value.

[0123] During verification, 20% of the sample set is randomly selected from the new energy multimodal data. The spatial distance h of the cross-modal sample pairs and the Hamming distance of the corresponding hash codes are calculated. The proportion of sample pairs that meet the above thresholds is statistically analyzed (≥95% meet the spatial correlation threshold, ≥90% meet the small-scale variation threshold, and the overall dispersion is within the adaptation range). If the verification fails, return to S223 to readjust the Hamming distance distribution or hash bit length of the hash codes. If the verification passes, the final cross-modal hash code containing the associated device identifier, spatial coordinates, and modality type is output to ensure that it accurately reflects the spatial correlation of the multimodal data and provides a reliable binary quantization basis for the subsequent generation of cross-modal spatial constraint rules.

[0124] S23. Based on the correlation between the multimodal cross-semivariance function and the cross-modal hash code, generate cross-modal space constraint rules.

[0125] In a preferred embodiment, generating cross-modal space constraint rules based on the correlation between the multimodal cross-semivariance function and the cross-modal hash code includes the following steps:

[0126] S231. Map the spatial statistical parameters of the multimodal cross semivariogram function to the spatial constraint dimension of the cross-modal hash code to obtain the parameter mapping result;

[0127] It should be noted that the spatial statistical parameters of the multimodal cross-semivariance function are transformed into spatial constraint dimensions (such as distance thresholds and difference tolerance) that can be directly applied to cross-modal hash codes. This establishes a quantitative correspondence between spatial statistical characteristics and hash code constraints, providing a parameter basis for rule generation. Specifically, the range is mapped to the upper bound of the spatial correlation Hamming distance of the hash code. The nugget constant is mapped to the hash code difference tolerance threshold for close sample pairs. The sill value is mapped to the overall dispersion interval of the hash code. The mapping result is a set of correlation tables of parameters, constraint dimensions, and thresholds.

[0128] S232. Based on the parameter mapping results and combined with the Hamming distance distribution of cross-modal hash codes, preliminary spatial constraint rules are generated.

[0129] It should be noted that preliminary spatial constraint rules are generated based on the parameter mapping results and the Hamming distance distribution characteristics of cross-modal hash codes. For example, when the correlation coefficient between the Hamming distance and the spatial distance is ≥0.8, a strong spatial association rule is defined; when the correlation coefficient is <0.8, a weak association rule is triggered. The rules clearly define the inter-modal matching threshold, spatial range priority, etc., providing a benchmark for subsequent dynamic optimization.

[0130] S233. Based on the physical characteristics and real-time operating data of new energy equipment, the preliminary spatial constraint rules are dynamically optimized to obtain the optimized spatial constraint rules.

[0131] It should be noted that, by combining the physical characteristics of new energy equipment (such as the vibration attenuation law of wind turbine blades and the hot spot diffusion characteristics of photovoltaic panels) and real-time operating data, including voltage / current fluctuations monitored in real time by the SCADA system and wind speed / solar illumination data from meteorological stations, the initial rules are dynamically optimized. For example, in wind power scenarios, if an abnormal increase in wind turbine bearing temperature is detected in real time, the range needs to be adjusted to a smaller range (such as from 500m to 300m) to strengthen the cross-modal correlation of local fault areas; in photovoltaic scenarios, if a sudden change in sunlight intensity causes a change in hot spot distribution, the base value needs to be dynamically adjusted to match the new total variation intensity, so that the rules fit the actual operating state of the equipment.

[0132] S234. Verify the optimized spatial constraint rules, and output the final cross-modal spatial constraint rules after the verification is passed.

[0133] It should be noted that during verification, cross-validation (such as leave-one-out method), GIS spatial consistency analysis (such as comparing the spatial distribution of hash codes with the actual installation location of the equipment), and engineering scenario backtracking (such as historical fault case matching test) can be used to verify the effectiveness of the optimized rules. If the verification passes, for example, if the rule matching accuracy is ≥95%, the final cross-modal spatial constraint rules are output; if it fails, a feedback mechanism is triggered to adjust the semivariance parameter or hash algorithm parameter, and the S231-S234 process is re-executed to form a closed-loop optimization system of rule generation, verification, and feedback.

[0134] S3. Based on cross-modal space constraint rules, the multimodal data of new energy equipment is labeled and classified to generate the labeling results of new energy multimodal data.

[0135] Specifically, by applying the cross-modal spatial constraint rules generated in the preceding steps, spatial-semantic collaborative annotation and classification of multimodal data of new energy equipment is performed. Specifically, by matching the spatial association range (variable range a constraint) and semantic similarity (hash code Hamming distance) of different modal features through rules, it is ensured that the annotation results not only conform to the physical characteristics of the equipment (hot spot diffusion law, etc.) but also meet the constraints of real-time operating data. Finally, accurate new energy multimodal annotation results are output to support efficient decision-making in application scenarios such as intelligent operation and maintenance and fault detection.

[0136] like Figure 2 As shown, according to a second embodiment of the present invention, a labeling system for multimodal data of new energy is provided, the system comprising:

[0137] Data acquisition and processing module 1 is used to acquire multimodal data of new energy equipment, preprocess the multimodal data, and map the preprocessed multimodal data to the same feature space to obtain mapped multimodal data;

[0138] The constraint rule generation module 2 is used to perform multimodal spatial statistical analysis on the feature vectors of the mapped multimodal data based on the type of multimodal data, using the semivariance function, and to generate cross-modal spatial constraint rules in combination with the cross-modal hashing algorithm.

[0139] Data annotation module 3 is used to annotate and classify multimodal data of new energy equipment based on cross-modal space constraint rules, and generate annotation results for new energy multimodal data.

[0140] According to a third embodiment of the present invention, an electronic device is provided, the electronic device comprising: one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the steps in any of the above method embodiments.

[0141] According to a fourth embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to perform the steps in any of the above method embodiments.

[0142] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0143] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0144] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A labeling method for multimodal data of new energy sources, characterized in that, Includes the following steps: S1. Acquire multimodal data of new energy equipment, preprocess the multimodal data, and map the preprocessed multimodal data to the same feature space to obtain the mapped multimodal data; S2. Based on the type of multimodal data, the semivariance function is used to perform multimodal spatial statistical analysis on the feature vectors of the mapped multimodal data, and cross-modal hashing algorithm is combined to generate cross-modal spatial constraint rules. S3. Based on cross-modal space constraint rules, the multimodal data of new energy equipment is labeled and classified to generate the labeling results of new energy multimodal data.

2. The annotation method for multimodal data of new energy sources according to claim 1, characterized in that, The process of performing multimodal spatial statistical analysis on the feature vectors of the mapped multimodal data using a semivariance function based on the type of multimodal data, and generating cross-modal spatial constraint rules by combining a cross-modal hashing algorithm, includes the following steps: S21. Based on the type of multimodal data and combined with the preset feature extraction method library, feature vectors are extracted for each type of mapped multimodal data, and the feature vectors of each type of data are associated with spatial coordinates to construct a multimodal cross semivariance function. S22. Generate cross-modal hash codes based on the multimodal cross semivariance function and the cross-modal hashing algorithm; S23. Based on the correlation between the multimodal cross-semivariance function and the cross-modal hash code, generate cross-modal space constraint rules.

3. The annotation method for multimodal data of new energy sources according to claim 2, characterized in that, The steps of extracting feature vectors for each type of mapped multimodal data based on the type of multimodal data and in conjunction with a pre-defined feature extraction method library, and then associating the feature vectors of each modality with spatial coordinates to construct a multimodal cross-semivariance function include the following: S211. Based on the type of multimodal data, extract the corresponding feature extraction method from the preset feature extraction method library, and perform feature extraction on each type of mapped modal data to obtain the feature vector of each modality. S212. Based on the spatial attributes of each modality data, configure a spatial coordinate system for each modality data, and bind the feature vector of each modality to the corresponding spatial coordinate system to form a structured association pair; S213. Based on the structured association pairs, construct the initial multimodal cross semivariogram function, and optimize the initial multimodal cross semivariogram function by extracting spatial statistical parameters to obtain the final multimodal cross semivariogram function.

4. The annotation method for multimodal data of new energy sources according to claim 3, characterized in that, The steps involved in constructing an initial multimodal cross-semivariance function based on structured association pairs, and then optimizing the initial multimodal cross-semivariance function by extracting spatial statistical parameters to obtain the final multimodal cross-semivariance function. S2131. Based on the spatial coordinate system of structured association pairs, calculate the cross semivariance value between feature vectors of different modes with a preset spatial distance step size as the interval. S2132. Use a spherical model to fit the cross semivariance values ​​and generate the initial multimodal cross semivariance function; S2133. Extract spatial statistical parameters from the initial semivariance function, and adjust and optimize the spatial statistical parameters based on the physical characteristics of the new energy equipment to obtain optimized spatial statistical parameters; the spatial statistical parameters include nugget constant, sill value and range. S2134. Substitute the optimized spatial statistical parameters into the initial multimodal cross semivariogram function to generate the final multimodal cross semivariogram function.

5. The annotation method for multimodal data of new energy sources according to claim 4, characterized in that, The method of generating cross-modal hash codes based on the multimodal cross semivariance function and combined with the cross-modal hashing algorithm includes the following steps: S221. Map the parameters of the multimodal cross semivariogram function to the input layer of the cross-modal hashing algorithm to impose space constraints on the cross-modal hashing algorithm; S222. Based on the feature vector of each modality data, combined with spatial constraints, feature standardization and hash encoding are performed to generate an initial hash code; S223. A cross-modal comparison learning strategy based on the FLIP model is used to optimize the initial hash code to obtain an optimized hash code. S224. Based on the preset spatial parameter threshold of the multimodal cross semivariogram function, verify the spatial constraint satisfaction of the optimized hash code, and output the final cross-modal hash code after the verification is passed.

6. The annotation method for multimodal data of new energy sources according to claim 5, characterized in that, The cross-modal comparison learning strategy based on the FLIP model optimizes the initial hash code to obtain the optimized hash code, including the following steps: S2231. Based on the nugget constant and range in the multimodal cross-variance function, adjust the Hamming distance distribution of the initial hash code to obtain the adjusted hash code; S2232. Using the cross-modal comparative learning strategy of the FLIP model, semantic alignment is performed on the feature vectors of each modality to generate semantically aligned feature vectors. S2233. The hash bit length is dynamically adjusted based on the base value, and the hash code is regenerated by combining the semantically aligned feature vector to obtain the optimized hash code.

7. The annotation method for multimodal data of new energy sources according to claim 2, characterized in that, The generation of cross-modal space constraint rules based on the correlation between the multimodal cross-semivariance function and the cross-modal hash code includes the following steps: S231. Map the spatial statistical parameters of the multimodal cross semivariogram function to the spatial constraint dimension of the cross-modal hash code to obtain the parameter mapping result; S232. Based on the parameter mapping results and combined with the Hamming distance distribution of cross-modal hash codes, preliminary spatial constraint rules are generated. S233. Based on the physical characteristics and real-time operating data of new energy equipment, the preliminary spatial constraint rules are dynamically optimized to obtain the optimized spatial constraint rules. S234. Verify the optimized spatial constraint rules, and output the final cross-modal spatial constraint rules after the verification is passed.

8. A labeling system for multimodal data of new energy sources, used to implement the labeling method for multimodal data of new energy sources as described in any one of claims 1-7, characterized in that, The system includes: The data acquisition and processing module is used to acquire multimodal data of new energy equipment, preprocess the multimodal data, and map the preprocessed multimodal data to the same feature space to obtain the mapped multimodal data. The constraint rule generation module is used to perform multimodal spatial statistical analysis on the feature vectors of the mapped multimodal data based on the type of multimodal data, using the semivariance function, and to generate cross-modal spatial constraint rules in combination with the cross-modal hashing algorithm. The data annotation module is used to annotate and classify multimodal data of new energy equipment based on cross-modal space constraint rules, and generate annotation results for new energy multimodal data.

9. An electronic device, characterized in that, The electronic device includes: one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the steps of the method according to any one of claims 1 to 7 are implemented when the computer program controls the device containing the computer-readable storage medium to execute during runtime.

Images