Electrical component identification method and device based on multi-source operation data fusion, computer equipment, readable storage medium and program product

Abstract

The invention relates to an electrical component identification method and device based on multi-source operation data fusion, computer equipment, a computer readable storage medium and a computer program product, relates to the technical field of equipment identification, and can improve the accuracy of electrical component identification. The method comprises the following steps: synchronously triggering different types of sensors to obtain various operation data in the operation process of the electrical component, and determining physical position association of the operation data through spatial registration to obtain electrical component identification data corresponding to each operation data; inputting the electrical component identification data into an identification model, determining feature information, and obtaining a dependency relationship between the feature information to generate interactive feature information; fusing the interactive feature information of the identification data of each electrical component based on the dynamic feature fusion weight to obtain fused feature information; and outputting a type identification result of the electrical component and a current operation state of the electrical component from the identification model based on the fused feature information.

Description

Technical Field

[0001] The present application relates to the field of device identification technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for identifying electrical components based on multi-source operating data fusion. Background Art

[0002] With the continuous improvement of the intelligent level of power systems, online identification and operating status monitoring of electrical components have become an important basis for ensuring the safe operation of equipment. Traditional identification methods mainly rely on single sensors or single-modal data. In recent years, multimodal fusion recognition has become a research hotspot. The combined use of multiple sensor data types such as images, sound, temperature, and current can improve the accuracy and stability of recognition. However, existing multimodal methods still face the problem of limited effective interaction and difficulty in fusion of multi-source data features in practical engineering applications, resulting in low recognition accuracy of electrical components. Summary of the Invention

[0003] Based on this, it is necessary to provide an electrical component identification method, device, computer equipment, computer-readable storage medium and computer program product based on multi-source operation data fusion to address the above technical problems.

[0004] In a first aspect, the present application provides an electrical component identification method based on multi-source operation data fusion, comprising:

[0005] By synchronously triggering multiple sensors of different types, a variety of operating data during the operation of the electrical component is acquired, and the physical position associations between the various operating data are determined through spatial registration to obtain electrical component identification data corresponding to each type of operating data; the electrical component identification data corresponding to each type of operating data includes the operating data and the physical position associations of the operating data with other types of operating data; the various types of operating data include image sensor data and time series sensor data;

[0006] Inputting the electrical component identification data into a pre-trained recognition model, determining feature information of each electrical component identification data, and obtaining dependency relationships between different feature information to generate interactive feature information between each electrical component identification data;

[0007] Determining a dynamic feature fusion weight, and based on the dynamic feature fusion weight, fusing the interactive feature information of each electrical component identification data to obtain fused feature information;

[0008] Based on the fused feature information, the identification model outputs a type identification result of the electrical component and a current operating status of the electrical component.

[0009] In one embodiment, after acquiring a plurality of operating data during the operation of the electrical component, and before determining the spatial position association between the plurality of operating data through spatial registration and obtaining the electrical component identification data corresponding to each operating data, the method further includes:

[0010] For any two types of sensors, if the sampling timestamps of the operating data acquired by the two types of sensors are different, interpolation calculation is performed on the operating data according to the different sampling timestamps to obtain time-aligned operating data;

[0011] The determining of the physical position association between the operating data through spatial registration processing to obtain electrical component identification data corresponding to each type of operating data includes:

[0012] Through the spatial calibration matrix, the target area of the image sensor data in the time-aligned operating data and the preset physical position of the corresponding sensor of the timing sensor data in the time-aligned operating data are spatially aligned to obtain electrical component identification data.

[0013] In one embodiment, the recognition model includes multiple convolutional neural networks, and determining the characteristic information of each electrical component identification data includes:

[0014] Inputting the image sensor data of the electrical component recognition data into a convolutional neural network including a plurality of residual blocks, and obtaining the attention weights of the image sensor data in the image channel direction and the physical position direction through the coordinate attention unit embedded in each residual block;

[0015] Determining feature information corresponding to the image sensor data based on the attention weight and the current output of the residual block;

[0016] dividing the time series sensor data of the electrical component identification data into time series segments of different time lengths;

[0017] Each of the time series segments is input into a convolutional neural network with different convolution kernel sizes to determine the characteristic information of the time series sensor data at different time lengths.

[0018] In one embodiment, performing cross-modal feature interactive learning on the feature information to determine the dependency between different feature information to generate interactive feature information between the electrical component identification data includes:

[0019] Determining, based on the physical location association, a characteristic correlation between characteristic information corresponding to each of the electrical component identification data; the characteristic correlation is used to characterize a dependency relationship between each of the electrical component identification data;

[0020] Determine interactive feature information of each electrical component identification data according to the feature association degree and the feature information.

[0021] In one embodiment, determining the dynamic feature fusion weight includes:

[0022] splicing the interactive feature information of the identification data of each electrical component to obtain joint feature information;

[0023] Inputting the joint feature information into a preset gated network structure to obtain a network output result; the gated network structure uses adjustable weight parameters and bias parameters to perform a linear transformation on the joint feature information;

[0024] According to the network output result, the dynamic feature fusion weight is determined.

[0025] In one embodiment, the recognition model is trained by the following steps, including:

[0026] Obtain historical electrical component identification data;

[0027] Inputting part of the identification data in the historical electrical component identification data into the identification model to be trained, extracting training interaction feature information of each part of the identification data, and performing data fusion on each part of the identification data based on the training interaction feature information to obtain training fusion feature information;

[0028] According to the fusion feature information, the recognition model to be trained outputs a training type recognition result and a training running status;

[0029] Determining a loss value of a preset loss function based on the training type identification result and the training running status;

[0030] According to the loss value, the parameters of the recognition model to be trained are updated to determine the current model training difficulty, and based on the current model training difficulty, historical recognition data are added to the partial recognition data to obtain updated partial recognition data. The recognition model to be trained is trained again based on the updated partial recognition data. When the training end conditions are met, the trained recognition model is obtained.

[0031] In a second aspect, the present application further provides an electrical component identification device based on multi-source operation data fusion, comprising:

[0032] An electrical component identification data acquisition module is configured to synchronously trigger multiple sensors of different types to acquire various operating data during the operation of the electrical component, and determine the physical position associations between the various operating data through spatial registration to obtain electrical component identification data corresponding to each type of operating data; the electrical component identification data corresponding to each type of operating data includes the operating data and the physical position associations between the operating data and other types of operating data; the various types of operating data include image sensor data and time series sensor data;

[0033] an interactive feature information determination module, configured to input the electrical component identification data into a pre-trained recognition model, determine feature information of each electrical component identification data, and obtain dependency relationships between different feature information to generate interactive feature information between each electrical component identification data;

[0034] an interactive feature information fusion module, which determines a dynamic feature fusion weight and fuses the interactive feature information of each electrical component identification data based on the dynamic feature fusion weight to obtain fused feature information;

[0035] The electrical component identification module is used to output the type identification result of the electrical component and the current operating status of the electrical component from the identification model based on the fused feature information.

[0036] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0037] By synchronously triggering multiple sensors of different types, a variety of operating data during the operation of the electrical component is acquired, and the physical position associations between the various operating data are determined through spatial registration to obtain electrical component identification data corresponding to each type of operating data; the electrical component identification data corresponding to each type of operating data includes the operating data and the physical position associations of the operating data with other types of operating data; the various types of operating data include image sensor data and time series sensor data;

[0038] Inputting the electrical component identification data into a pre-trained recognition model, determining feature information of each electrical component identification data, and obtaining dependency relationships between different feature information to generate interactive feature information between each electrical component identification data;

[0039] Determining a dynamic feature fusion weight, and based on the dynamic feature fusion weight, fusing the interactive feature information of each electrical component identification data to obtain fused feature information;

[0040] Based on the fused feature information, the identification model outputs a type identification result of the electrical component and a current operating status of the electrical component.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0042] By synchronously triggering multiple sensors of different types, a variety of operating data during the operation of the electrical component is acquired, and the physical position associations between the various operating data are determined through spatial registration to obtain electrical component identification data corresponding to each type of operating data; the electrical component identification data corresponding to each type of operating data includes the operating data and the physical position associations of the operating data with other types of operating data; the various types of operating data include image sensor data and time series sensor data;

[0043] Inputting the electrical component identification data into a pre-trained recognition model, determining feature information of each electrical component identification data, and obtaining dependency relationships between different feature information to generate interactive feature information between each electrical component identification data;

[0044] Determining a dynamic feature fusion weight, and based on the dynamic feature fusion weight, fusing the interactive feature information of each electrical component identification data to obtain fused feature information;

[0045] Based on the fused feature information, the identification model outputs a type identification result of the electrical component and a current operating status of the electrical component.

[0046] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0047] By synchronously triggering multiple sensors of different types, a variety of operating data during the operation of the electrical component is acquired, and the physical position associations between the various operating data are determined through spatial registration to obtain electrical component identification data corresponding to each type of operating data; the electrical component identification data corresponding to each type of operating data includes the operating data and the physical position associations of the operating data with other types of operating data; the various types of operating data include image sensor data and time series sensor data;

[0048] Inputting the electrical component identification data into a pre-trained recognition model, determining feature information of each electrical component identification data, and obtaining dependency relationships between different feature information to generate interactive feature information between each electrical component identification data;

[0049] Determining a dynamic feature fusion weight, and based on the dynamic feature fusion weight, fusing the interactive feature information of each electrical component identification data to obtain fused feature information;

[0050] Based on the fused feature information, the identification model outputs a type identification result of the electrical component and a current operating status of the electrical component.

[0051] The above-mentioned electrical component identification method, device, computer equipment, computer-readable storage medium and computer program product based on multi-source operation data fusion obtains various operation data during the operation of the electrical component by synchronously triggering multiple sensors of different types, and determines the physical position association between the multiple operation data through spatial alignment to obtain the electrical component identification data corresponding to each operation data; inputs the electrical component identification data into a pre-trained recognition model to determine the feature information of each electrical component identification data, and obtains the dependency relationship between different feature information to generate the interactive feature information between each electrical component identification data; determines the dynamic feature fusion weight, and based on the dynamic feature fusion weight, fuses the interactive feature information of each electrical component identification data to obtain fused feature information; based on the fused feature information, the recognition model outputs the type recognition result of the electrical component and the current operation status of the electrical component. In this application, by synchronously triggering the acquisition of image and time series sensor data, establishing a spatial correspondence between the data, and constructing recognition data under a unified reference system, the foundation for multimodal collaborative analysis is laid; the data is input into a pre-trained recognition model to extract the feature information of multi-source data respectively, and further explore the interaction between different modalities in structural position and dynamic response to form a feature interaction expression that can be used for fusion; on this basis, a fusion weight mechanism is determined to fuse multi-source features into a unified representation for outputting the type recognition and operation status judgment of electrical components, thereby effectively improving the accuracy of electrical component recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 FIG2 is an application environment diagram of an electrical component identification method based on multi-source operation data fusion in one embodiment;

[0054] Figure 2 is a flow chart of an electrical component identification method based on multi-source operation data fusion in another embodiment;

[0055] Figure 3 is a structural block diagram of an electrical component identification device based on multi-source operation data fusion in one embodiment;

[0056] Figure 4 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0058] In one embodiment, Figure 1 As shown, a method for identifying electrical components based on the fusion of multi-source operating data is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0059] In step S102 , a plurality of different types of sensors are synchronously triggered to obtain a variety of operating data during the operation of the electrical component, and the physical position association between the various operating data is determined through spatial registration to obtain electrical component identification data corresponding to each type of operating data.

[0060] Among them, the electrical component identification data corresponding to each type of operating data includes the operating data and the physical location association between the operating data and other operating data; the various operating data include image sensor data and time series sensor data.

[0061] Synchronous triggering can be achieved by using a specific control mechanism to enable multiple independent or distributed sensors to start collecting data at the same time or with a pre-set and known time relationship.

[0062] Spatial registration can be the process of converting data about the same electrical component or its scene obtained from different sensors, different perspectives or different times into the same unified spatial coordinate system. It involves the calculation and application of parameters of geometric transformations (such as translation, rotation, scaling, affine transformation, and projection transformation) to establish the correspondence between different data points or areas in physical space, enabling accurate comparison and fusion of spatial information from different sources.

[0063] The electrical component identification data can be a data unit formed by encapsulating or combining the data content of each type of operating data with the physical location association information between it and other operating data, and is used to provide a regular input containing spatial context for subsequent feature extraction and fusion steps.

[0064] Image-based sensor data can be sensor data presented in the form of images, such as a two-dimensional or three-dimensional pixel array, where each pixel value represents light intensity, temperature, depth, or other physical quantities at a specific spatial location. For example, visible light images captured by industrial cameras can be used to identify signs of faults such as localized overheating and abnormal heat dissipation, while infrared thermal images captured by infrared thermal imagers can be used to detect abnormalities such as structural damage, discoloration, and foreign matter.

[0065] Time-series sensor data can be a series of data points recorded in chronological order, each representing the measured value of a physical quantity at a specific moment. For example, current signals collected by current transformers, such as Hall effect current sensors, monitor operating current changes in real time, reflecting electrical load and abnormal fluctuations; sound signals collected by microphones are used to identify abnormal audio features such as breakdown sounds and arcing sounds; and temperature curves collected by temperature sensors can be used to monitor the temperature rise of key components using contact or non-contact temperature sensors.

[0066] Optionally, adaptive histogram equalization (AHE) is performed on the captured infrared and visible light images, respectively. This operation enhances the local contrast of the image, allowing for better visualization of details in both bright and dark areas. This helps highlight the key structural contours and potential subtle anomalies of electrical components. Discrete wavelet transform-based image denoising techniques are then applied to the contrast-enhanced image data. By decomposing the image into wavelet coefficients of different scales and directions and performing thresholding on these coefficients (e.g., soft thresholding or hard thresholding denoising), background random noise introduced during image acquisition (e.g., sensor thermal noise and noise generated by environmental electromagnetic interference) can be effectively suppressed while effectively preserving image edges and detail information.

[0067] Optionally, the collected raw sound signal can first be subjected to spectral subtraction for noise reduction. By estimating the power spectrum of the background noise (for example, from sound clips when the device is not operating or operating smoothly) and subtracting this noise spectrum from the power spectrum of each frame of the noisy sound signal, the signal-to-noise ratio of fault-related sound features (such as discharge sounds and abnormal noises) can be effectively improved.

[0068] The terminal further converts the denoised sound signal into a Mel-spectrogram. This process involves framing the signal, windowing it, and performing a fast Fourier transform to obtain the spectrum. The spectrum is then mapped to the Mel-frequency scale using a Mel filter bank, and the logarithmic energy is taken for use in sound event recognition and fault diagnosis.

[0069] Optionally, for time series data such as current signals and temperature signals collected by temperature sensors, the terminal first performs sliding window segmentation processing and normalizes the maximum and minimum values of the data within each window to improve the uniformity of the characteristics of different sensors.

[0070] Exemplarily, a terminal integrates, connects, and controls multiple different types of sensors. First, the terminal uses its internal synchronization control mechanism or issues commands to external synchronization devices to enable various sensors deployed within the electrical component or its monitoring environment (for example, but not limited to, at least one image sensor and one timing sensor) to initiate data collection based on a time base that is as consistent as possible. These sensors capture the operating data of the electrical component within a set monitoring period, forming a preliminary multimodal dataset. Subsequently, the terminal spatially registers the collected operating data from different sources based on preset sensor layout information, loaded calibration parameters, or real-time reference object information. The goal of spatial registration is to determine a unified reference coordinate system and map the spatial information of all data sources into this coordinate system, thereby clarifying the relative positional relationships (i.e., physical location associations) between the specific physical locations or regions of the electrical component corresponding to different data streams. Finally, for each sensor-generated operating data, the terminal combines the raw data content with the calculated physical location association information to form structured electrical component identification data for use in subsequent steps.

[0071] Step S104 , inputting the electrical component identification data into a pre-trained identification model, determining feature information of each electrical component identification data, and obtaining dependency relationships between different feature information to generate interactive feature information between each electrical component identification data.

[0072] In the context of multimodal feature information, dependency refers to one or more of the following: correlation, influence, or conditional probability relationships between feature information from different sources or types. For example, changes in a particular image feature can be highly correlated with the pattern of a specific temporal feature, or the presence of one feature can indicate the probability of the presence of another.

[0073] For example, the terminal first feeds the electrical component identification data acquired in the previous step, including operational data and its associated physical location, into the corresponding input interfaces of one or a series of pre-trained recognition models, either individually or jointly. This recognition model internally comprises at least one feature extraction phase and one feature interaction phase. In the feature extraction phase, the model uses its internal feature extractors (e.g., different branches or modules of a deep neural network) to independently or in parallel calculate and output initial feature information for each type of electrical component identification data (e.g., image data or various types of time series data). This initial feature information effectively represents the original input data at different levels of abstraction. Next, in the feature interaction phase, the model analyzes and quantifies the interdependencies between these initial feature information extracted from different data sources. Based on the identified dependencies, the model adjusts, enhances, or combines the initial feature information to generate interactive feature information. This interactive feature information aims to capture cross-modal synergies or indicative associations, providing richer input for subsequent fusion and decision-making.

[0074] Step S106 , determining a dynamic feature fusion weight, and based on the dynamic feature fusion weight, fusing the interactive feature information of each electrical component identification data to obtain fused feature information.

[0075] For example, a pre-trained recognition model deployed within the terminal receives as input the interactive feature information from the previous step, which belongs to the identification data of different electrical components. Based on the content and characteristics of this input interactive feature information, the weight determination mechanism dynamically calculates a set of dynamic feature fusion weights, each of which corresponds to one or a group of interactive feature information, reflecting its relative importance or contribution in the current context. The recognition model then performs a weighted combination or integration of the corresponding interactive feature information based on this set of dynamic feature fusion weights, ultimately outputting a single, more information-dense fused feature information vector or tensor.

[0076] Step S108: Based on the fused feature information, the recognition model outputs the type recognition result of the electrical component and the current operating status of the electrical component.

[0077] The type recognition result can be a judgment or classification of the monitored electrical component's identity category, determined by a pre-trained recognition model after processing the fused feature information. This result can be a discrete label or a probability distribution over a predefined list of component types, indicating the specific type the model identifies (e.g., circuit breaker, transformer, disconnector, capacitor, etc., or even more specific models). The current operating status is the recognition model's assessment and determination of the operating condition and health level of the electrical component at the moment of monitoring. This can take various forms, such as discrete category labels indicating normal, warning, and fault levels; labels describing specific failure modes (e.g., partial discharge, mechanical wear, overheating); continuous numerical values representing health status (e.g., health index or remaining life prediction); or health state probability distribution, i.e., the probability of a component belonging to each predefined health state (e.g., normal, caution, minor fault, major fault, etc.).

[0078] For example, the terminal uses the fused feature information generated in the previous step as input to a pre-trained recognition model. This fused feature information is then analyzed and mapped to produce two primary results: the electrical component type, which clearly indicates the predefined category to which the monitored electrical component belongs; and the electrical component's current operating status, which describes the component's current health. Together, these two results constitute a complete identification and assessment of the electrical component.

[0079] In this embodiment, image and time series sensor data are collected by synchronous triggering, and a spatial correspondence between the data is established to construct recognition data under a unified reference system, laying the foundation for multimodal collaborative analysis; the data is input into a pre-trained recognition model to extract feature information of multi-source data respectively, and further explore the interaction between different modalities in structural position and dynamic response to form a feature interaction expression that can be used for fusion; on this basis, a fusion weight mechanism is determined to fuse multi-source features into a unified representation for outputting type recognition and operating status judgment of electrical components, thereby effectively improving the accuracy of electrical component recognition.

[0080] In an exemplary embodiment, after acquiring a plurality of operating data during the operation of the electrical component, determining the spatial position association between the plurality of operating data by spatial registration, and before obtaining the electrical component identification data corresponding to each operating data, the method further includes:

[0081] For any two types of sensors, if the sampling timestamps of the operation data acquired by the two types of sensors are different, the operation data are interpolated according to the different sampling timestamps to obtain time-aligned operation data.

[0082] Among them, the sampling timestamp can be metadata associated with each data point or data sample collected by the sensor, which can mark the absolute time (for example, year, month, day, hour, minute, second, millisecond, microsecond) when the data point is captured or recorded by the sensor or the time relative to a reference starting point.

[0083] Interpolation calculation is used to estimate or construct the value of new unknown data points located between a series of known discrete data points (for example, sensor readings at different sampling timestamps). In some embodiments, interpolation can be based on a variety of algorithms, such as linear interpolation, polynomial interpolation, spline interpolation, etc., allowing data sampled at different time intervals or non-uniform intervals to be converted into a representation on a unified, more dense sequence of time points, or to resample the data to a specific required time point to achieve time alignment.

[0084] Specifically, for the operating data collected by any two or more types of sensors, if the terminal detects that there is a difference in their respective sampling timestamp sequences (ie, not all corresponding observation points have the same timestamp), the terminal can start an interpolation calculation program.

[0085] In a specific embodiment, the terminal first analyzes all the running data streams to be aligned (for example, the data stream from sensor A). , whose timestamp is ;Data stream from sensor B , whose timestamp is .if and If they are inconsistent, the terminal will determine a unified target timestamp sequence As an alignment reference. Optionally, It can be a timestamp sequence of one of the running data streams. For example, the timestamp of the data stream with the densest sampling points (highest sampling rate) or the most reliable time base (such as a sensor with external precise timing) is selected as . Optionally, It can be a newly generated, equally spaced time series covering the time range of all data streams with a preset time interval (for example, every 1 millisecond, every 10 milliseconds, or other intervals set according to specific analysis requirements). It can also be a set of time points triggered by a specific event (such as a key signal exceeding a threshold).

[0086] For those original timestamp sequences with Inconsistent data streams, the terminal will At each time point On the other hand, the estimated value is calculated using the selected interpolation algorithm, for example, the calculation is completed by formula (1). After the above calculation is completed for all data streams that need to be interpolated, the original data and the interpolated data together constitute the unified time base. This means that for At any point in time in the dataset, we now have data values (raw or interpolated) from all relevant sensors, and these data can be considered to be acquired at the same time.

[0087]

[0088] in is the data aligned at timestamp t, represents the interpolation operation, The original timestamp of the i-th sensor Collected data.

[0089] In one specific embodiment, when monitoring an electrical component, a current sensor records the current waveform at a frequency of 5,000 points per second (5 kHz), while a temperature sensor records the component surface temperature at a frequency of 1 point per second (1 Hz). If the terminal decides to align the data at 1 millisecond (1 kHz) intervals, the current data may require some decimation or smoothing (if the original points are too dense), while the temperature data requires interpolation of 999 new temperature estimates between every two original sample points to match the 1 millisecond time grid. The terminal automatically selects the appropriate interpolation method (such as linear or spline interpolation for slowly changing quantities such as temperature) to accomplish this based on a pre-defined strategy or data characteristics.

[0090] In this embodiment, interpolation calculations are performed on operating data obtained by different sensors with different sampling timestamps, thereby converting and unifying multi-source raw operating data with heterogeneous time sampling characteristics onto a common and precise time base. This process compensates for the discreteness and misalignment of the raw data on the time axis caused by factors such as inherent sampling rate differences of sensors, asynchronous startup, or transmission jitter through mathematical estimation.

[0091] In an exemplary embodiment, determining the physical position association between the operating data through spatial registration processing to obtain electrical component identification data corresponding to each type of operating data includes:

[0092] Through the spatial calibration matrix, the target area of the image sensor data in the time-aligned operation data and the preset spatial position of the corresponding sensor of the timing sensor data in the time-aligned operation data are spatially aligned to obtain the electrical component identification data.

[0093] A spatial calibration matrix is a set or a single mathematical matrix containing parameters that describe the geometric characteristics of one or more sensors and their position and orientation in three-dimensional space. For point-type or area-type time-series sensors, the associated calibration information may simply be their precise three-dimensional coordinates in a unified coordinate system or a geometric description of their sensing range. These matrices and parameters are precisely determined through a pre-calibration process using specific calibration objects or known geometric scenes.

[0094] A target region can be a specific portion or area within image data acquired by an imaging sensor (such as a visible light camera or infrared thermal imager) designated for analysis. This region may encompass an entire electrical assembly, a critical component within the assembly, or a portion of the image displaying a specific feature (such as a hotspot or crack). During spatial registration, the pixel coordinates of the target region, or the geometric shape it defines in the image (such as a rectangle or polygon), are mapped to a real-world physical coordinate system.

[0095] The preset physical location may be a physical installation location of a non-imaging timing sensor (such as a temperature sensor, a microphone, a measurement point of a current transformer, etc.) on an electrical component or in its surrounding environment that is pre-set or precisely measured.

[0096] Specifically, all sensors involved in data acquisition (especially image sensors and timing sensors that require precise positioning) can be spatially calibrated through terminals or external calibration equipment to obtain a spatial calibration matrix that describes their internal geometric parameters and / or external parameters (position and posture) relative to a common coordinate system or the coordinate system of the electrical component body.

[0097] In some optional embodiments, a target area is first determined in the image. This target area can be the entire image frame, or a region of interest in the image pre-circled by the user or automatically identified by other algorithms (such as target detection algorithms), such as a key component of an electrical assembly, a connection point, or a suspected abnormal area. The target area is defined in the image by a set of pixel coordinates (such as the coordinates of the upper left and lower right corners of a rectangle or the coordinates of the vertices of a polygon). The terminal then uses the spatial calibration matrix corresponding to the image sensor (particularly the internal and external parameters) to map the two-dimensional image coordinates of the target area to a three-dimensional world coordinate system.

[0098] The terminal processes time-aligned, time-series sensor data. For each time-series data stream, the terminal obtains the preset physical location of the corresponding sensor. These locations are points or small regions in three-dimensional space and are defined in the same world coordinate system to which the image data is ultimately mapped, or they can be converted to that world coordinate system using a known transformation relationship. Through the above steps, the target area of the image data and the spatial location of the time-series sensor are both expressed in the same unified world coordinate system.

[0099] Ultimately, the terminal encapsulates the time-aligned raw operational data (image data and various time series data) with the precise physical location association information established through the spatial calibration matrix-based registration process (for example, the mapping relationship between image pixels and 3D world coordinates, parameters indicating the 3D position of the time series sensor and its corresponding projected position in each image), forming structured electrical component identification data. Each piece of data not only contains its content but also carries its precise correspondence with other data sources in the shared physical space.

[0100] In this embodiment, a spatial calibration matrix is used to spatially align the target area of time-aligned image sensor data and the preset spatial position of the corresponding sensor of time series sensor data, thereby unifying multi-source sensor data from different physical principles and different spatial perception methods into a common three-dimensional physical space coordinate system with high precision. This provides input data with highly accurate spatial context and known geometric relationships for subsequent feature extraction, cross-modal feature interaction, and data fusion, enabling the model to more effectively learn and utilize the coordinated changes and complementary features of multi-source information at fine-grained physical locations, thereby significantly improving the accuracy of fault location, the accuracy of state assessment, and the ability to comprehensively analyze and fine-tune the understanding of local abnormal phenomena of electrical components.

[0101] In an exemplary embodiment, the recognition model includes multiple convolutional neural networks to determine feature information of each electrical component identification data, including:

[0102] The image sensor data of the electrical component recognition data is input into a convolutional neural network including multiple residual blocks. The attention weights of the image sensor data in the image channel direction and the physical position direction are obtained through the coordinate attention unit embedded in each residual block. The feature information corresponding to the image sensor data is determined based on the attention weight and the current output of the residual block. The time series sensor data of the electrical component recognition data is divided into time series segments of different time lengths. Each time series segment is input into a convolutional neural network with different convolution kernel sizes to determine the feature information of the time series sensor data at different time lengths.

[0103] Among them, the coordinate attention unit, unlike the attention mechanism that only focuses on the relationship between channels (such as SEAttention), not only captures the dependencies between channels, but also effectively encodes the long-range spatial dependencies with precise location information. It is achieved by decomposing the channel attention into two parallel one-dimensional attention processes that perform feature encoding along the image width (X direction) and height (Y direction). The attention maps in these two directions are then combined and applied to the input feature map, enabling the model to simultaneously focus on which feature channels are more important (channel attention) and which positions in space are more critical, thereby more accurately locating and identifying targets or regions of interest.

[0104] Time series segments can be subsequences of varying durations extracted from raw, potentially longer time series sensor data (such as current, sound, or temperature data). For example, a one-minute current signal can be segmented into multiple segments: short, 1-second segments, medium, 5-second segments, and long, 10-second segments. Functionally, this segmentation enables analysis of data at different time scales, capturing events and patterns of varying duration or frequency.

[0105] In convolutional neural networks processing one-dimensional time series data, the kernel size can be its length in the time dimension. For example, a kernel size of 3 will observe three consecutive time points simultaneously. Using kernels of different sizes allows the network to learn local temporal patterns corresponding to the kernel length.

[0106] Specifically, the terminal inputs image-based sensor data (e.g., infrared thermal images, visible light defect images) from electrical component recognition data into a dedicated image processing branch within the recognition model. This branch is a deep convolutional neural network (CNN) consisting of multiple sequentially connected residual blocks, such as a modified ResNet (residual network) architecture. Within each or selected residual blocks of this image processing branch, a coordinate attention unit is embedded. When an image feature map flows through a residual block, the coordinate attention unit performs the following operations: First, the input feature map is subjected to one-dimensional feature average pooling or more complex encoding along the width (X) and height (Y) directions, compressing the spatial information of each channel into two independent direction-aware feature maps. These two direction-aware feature maps are then concatenated and transformed through a shared convolutional layer (e.g., a 1x1 convolution) and nonlinear activation (e.g., a sigmoid or hard-sigmoid activation) to generate attention maps corresponding to the X and Y directions, respectively.

[0107] The attention maps in these two directions are then extended and jointly act on the input feature map or intermediate feature map of the residual block for feature weight calibration. This enables the model to capture spatial long-range dependencies with precise direction awareness and highlight features that are more important both in specific channels and specific spatial positions.

[0108] The feature map after being weighted and adjusted by the coordinate attention unit will continue to perform subsequent operations such as convolution in the residual block. Based on the current output of the residual block after being adjusted by the attention weights (for example, the final output feature map of this residual block, or in some designs, the feature map directly adjusted by the attention module), the feature information of this image class sensing data at this processing stage is determined and can be reflected by formula (2). The entire image processing branch stacks multiple such residual blocks and coordinate attention units, layer by layer extracting and optimizing the final image feature information that is highly sensitive to both spatial position and channel information.

[0109]

[0110] Among them, is the output of the residual block, is the feature after convolution and coordinate attention module.

[0111] Specifically, for time-series sensing data, the terminal inputs the time-series sensing data in the electrical component recognition data (such as current waveforms, sound signals, temperature sequences, etc.) into one or more dedicated time-series processing branches in the recognition model. First, the terminal (or the preprocessing module inside the model) divides the original, possibly long time-series sensing data into multiple time-series segments with different fixed time lengths according to a preset rule. For example, an original time-series signal with a length of T can be divided into a set of segments S1 with a length of L1, a set of segments S2 with a length of L2,..., a set of segments Sk with a length of Lk (where L1 < L2 <... < Lk). These segments can overlap or not. These groups of time-series segments with different time lengths are respectively input into convolutional neural networks with different convolutional kernel sizes (kernelsize).

[0112] For example, for short time-series segments with a length of L1, a CNN with a smaller convolutional kernel (such as size 3 or 5) may be used to capture high-frequency, rapidly changing local patterns in the signal. For long time-series segments with a length of Lk, a CNN with a larger convolutional kernel (such as size 15 or 31) may be used to capture low-frequency, trend-based long-term dependency patterns in the signal. These CNNs in different branches can share some underlying parameters or be completely independent network structures. Each CNN branch extracts features from the time-series segments of the corresponding time length through its specific convolutional kernel size and subsequent pooling, activation, etc. operations.

[0113] After each CNN branch processes all time series segments of its corresponding time length, its output (e.g., a feature vector after global average pooling or flattening) constitutes the feature information of the time series sensor data at that specific time length (or time scale). Ultimately, feature information from different time scales (CNN branches with different convolution kernel sizes) can be spliced or further fused through a nested temporal attention mechanism to form a multi-scale comprehensive feature representation of the time series signal.

[0114] In this embodiment, firstly, by dual dynamic attention and information screening of information channels and spatial position dimensions in the process of image feature extraction, and secondly by dividing it into time series segments of different time lengths and inputting them into convolutional neural networks with different convolution kernel sizes to determine feature information, the model's perception of local anomalies of electrical components, subtle structural defects or specific spatial patterns, as well as the comprehensive perception and understanding of time series events and state evolution patterns with different durations and frequency characteristics are enhanced, thereby improving the learning efficiency and generalization ability of the model in practical applications.

[0115] In an exemplary embodiment, cross-modal feature interaction learning is performed on feature information to determine dependencies between different feature information to generate interactive feature information between identification data of various electrical components, including:

[0116] Based on the physical position association, the characteristic association degree between the characteristic information corresponding to each electrical component identification data is determined; based on the characteristic association degree and the characteristic information, the interactive characteristic information of each electrical component identification data is determined.

[0117] Feature correlation describes the degree of interrelationship, influence, or coupling between feature information extracted from different sensors (corresponding to identification data of different electrical components) with known connections in physical space. It can be a learned weight, a calculated similarity score, or a numerical value representing conditional probability. Feature correlation accurately reflects the dependencies between different modal features within a specific physical spatial correspondence, specifically the extent to which changes or states of one feature are synchronized, correlated, or indicative of changes or states of features associated with another physical location.

[0118] Specifically, the terminal retrieves the physical location association information established in the previous step for each electrical component identification data (i.e., each modal data). This information specifies, for example, which region (pixel set) in image A corresponds to which region in image B, or whether a region in image A is spatially adjacent to or overlaps with the physical probe position of timing sensor C. Based on this physical location association, the recognition model within the terminal employs a specific mechanism to determine the degree of feature correlation between physically associated feature information from different modalities.

[0119] In one embodiment, when adopting a framework similar to the cross-modal attention mechanism, the physical location association information can be used to adjust the calculation of the attention weight. For example, for a query feature (from modal , corresponding to the physical region A) and a set of key features (from mode j, corresponding to physical areas B, C, D, etc.), if the physical location association shows that area A is highly related to area B (such as overlapping or very close), the model will calculate and When paying attention to each other, a higher initial bias or a stronger interaction function can be given. Alternatively, the position information itself (such as the encoding of relative coordinates and distance) can be input into the attention mechanism as part of the feature, allowing the model to naturally take physical position associations into account during the learning process, so that the calculated attention weights (i.e., the feature association here) can reflect the spatial dependency. For example, if dot-product-based attention is used, its calculation can be expressed as .in and The position information has been incorporated into the representation of or its calculation has been constrained by the position relationship. That is the feature correlation.

[0120] After determining the feature correlations that characterize the dependency relationships, the terminal uses these correlations (expressed as a set of weights or modulation coefficients) and the original feature information of each modality to generate the interactive feature information of each modality, as shown in formula (3).

[0121]

[0122] in, is the query matrix of the i-th modal feature, is the key matrix and value matrix of the j-th modal feature, is the dimension of the key matrix.

[0123] For example, in one application scenario, the terminal processes temperature distribution feature information extracted from an infrared image of an electrical connector, along with surface texture and color feature information extracted from a visual light image of the same connector, accurately identified through spatial alignment. These two sets of feature information share a clear physical location relationship due to their source data (i.e., they describe different aspects of the same physical object at the same location). The feature interaction module within the recognition model then calculates the correlation between the two sets of features in the corresponding regions. If the infrared feature indicates a high temperature at a certain point, while the visible light feature at the same point shows a darkened or oxidized material texture, the model will calculate a high positive correlation between the two. Subsequently, when generating the interactive feature information, the high-temperature infrared feature may be enhanced due to this high correlation (because it provides visual evidence). Furthermore, the visual feature may also receive a higher weight when characterizing an anomaly due to the high temperature information at the corresponding location. This generated interactive feature information is more powerful in indicating a potential fault at the connector than temperature or visual information alone.

[0124] In this embodiment, by utilizing the feature correlation calculated based on the physical position association to generate interactive feature information, a more accurate and reliable integration and enhancement of multi-physical quantity feature information that co-occurs, couples, or confirms each other in space is achieved, so that the generated interactive feature information can more deeply reflect the comprehensive state of electrical components under the joint action of specific local and multi-faceted factors.

[0125] In an exemplary embodiment, determining a dynamic feature fusion weight includes:

[0126] The interactive feature information of the identification data of each electrical component is spliced to obtain joint feature information; the joint feature information is input into a preset gated network structure to obtain the network output result; the gated network structure uses adjustable weight parameters and bias parameters to perform a linear transformation on the joint feature information; and the dynamic feature fusion weight is determined based on the network output result.

[0127] The joint feature information may be a higher-dimensional comprehensive feature representation formed by combining interactive feature information from different electrical component identification data (i.e., different modalities) that has undergone interactive processing through a splicing operation.

[0128] The preset gated network structure refers to a specific type of neural network layer or small network module that is pre-defined in the recognition model, which can be used as a gating mechanism to dynamically generate weights. Although the structure is preset, the adjustable weight parameters and bias parameters contained in it are optimized through learning during the model training phase. It is used to receive joint feature information as input and output a set of dynamic feature fusion weights for adjusting the contribution of each modal feature.

[0129] The weight parameter (is a matrix such as ) determines how each element of the input feature (here the joint feature information) linearly contributes to the calculation of the next layer; the bias parameter (is a vector, such as ) provides an additional learnable offset for the linear transformation. The values of these parameters are iteratively updated during model training through the back-propagation algorithm and optimizer to minimize the model's loss function, allowing the gating network to more accurately generate effective dynamic fusion weights.

[0130] Specifically, the terminal first collects the interactive feature information of each electrical component identification data (i.e., each modality) from the previous step (feature interaction stage). For example, , these interactive feature information can be in the form of vectors or tensors, , , , , which are visual features, infrared features, sound features, and current features. The terminal splices these interactive feature information along a preset dimension to obtain , [;] represents the splicing operation.

[0131] The terminal will obtain the joint feature information from the previous step As input, it is fed into a preset gated network structure inside the recognition model. The gated network structure uses adjustable (i.e., learned during the model training phase) weight parameters Wg and bias parameters Bg to adjust the joint feature information of the input. Perform a linear transformation. For example, the linear transformation can be expressed as: , this linear transformation can be regarded as the result of the gating network's preliminary analysis and mapping of the joint feature information, and its dimension is designed to match the number of dynamic fusion weights that need to be generated (for example, if a weight needs to be generated for each of N modalities, the linear change result may be an N-dimensional vector). After obtaining the above linear transformation result, the gating network will further process it to generate the final dynamic feature fusion weight G. Exemplarily, the terminal will process the linear transformation result Z through an activation function (such as a sigmoid function or a softmax function, depending on the desired characteristics of the weight) to obtain the dynamic feature fusion weight G. Optionally, the calculation of G is completed by formula (4).

[0132]

[0133] in is the sigmoid function, 、 is a trainable parameter, [;] represents the feature concatenation operation, , , , , respectively visual features, infrared features, sound features, and current features.

[0134] At this point, all element values in the output G are also in the range (0, 1), and their sum is 1, which allows G to be directly used as a normalized weight for proportionally distributing contributions. This generated set of G (for example, a vector containing N elements, [g1, g2, ..., gN]) is the final dynamic feature fusion weight corresponding to each source modality of the original interactive feature information.

[0135] In this embodiment, the interaction feature information of each modality is spliced and input into a preset gated network structure that uses adjustable parameters for linear transformation (and combines an activation function), so as to determine the dynamic feature fusion weight according to its output. This lays a solid foundation for forming a final fusion feature that can more accurately and robustly characterize the complex state of electrical components, and indirectly improves the diagnostic accuracy of the entire recognition model and its generalization ability for variable working conditions.

[0136] In an exemplary embodiment, the recognition model is trained by the following steps, including:

[0137] Obtain historical electrical component recognition data; input part of the recognition data in the historical electrical component recognition data into the recognition model to be trained, extract the training interaction feature information of each part of the recognition data, and fuse the data of each part of the recognition data according to the training interaction feature information to obtain training fusion feature information; according to the fusion feature information, the recognition model to be trained outputs the training type recognition result and the training operation status; based on the training type recognition result and the training operation status, determine the loss value of the preset loss function; according to the loss value, update the parameters of the recognition model to be trained, determine the current model training difficulty, and according to the current model training difficulty, add historical recognition data to the partial recognition data to obtain updated partial recognition data, and train the recognition model to be trained again according to the updated partial recognition data. When the training end condition is met, a trained recognition model is obtained.

[0138] The historical electrical component identification data can be a batch of data sets collected and prepared before model training begins. This data set contains a large amount of operating data of electrical components at different times and under different operating conditions in the past. This historical data has been processed into an electrical component identification data format that associates raw sensor readings with corresponding physical locations. The historical data is also accurately labeled, indicating the actual electrical component type corresponding to each data sample and its actual operating status or fault type at the time.

[0139] Model training difficulty is an assessment of the degree of learning challenge faced by the current recognition model during the training process. It can be based on multiple indicators, such as the average loss of current historical electrical component recognition data on the model, the number of training rounds, etc., as a key basis for adjusting data supply in the dynamic course learning strategy. In some embodiments, the dynamic course learning strategy starts with simple samples or tasks, and gradually increases the difficulty and complexity of training samples after the model masters the basic capabilities. Among them, the difficulty assessment and sample selection are adaptively adjusted according to the actual performance of the model during the training process.

[0140] Specifically, the terminal first accesses or loads a pre-collected and annotated historical electrical component identification dataset. This dataset contains multi-source sensor data of numerous electrical components in various known operating states and their corresponding true labels (i.e., the correct component type and operating state).

[0141] The terminal adopts a dynamic course learning strategy. At the beginning of training, a portion of relatively simple recognition data is selected from all historical data as part of the recognition data of the current training iteration and input into the recognition model to be trained. After receiving these data, the recognition model to be trained will sequentially execute the various processing stages defined in the aforementioned embodiment: extract initial feature information for each input sample data. Based on the initial feature information, cross-modal feature interaction learning is performed to determine the dependency between features and generate training interaction feature information. Based on the training interaction feature information, the dynamic feature fusion weight is determined, and based on this weight, the training interaction feature information is subjected to data fusion to obtain training fusion feature information.

[0142] Based on the obtained training fusion feature information, the output layer of the trained recognition model outputs the training type recognition results and training operating states (which can represent the probability distribution of unpredicted operating state categories) corresponding to the batch of recognition data. The terminal compares these model predictions with the true labels of the corresponding samples and calculates a total loss value using one or more pre-set loss functions. Optionally, the pre-set loss function is a multi-task loss function, a weighted sum of multiple single loss terms, designed to simultaneously optimize multiple model objectives. For example, the model optimization training component may include a cross-entropy classification loss, which is used to evaluate the difference between the training type recognition results and the true type labels, and the difference between the training operating states and the true state labels. A contrastive learning loss can be applied, for example, to the training interaction feature information or the training fusion feature information, to ensure that the feature representations of samples from the same category (e.g., the same component type or the same fault state) are closer in feature space, while the feature representations of samples from different categories are more distant. This helps learn more discriminative features. Feature disentanglement loss can be introduced when processing multimodal interactive features or fused features to encourage features from different modalities to learn more independent and complementary information, reducing redundancy between features. For example, this can be achieved by minimizing the mutual information between different feature subsets or using adversarial learning.

[0143] Based on the calculated total loss value, the terminal uses an optimization algorithm to calculate the gradient of the loss relative to all adjustable parameters of the model through the back-propagation mechanism, and updates these parameters along the negative gradient direction in order to reduce the loss in the next iteration. After the parameters are updated, the terminal will evaluate the current model training difficulty. In some embodiments, if the average loss of the current training batch decreases significantly or the performance on the validation data set improves, it is considered that the model has adapted well to the data of the current difficulty and the difficulty can be appropriately increased; if the loss decreases slowly or the performance stagnates, it is possible that the current data difficulty is too high or the model is stuck in a local optimum. Based on the assessed training difficulty, the terminal executes a dynamic course learning strategy: on the basis of the current partial recognition data, a new part of historical recognition data from the total historical data set that has not been fully trained before and may be more complex or more challenging is added to form updated partial recognition data. For example, in the early stage, only data with common fault types and clear signals are used, and in the later stage, rare faults, noisy data or data under boundary conditions are gradually added.

[0144] The terminal repeats the above steps using the updated partial recognition data (or the original partial recognition data or a new random subset thereof when the difficulty has not been adjusted). When one or more preset training end conditions are met, the training process terminates, and the recognition model is saved and deployed to the actual electrical component monitoring terminal for performing online recognition tasks.

[0145] In this embodiment, through the application of dynamic course learning strategies and by providing training data from easy to difficult, the model is able to learn more stably and efficiently, avoiding early difficulties with complex samples, and helping the model gradually build a cognitive level of various states and state patterns of electrical components.

[0146] In some embodiments, electrical component condition monitoring and fault diagnosis technology is evolving from reliance on a single or small amount of sensor information (such as infrared images, visible light images, current, and temperature signals) to multi-sensor data fusion, in which deep learning models, such as convolutional neural networks (CNNs) for image analysis and recurrent neural networks (RNNs) or temporal convolutional networks (TCNs) for processing time series signals (such as current and temperature), as well as feature-level or decision-level fusion strategies (such as cross-modal self-attention mechanisms), have been initially applied. However, existing technologies still face many challenges: First, the heterogeneity of multi-source data (including significant differences in sampling frequency and data format) and the lack of unified data models and semantic standards make it difficult to accurately align cross-modal data in the spatiotemporal dimensions, resulting in low fusion efficiency and difficulty in adapting to the expansion of data requirements in scenarios such as the access of new energy equipment; second, the deep learning model's reliance on a large amount of high-quality labeled data is in conflict with the current situation where real fault samples of power equipment are scarce and data distribution is uneven (normal state data far outnumbers fault data), resulting in insufficient generalization ability of the model under new equipment or complex extreme working conditions; third, existing multi-source information fusion strategies, especially dynamic weight allocation mechanisms (such as gating networks based on fixed rules), are often difficult to adapt to the changes in the importance of each sensor information in different operating scenarios, and the interpretability of their fusion process needs to be improved.

[0147] In an exemplary embodiment, the present application provides an electrical component identification method that can solve the above problems, such as Figure 2 As shown, the specific steps include:

[0148] Step S201 : collecting and preprocessing multi-source operating data of electrical components.

[0149] During the data collection phase, the terminal uses a synchronized triggering mechanism to ensure that multiple different types of sensors (e.g., at least imaging and timing sensors) deployed within the target electrical component or its monitoring environment initiate data collection based on a unified time base. These sensors may include: an infrared thermal imager to capture infrared images of the temperature distribution on the electrical component's surface, which can be used to identify potential localized overheating or thermal anomalies; a visible light industrial camera to obtain high-resolution visible light images of the component's surface for detecting structural integrity or surface anomalies; one or a group of high-sensitivity microphones to capture acoustic signals generated during electrical component operation to identify specific sound frequencies, such as mechanical sounds, abnormal vibrations, or partial discharges; and current sensors connected to the component's relevant circuits to monitor parameters such as the operating current waveform, amplitude, phase, frequency, and harmonics in real time. Furthermore, multiple point or small-area temperature sensors may be included to accurately measure the real-time temperature of key locations within the electrical component. The terminal ensures that the data collected by all sensors is highly time-synchronized, for example, with timestamp errors within one millisecond, forming a preliminary time-aligned multimodal dataset. These data constitute a variety of operational data, including image sensor data and time series sensor data.

[0150] The acquired multi-source operating data is pre-processed. For example, for infrared and visible light image data, the terminal can perform adaptive histogram equalization to enhance the local contrast of the image, and use an image denoising algorithm based on discrete wavelet transform to suppress background noise. For sound signal data, the terminal can first use spectral subtraction to remove steady-state background noise, and then convert it into a Mel-spectrogram to extract time-frequency features that are more consistent with auditory perception. For time series signal data such as current and point temperature, the terminal can perform sliding window segmentation and perform maximum and minimum value normalization on the sample segment data in each window. The calculation can be described as: the normalized value is equal to the result of (original value minus the minimum value in the window) divided by (maximum value in the window minus minimum value in the window).

[0151] Step S202 : performing time alignment and control registration on the pre-processed multi-source operation data to obtain electrical component identification data.

[0152] After obtaining the preliminarily synchronized multi-source operating data, the terminal performs a refined time alignment on these data. Specifically, for any two or more types of sensors, if the terminal detects that there is an inconsistency in the sampling timestamp sequences of the operating data they each obtain, the terminal will interpolate the operating data that needs to be resampled based on these different sampling timestamps. For example, the terminal can select a reference timestamp sequence and then use a linear interpolation method, such as linear interpolation, on other data streams to estimate the data value at the reference timestamp based on adjacent original data points. Through this interpolation process, the operating data of all modes are processed to obtain time-aligned operating data on a unified high-precision time base.

[0153] The terminal then uses a pre-acquired and stored spatial calibration matrix to accurately spatially register the target region of the image sensor data within the time-aligned operational data, as well as the preset spatial locations of the corresponding sensors for the time-series sensor data within the time-aligned operational data. For image sensor data, the terminal first determines the target region for analysis within the image. Then, using the spatial calibration matrix corresponding to the image sensor, the terminal accurately maps the 2D image pixel coordinates of the target region to the 3D world coordinate system. For time-series sensor data, the terminal retrieves the preset 3D spatial locations of the corresponding sensors (for example, the 3D coordinates of a microphone unit or temperature probe), which were accurately measured and recorded during sensor installation and system calibration. By applying the transformation relationships contained in the spatial calibration matrix, the terminal can unify all image information and time-series sensor point information into a common reference coordinate system, thereby clarifying the physical location relationships between different operational data.

[0154] Step S203: input the obtained electrical component identification data into a pre-trained identification model to identify the electrical component and determine the current operating status of the electrical component.

[0155] During the feature extraction phase of the recognition model, specific network structures are employed for different types of data. For input image sensor data, the terminal feeds it into a convolutional neural network branch within the model, consisting of multiple residual blocks, such as a modified residual network architecture. Within each of these residual blocks, a coordinate attention unit is embedded. This unit captures long-range spatial dependencies with precise positional information through a specific one-dimensional feature encoding process and generates attention weights in both the image channel and physical location directions. These attention weights are then used to dynamically recalibrate the output feature maps of the residual blocks, extracting image features that are highly sensitive to both spatial location and content. For input time series sensor data, the terminal first segments it into time series segments of varying preset time lengths, as needed. These segments are then fed into one-dimensional convolutional neural network branches or modules within the recognition model with varying convolution kernel sizes to extract a set of feature information that characterizes the dynamic characteristics of the time series sensor data at multiple time scales.

[0156] After obtaining the initial feature information for electrical component identification data from each modality, the terminal utilizes the model's internal feature interaction module to determine the feature correlation between these initial feature information from different modalities based on the physical location associations determined during the data preparation phase. This feature correlation is used to characterize the dependencies between them. Subsequently, based on this feature correlation and the original (or initially transformed) feature information, interactive feature information for each modality is determined. This process is implemented using a cross-modal attention mechanism. The core computation can be understood as follows: the interactive context information is calculated by first performing matrix multiplication and transposition on the query vector from modality i and the key vector from modality j. The result is then scaled by dividing it by the square root of the key vector's dimension. The result is then processed through a normalized exponential function (softmax function) to obtain an attention weight (i.e., feature correlation). Finally, the resulting attention weight is multiplied by the value vector from modality j. The query vector is derived from the features of one modality, while the key and value vectors are derived from the features of another modality that is physically associated with that modality. The terminal repeats this process for multiple important modal pairs, generating one or a set of enhanced interactive feature information that incorporates the context of other modalities for each original modal feature information.

[0157] After obtaining the interactive feature information from each modality, the terminal further performs dynamic feature fusion. First, it determines the dynamic feature fusion weights. This is achieved by concatenating the interactive feature information from each modality (for example, visual feature information, infrared feature information, sound feature information, current feature information, and temperature feature information after interaction) to generate a high-dimensional joint feature information vector. This joint feature information vector is then input into a preset gated network structure within the recognition model. This gated network structure uses adjustable weight and bias parameters to perform a linear transformation on the input joint feature information vector. Specifically, the weight matrix multiplies the joint feature information vector plus the bias parameter vector. The result of this linear transformation is processed through an activation function (sigmoid function) to determine the dynamic feature fusion weights for each modality. These weights are a vector containing the weights for each modality. After determining the dynamic feature fusion weights, the terminal performs a weighted fusion of the interactive feature information from each modality based on these weights. For example, this is done by multiplying the interactive feature information from each modality by its corresponding weight and then summing all these products to obtain the final fused feature information.

[0158] Finally, the terminal inputs the fused feature information into the recognition model's output layer. This layer performs calculations based on the fused feature information and outputs two main results: the electrical component type identification result (e.g., a classification label indicating the current component type or a probability distribution of each category); and the current operating state of the electrical component (e.g., a classification label indicating the component's health status or a probability distribution of these states).

[0159] The pre-trained recognition model is obtained through a process that includes obtaining historical electrical component recognition data, inputting part of the historical data into the to-be-trained model in stages for forward propagation (extracting training interaction feature information, performing data fusion to obtain training fusion feature information, and outputting training recognition results and operating status), determining a preset loss function based on the training results and the true label (for example, a multi-task loss function, which may include a classification loss for evaluating the difference between the type recognition result and the true type label, a classification loss for evaluating the difference between the operating status and the true status label, and optionally a contrastive learning loss or feature disentanglement loss for optimizing feature representation. The total loss value is the weighted sum of these individual loss terms, for example, the total loss is equal to the first weight coefficient multiplied by the type recognition cross entropy loss plus the second weight coefficient multiplied by the state evaluation cross entropy loss plus the third weight coefficient multiplied by the contrastive learning loss plus the fourth weight coefficient multiplied by the feature disentanglement loss), backpropagating and updating the model parameters according to the loss value, and iteratively training in combination with a dynamic curriculum learning strategy until the preset training end conditions are met.

[0160] In this embodiment, a high-quality and highly consistent multimodal data input foundation is obtained through precise spatiotemporal alignment and preprocessing; by adopting advanced deep learning structures designed for modal characteristics (such as residual networks with coordinate attention and multi-scale temporal convolutional networks), initial features that are more comprehensive in representing the state of electrical components and more sensitive to subtle changes are extracted; by introducing a feature interaction mechanism based on physical location association, the inherent correlation and information complementarity between different data sources are effectively mined and utilized, generating interactive features with richer information content; and through a learnable dynamic gating network, intelligent weighting and optimized fusion of the contribution of each modal information according to the real-time data characteristics are achieved.

[0161] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in their order. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with the steps or at least a portion of the steps or stages in the steps.

[0162] Based on the same inventive concept, embodiments of the present application also provide an electrical component identification device based on multi-source operational data fusion for implementing the aforementioned electrical component identification method based on multi-source operational data fusion. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the electrical component identification device based on multi-source operational data fusion provided below can be found in the above-mentioned limitations of the electrical component identification method based on multi-source operational data fusion, and will not be repeated here.

[0163] In an exemplary embodiment, Figure 3 As shown, an electrical component identification device based on multi-source operation data fusion is provided, including: an electrical component identification data acquisition module 310, an interactive feature information determination module 320, an interactive feature information fusion module 330 and an electrical component identification module 340, wherein:

[0164] The electrical component identification data acquisition module 310 is configured to synchronously trigger multiple sensors of different types to acquire various operating data during the operation of the electrical component, and determine the physical positional associations between the various operating data through spatial registration to obtain electrical component identification data corresponding to each type of operating data; the electrical component identification data corresponding to each type of operating data includes the operating data and the physical positional associations between the operating data and other types of operating data; the various types of operating data include image sensor data and time series sensor data;

[0165] The interactive feature information determination module 320 is configured to input the electrical component identification data into a pre-trained recognition model, determine feature information of each electrical component identification data, and obtain dependency relationships between different feature information to generate interactive feature information between the electrical component identification data;

[0166] The interactive feature information fusion module 330 determines a dynamic feature fusion weight, and based on the dynamic feature fusion weight, fuses the interactive feature information of each electrical component identification data to obtain fused feature information;

[0167] The electrical component identification module 340 is configured to output the type identification result of the electrical component and the current operating status of the electrical component from the identification model based on the fused feature information.

[0168] In one embodiment, the electrical component identification data acquisition module 310 is also used to, for any two types of sensors, if the sampling timestamps of the operating data acquired by the two types of sensors are different, perform interpolation calculation on the operating data according to the different sampling timestamps to obtain time-aligned operating data.

[0169] In one embodiment, the electrical component identification data acquisition module 310 is also used to spatially align the target area of the image-type sensor data in the time-aligned operating data and the preset physical position of the sensor corresponding to the timing-type sensor data in the time-aligned operating data through a spatial calibration matrix to obtain electrical component identification data.

[0170] In one embodiment, the interactive feature information determination module 320 is also used to input the image-type sensor data of the electrical component identification data into a convolutional neural network including multiple residual blocks, and obtain the attention weight of the image-type sensor data in the image channel direction and the physical position direction through the coordinate attention unit embedded in each residual block; determine the feature information corresponding to the image-type sensor data based on the attention weight and the current output of the residual block; divide the time series sensor data of the electrical component identification data into time series segments of different time lengths; and input each of the time series segments into a convolutional neural network with different convolution kernel sizes to determine the feature information of the time series sensor data at different time lengths.

[0171] In one embodiment, the interactive characteristic information determination module 320 is also used to determine the characteristic correlation between the characteristic information corresponding to each of the electrical component identification data based on the physical location association; the characteristic correlation is used to characterize the dependency relationship between each of the electrical component identification data; and based on the characteristic correlation and the characteristic information, the interactive characteristic information of each of the electrical component identification data is determined.

[0172] In one embodiment, the interactive feature information fusion module 330 is also used to splice the interactive feature information of each electrical component identification data to obtain joint feature information; input the joint feature information into a preset gated network structure to obtain a network output result; the gated network structure uses adjustable weight parameters and bias parameters to perform a linear transformation on the joint feature information; and determine the dynamic feature fusion weight based on the network output result.

[0173] In one embodiment, the interactive feature information determination module 320 is also used to obtain historical electrical component identification data; input part of the identification data in the historical electrical component identification data into the recognition model to be trained, extract the training interactive feature information of each part of the identification data, and fuse the part of the identification data according to the training interactive feature information to obtain training fusion feature information; according to the fusion feature information, the recognition model to be trained outputs the training type identification result and the training operation status; based on the training type identification result and the training operation status, the loss value of the preset loss function is determined; according to the loss value, the parameters of the recognition model to be trained are updated to determine the current model training difficulty, and according to the current model training difficulty, historical identification data is added to the partial identification data to obtain updated partial identification data, and the recognition model to be trained is trained again according to the updated partial identification data, and when the training end condition is met, the trained recognition model is obtained.

[0174] Each module in the aforementioned electrical component identification device based on multi-source operational data fusion can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a computer device memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0175] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0176] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0177] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0178] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be achieved through WIFI, a mobile cellular network, near field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an electrical component identification method based on multi-source operation data fusion.

[0179] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0180] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0181] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to a memory, a database, or its medium used in the embodiments provided in this application can include at least one of a non-volatile memory and a volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0182] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0183] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An electrical component identification method based on multi-source operation data fusion, characterized in that: The method comprises: By synchronously triggering multiple sensors of different types, a variety of operating data during the operation of the electrical component is acquired, and the physical position associations between the various operating data are determined through spatial registration to obtain electrical component identification data corresponding to each type of operating data; the electrical component identification data corresponding to each type of operating data includes the operating data and the physical position associations of the operating data with other types of operating data; the various types of operating data include image sensor data and time series sensor data; Inputting the electrical component identification data into a pre-trained recognition model, determining feature information of each electrical component identification data, and obtaining dependency relationships between different feature information to generate interactive feature information between each electrical component identification data; Determining a dynamic feature fusion weight, and based on the dynamic feature fusion weight, fusing the interactive feature information of each electrical component identification data to obtain fused feature information; Based on the fused feature information, the identification model outputs a type identification result of the electrical component and a current operating status of the electrical component.

2. The method according to claim 1, characterized in that After acquiring a plurality of operation data during the operation of the electrical component, and before determining the spatial position association between the plurality of operation data through spatial registration and obtaining the electrical component identification data corresponding to each operation data, the method further includes: For any two types of sensors, if the sampling timestamps of the operating data acquired by the two types of sensors are different, interpolation calculation is performed on the operating data according to the different sampling timestamps to obtain time-aligned operating data; The determining of the physical position association between the operating data through spatial registration processing to obtain electrical component identification data corresponding to each type of operating data includes: Through the spatial calibration matrix, the target area of the image sensor data in the time-aligned operating data and the preset physical position of the corresponding sensor of the timing sensor data in the time-aligned operating data are spatially aligned to obtain electrical component identification data.

3. The method according to claim 1, characterized in that The recognition model includes a plurality of convolutional neural networks, and determining the characteristic information of each electrical component identification data includes: Inputting the image sensor data of the electrical component recognition data into a convolutional neural network including a plurality of residual blocks, and obtaining the attention weights of the image sensor data in the image channel direction and the physical position direction through the coordinate attention unit embedded in each residual block; Determining feature information corresponding to the image sensor data based on the attention weight and the current output of the residual block; dividing the time series sensor data of the electrical component identification data into time series segments of different time lengths; Each of the time series segments is input into a convolutional neural network with different convolution kernel sizes to determine the characteristic information of the time series sensor data at different time lengths.

4. The method according to claim 3, characterized in that The step of performing cross-modal feature interactive learning on the feature information to determine the dependency between different feature information to generate interactive feature information between the electrical component identification data includes: Determining, based on the physical location association, a characteristic correlation between characteristic information corresponding to each of the electrical component identification data; the characteristic correlation is used to characterize a dependency relationship between each of the electrical component identification data; Determine interactive feature information of each electrical component identification data according to the feature association degree and the feature information.

5. The method according to claim 4, characterized in that Determining the dynamic feature fusion weight includes: splicing the interactive feature information of the identification data of each electrical component to obtain joint feature information; Inputting the joint feature information into a preset gated network structure to obtain a network output result; the gated network structure uses adjustable weight parameters and bias parameters to perform a linear transformation on the joint feature information; According to the network output result, the dynamic feature fusion weight is determined.

6. The method according to any one of claims 1 to 5, characterized in that The recognition model is trained through the following steps, including: Obtain historical electrical component identification data; Inputting part of the identification data in the historical electrical component identification data into the identification model to be trained, extracting training interaction feature information of each part of the identification data, and performing data fusion on each part of the identification data based on the training interaction feature information to obtain training fusion feature information; According to the fusion feature information, the recognition model to be trained outputs a training type recognition result and a training running status; Determining a loss value of a preset loss function based on the training type identification result and the training running status; According to the loss value, the parameters of the recognition model to be trained are updated to determine the current model training difficulty, and based on the current model training difficulty, historical recognition data are added to the partial recognition data to obtain updated partial recognition data. The recognition model to be trained is trained again based on the updated partial recognition data. When the training end conditions are met, the trained recognition model is obtained.

7. An electrical component identification device based on multi-source operation data fusion, characterized in that: The device comprises: An electrical component identification data acquisition module is configured to synchronously trigger multiple sensors of different types to acquire various operating data during the operation of the electrical component, and determine the physical position associations between the various operating data through spatial registration to obtain electrical component identification data corresponding to each type of operating data; the electrical component identification data corresponding to each type of operating data includes the operating data and the physical position associations between the operating data and other types of operating data; the various types of operating data include image sensor data and time series sensor data; an interactive feature information determination module, configured to input the electrical component identification data into a pre-trained recognition model, determine feature information of each electrical component identification data, and obtain dependency relationships between different feature information to generate interactive feature information between each electrical component identification data; an interactive feature information fusion module, which determines a dynamic feature fusion weight and fuses the interactive feature information of each electrical component identification data based on the dynamic feature fusion weight to obtain fused feature information; The electrical component identification module is used to output the type identification result of the electrical component and the current operating status of the electrical component from the identification model based on the fused feature information.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images