Cross-domain internet of things device intelligent collaboration method and system based on semantic knowledge graph

Abstract

The application provides a cross-domain Internet of Things device intelligent cooperation method and system based on a semantic knowledge graph. The method comprises: adjusting the sampling frequency of the cross-domain Internet of Things device according to the level value; after the sampling frequency adjustment is completed, obtaining the image texture data and sound spectrum data of the monitored device collected by the cross-domain Internet of Things device; based on the image texture data and sound spectrum data, performing cross-modal feature space alignment using a feature-level fusion classifier to generate a joint semantic feature vector representing the state of the monitored device; based on the joint semantic feature vector, constructing a semantic knowledge graph; and dynamically optimizing the feature weight of the cross-domain Internet of Things device in the semantic knowledge graph according to the level value, so as to generate a cross-domain Internet of Things device cooperation control instruction according to the dynamically optimized feature weight. The application realizes multi-modal data cooperative perception and dynamic weight optimization driven by vibration signals, and improves the cross-domain device state monitoring precision and control efficiency.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to a cross-domain IoT device intelligent collaboration method and system based on semantic knowledge graphs. Background Technology

[0002] In the field of industrial equipment condition monitoring, for cross-domain IoT devices that rotate or vibrate at high speeds, it is necessary to achieve multimodal collaborative sensing in order to overcome the time-varying characteristics of mechanical motion, avoid oversampling or feature omission caused by fixed sampling frequency, and support interpretable diagnosis of equipment degradation status.

[0003] The existing solution adopts adaptive sampling and multimodal decision fusion based on vibration sensors. The specific process is as follows: the vibration signal of the equipment is monitored in real time by piezoelectric sensors. When the vibration intensity exceeds the preset vibration intensity threshold, the camera and microphone are dynamically triggered to sample. Then, image texture features and sound spectrum data features are extracted respectively. Finally, the equipment status classification result is generated through decision-level fusion.

[0004] However, existing solutions have two key drawbacks. First, visual and acoustic sensors are triggered independently by vibration signals. Due to hardware response delays, there is a millisecond-level time difference in cross-domain data. In high-speed vibration scenarios, this time difference can cause motion artifacts and spectral phase shifts. Second, when performing post-fusion of image and sound features, there is a lack of cross-modal feature space alignment mechanism, which leads to the forced aggregation of features with weak semantic correlation, weakening the physical interpretability of state representation and hindering accurate diagnosis. Summary of the Invention

[0005] This application provides a cross-domain IoT device intelligent collaboration method and system based on semantic knowledge graphs to solve the problems of asynchronous data fragmentation and semantic mismatch of multimodal features in the prior art.

[0006] Firstly, this application provides a cross-domain IoT device intelligent collaboration method based on semantic knowledge graphs, including:

[0007] The mechanical vibration energy of the monitored equipment is converted into an electrical signal, and a corresponding level value is generated synchronously.

[0008] Based on the stated level value, the sampling frequency of the cross-domain IoT device is adjusted. After the sampling frequency adjustment is completed, the image texture data and sound spectrum data of the monitored device collected by the cross-domain IoT device are obtained.

[0009] Based on the image texture data and sound spectrum data, a feature-level fusion classifier is used to perform cross-modal feature space alignment to generate a joint semantic feature vector representing the state of the monitored device.

[0010] Based on the joint semantic feature vector, a semantic knowledge graph is constructed;

[0011] Based on the level value, the feature weights of cross-domain IoT devices in the semantic knowledge graph are dynamically optimized, so as to generate cross-domain IoT device collaborative control instructions based on the dynamically optimized feature weights.

[0012] Optionally, the step of aligning cross-modal feature spaces using a feature-level fusion classifier based on the image texture data and sound spectrum data to generate a joint semantic feature vector representing the state of the monitored device includes:

[0013] The image texture data is input into the image projection function to perform a mapping operation, generating projected image features; the sound spectrum data is input into the sound projection function to perform a mapping operation, generating projected sound features.

[0014] Based on the projected image features and projected sound features, intermodal similarity calculation is performed in the shared latent space to generate a set of intermodal similarity measurement values.

[0015] Based on the set of intermodal similarity metrics, a feature-level fusion classifier is used to construct an association mapping to generate a cross-modal feature association mapping table.

[0016] Feature reconstruction is performed based on the cross-modal feature association mapping table to generate feature slices of a unified dimension;

[0017] The feature slices are input into the bidirectional feature interaction module for multiple rounds of aggregation to generate a joint semantic feature vector representing the state of the monitored device.

[0018] Optionally, the step of reconstructing features based on the cross-modal feature association mapping table to generate feature slices of a unified dimension includes:

[0019] The projected image features are segmented to generate a set of image feature segments, and the projected sound features are segmented to generate a set of sound feature segments.

[0020] Based on the image feature segment set, the sound feature segment set, and the cross-modal feature association mapping table, a key-value matching operation is performed to generate a set of matching feature pairs;

[0021] Perform a cross-aggregation operation on the set of matching feature pairs to generate a sequence of fused feature units;

[0022] Based on the set of image feature segments and the set of sound feature segments, perform the unmatched identifier extraction operation to generate a set of unmatched image identifiers and a set of unmatched sound identifiers;

[0023] Based on the set of unmatched image identifiers and the set of unmatched sound identifiers, an interpolation compensation operation is performed to generate compensated image feature units and compensated sound feature units;

[0024] The fused feature unit sequence, the compensated image feature unit, and the compensated sound feature unit are subjected to a multidimensional structure recombination operation to generate a feature slice of uniform dimension.

[0025] Optionally, the step of generating a cross-modal feature association mapping table by constructing an association mapping based on the set of inter-modal similarity metrics using a feature-level fusion classifier includes:

[0026] In the feature-level fusion classifier, a threshold filtering operation is performed on the set of inter-modal similarity metrics to generate a set of candidate mapping pairs;

[0027] Perform feature indexing and grouping operations on the projected image features and projected sound features to generate image feature index groups and sound feature index groups;

[0028] A bidirectional correlation constraint is applied to the image feature index group and the sound feature index group to generate an effective mapping rule set;

[0029] The candidate mapping pair set is subjected to rule verification with the preset valid mapping rule set to generate verified mapping pairs;

[0030] A key-value table construction operation is performed on the verified mapping pairs to generate a cross-modal feature association mapping table.

[0031] Optionally, constructing a semantic knowledge graph based on the joint semantic feature vector includes:

[0032] State nodes are extracted from the joint semantic feature vector to generate a device state node set;

[0033] Perform physical relationship modeling operations on the set of device status nodes to generate a physical topology table;

[0034] Based on the physical topology table, perform edge connection operations to generate an initial knowledge graph skeleton;

[0035] The joint semantic feature vector is parsed to generate a set of vibration feature values;

[0036] The set of vibration feature values ​​is injected into the initial knowledge graph skeleton to assign edge weights and generate a weighted knowledge graph.

[0037] The weighted knowledge graph is subjected to spatiotemporal constraint binding operations to obtain a semantic knowledge graph.

[0038] Optionally, the step of injecting the set of vibration feature values ​​into the initial knowledge graph skeleton to assign edge weights and generate a weighted knowledge graph includes:

[0039] Perform edge type identification on the initial knowledge graph skeleton to generate a spatial edge set and a temporal edge set;

[0040] Perform a feature dimension separation operation on the vibration feature value set to generate an amplitude feature set and an attenuation feature set;

[0041] Perform feature allocation operations on the spatial edge set and the amplitude feature set, and on the temporal edge set and the attenuation feature set, respectively, to generate a spatial edge amplitude mapping table and a temporal edge attenuation mapping table;

[0042] Based on the spatial edge amplitude mapping table and the temporal edge decay mapping table, spatial weight and temporal weight calculation operations are performed to generate spatial edge weight values ​​and edge temporal weight values.

[0043] The spatial edge weights and temporal edge weights are injected into the initial knowledge graph skeleton for weight update, generating a weighted knowledge graph.

[0044] Optionally, the step of dynamically optimizing the feature weights of cross-domain IoT devices in the semantic knowledge graph based on the level value, so as to generate cross-domain IoT device collaborative control instructions based on the dynamically optimized feature weights, includes:

[0045] The vibration mode is analyzed based on the level value to generate a dominant vibration mode identifier;

[0046] The semantic knowledge graph is subjected to edge weight extraction to generate an original weight distribution table;

[0047] The dominant vibration mode identifier is input into the original weight distribution table for pattern matching, generating a matching weight subset. The matching weight subset is then input into the semantic knowledge graph for weight update operations, generating an optimized knowledge graph.

[0048] The optimized knowledge graph is traversed to retrieve control rules and generate an initial control instruction set.

[0049] The initial control instruction set is subjected to device coordination constraint operations to generate cross-domain IoT device coordination control instructions.

[0050] Secondly, this application provides a cross-domain IoT device intelligent collaboration system based on semantic knowledge graphs, including:

[0051] The conversion module is used to convert the mechanical vibration energy of the monitored equipment into an electrical signal and synchronously generate the corresponding level value;

[0052] The acquisition module is used to adjust the sampling frequency of the cross-domain IoT device according to the level value, and after the sampling frequency adjustment is completed, acquire the image texture data and sound spectrum data of the monitored device collected by the cross-domain IoT device.

[0053] The generation module is used to generate a joint semantic feature vector representing the state of the monitored device by aligning the image texture data and sound spectrum data across the modal feature space using a feature-level fusion classifier.

[0054] A construction module is used to construct a semantic knowledge graph based on the joint semantic feature vector;

[0055] The dynamic optimization module is used to dynamically optimize the feature weights of cross-domain IoT devices in the semantic knowledge graph based on the level value, so as to generate cross-domain IoT device collaborative control instructions based on the dynamically optimized feature weights.

[0056] Thirdly, this application provides a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are to be invoked and executed by the processing component to implement the cross-domain IoT device intelligent collaboration method based on semantic knowledge graph as described in any of the first aspects.

[0057] Fourthly, this application provides a computer storage medium storing a computer program, which, when executed by a computer, implements the cross-domain IoT device intelligent collaboration method based on semantic knowledge graph as described in any of the first aspects.

[0058] This application provides a cross-domain IoT device intelligent collaboration method based on a semantic knowledge graph. The method includes: converting the mechanical vibration energy of the monitored device into an electrical signal and synchronously generating a corresponding level value; adjusting the sampling frequency of the cross-domain IoT device according to the level value; after the sampling frequency adjustment is completed, acquiring image texture data and sound spectrum data of the monitored device collected by the cross-domain IoT device; based on the image texture data and sound spectrum data, performing cross-modal feature space alignment using a feature-level fusion classifier to generate a joint semantic feature vector representing the state of the monitored device; constructing a semantic knowledge graph based on the joint semantic feature vector; and dynamically optimizing the feature weights of the cross-domain IoT device in the semantic knowledge graph according to the level value, so as to generate cross-domain IoT device collaborative control commands based on the dynamically optimized feature weights.

[0059] This application achieves self-powered sensing and quantitative characterization of vibration intensity by converting mechanical vibration energy into electrical signals and simultaneously generating level values. Based on the level values, the sampling frequency is dynamically adjusted to ensure that cross-domain equipment automatically optimizes data acquisition efficiency when vibration conditions change. After simultaneously acquiring image texture and sound spectrum data, a feature-level fusion classifier is used for cross-modal space alignment, eliminating the semantic gap between heterogeneous data and generating a highly discriminative joint semantic feature vector. Based on this vector, a semantic knowledge graph is constructed, forming an interpretable equipment state topology network. Finally, the equipment feature weights in the graph are dynamically optimized according to the level values, enabling collaborative control commands to adapt to the current vibration intensity in real time, achieving a closed-loop response of state perception and equipment control.

[0060] Furthermore, firstly, texture data and spectral data are mapped to projected features using image and sound projection functions, respectively; then, a set of intermodal similarity metrics is calculated in the shared latent space, and a cross-modal feature association mapping table is constructed accordingly; image segments are generated by dividing the projected image features into blocks, and sound segments are generated simultaneously by dividing the sound features into blocks; key-value matching is performed on the two sets of segments based on the association mapping table to obtain a set of matching feature pairs, which are then cross-aggregated to generate a fused feature unit sequence; simultaneously, unmatched image and sound identifier sets are extracted, and compensation feature units are generated through interpolation compensation; finally, the fused sequence and compensation units are multi-dimensionally recombined to generate feature slices of a unified dimension, which are input into a bidirectional feature interaction module for multi-round aggregation to output a joint semantic feature vector. This application enables fine-grained cross-modal feature alignment and information preservation. Projection function mapping ensures that heterogeneous data are represented in a comparable latent space. The key-value matching mechanism realizes local semantic association between image texture blocks and sound spectrum segments, solving the feature misalignment problem under non-steady-state signals. Unmatched identifier extraction and interpolation compensation preserve unique features of a single modality, avoiding information loss caused by cross-modal matching. Multidimensional recombination generates unified-dimensional feature slices, providing structurally aligned input for subsequent bidirectional interaction, improving the physical interpretability and state discrimination robustness of the joint feature vector.

[0061] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 A flowchart of a cross-domain IoT device intelligent collaboration method based on semantic knowledge graph provided in this application embodiment;

[0064] Figure 2 This is a schematic diagram of the structure of a cross-domain IoT device intelligent collaboration system based on semantic knowledge graphs provided in an embodiment of this application;

[0065] Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0066] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0067] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 11, 12, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0068] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] To address the issues of asynchronous data fragmentation and semantic mismatch in multimodal features in existing technologies, this application provides a cross-domain IoT device intelligent collaboration method based on a semantic knowledge graph. This method employs the following concept: Real-time level signals are generated through the conversion between vibration energy and electrical energy. Using these signals as a control benchmark, the sampling frequency of the cross-domain IoT devices is dynamically adjusted to ensure data acquisition matches the device's vibration state. Simultaneously, optimized image texture and sound spectrum data are acquired and input into a feature-level fusion classifier for cross-modal spatial alignment, generating a joint semantic feature vector representing the device's health status. An interpretable semantic knowledge graph is constructed based on this vector, and the feature weights of different devices in the graph are dynamically optimized using the level signals. Finally, precise cross-domain device collaborative control commands are generated, achieving adaptive energy consumption and closed-loop status monitoring.

[0070] Figure 1A flowchart of the cross-domain IoT device intelligent collaboration method based on semantic knowledge graph provided in the embodiments of this application is shown below. Figure 1 As shown, the method includes:

[0071] S11. Convert the mechanical vibration energy of the monitored equipment into an electrical signal and synchronously generate the corresponding level value.

[0072] The monitored equipment can be various types of industrial equipment. The mechanical vibration energy of the monitored equipment can refer to the kinetic energy generated by bearing wear or rotor imbalance in rotating equipment, which is converted into an electrical signal through piezoelectric materials. The electrical signal is a continuous analog current / voltage waveform generated from the mechanical vibration energy via piezoelectric conversion, reflecting the instantaneous vibration state of the equipment. The level value refers to the discrete digital quantity generated after analog-to-digital conversion of the electrical signal, and its range is positively correlated with the vibration intensity.

[0073] In this embodiment, the mechanical vibration energy of the monitored equipment is first captured by a piezoelectric conversion device and converted into a continuous analog electrical signal; then, the electrical signal is quantized by an analog-to-digital converter to synchronously generate a digital level value characterizing the vibration intensity.

[0074] S12. Adjust the sampling frequency of the cross-domain IoT device according to the level value. After the sampling frequency adjustment is completed, acquire the image texture data and sound spectrum data of the monitored device collected by the cross-domain IoT device.

[0075] In this context, cross-domain IoT devices refer to terminals integrating multimodal sensors such as vision and acoustics, including industrial cameras and directional microphones, and may also include other types of monitoring devices; this application does not specifically limit this. Sampling frequency refers to the rate at which the IoT device collects data, dynamically adjusted based on the level value to avoid aliasing distortion under high-frequency vibrations. Image texture data can refer to the grayscale distribution matrix of the device surface captured by the camera, including degradation features such as cracks and corrosion.

[0076] In this embodiment, the current vibration intensity level is first determined by parsing the level value; then, a sampling frequency adjustment command is sent to the cross-domain IoT device according to the preset intensity-frequency mapping relationship; subsequently, after the device completes the sampling frequency reset, the camera is synchronously triggered to collect image texture data on the surface of the device, and the microphone is activated to collect sound spectrum data of the device operation.

[0077] S13. Based on image texture data and sound spectrum data, a feature-level fusion classifier is used to perform cross-modal feature space alignment to generate a joint semantic feature vector representing the state of the monitored device.

[0078] Among them, sound spectrum data refers to the frequency domain energy distribution generated by the Fourier transform of sound waves collected by the microphone, representing abnormal noise characteristics. Feature-level fusion classifier refers to a neural network model that uses a shared latent space mapping to achieve cross-modal feature alignment and fusion. Cross-modal feature space alignment refers to the operation of mapping image texture and sound spectrum to a unified vector space to eliminate semantic ambiguity. Joint semantic feature vector refers to a dense vector that fuses bimodal features, with dimensions including joint semantic information about the device's degradation state.

[0079] In this embodiment, image texture data is first input into a pre-trained convolutional neural network to extract spatial features and generate projected image features; simultaneously, sound spectrum data is input into a temporal encoder to extract frequency domain features and generate projected sound features; secondly, the cosine similarity matrix of the two types of features is calculated in the shared latent space to generate a set of intermodal similarity metrics; then, a cross-modal feature association mapping table is constructed through a feature-level fusion classifier; finally, the features are reconstructed and aligned based on this mapping table, and a joint semantic feature vector is generated by aggregation through a bidirectional long short-term memory network.

[0080] S14. Construct a semantic knowledge graph based on joint semantic feature vectors.

[0081] Among them, semantic knowledge graph refers to a topological network with device status nodes as entities and physical relationships as edges, and edge weights represent the strength of fault association.

[0082] In this embodiment, firstly, graph node modeling is performed on the joint semantic feature vector to extract the device state node set; secondly, node connection edges are constructed based on the physical topology of the device to form an initial knowledge graph skeleton; then, the vibration feature values ​​in the vector are parsed and injected into the edge weight attributes of the skeleton; finally, spatiotemporal constraint rules are superimposed to generate a weighted semantic knowledge graph.

[0083] S15. Based on the level value, dynamically optimize the feature weights of cross-domain IoT devices in the semantic knowledge graph, so as to generate cross-domain IoT device collaborative control instructions based on the dynamically optimized feature weights.

[0084] Feature weights refer to the contribution parameters of nodes and edges in the knowledge graph, which are dynamically adjusted based on vibration intensity to adapt to changes in operating conditions. Cross-domain IoT device collaborative control commands refer to device linkage commands generated based on the optimized graph, such as adjusting fan speed or triggering alarms.

[0085] In this embodiment, the current dominant vibration mode is first analyzed based on the level value to generate a mode identifier; then, a feature weight distribution table is extracted from the semantic knowledge graph; subsequently, the weight subset corresponding to the identifier is matched for dynamic optimization and updating; finally, the optimized graph is traversed to generate device control rules, and cross-domain IoT device collaborative control instructions are output after collaborative constraint verification.

[0086] Here's a specific example: In the scenario of bearing monitoring for industrial fans, the mechanical vibration of the fan, such as vibration caused by bearing wear, is first captured by a vibration sensor and converted into a continuous voltage or current signal. Assume the original signal voltage is measured to be 1.5 volts. Next, a digital converter quantizes this analog signal into discrete digital values, i.e., level values. The specific formula for calculating the level value is: the level value equals the input voltage divided by the maximum voltage range, then multiplied by the maximum digital value. Here, the maximum voltage range is set to 3 volts, and the digital converter has 8 bits, so its maximum digital value is 255. According to the formula: the input voltage of 1.5 volts divided by the maximum voltage range of 3 volts equals 0.5, 0.5 multiplied by the maximum digital value of 255 equals 127.5, which, after rounding, is approximately 128. This level value is proportional to the vibration intensity; the larger the value, the stronger the vibration.

[0087] Secondly, the preset intensity-frequency relationship graph is queried based on the calculated level value of 128. This graph defines the sensor sampling rate corresponding to different level value ranges: for example, a sampling rate of 50 Hz is used when the level value is between 0 and 100, a sampling rate of 100 Hz is used when the level value is between 100 and 200, and a sampling rate of 200 Hz is used when the level value exceeds 200. Therefore, the current level value of 128 falls within the range of 100 to 200, and the sampling rate of the visual and sound sensors is adjusted to 100 Hz accordingly. After the adjustment is completed, the intelligent monitoring device integrating a camera and microphone is activated. The camera begins to capture image data of the surface of the wind turbine bearing, which is a grayscale matrix containing detailed information such as cracks. The microphone captures the sound waveform of the bearing operation and converts it into a frequency domain spectrum displaying abnormal vibration frequencies through Fourier transform.

[0088] Subsequently, the acquired image data is input into an image transformation model for processing. The model's computation process is defined as follows: image features are obtained by max pooling the result of a two-dimensional convolution operation processed by a modified linear unit function, ultimately generating a feature vector containing spatial texture information. Simultaneously, the acquired sound spectrum data Y is input into a sound coding model. This model first converts the original spectrum into a Mel spectrum that better matches human hearing characteristics using a Mel filter bank, then processes it through a gated recurrent unit, and finally applies layer normalization to generate a vector reflecting time-frequency domain features. Next, cross-modal feature fusion is performed: the generated image and sound feature vectors are mapped to a shared feature space, and their cosine similarity is calculated within this space. Based on the calculated similarity value, highly matched image block features and sound segment features are recombined into standardized feature fragments. For unmatched local feature parts, interpolation and padding are performed using their neighboring spatial or temporal feature values. Finally, all recombined and padded feature fragments are integrated into a unified joint semantic feature vector, which fuses key information from both image and sound modalities, jointly representing the current wear state of the bearing.

[0089] Next, a semantic knowledge graph is constructed based on this joint semantic feature vector: First, key equipment status nodes are extracted, such as the node representing "inner ring crack" and the node representing "normal wear". Then, according to the actual physical structure diagram of the wind turbine, connection rules are established between the nodes. For example, edges reflecting their mechanical connection relationship are added between the node representing "inner ring" and the node representing "ball bearing". Vibration feature values ​​parsed from the joint feature vector are then injected into these connection edges, for example, an amplitude value of 0.8 is extracted. This amplitude value is used to calculate the edge weight, specifically: the weight equals the amplitude divided by the distance between the physical components represented by the nodes. Assuming the physical distance between the node representing the inner ring crack and the associated ball bearing node is 10 units, and the amplitude value is 0.8, then the weight of this edge is 0.8 divided by 10, which equals 0.08.

[0090] Finally, based on the current vibration signal level of 128, in-depth spectrum analysis is performed to identify the dominant vibration mode, such as identifying it as a "shaft imbalance mode" through frequency feature matching. Based on this mode identifier, all edges strongly related to this mode and their current weight values ​​are retrieved from the constructed semantic knowledge graph, forming a subset. Then, the weight values ​​of these edges are updated according to the energy intensity coefficient (e.g., coefficient 0.6) of the dominant vibration mode. The update formula is: the new weight equals the original weight multiplied by 1 plus the intensity coefficient. For example, the original edge weight is 0.08 × (1 + 0.6) = 0.08 × 1.6 = 0.128. After the weight update, an optimized knowledge graph is obtained. This optimized graph is traversed in a depth-first manner, triggering a preset control rule base for matching, thereby generating a preliminary set of equipment control instructions, such as issuing instructions to "reduce the fan speed by 10%" and "start the lubrication system". To ensure that operational commands from different devices do not conflict or exceed system limits, the system verifies and optimizes these initial commands based on preset collaborative constraint rules (such as the maximum allowable speed difference limit for the fan and the system's total power limit). Ultimately, it generates and outputs consistent cross-domain IoT device collaborative control commands, such as adjusting them to "reduce the fan speed by 8% while simultaneously activating the fixed-point lubrication system." This entire process achieves a closed loop from adaptive vibration data acquisition, multimodal information fusion, dynamic construction of state knowledge graphs to intelligent collaborative control decision-making, effectively improving the monitoring accuracy and maintenance efficiency of industrial equipment.

[0091] By executing S11~S15, the embodiments of this application adaptively adjust the data acquisition efficiency through vibration intensity to eliminate feature omissions under high-speed operating conditions; utilize cross-modal feature alignment to construct a highly discriminative equipment state representation to support the construction of an interpretable knowledge graph; and generate precise collaborative instructions based on dynamic weight optimization to improve equipment fault response speed and maintenance efficiency.

[0092] In one possible embodiment, S13, based on image texture data and sound spectrum data, a feature-level fusion classifier is used to perform cross-modal feature space alignment to generate a joint semantic feature vector representing the state of the monitored device, including:

[0093] Step 131: Input the image texture data into the image projection function and perform a mapping operation to generate the projected image features. Input the sound spectrum data into the sound projection function and perform a mapping operation to generate the projected sound features.

[0094] The image projection function refers to a mapping model implemented through a convolutional neural network, used to convert two-dimensional image textures into high-dimensional feature vectors, including edge detection and texture encoding operations. The projected image features are tensors generated from image data processed by the projection function, with dimensions reflecting the abstract semantic level of the spatial texture. The sound projection function refers to a mapping model based on a temporal encoder, used to convert spectral sequences into joint time-frequency representations, including Mel filter banks and temporal convolution operations. The projected sound features are vector sequences generated from spectral data processed by the sound projection function, with length corresponding to the number of time windows. For example, the formula for the image projection function can be:

[0095] ,in, Features of the projected image To correct the linear unit, For max pooling operation, This is a two-dimensional convolution operation. For image texture data, These are the weight parameters of the convolutional layer. These are the bias parameters for the convolutional layer.

[0096] The formula for the sound projection function can be:

[0097] ,in, The characteristics of the sound after projection. For layer normalization, For gated loop unit, To convert the original spectrum to a Mel spectrum, For the sound spectrum, For projection weights.

[0098] In this embodiment, image texture data is first input into an image projection function based on a convolutional neural network to perform spatial feature extraction, generating projected image features containing local texture semantics; simultaneously, sound spectrum data is input into a sound projection function based on a gated recurrent unit to perform time-frequency feature encoding, generating projected sound features characterizing the evolution of the spectrum.

[0099] Step 132: Based on the projected image features and projected sound features, perform inter-modal similarity calculation operations in the shared latent space to generate a set of inter-modal similarity metrics.

[0100] The shared latent space refers to the mathematical space constructed by the fully connected layers of a neural network, used to uniformly represent distance metrics for heterogeneous modal features. The inter-modal similarity calculation operation refers to the process of performing vector dot product and norm calculations within the latent space, with output values ​​ranging from -1 to 1. The inter-modal similarity metric set is a matrix recording the similarity of all cross-modal feature pairs, with rows and columns corresponding to image feature indices and sound feature indices, respectively.

[0101] In this embodiment of the application, image texture data is first input into an image projection function based on a convolutional neural network to perform spatial feature extraction, generating projected image features containing local texture semantics; simultaneously, sound spectrum data is input into a sound projection function based on a gated recurrent unit to perform time-frequency feature encoding, generating projected sound features characterizing the evolution of the spectrum.

[0102] Step 132: Based on the projected image features and projected sound features, perform inter-modal similarity calculation operations in the shared latent space to generate a set of inter-modal similarity metrics.

[0103] The shared latent space refers to the mathematical space constructed by the fully connected layers of a neural network, used to uniformly represent distance metrics for heterogeneous modal features. The inter-modal similarity calculation operation refers to the process of performing vector dot product and norm calculations within the latent space, with output values ​​ranging from -1 to 1. The inter-modal similarity metric set is a matrix recording the similarity of all cross-modal feature pairs, with rows and columns corresponding to image feature indices and sound feature indices, respectively.

[0104] In this embodiment, in a practical application scenario of industrial fan bearing monitoring, the system first collects the mechanical vibration signal generated by bearing wear during fan operation and converts it into a voltage signal. Assume the voltage detected by the sensor is 1.5 volts. This analog signal is then input to an analog-to-digital converter for quantization, and a digital level value of 128 is calculated according to a preset conversion rule. This quantization process follows a standard conversion formula, where 3 volts is the maximum range of the device, 255 corresponds to the maximum digital value of the 8-bit converter, and the final value is calculated based on the ratio of the actual voltage to the range.

[0105] When the system begins processing sensor data, it performs crucial feature extraction operations: image data collected from the surface of the wind turbine bearing is input into a dedicated image processing model. This model performs spatial feature extraction through a multi-layer neural network, including convolution calculations and feature compression. For example, after processing a bearing crack image with a resolution of 1920 x 1080 pixels, the system generates a vector containing 256 feature dimensions. Simultaneously, the collected sound spectrum data is parsed by a specialized sound processing model. This model converts the raw audio data into a time-frequency feature sequence through operations such as spectrum transformation and temporal coding. Taking 10 seconds of audio data as an example, after a 128th-order spectrum transformation, it forms serialized data containing 80 dimensions of features.

[0106] Subsequently, the system performs cross-modal correlation analysis within a shared feature space: features extracted from both types of sensor data are mapped to a unified dimensional space. Within this space, the system automatically calculates the similarity index between each image feature block and the sound feature segment. Taking an image feature block reflecting an inner ring crack as an example, its similarity to the sound feature segment representing a 12 kHz vibration noise reaches 92%, and the system records this to form a 32-row, 40-column similarity comparison matrix.

[0107] Next, a feature association table is constructed based on the similarity results. Specifically, the system sets an 80% similarity threshold as the association threshold to filter out reliable feature correspondences. Simultaneously, a neural network is used to analyze the physical correlations between features, such as automatically identifying the correspondence between surface crack textures and high-frequency abnormal noises. Finally, a feature mapping table containing 256 sets of correspondences is constructed, recording detailed feature block pairing information.

[0108] Then, a multimodal feature reconstruction operation is performed. Specifically, the image features are divided into 32 blocks according to spatial relationships, and the sound features are divided into 40 segments according to temporal relationships. Then, the matching feature pairs are fused. The 5 unmatched image blocks are filled with the feature values ​​of the surrounding blocks, and the 8 unmatched sound segments are filled with the average value of the adjacent time periods. Finally, a standardized 32x40 feature matrix is ​​generated, and the features in each dimension are kept in spatiotemporal alignment.

[0109] Finally, the device state vector is synthesized through a deep network. Specifically, the reconstructed feature input is a neural network module with bidirectional analysis function. This module captures the crack development trend through forward temporal analysis and predicts the wear development path through reverse analysis. After three iterations of optimization, a 32-dimensional joint feature vector is finally generated. Each dimension of this vector reflects a specific state feature. For example, the value of 0.83 in the seventh dimension represents the comprehensive probability of inner ring wear matching high-frequency abnormal noise.

[0110] By executing steps 131 to 135, this embodiment of the application eliminates the semantic gap between image texture and sound spectrum through shared latent space; retains the correspondence of local features using an association mapping table; and generates a highly discriminative joint representation of device status through bidirectional interactive aggregation, providing high-precision feature support for fault diagnosis.

[0111] In one possible embodiment, step 134, reconstructing features based on the cross-modal feature association mapping table to generate feature slices of a unified dimension, includes:

[0112] Step a1: Perform a block operation on the projected image features to generate a set of image feature segments, and perform a block operation on the projected sound features to generate a set of sound feature segments.

[0113] The block segmentation operation refers to the technique of dividing the image feature matrix into local blocks according to a preset grid size, with the grid parameters set based on the feature map resolution. The image feature segment set is a container storing the local features after block segmentation, with elements containing block coordinates and feature vector key-value pairs. The segmentation operation refers to the processing of dividing the audio temporal features according to a fixed time window length, with the window length dynamically adjusted by the spectral variation period. The audio feature segment set is a container storing the temporal features after segmentation, with elements containing the time window start point and feature vectors. It should be noted that the embodiments of this application do not specifically limit the size of the preset grid size, fixed time window length, etc.

[0114] In this embodiment, a grid-based block division operation is first performed on the projected image features to divide the feature tensor into several local blocks along the spatial dimension, generating a set of image feature segments containing position indices; at the same time, a time window sliding segmentation operation is performed on the projected sound features to divide the temporal features into equal-length segments, generating a set of sound feature segments with timestamps.

[0115] Step a2: Perform key-value matching based on the image feature segment set, the sound feature segment set, and the cross-modal feature association mapping table to generate a set of matching feature pairs.

[0116] The key-value matching operation refers to the query process of retrieving associated audio segment identifiers based on image block identifiers using a mapping table as the rule. The matching feature pair set is a list of tuples recording successfully matched combinations of image blocks and audio segment identifiers.

[0117] In this embodiment, the key-value correspondence rule is first established by reading the cross-modal feature association mapping table; then, each block identifier of the image feature segment set is traversed as the key, and the associated sound feature segment identifier is retrieved in the mapping table as the value; then, a matching record is generated for the successfully retrieved key-value pairs; finally, a set of matching feature pairs containing all matching relationships is output.

[0118] Step a3: Perform a cross-aggregation operation on the set of matching feature pairs to generate a sequence of fused feature units.

[0119] The cross-aggregation operation refers to the weighted fusion calculation performed on the matching feature pairs, with the weights determined by cross-modal similarity. The fused feature unit sequence refers to the vector sequence generated by aggregation, maintaining the original spatial or temporal relationship.

[0120] In this embodiment, each image segment and sound segment in the matching feature pair set is first extracted; then, the feature interaction weights are calculated through a cross-modal attention mechanism; subsequently, a weighted summation operation is performed on the image feature vector and the sound feature vector according to the weights; finally, the aggregation results are used to generate a fused feature unit sequence in sequence order.

[0121] Step a4: Based on the set of image feature segments and the set of sound feature segments, perform the unmatched identifier extraction operation to generate a set of unmatched image identifiers and a set of unmatched sound identifiers.

[0122] The unmatched identifier extraction operation refers to identifying feature segment identifiers not covered by the mapping table by calculating the set difference. The unmatched image identifier set refers to the set of coordinate indices of unmatched blocks in the image feature segment set. The unmatched sound identifier set refers to the set of starting point indices of unmatched time windows in the sound feature segment set.

[0123] In this embodiment, firstly, unmatched block identifiers in the image feature segment set are scanned to generate an unmatched image identifier set; secondly, unmatched segment identifiers in the sound feature segment set are scanned to generate an unmatched sound identifier set; finally, redundant identifier deduplication is performed on the two sets.

[0124] Step a5: Perform interpolation compensation operation based on the unmatched image identifier set and the unmatched sound identifier set to generate compensated image feature units and compensated sound feature units.

[0125] Interpolation compensation refers to the technique of generating alternative vectors for unmatched features based on spatial or temporal proximity principles. Compensated image feature units are virtual features generated by interpolating feature vectors from adjacent blocks, preserving local texture characteristics. Compensated audio feature units are compensation vectors generated based on the average features of preceding and following time windows, maintaining frequency domain energy continuity.

[0126] In this embodiment, firstly, the corresponding feature segments are located based on the set of unmatched image identifiers, and compensated image feature units are generated by linear interpolation of adjacent block features; secondly, the corresponding feature segments are located based on the set of unmatched sound identifiers, and compensated sound feature units are generated by filling with the mean of temporally adjacent segments; finally, feature dimension standardization processing is performed on the compensated units.

[0127] Step a6: Perform a multi-dimensional structure recombination operation on the fused feature unit sequence, the compensated image feature unit, and the compensated sound feature unit to generate a feature slice of uniform dimension.

[0128] Among them, multidimensional structure recombination operation refers to the process of recombinating heterogeneous features into a unified dimension tensor according to the original structure, including channel alignment and position calibration.

[0129] In this embodiment, the fused feature unit sequence is first sorted according to its original position; then, the compensated image feature unit is inserted into the missing position of the image feature; then, the compensated sound feature unit is inserted into the missing position of the sound feature; finally, a feature slice with uniform channel dimension is generated through tensor splicing operation.

[0130] Here is a specific example: In the feature reconstruction stage of an industrial fan bearing monitoring scenario, the system performs structured processing on the extracted image and sound features. First, feature segmentation is performed: the 256-dimensional feature vector generated from the bearing surface image is divided into 32 blocks according to spatial location. Each block corresponds to a 60×60 pixel area and is assigned a unique identifier, such as A. 32 This represents the block located at coordinates (120, 90); simultaneously, the 80-dimensional feature sequence generated from the sound spectrum is divided into 40 segments along the time axis, each segment covering 0.25 seconds of audio data, with identifiers such as S. 28 Mark the segment starting at 7.0 seconds.

[0131] Then, key-value matching and association are performed: the system calls the pre-stored 256 sets of feature association mapping tables and traverses all 32 image block identifiers for association retrieval. For example, block A is identified. 15 With sound fragment S 22 There are corresponding relationships, ultimately forming a set containing 28 valid matches, such as (A 15 ,S 22 ), (A 21 ,S 30 Pairing and combining features, etc. In the key feature fusion stage, cross-aggregation is implemented for the 28 sets of matching features: weight coefficients are calculated based on cross-modal similarity, such as A... 15 With S 22 The similarity was 0.92. A weighted fusion formula was used to synthesize new feature units from image features and sound features at a ratio of 0.6:0.4, generating a 28-dimensional fusion sequence that maintained the original spatiotemporal order. Compensation processing was performed on unmatched features. Specifically, the system identified four unrelated image blocks, such as A3, A7, and A... 11 A 28Compensation values ​​are generated through linear interpolation of features from adjacent blocks. For example, the compensation value for block A7 is the average of the features from adjacent blocks A6 and A8. Simultaneously, 12 unmatched audio segments are identified, such as S1, S8, and S... 16 S 32 The average value of the time-adjacent segments is used for filling, such as the S8 compensation value being the arithmetic mean of the features of S7 and S9.

[0132] Finally, multi-dimensional reconstruction is performed. Specifically, the 28 fusion units are reordered according to spatial coordinates, and compensation units are inserted at positions 3, 7, 11, and 28 of the image block sequence, and sound compensation units are inserted at positions 1, 8, 16, and 32 of the sound segment sequence, forming a standardized three-dimensional feature slice of 40 time segments × 256 feature channels. Taking the monitoring of bearing inner ring cracks as an example, when the crack region block A9 successfully matches the high-frequency abnormal noise segment S... 14 At that time, the non-cracked region block A5 is compensated by feature interpolation of the neighboring blocks A4 and A6, and finally a feature matrix that fully represents the spatiotemporal state of the equipment is constructed.

[0133] By executing steps a1 to a6, the embodiments of this application establish local feature semantic associations through key-value matching; avoid information loss by using an interpolation compensation mechanism; and generate structured fusion inputs through multi-dimensional recombination, thereby improving the discriminative power and robustness of state representation.

[0134] In one possible embodiment, step 133, based on the set of inter-modal similarity metrics, utilizes a feature-level fusion classifier to perform an association mapping construction operation to generate a cross-modal feature association mapping table, includes:

[0135] Step b1: In the feature-level fusion classifier, a threshold filtering operation is performed on the set of inter-modal similarity metrics to generate a set of candidate mapping pairs.

[0136] The candidate mapping pair set refers to the set of potential matching pairs obtained by thresholding the set of intermodal similarity metrics. It includes pairings where the similarity between image features and sound features is higher than the threshold. It is used to reflect the preliminary screening results and as input for subsequent verification.

[0137] In this embodiment, firstly, in the feature-level fusion classifier, a threshold filtering operation is performed on the set of inter-modal similarity metrics, wherein the set of inter-modal similarity metrics includes the similarity calculation results between image features and sound features; secondly, each similarity value is compared by a preset threshold, and feature pairs corresponding to values ​​higher than the threshold are filtered out; subsequently, these feature pairs are integrated to generate a candidate mapping pair set, which includes potentially matching combinations of image features and sound features.

[0138] Step b2: Perform feature indexing and grouping operations on the projected image features and projected sound features to generate image feature index groups and sound feature index groups.

[0139] The image feature index group refers to the set of indexes formed by grouping the projected image features, including subsets of image features partitioned based on feature similarity or spatial index, used to represent the group structure of image features in bidirectional association constraints. The sound feature index group refers to the set of indexes formed by grouping the projected sound features, including subsets of sound features partitioned based on feature similarity or temporal index, used to represent the group structure of sound features in bidirectional association constraints.

[0140] In this embodiment, firstly, feature indexing and grouping operations are performed on the projected image features and projected sound features; secondly, a clustering algorithm and index-based grouping technique are used to group the projected image features according to their index values ​​to form image feature index groups; simultaneously, the projected sound features are grouped according to their index values ​​to form sound feature index groups; subsequently, image feature index groups and sound feature index groups are generated, and these groups are used for subsequent association analysis.

[0141] Step b3: Apply bidirectional correlation constraints to the image feature index group and the sound feature index group to generate an effective mapping rule set.

[0142] The bidirectional association constraint refers to the bidirectional consistency requirement imposed on the image feature index group and the sound feature index group, including ensuring that each image group uniquely maps to the sound group and vice versa, used to generate valid rules based on inter-group association. The valid mapping rule set refers to the set of rules generated through the bidirectional association constraint operation, including logical rules defining the valid mapping between the image feature index group and the sound feature index group, used to evaluate candidate pairs in rule validation.

[0143] In this embodiment, firstly, bidirectional association constraints are applied to the image feature index group and the sound feature index group; secondly, association rule mining or optimization algorithms are applied to ensure that each group in the image feature index group and the corresponding group in the sound feature index group satisfy bidirectional mapping consistency; subsequently, an effective mapping rule set is generated by analyzing the relationship between the groups, which defines the effective mapping rules between the image feature index group and the sound feature index group.

[0144] Step b4: Perform rule verification on the candidate mapping pair set and the preset valid mapping rule set to generate verified mapping pairs.

[0145] The preset valid mapping rule set refers to a predefined set of valid mapping rules, including standard rules based on domain knowledge or historical data, used as a reference benchmark in the rule verification step. The verified mapping pairs refer to a set of reliable mapping pairs after rule verification, including candidate mapping pairs that conform to the preset valid mapping rule set, used as input data for key-value table construction.

[0146] In this embodiment of the application, firstly, the candidate mapping pair set is verified against the preset valid mapping rule set; secondly, each candidate mapping pair is checked for compliance with the rules in the preset valid mapping rule set by a rule matching algorithm, and invalid pairs that violate the rules are filtered out; subsequently, verified mapping pairs are generated, which include verified image feature and sound feature mapping pairs.

[0147] Step b5: Perform a key-value table construction operation on the verified mapping pairs to generate a cross-modal feature association mapping table.

[0148] Among them, the key-value table construction operation refers to the operation of converting the verified mapping pairs into a key-value structure, including constructing a mapping with features as keys and associated features as values, which is used to generate a cross-modal association table for structured storage.

[0149] In this embodiment, firstly, a key-value table is constructed for the verified mapping pairs; secondly, image features in each verified mapping pair are used as keys and sound features as values, or vice versa, to construct key-value pairs; subsequently, a cross-modal feature association mapping table is generated through data structure operations such as hash tables, which stores the association relationships between features.

[0150] Here is a specific example: First, in the feature-level fusion classifier, a threshold filtering operation is performed on the set of inter-modal similarity metrics for image and sound features to generate a set of candidate mapping pairs. Second, feature indexing is performed on the projected image and sound features to form image feature index groups and sound feature index groups. Then, bidirectional association constraints are applied to the image and sound feature index groups to generate a set of valid mapping rules. Next, the set of candidate mapping pairs is validated against the preset set of valid mapping rules to generate validated mapping pairs. Finally, a key-value table construction operation is performed on the validated mapping pairs to generate a cross-modal feature association mapping table.

[0151] By executing steps b1 to b5, this embodiment of the application realizes intermodal similarity measurement and threshold screening through a feature-level fusion classifier, generates candidate mapping pairs, and then forms an effective rule set by combining feature index grouping and bidirectional association constraints. The mapping accuracy is improved through rule verification, and finally a cross-modal association mapping in the form of a key-value table is constructed, thereby enhancing the robustness and reliability of cross-modal feature matching and optimizing the efficiency of multimedia data processing.

[0152] In one possible embodiment, S14, based on the joint semantic feature vector, construct a semantic knowledge graph, including:

[0153] Step 141: Extract state nodes from the joint semantic feature vector to generate a device state node set.

[0154] State node extraction refers to identifying dense regions representing key device states in the joint feature vector using clustering algorithms. Each node corresponds to a specific fault mode or operating state. The device state node set is a container that stores the core entities of the knowledge graph, and the node attributes include state type and confidence parameters.

[0155] In this embodiment, the high-dimensional feature distribution in the joint semantic feature vector is first analyzed by clustering algorithm; then, regions with concentrated density in the feature space are identified as key state representation points; then, a unique node identifier is assigned to each representation point; finally, a device state node set containing all key state nodes is generated.

[0156] Step 142: Perform physical relationship modeling operation on the device status node set to generate a physical topology relationship table.

[0157] The physical topology table is a two-dimensional table structure that records the rules for connecting nodes. Its fields include the starting node, the ending node, the relationship type, and the maximum effective distance.

[0158] In this embodiment, the physical structure topology diagram of the monitored device is first loaded; then the spatial positional relationship of the device status node set in the physical topology is parsed; subsequently, the connection rules between nodes, including mechanical transmission relationships and electrical coupling relationships, are established; finally, a physical topology relationship table recording the physical dependencies between nodes is generated.

[0159] Step 143: Perform edge connection operations based on the physical topology table to generate the initial knowledge graph skeleton.

[0160] The edge connection operation refers to the process of adding directed edges between graph nodes based on the physical topology table. Edge types include spatial adjacency and functional coupling. The initial knowledge graph skeleton refers to the topological framework that only contains nodes and basic connection edges, without the injection of dynamic feature weights.

[0161] In this embodiment, the node connection rules in the physical topology table are first read; then, the device status nodes are used as graph nodes; then, directed connection edges are added according to physical dependencies; finally, an initial knowledge graph skeleton containing nodes and edges is constructed.

[0162] Step 144: Perform eigenvalue parsing on the joint semantic feature vector to generate a set of vibration feature values.

[0163] Eigenvalue analysis refers to the technique of separating vibration-related dimensions from joint semantic feature vectors. The analysis objects include time-domain peak values ​​and frequency-domain resonant components. A vibration eigenvalue set is a structured container that stores multi-dimensional vibration parameters, with elements including mean amplitude, attenuation coefficient, and frequency band energy proportion.

[0164] In this embodiment, the vibration-related feature dimension in the joint semantic feature vector is first located; then the time-domain amplitude feature and frequency-domain energy feature are extracted; then the feature values ​​are standardized; and finally, a set of vibration feature values ​​containing multi-dimensional vibration characteristics is generated.

[0165] Step 145: Inject the set of vibration feature values ​​into the initial knowledge graph skeleton, assign edge weights, and generate a weighted knowledge graph.

[0166] Weighted knowledge graphs refer to dynamic topological networks formed after the initial skeleton is assigned edge weights, where edge weights reflect the strength of state transmission between nodes.

[0167] In this embodiment, the connection edge types in the initial knowledge graph skeleton are first parsed; then, the amplitude features in the vibration feature value set are mapped to spatial edges; at the same time, the attenuation features are mapped to temporal edges; then, the spatial edge weight values ​​and temporal edge weight values ​​are calculated; finally, the skeleton edge weights are updated to generate a weighted knowledge graph.

[0168] Step 146: Perform spatiotemporal constraint binding operations on the weighted knowledge graph to obtain the semantic knowledge graph.

[0169] Among them, the spatiotemporal constraint binding operation refers to the process of binding geographic location coordinates to nodes and timestamps to edge weights, so as to achieve spatiotemporal alignment between the graph and the physical world.

[0170] In this embodiment, the geographical coordinates and timestamp information of the device operation are first obtained; then the spatial coordinates are bound to the location attributes of the graph nodes; then the timestamps are bound to the edge weight update time attributes; finally, a semantic knowledge graph that integrates spatiotemporal attributes is generated.

[0171] Here is a specific example: First, extract the inner ring crack and ball bearing peeling state nodes from the joint feature vector of wind turbine bearing wear to form a set of equipment state nodes. Second, establish a physical relationship table between nodes based on the wind turbine drive chain topology. Next, construct an initial knowledge graph skeleton using state nodes as graph nodes and physical relationships as connecting edges. Then, parse the vibration features in the joint vector to generate a set of vibration feature values ​​including impact amplitude and high-frequency attenuation. Then, inject amplitude features into spatial edges to calculate weights, and inject attenuation features into temporal edges to calculate weights, generating a weighted knowledge graph. Finally, bind the wind turbine's geographical coordinates to the nodes and the sampling timestamp to the edge weights to form a spatiotemporally traceable semantic knowledge graph.

[0172] By executing steps 141 to 146, the embodiments of this application ensure that the graph structure conforms to the actual structure of the device through physical topology constraints; vibration feature injection gives the edge weights physical meaning; and spatiotemporal binding enhances the decision support capability of the graph in predictive maintenance.

[0173] In one possible embodiment, step 145, injecting the vibration feature value set into the initial knowledge graph skeleton to assign edge weights and generate a weighted knowledge graph, includes:

[0174] Step c1: Perform edge type identification on the initial knowledge graph skeleton to generate a spatial edge set and a temporal edge set.

[0175] Among them, edge type identification refers to the technique of classifying edges based on their relational type attributes. Spatial edges represent physical location relationships, while temporal edges represent the order of state evolution. A spatial edge set is a container that stores connected edges describing the spatial location relationships of device components in a knowledge graph; edge attributes include distance and direction parameters. A temporal edge set is a container that stores connected edges describing the evolution of device states over time; edge attributes include time intervals and trends of change.

[0176] In this embodiment, all connecting edges of the initial knowledge graph skeleton are traversed first; then, spatial edges representing spatial location relationships and temporal edges representing temporal evolution relationships are identified based on the relation type field in the edge attributes; then, spatial edges are assigned to the spatial edge set and temporal edges are assigned to the temporal edge set; finally, two independent edge sets are output.

[0177] Step c2: Perform feature dimension separation operation on the vibration feature value set to generate amplitude feature set and attenuation feature set.

[0178] The feature dimension separation operation refers to the process of dividing vibration features into independent subsets according to physical meaning. The amplitude feature set includes peak value and root mean square value, while the attenuation feature set includes damping coefficient and energy attenuation rate. The amplitude feature set is a container for storing vibration intensity parameters, and its elements are strongly correlated with the spatial location of the equipment. The attenuation feature set is a container for storing vibration energy attenuation parameters, and its elements are strongly correlated with the time evolution process.

[0179] In this embodiment, the dimension labels of the vibration feature value set are first parsed; then, the amplitude-related features describing the vibration intensity are separated to form an amplitude feature set; at the same time, the time-domain features describing energy decay are separated to form a decay feature set; finally, two feature subsets are output.

[0180] Step c3: Perform feature assignment operations on the spatial edge set and amplitude feature set, and the temporal edge set and attenuation feature set, respectively, to generate the spatial edge amplitude mapping table and the temporal edge attenuation mapping table.

[0181] Feature assignment refers to the process of establishing a mapping relationship between spectral edges and vibrational features. Spatial edges are assigned amplitude features, and temporal edges are assigned attenuation features. The spatial edge amplitude mapping table is a two-dimensional mapping table recording spatial edge identifiers and associated amplitude feature values, supporting weighted calculation and retrieval. The temporal edge attenuation mapping table is a two-dimensional mapping table recording temporal edge identifiers and associated attenuation feature values, supporting weighted calculation and retrieval.

[0182] In this embodiment, firstly, a correspondence rule between spatial edges and amplitude features is established; secondly, each edge in the set of spatial edges is traversed, and associated feature values ​​are extracted from the set of amplitude features to generate a spatial edge amplitude mapping table; simultaneously, a correspondence rule between temporal edges and attenuation features is established; subsequently, each edge in the set of temporal edges is traversed, and associated feature values ​​are extracted from the set of attenuation features to generate a temporal edge attenuation mapping table.

[0183] Step c4: Based on the spatial edge amplitude mapping table and the temporal edge decay mapping table, perform spatial weight and temporal weight calculation operations to generate spatial edge weight values ​​and edge temporal weight values.

[0184] Spatial weight calculation refers to the process of calculating edge weights by dividing the amplitude characteristic value by the spatial distance. The weight value is positively correlated with vibration intensity and negatively correlated with distance. Temporal weight calculation refers to the process of calculating edge weights by multiplying the attenuation coefficient by the time interval. The weight value reflects the rate of state change. Spatial edge weight value refers to the dynamic weight parameter of a spatial edge; the larger the value, the stronger the influence of the spatial relationship on the current state. Temporal edge weight value refers to the dynamic weight parameter of a temporal edge; the larger the value, the faster the state evolution speed.

[0185] In this embodiment, the spatial edge amplitude mapping table is first read; then the spatial edge weight value is calculated by the ratio of the amplitude characteristic value to the spatial distance; the temporal edge attenuation mapping table is read at the same time; then the temporal edge weight value is calculated by the product of the attenuation coefficient and the time interval; finally, two types of weight values ​​are output.

[0186] Step c5: Inject the spatial edge weights and temporal edge weights into the initial knowledge graph skeleton to update the weights and generate a weighted knowledge graph.

[0187] Among them, weight update is the operation of injecting the calculated weight values ​​into the edge attributes of the knowledge graph to realize the dynamic construction of the graph.

[0188] In this embodiment, the spatial edges of the initial knowledge graph skeleton are first located; then the calculated spatial edge weights are injected to update the edge attributes; simultaneously, temporal edges are located; subsequently, the calculated temporal edge weights are injected to update the edge attributes; finally, a weighted knowledge graph is generated.

[0189] Here's a specific example: First, identify the spatial edges describing component locations and the temporal edges describing wear processes in the wind turbine bearing knowledge graph, forming independent sets. Next, separate the amplitude and attenuation parameters from the bearing vibration characteristics to generate corresponding feature sets. Then, establish a mapping table between spatial edges and bearing housing amplitude, and a mapping table between temporal edges and wear attenuation. Subsequently, calculate the weight of the spatial edges by dividing the amplitude by the component distance, and calculate the weight of the temporal edges by multiplying the attenuation coefficient by the time interval. Finally, inject the weight values ​​into the corresponding edges of the graph to generate a dynamically weighted knowledge graph.

[0190] By executing steps c1 to c5, the embodiments of this application accurately quantify the influence intensity of spatial relationships through amplitude features and capture the state evolution rate through attenuation features, so that the graph edge weights truly reflect the operating conditions of the equipment.

[0191] In one possible embodiment, S15, based on the level value, dynamically optimize the feature weights of cross-domain IoT devices in the semantic knowledge graph, so as to generate cross-domain IoT device collaborative control instructions based on the dynamically optimized feature weights, including:

[0192] Step 151: Analyze the vibration mode of the level value and generate the dominant vibration mode identifier.

[0193] Vibration mode analysis refers to identifying the dominant vibration characteristics in the level values ​​through spectrum analysis and mode matching techniques, including the extraction of fundamental frequency resonance and harmonic component characteristics. The dominant vibration mode identifier represents a classification label for the core vibration type of the equipment, and is generated by combining the frequency range code and energy percentage level.

[0194] In this embodiment, the spectral features are first extracted by performing a fast Fourier transform on the level value; then, the dominant frequency component is identified by a pre-trained vibration mode classifier; subsequently, feature templates in the device vibration mode library are matched; and finally, a dominant vibration mode identifier representing the current core vibration type is generated.

[0195] Step 152: Perform edge weight extraction on the semantic knowledge graph to generate the original weight distribution table.

[0196] The edge weight extraction operation refers to the process of traversing the edge attributes of the knowledge graph and storing the weights in a structured manner, outputting a set of triples containing the edge identifier, weight value, and update time. The original weight distribution table is a two-dimensional relational table recording all edge weights of the knowledge graph, supporting fast retrieval based on vibration mode features.

[0197] In this embodiment, all connecting edges of the semantic knowledge graph are traversed first; then the dynamic weight attribute value of each edge is extracted; subsequently, the correspondence between edge identifiers and weight values ​​is recorded; and finally, an original weight distribution table containing the weight distribution of the entire graph is generated.

[0198] Step 153: Input the dominant vibration mode identifier into the original weight distribution table for pattern matching, generate a matching weight subset, and input the matching weight subset into the semantic knowledge graph for weight update operation to generate the optimized knowledge graph.

[0199] Pattern matching refers to the technique of comparing vibration mode identifiers with feature fields of the weight distribution table, with matching criteria including frequency coverage and energy correlation. The matched weight subset refers to the subset of weight records in the original weight distribution table that are strongly correlated with the dominant vibration mode, used for targeted optimization of the knowledge graph. Weight update operation refers to the process of dynamically adjusting the edge weights of the knowledge graph based on the vibration mode intensity coefficients; the adjustment amount is determined by multiplying the mode energy proportion by the weight sensitivity parameter. The optimized knowledge graph is a topological network whose edge weights are adaptively updated by the vibration modes, and its edge weight distribution reflects the state correlation strength under the current operating conditions.

[0200] In this embodiment, the characteristic frequency range corresponding to the dominant vibration mode identifier is first parsed; then, the edge weight subset associated with the frequency range is retrieved from the original weight distribution table; subsequently, the matching weight subset is input into the graph update module; and finally, the corresponding edge weight values ​​are adjusted according to the vibration mode intensity coefficient to generate an optimized knowledge graph.

[0201] Step 154: Traverse the optimized knowledge graph to retrieve control rules and generate an initial control instruction set.

[0202] Among them, control rule retrieval refers to the query process of traversing a pre-set rule base based on the state of graph nodes. The rule conditions include node threshold states and edge weight thresholds. The initial control instruction set refers to the set of basic device operation instructions generated by rule matching, and the elements include device identifier, operation type, and intensity parameters.

[0203] In this embodiment, a pre-set device control rule library is first loaded; then, the node paths of the optimized knowledge graph are traversed using a depth-first approach; subsequently, node states and rule conditions are matched; and finally, an initial control instruction set containing device operation instructions is generated.

[0204] Step 155: Perform device coordination constraint operations on the initial control instruction set to generate cross-domain IoT device coordination control instructions.

[0205] Among them, device coordination constraint operation refers to the optimization process of resolving conflicts between multiple device commands. The constraints include physical space mutual exclusion, energy supply limit and execution timing dependency.

[0206] In this embodiment, device operation conflicts in the initial control instruction set are first parsed; then, safety distance constraints and energy allocation constraints between devices are applied; subsequently, conflicting instructions are resolved through constraint satisfaction algorithms; and finally, cross-domain IoT device collaborative control instructions that can be executed in parallel are generated.

[0207] Here is a specific example: First, spectral analysis of the wind turbine bearing voltage levels generates shaft imbalance vibration mode identifiers. Second, all edge weights are extracted from the semantic knowledge graph to generate an initial weight distribution table. Next, a subset of weights related to the imbalance vibration is matched, and the graph is updated based on the vibration intensity to generate an optimized knowledge graph. Then, the graph is traversed to retrieve nodes exhibiting excessive vibration, generating speed reduction and lubrication initiation commands. Finally, equipment safety speed difference constraints and total power limits are applied to generate a coordinated control command that reduces speed by 30% while simultaneously initiating fixed-point lubrication.

[0208] By executing steps 151 to 155, this embodiment of the application accurately captures the influence path of vibration modes through dynamic optimization of the knowledge graph; and generates safe and efficient cross-device linkage instructions based on constraint resolution, thereby improving the reliability of system response under complex working conditions.

[0209] Figure 2 This is a schematic diagram of the structure of a cross-domain IoT device intelligent collaboration system based on semantic knowledge graphs provided in an embodiment of this application, as shown below. Figure 2 As shown, the system includes:

[0210] The conversion module 21 is used to convert the mechanical vibration energy of the monitored equipment into an electrical signal and synchronously generate the corresponding level value.

[0211] The acquisition module 22 is used to adjust the sampling frequency of the cross-domain IoT device according to the level value, and after the sampling frequency adjustment is completed, acquire the image texture data and sound spectrum data of the monitored device collected by the cross-domain IoT device.

[0212] The generation module 23 is used to generate a joint semantic feature vector representing the state of the monitored device by aligning cross-modal feature space based on image texture data and sound spectrum data using a feature-level fusion classifier.

[0213] Module 24 is used to construct a semantic knowledge graph based on joint semantic feature vectors.

[0214] The dynamic optimization module 25 is used to dynamically optimize the feature weights of cross-domain IoT devices in the semantic knowledge graph based on the level value, so as to generate cross-domain IoT device collaborative control instructions based on the dynamically optimized feature weights.

[0215] Figure 2 The aforementioned cross-domain IoT device intelligent collaboration system based on semantic knowledge graphs can execute... Figure 1The implementation principle and technical effects of the cross-domain IoT device intelligent collaboration method based on semantic knowledge graphs described in the illustrated embodiments will not be repeated here. The specific methods by which each module and unit of the cross-domain IoT device intelligent collaboration system based on semantic knowledge graphs in the above embodiments perform operations have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0216] In one possible design, Figure 2 The cross-domain IoT device intelligent collaboration system based on semantic knowledge graphs in the illustrated embodiment can be implemented as a computing device, such as... Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32.

[0217] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component 32.

[0218] The processing component 32 is used to: convert the mechanical vibration energy of the monitored device into an electrical signal and synchronously generate a corresponding level value; adjust the sampling frequency of the cross-domain IoT device according to the level value; after the sampling frequency adjustment is completed, acquire the image texture data and sound spectrum data of the monitored device collected by the cross-domain IoT device; based on the image texture data and sound spectrum data, use a feature-level fusion classifier to perform cross-modal feature space alignment to generate a joint semantic feature vector representing the state of the monitored device; construct a semantic knowledge graph based on the joint semantic feature vector; and dynamically optimize the feature weights of the cross-domain IoT device in the semantic knowledge graph according to the level value, so as to generate cross-domain IoT device collaborative control commands according to the dynamically optimized feature weights.

[0219] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0220] Storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as Random Access Memory (RAM), Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0221] Of course, computing devices may also include other components, such as input / output interfaces, display components, communication components, etc.

[0222] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc.

[0223] The communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0224] The computing device can be a physical device or an elastic computing host provided by a cloud computing platform. In this case, the computing device can refer to a cloud server, and the aforementioned processing components, storage components, etc., can be basic server resources rented or purchased from the cloud computing platform.

[0225] This application also provides a computer storage medium storing a computer program, which, when executed by a computer, can perform the above-described functions. Figure 1 The illustrated embodiment is a cross-domain IoT device intelligent collaboration method based on semantic knowledge graph.

[0226] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0227] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0228] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0229] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A cross-domain IoT device intelligent collaboration method based on semantic knowledge graph, characterized in that, include: The mechanical vibration energy of the monitored equipment is converted into an electrical signal, and a corresponding level value is generated synchronously. Based on the stated level value, the sampling frequency of the cross-domain IoT device is adjusted. After the sampling frequency adjustment is completed, the image texture data and sound spectrum data of the monitored device collected by the cross-domain IoT device are obtained. Based on the image texture data and sound spectrum data, a feature-level fusion classifier is used to perform cross-modal feature space alignment to generate a joint semantic feature vector representing the state of the monitored device. Based on the joint semantic feature vector, a semantic knowledge graph is constructed; Based on the level value, the feature weights of cross-domain IoT devices in the semantic knowledge graph are dynamically optimized, so as to generate cross-domain IoT device collaborative control instructions based on the dynamically optimized feature weights. The step of aligning cross-modal feature spaces using a feature-level fusion classifier based on the image texture data and sound spectrum data to generate a joint semantic feature vector representing the state of the monitored device includes: The image texture data is input into the image projection function to perform a mapping operation, generating projected image features; the sound spectrum data is input into the sound projection function to perform a mapping operation, generating projected sound features. Based on the projected image features and projected sound features, intermodal similarity calculation is performed in the shared latent space to generate a set of intermodal similarity measurement values. Based on the set of intermodal similarity metrics, a feature-level fusion classifier is used to construct an association mapping to generate a cross-modal feature association mapping table. The projected image features are segmented to generate a set of image feature segments, and the projected sound features are segmented to generate a set of sound feature segments. Based on the image feature segment set, the sound feature segment set, and the cross-modal feature association mapping table, a key-value matching operation is performed to generate a set of matching feature pairs; Perform a cross-aggregation operation on the set of matching feature pairs to generate a sequence of fused feature units; Based on the set of image feature segments and the set of sound feature segments, perform the unmatched identifier extraction operation to generate a set of unmatched image identifiers and a set of unmatched sound identifiers; Based on the set of unmatched image identifiers and the set of unmatched sound identifiers, an interpolation compensation operation is performed to generate compensated image feature units and compensated sound feature units; The fused feature unit sequence, the compensated image feature unit, and the compensated sound feature unit are subjected to a multi-dimensional structure recombination operation to generate feature slices of a unified dimension. The feature slices are input into the bidirectional feature interaction module for multiple rounds of aggregation to generate a joint semantic feature vector representing the state of the monitored device; The step of adjusting the sampling frequency of the cross-domain IoT device according to the level value includes: The vibration intensity level is determined by analyzing the voltage level. Based on the preset intensity-frequency mapping relationship, a sampling frequency adjustment command is sent to cross-domain IoT devices; After the device completes the sampling frequency reset, the camera is simultaneously triggered to collect image texture data on the surface of the device, and the microphone is activated to collect sound spectrum data of the device operation.

2. The method according to claim 1, characterized in that, The step of constructing a cross-modal feature association mapping table based on the set of inter-modal similarity metrics, using a feature-level fusion classifier, includes: In the feature-level fusion classifier, a threshold filtering operation is performed on the set of inter-modal similarity metrics to generate a set of candidate mapping pairs; Perform feature indexing and grouping operations on the projected image features and projected sound features to generate image feature index groups and sound feature index groups; A bidirectional correlation constraint is applied to the image feature index group and the sound feature index group to generate an effective mapping rule set; The candidate mapping pair set is subjected to rule verification with the preset valid mapping rule set to generate verified mapping pairs; A key-value table construction operation is performed on the verified mapping pairs to generate a cross-modal feature association mapping table.

3. The method according to claim 1, characterized in that, The construction of a semantic knowledge graph based on the joint semantic feature vector includes: State nodes are extracted from the joint semantic feature vector to generate a device state node set; Perform physical relationship modeling operations on the set of device status nodes to generate a physical topology table; Based on the physical topology table, perform edge connection operations to generate an initial knowledge graph skeleton; The joint semantic feature vector is parsed to generate a set of vibration feature values; The set of vibration feature values ​​is injected into the initial knowledge graph skeleton to assign edge weights and generate a weighted knowledge graph. The weighted knowledge graph is subjected to spatiotemporal constraint binding operations to obtain a semantic knowledge graph.

4. The method according to claim 3, characterized in that, The step of injecting the vibration feature value set into the initial knowledge graph skeleton, assigning edge weights, and generating a weighted knowledge graph includes: Perform edge type identification operation on the initial knowledge graph skeleton to generate a spatial edge set and a temporal edge set; Perform a feature dimension separation operation on the vibration feature value set to generate an amplitude feature set and an attenuation feature set; Perform feature allocation operations on the spatial edge set and the amplitude feature set, and on the temporal edge set and the attenuation feature set, respectively, to generate a spatial edge amplitude mapping table and a temporal edge attenuation mapping table; Based on the spatial edge amplitude mapping table and the temporal edge decay mapping table, spatial weight and temporal weight calculation operations are performed to generate spatial edge weight values ​​and edge temporal weight values. The spatial edge weights and temporal edge weights are injected into the initial knowledge graph skeleton for weight update, generating a weighted knowledge graph.

5. The method according to claim 1, characterized in that, The step of dynamically optimizing the feature weights of cross-domain IoT devices in the semantic knowledge graph based on the level value, and generating cross-domain IoT device collaborative control instructions based on the dynamically optimized feature weights, includes: The vibration mode is analyzed based on the level value to generate a dominant vibration mode identifier; The semantic knowledge graph is subjected to edge weight extraction to generate an original weight distribution table; The dominant vibration mode identifier is input into the original weight distribution table for pattern matching, generating a matching weight subset. The matching weight subset is then input into the semantic knowledge graph for weight update operations, generating an optimized knowledge graph. The optimized knowledge graph is traversed to retrieve control rules and generate an initial control instruction set. The initial control instruction set is subjected to device coordination constraint operations to generate cross-domain IoT device coordination control instructions.

6. A cross-domain IoT device intelligent collaboration system based on semantic knowledge graphs, characterized in that, A method for executing a cross-domain IoT device intelligent collaboration method based on semantic knowledge graph as described in claim 1 includes: The conversion module is used to convert the mechanical vibration energy of the monitored equipment into an electrical signal and synchronously generate the corresponding level value; The acquisition module is used to adjust the sampling frequency of the cross-domain IoT device according to the level value, and after the sampling frequency adjustment is completed, acquire the image texture data and sound spectrum data of the monitored device collected by the cross-domain IoT device. The generation module is used to generate a joint semantic feature vector representing the state of the monitored device by aligning the image texture data and sound spectrum data across the modal feature space using a feature-level fusion classifier. A construction module is used to construct a semantic knowledge graph based on the joint semantic feature vector; The dynamic optimization module is used to dynamically optimize the feature weights of cross-domain IoT devices in the semantic knowledge graph based on the level value, so as to generate cross-domain IoT device collaborative control instructions based on the dynamically optimized feature weights.

7. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the cross-domain IoT device intelligent collaboration method based on semantic knowledge graph as described in any one of claims 1 to 5.

8. A computer storage medium, characterized in that, The device stores a computer program, which, when executed by a computer, implements a cross-domain IoT device intelligent collaboration method based on semantic knowledge graphs as described in any one of claims 1 to 5.

Images