Process layer device multi-modal perception-based anomaly detection and acousto-optic interaction method and system

Abstract

The application provides an abnormality detection and acousto-optic interaction method and system for multi-modal perception of process layer equipment, relates to the technical field of industrial automation, and comprises the following steps: collecting energy consumption fluctuation data and wireless communication quality data of a process layer equipment cluster, combining with equipment identification to form an original operation data set and labeling the original operation data set with equipment-associated identification to form an associated data set, adjusting and verifying initial filter gain parameters of an adaptive filter circuit to determine target filter gain parameters, filtering signals of the associated data set according to the target filter gain parameters, finally generating an equipment operation link, identifying abnormal equipment with parameters deviating from a normal range in the link, generating an equipment abnormality judgment result, determining an alarm level, and generating corresponding alarm audio signals, optical signals and ultrasonic carrier waves; and modulating characteristic parameters to the ultrasonic carrier waves to complete directional acousto-optic prompting of an abnormal area in combination with the optical signals, so that the application realizes intelligent and accurate identification, detection and acousto-optic prompting of implicit abnormalities under complex working conditions.

Description

Technical Field

[0001] This application relates to the field of industrial automation technology, and in particular to anomaly detection and audio-visual interaction methods and systems for multimodal sensing of process-level equipment. Background Technology

[0002] In complex systems such as industrial automation and smart grids, process-layer device clusters undertake critical tasks such as data acquisition, command execution, and real-time control. Their operational status directly affects the stability and security of the entire system. As device integration and system complexity continue to increase, the coupling between devices becomes increasingly tight. An anomaly in a single device can spread rapidly through physical connections or data interactions, triggering a chain reaction and leading to local or even global functional failures.

[0003] Existing solutions attempt to collect operating parameters of equipment, such as vibration, temperature, and current, by deploying distributed sensor networks. Machine learning algorithms are then used to perform pattern recognition on the collected data to determine if any operational anomalies exist. This approach establishes a feature model of the equipment's normal state and compares real-time data with the model, triggering an alarm when the deviation exceeds a set threshold. However, existing solutions reveal significant shortcomings in practical applications. For example, their data processing typically employs fixed thresholds or static filtering strategies, making it difficult to adapt to the dynamic changes in signal interference under complex electromagnetic environments. This results in the inability to effectively separate effective signals from noise, affecting the accuracy of anomaly identification. Furthermore, the lack of modeling of the collaborative operating relationships between devices makes it impossible to capture potential faults caused by abnormal linkages between related equipment, easily overlooking early signs. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for anomaly detection and audio-visual interaction of multimodal sensing of process layer equipment, so as to solve the problems of signal separation, early fault leakage and inaccurate anomaly identification caused by the use of fixed thresholds or static filtering and lack of equipment collaborative modeling in the prior art.

[0005] To address the aforementioned technical problems, in a first aspect, this application provides an anomaly detection and audio-visual interaction method for multimodal sensing of process-level devices, comprising:

[0006] Collect energy consumption fluctuation data and wireless communication quality data generated by each device in the process layer device cluster during operation, and combine them with the device identifier of each device to form the original operation dataset of the process layer device cluster;

[0007] The frequency of physical connections and the amount of data interaction between each device are counted to label the original running dataset with device association identifiers, forming an associated dataset including device association identifiers;

[0008] Based on the associated dataset, the initial filter gain parameters of the adaptive filter circuit are adjusted and verified to obtain the target filter gain parameters, so as to perform signal filtering operation on the associated dataset and generate the processed associated dataset.

[0009] Based on the processed associated dataset, a device operation link is generated. A graph neural network is used to identify abnormal devices in the device operation link whose energy consumption fluctuation data and wireless communication quality data deviate from the corresponding normal range, so as to generate a device abnormality judgment result.

[0010] Based on the device anomaly determination result, an alarm level is determined to generate an alarm audio signal, an optical signal, and an ultrasonic carrier wave corresponding to the alarm level.

[0011] The characteristic parameters of the alarm audio signal are modulated onto the ultrasonic carrier to form a directional ultrasonic carrier signal, which, combined with the optical signal, completes a directional audio-visual alert for the specific area where the abnormal device is located.

[0012] Optionally, the characteristic parameters of the alarm audio signal are modulated onto the ultrasonic carrier to form a directional ultrasonic carrier signal, which, combined with the optical signal, provides a directional audio-visual alert for the specific area where the malfunctioning device is located, including:

[0013] The feature parameters used to modulate the ultrasonic carrier are extracted from the alarm audio signal. The feature parameters include the signal frequency, playback duration, and electrical signal amplitude corresponding to the pitch.

[0014] The characteristic parameters are modulated onto the ultrasonic carrier wave to form a modulated ultrasonic signal;

[0015] Based on the abnormal devices and their corresponding location information in the device anomaly determination results, the modulated ultrasonic signal is directionally transmitted through an ultrasonic transmitting device to form a focused sound field in the specific physical area where each abnormal device is located. Combined with the modulated ultrasonic signal, a directional ultrasonic carrier signal is formed.

[0016] Based on the optical signal parameters and the optical signal, an optimized optical signal is generated with the transmission time synchronized with the directional ultrasonic carrier signal and the optical coverage range matching the range of the focused sound field.

[0017] When the directional ultrasonic carrier signal is received within the focused sound field, an auditory cues corresponding to the alarm audio signal and a visual cues corresponding to the optimized optical signal and the specific physical area where each abnormal device is located are generated, so as to complete the directional audio-visual cues for the specific area where the abnormal device is located.

[0018] Optionally, based on the abnormal devices and their corresponding location information in the device anomaly determination result, the modulated ultrasonic signal is directionally transmitted through an ultrasonic transmitting device to form a focused sound field in a specific physical area where each abnormal device is located. Combined with the modulated ultrasonic signal, a directional ultrasonic carrier signal is formed, including:

[0019] Based on the abnormal equipment location information in the equipment abnormality determination result, the relative orientation of the abnormal equipment and the ultrasonic transmitting device is determined, and the relative orientation is converted into a directional transmission angle;

[0020] The directional transmission power required to maintain signal strength is determined based on the spatial distance between the specific physical area where the abnormal device is located and the ultrasonic transmitting device.

[0021] Based on the directional emission angle and the directional emission power, the modulated ultrasonic signal is emitted using an ultrasonic emission device to form a focused sound field with a coverage area matching the specific physical area;

[0022] Based on the spatial constraint effect and directional propagation properties of the focused sound field on the modulated ultrasonic signal, a directional ultrasonic carrier signal is formed.

[0023] Secondly, this application provides an anomaly detection and audio-visual interaction system for multimodal sensing of process-level devices, including:

[0024] The data acquisition module is used to collect energy consumption fluctuation data and wireless communication quality data generated by each device in the process layer device cluster during operation, and combine them with the device identifier of each device to form the original operation dataset of the process layer device cluster.

[0025] The annotation module is used to count the frequency of physical connections and the amount of data interaction between devices, so as to annotate the original running dataset with device association identifiers and form an association dataset including device association identifiers;

[0026] The filtering module is used to adjust and verify the initial filter gain parameters of the adaptive filtering circuit based on the associated dataset to obtain the target filter gain parameters, so as to perform signal filtering operation on the associated dataset and generate the processed associated dataset.

[0027] The identification module is used to generate a device operation link based on the processed associated dataset, and to identify abnormal devices in the device operation link whose energy consumption fluctuation data and wireless communication quality data deviate from the corresponding normal range through a graph neural network, so as to generate a device abnormality judgment result.

[0028] The determination module is used to determine the alarm level based on the device anomaly determination result, so as to generate an alarm audio signal, an optical signal, and an ultrasonic carrier wave corresponding to the alarm level;

[0029] The prompting module is used to modulate the characteristic parameters of the alarm audio signal onto the ultrasonic carrier to form a directional ultrasonic carrier signal, which, combined with the optical signal, completes directional audio-visual prompting for the specific area where the abnormal device is located.

[0030] Thirdly, this application provides an electronic device, comprising:

[0031] Memory, used to store computer programs;

[0032] A processor, used to execute the computer program to implement the steps of the anomaly detection and audio-visual interaction method for process layer device multimodal sensing as described in the first aspect above.

[0033] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the anomaly detection and audio-visual interaction method for process layer device multimodal perception as described in the first aspect above.

[0034] The anomaly detection and audio-visual interaction method for multimodal perception of process-level equipment provided in this application collects energy consumption fluctuation data and wireless communication quality data of each device in the process-level equipment cluster during operation, and combines this with device identifiers to form an original operating dataset. This enables multi-dimensional dynamic perception of the device's operating status, providing a traceable foundation for subsequent analysis. Furthermore, it statistically analyzes the frequency of physical connections and data interaction between devices, annotating the original dataset with device association identifiers, and constructing an associated dataset including relationships. This overcomes the limitations of isolated analysis of single devices and supports the discovery of collaborative operation patterns and potential fault propagation paths. Based on this, the initial filter gain parameters of the adaptive filter circuit are adjusted and verified according to the associated dataset to obtain target filter gain parameters, which are then used to filter signals in the associated dataset, generating a processed associated dataset to adapt to changes in complex electromagnetic environments, suppress environmental noise interference, and improve the quality and reliability of multi-source data. After processing, the associated dataset generates a device operation link. Graph neural networks are used to analyze the deviation of device operating parameters in the associated network, identifying abnormal devices whose operating parameters deviate from the normal range, generating device anomaly judgment results, and improving the detection capability of latent faults and early anomalies caused by system linkage. Based on the device anomaly judgment results, the alarm level is determined by combining the location, type, and parameter deviation degree of the abnormal device, and matching alarm audio signals, optical signals, and ultrasonic carrier waves are generated, so that the alarm information can reflect the severity of the fault and assist in operation and maintenance decisions. Finally, the characteristic parameters of the alarm audio signal are modulated onto the ultrasonic carrier to form a directional ultrasonic carrier signal, and combined with the optical signal to achieve temporal and spatial coordinated output, forming a focused sound field and matching visual cues in the specific area where the abnormal device is located, completing directional audio-visual cues, solving the problem of the lack of spatial directionality in traditional broadcast alarms, and improving the ability to quickly and accurately identify fault locations in complex industrial environments.

[0035] Furthermore, characteristic parameters such as signal frequency, playback duration, and electrical signal amplitude corresponding to pitch are extracted from the alarm audio signal and modulated onto an ultrasonic carrier wave to form a modulated ultrasonic signal including audio information. Based on the abnormal device location information in the device anomaly determination result, the ultrasonic transmitting device is controlled to directionally transmit the modulated ultrasonic signal, forming a focused sound field in the physical area where the abnormal device is located, thereby generating a directional ultrasonic carrier signal. An optimized optical signal is generated with a transmission time synchronized with the directional ultrasonic carrier signal and a light coverage range matching the focused sound field. When the directional ultrasonic carrier signal is within the focused area... Upon reception, the corresponding auditory cues are reproduced, while simultaneously presenting visual cues corresponding to specific physical areas, achieving spatially precise directional audio-visual cues. By modulating the alarm audio onto an ultrasonic carrier wave and achieving directional transmission, the sound cues are clearly perceived only in the target area, avoiding interference with other areas and improving the recognizability of auditory cues in noisy backgrounds. Combined with temporally and spatially coordinated optical signals, a dual-focused visual and auditory alarm effect is formed, enhancing the ability of maintenance personnel to quickly identify and respond to the location of abnormal equipment, and improving the accuracy and efficiency of human-machine interaction in complex industrial scenarios. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A flowchart illustrating the anomaly detection and audio-visual interaction method for multimodal perception of process layer devices provided in this application embodiment;

[0038] Figure 2 A schematic diagram illustrating a specific implementation of the anomaly detection and audio-visual interaction method for multimodal sensing of process layer devices provided in this application embodiment;

[0039] Figure 3 This is a schematic diagram of the structure of the multimodal sensing anomaly detection and audio-visual interaction system for process layer devices provided in this application embodiment. Detailed Implementation

[0040] To address the problem of difficulty in timely and accurate fault location when process-level equipment clusters operate in complex industrial environments due to tight coupling between equipment, variable electromagnetic interference, and insufficient directionality of abnormal alarms, existing methods can identify anomalies and trigger audible and visual prompts based on multiple parameters of a single device. However, they are limited by static signal processing mechanisms, have weak adaptability to dynamic interference, and lack in-depth modeling of the collaborative operation relationship of equipment, so alarm information cannot accurately point to the specific fault location. To improve the accuracy of anomaly detection and the localization of alarms, this application focuses on multimodal data fusion and intelligent interaction. It collects operational characteristics such as equipment energy consumption fluctuations and wireless communication quality, and constructs a correlation dataset by combining the physical connections between devices and the frequency of data interaction. An adaptive filter circuit with dynamic adjustment capabilities is used to suppress environmental interference and improve signal quality. Based on this, by analyzing the coordinated change patterns of equipment operating parameters, an equipment operation link reflecting the interconnected characteristics of the devices is constructed. A graph neural network is used to mine abnormal deviation behaviors between nodes, enabling the identification of hidden faults and generating equipment anomaly judgment results. Then, based on the location information of the abnormal equipment, combined with the equipment's functional attributes and the degree of parameter deviation, the alarm level is determined, generating corresponding alarm audio signals, optical signals, and ultrasonic carriers at preset frequencies. Finally, the characteristic parameters of the alarm audio signal are modulated onto the ultrasonic carrier to form a directional ultrasonic carrier signal. Combined with the temporal and spatial characteristics of the optical signal, this achieves coordinated output of sound field focusing and visual cues in a specific area where the abnormal equipment is located, realizing precise spatial delivery of audio and visual cues, thereby improving the ability of maintenance personnel to quickly perceive and locate abnormal equipment.

[0041] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] The core of this application is to provide an anomaly detection and audio-visual interaction method for multimodal sensing of process-level devices. A flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0043] Step 101: Collect energy consumption fluctuation data and wireless communication quality data generated by each device in the process layer device cluster during operation, and combine them with the device identifier of each device to form the original operation dataset of the process layer device cluster.

[0044] In this step, the process-layer device cluster refers to the collection of underlying devices in complex systems such as industrial automation or smart grids that directly undertake key tasks such as data acquisition, command execution, and real-time control. It is an organically interconnected whole of devices formed based on system functional requirements and physical layout. Energy consumption fluctuation data refers to the fluctuation data of energy consumption indicators such as power consumption and electrical energy consumption of each device in the process-layer device cluster over time during operation. Wireless communication quality data refers to data reflecting transmission performance when data is transmitted wirelessly between devices in the process-layer device cluster. This data includes, but is not limited to, signal transmission rate, packet loss rate, latency, and signal strength. Device identifier refers to the unique identification information assigned to each device in the process-layer device cluster. This identifier includes, but is not limited to, device number and Internet Protocol address. The original operating dataset refers to the initial dataset formed by integrating the energy consumption fluctuation data and wireless communication quality data of each device in the process-layer device cluster acquired by the acquisition module, and associating these data with the corresponding device identifiers.

[0045] In this embodiment, the real-time acquisition functions of the energy consumption acquisition module and the wireless communication monitoring module of each device deployed in the process layer equipment cluster are activated. The acquisition frequency is set, and the energy consumption fluctuation data of each device during operation is acquired in real time through the energy consumption acquisition module, specifically including the changes in the device's instantaneous power and average power over time. The wireless communication monitoring module acquires the wireless communication quality data of each device when it interacts with associated devices in real time. While acquiring the above two types of data, the unique device identifier of each device is recorded simultaneously. This identifier can be the media access control address fixed at the time of manufacture or a unique device number pre-assigned by the system, ensuring that each data entry can be traced back to a specific device. Subsequently, through a data association algorithm, the energy consumption fluctuation data and wireless communication quality data of the same device at the same acquisition time point are bound one-to-one with the device identifier of the device to obtain a complete data record. Finally, the associated data records of all devices are organized in the order of acquisition time to construct a structured dataset including the fields of device identifier, acquisition time, energy consumption fluctuation data, and wireless communication quality data, forming an original operating dataset covering the operating information of all devices in the process layer equipment cluster.

[0046] Step 102: Count the frequency of physical connections and the amount of data interaction between each device to label the original running dataset with device association identifiers, forming an association dataset including device association identifiers.

[0047] In this step, physical connection frequency refers to the number of times a physical connection actually occurs between any two devices in the process layer device cluster within a preset time period. Data exchange volume refers to the total amount of data transmitted between any two devices in the process layer device cluster within a preset time period. Device association identifier refers to the identification information that annotates the device's operating data based on the physical connection frequency and data exchange volume between devices in the process layer device cluster, reflecting the device's association relationship with other devices. Association dataset refers to the dataset formed after labeling the operating data of each device with device association identifiers, based on the original operating dataset, including the original operating data and the association relationship information between devices.

[0048] In this embodiment, based on the physical connection topology and data interaction records of the process layer device cluster in the industrial system, the number of times physical connections actually occur between any two devices within a preset time period is counted to obtain the physical connection frequency between each device. At the same time, the total amount of data transmitted between any two devices within the same preset time period is counted to obtain the data interaction volume between each device. Based on the counted physical connection frequency and data interaction volume, the degree of association between devices is determined. Device association identifiers that can reflect the relationship between each device and other devices are marked on the operation data of each device in the original operation dataset. The marked original operation datasets are integrated to form an association dataset including device association identifiers.

[0049] Step 103: Based on the associated dataset, adjust and verify the initial filter gain parameters of the adaptive filter circuit to obtain the target filter gain parameters, and perform signal filtering operation on the associated dataset to generate the processed associated dataset.

[0050] In this step, the adaptive filtering circuit refers to an electronic circuit that can automatically adjust filtering parameters based on the characteristics of the input signal to achieve optimal filtering results. The initial filtering parameters refer to the filtering parameters preset by the adaptive filtering circuit based on experience or typical scenarios before signal processing begins; specifically, this step involves initial filtering gain parameters. The target filtering gain parameters refer to the filtering gain parameters that meet the current signal processing requirements, obtained by adjusting and verifying the initial filtering gain parameters. The processed associated dataset refers to the dataset formed after the associated dataset has been filtered by the adaptive filtering circuit according to the target filtering gain parameters, removing interference signals while retaining valid equipment operating data and equipment association identifiers.

[0051] In this embodiment of the application, step 103 specifically includes the following steps:

[0052] Step 301: Select signal data within a preset time period from the associated dataset.

[0053] In this step, the preset time period refers to a continuous time interval pre-set for analyzing equipment operating signals, based on the typical operating cycle of the process-level equipment cluster or the common duration of interference signals. Signal data refers to data extracted from the associated dataset within the preset time period, which includes energy consumption fluctuation data, wireless communication quality data, and corresponding equipment association identifiers for each device within that time period.

[0054] In this embodiment, a preset time period is first set based on the typical operating cycle of the process layer equipment cluster or the common duration of interference signals. Then, the built-in timestamp matching function of the system is used to traverse the timestamps of all data entries in the associated dataset and filter out all data whose timestamps fall within the preset time period. During the filtering process, it is ensured that each data entry retains the corresponding device association identifier, locking the signal data and device identifier one by one to avoid the loss of the association between the signal and the device in subsequent analysis, and finally forming a set of signal data within the preset time period.

[0055] Step 302: The signal detection unit of the adaptive filter circuit identifies the target signal component in the signal data whose signal amplitude exceeds a preset amplitude range. The target signal component is the characteristic component corresponding to the clutter signal generated by environmental electromagnetic interference.

[0056] In this step, the signal detection unit refers to the functional module in the adaptive filter circuit used to identify signal characteristics. Signal amplitude refers to the peak value of voltage or power fluctuations in the time domain, reflecting the strength of the signal. The preset amplitude range refers to a reasonable amplitude interval pre-set based on the signal characteristics during normal equipment operation, determined based on the amplitude statistics of historical normal signals. The target signal component refers to the portion of the signal data whose amplitude exceeds the preset amplitude range. The characteristic components corresponding to clutter signals generated by environmental electromagnetic interference refer to the signal characteristics included in the target signal component that can reflect the characteristics of electromagnetic interference. Environmental electromagnetic interference refers to the interference of electromagnetic signals generated by power equipment, wireless communication, etc., in the industrial environment on the equipment's operating signals, which can lead to signal distortion or the introduction of clutter. Clutter signals refer to useless signals mixed into the equipment's operating signals due to environmental electromagnetic interference, which can affect the judgment of the equipment's true operating status. Characteristic components refer to signal segments or parameter sets in the target signal that can uniquely identify the electromagnetic interference attributes of the environment. They are generated by electromagnetic interference in the industrial environment where the process-level equipment cluster is located. They are the key part of the clutter signal that distinguishes it from the signal during normal equipment operation. They include the specific characteristics of the interference signal. These characteristics can be high-frequency pulses in a fixed frequency band, burst signal patterns in the time domain, irregular amplitude fluctuations, etc. These characteristics differ from the signal characteristics corresponding to energy consumption fluctuation data and wireless communication quality data during normal equipment operation.

[0057] In this embodiment, signal data is input into the signal detection unit of the adaptive filter circuit. The signal detection unit monitors the signal data point by point at a preset sampling frequency. First, the signal amplitude, i.e., the voltage or power value, of each sampling point is read. Then, a preset amplitude range determined based on historical normal signal amplitude statistics is retrieved. The signal amplitude of each sampling point is compared with this range in real time, and the signal portion corresponding to the sampling point whose amplitude exceeds the preset amplitude range is marked, forming a preliminary target signal segment. Subsequently, the preliminary target signal segment is verified by signal feature analysis, such as fluctuation frequency and pulse shape, to confirm that the part that conforms to the characteristics of electromagnetic interference clutter, and finally the target signal component is determined.

[0058] Step 303: Calculate the ratio of the target signal component to the sampling points of the signal data, and adjust and verify the initial filter gain parameters of the adaptive filter circuit in conjunction with the parameter configuration data corresponding to the process layer device cluster, so as to obtain the target filter gain parameters.

[0059] In this step, the sampling point ratio refers to the ratio of the number of signal sampling points for the target signal component to the total number of signal sampling points for the signal data. Parameter configuration data refers to the set of settings stored in the system that are related to the operating environment of the process-level equipment cluster. This data includes filtering parameter adjustment rules to cope with different electromagnetic interference intensities, pre-configured based on the equipment cluster's installation environment and historical interference data.

[0060] In this embodiment of the application, step 303 may optionally include the following steps:

[0061] Step 311: Count the total number of signal sampling points of the signal data and the number of signal sampling points corresponding to the target signal component to calculate the sampling point ratio.

[0062] In this step, the total number of signal sampling points refers to the total number of times the signal data is sampled within a preset time period. The number of signal sampling points refers to the number of times the target signal component is sampled within a preset time period.

[0063] In this embodiment, the total number of signal sampling points is obtained by multiplying the preset time period length and the signal sampling frequency; then, the target signal components are traversed to count the number of signal sampling points contained therein; finally, the sampling point ratio is obtained by calculating the quotient of the number of signal sampling points corresponding to the target signal component and the total number of signal sampling points of the signal data.

[0064] Step 312: Based on the electromagnetic interference environment of the process layer equipment cluster, retrieve the corresponding parameter configuration data. The parameter configuration data includes a preset first proportion value, a second proportion value, and the gain adjustment rules of the adaptive filter circuit.

[0065] In this step, the electromagnetic interference environment refers to the range of electromagnetic interference strength levels in the environment where the process layer equipment cluster is located, based on long-term monitoring data from environmental electromagnetic monitoring equipment. The preset first and second proportion values ​​refer to the proportion thresholds of two sampling points set in the parameter configuration data, with the first proportion value being greater than the second proportion value. The gain adjustment rules refer to the preset set of rules used to adjust the filter gain parameters. These rules include gain up adjustment step size, gain down adjustment step size, maximum filter gain threshold, and minimum filter gain threshold, set based on interference suppression effectiveness and signal fidelity requirements.

[0066] In this embodiment, the electromagnetic interference environment of the process layer equipment cluster is acquired in real time through the built-in electromagnetic monitoring module. For example, if the monitored value is 40dBμV / m, it corresponds to a high-intensity electromagnetic interference level range. Based on the electromagnetic interference environment level, the corresponding parameter configuration data is retrieved from the parameter configuration database stored in the system. The parameter configuration data is preset for different interference intensity ranges. Taking a high-intensity electromagnetic interference environment as an example, the corresponding parameter configuration data has a first proportion value set to 10%, a second proportion value set to 5%, and a gain adjustment rule of gain increase step size of 0.3, gain decrease step size of 0.1, maximum filter gain threshold of 5.0, and minimum filter gain threshold of 1.0.

[0067] Step 313: Based on the gain adjustment rule and the comparison results of the sampling point ratio with the first ratio value and the second ratio value, adjust the initial filter gain parameter of the adaptive filter circuit to obtain the adjusted gain parameter.

[0068] In this embodiment, the sampling point proportion is first compared with the first proportion value and the second proportion value to obtain the comparison result. If the sampling point proportion is greater than the first proportion value, the adjusted gain parameter is obtained by calculating the sum of the initial filter gain parameter and the gain up adjustment step size. If the sampling point proportion is less than the second proportion value, the adjusted gain parameter is obtained by calculating the difference between the initial filter gain parameter and the gain down adjustment step size. If the sampling point proportion is between the first proportion value and the second proportion value, the initial filter gain parameter is kept unchanged, and the initial value is directly used as the adjusted gain parameter.

[0069] In this step, the comparison result refers to the relationship judgment result obtained by comparing the sampling point proportion with the first proportion value and the second proportion value. The adjusted gain parameter refers to the temporary parameter obtained after adjusting the initial filter gain parameter according to the gain adjustment rule and the comparison result. It has not yet been verified by threshold and its rationality needs to be further verified.

[0070] Step 314: Verify the adjusted gain parameters to obtain the target filter gain parameters.

[0071] In this embodiment, the maximum and minimum filter gain thresholds in the parameter configuration data are retrieved, and the adjusted gain parameter is compared with these two thresholds. If the adjusted gain parameter is greater than the maximum filter gain threshold, the maximum filter gain threshold is taken as the target filter gain parameter; if the adjusted gain parameter is less than the minimum filter gain threshold, the minimum filter gain threshold is taken as the target filter gain parameter; if the adjusted gain parameter is between the maximum and minimum filter gain thresholds, the adjusted gain parameter is directly taken as the target filter gain parameter.

[0072] Step 304: Based on the target filter gain parameter, the signal processing unit of the adaptive filter circuit performs signal filtering operation on the associated dataset to form a processed associated dataset. In the filtering operation, the correspondence between the power consumption fluctuation data and wireless communication quality data of each device and the device association identifier are retained.

[0073] In this step, the signal processing unit refers to the functional module in the adaptive filtering circuit that performs signal filtering operations.

[0074] In this embodiment, the target filter gain parameter is input to the signal processing unit of the adaptive filter circuit. The signal processing unit adjusts the signal amplification or attenuation coefficient of the internal filter circuit according to the parameter. For example, when the target filter gain parameter is 1.8, an attenuation coefficient of 1.8 is applied to the clutter signal. Subsequently, the full associated dataset is input to the signal processing unit, and signal filtering is performed on each data entry to filter out clutter signals generated by environmental electromagnetic interference. During the filtering process, the time correspondence between the energy consumption fluctuation data and the wireless communication quality data of each device is maintained through a timestamp synchronization mechanism. For example, power data and transmission rate data under the same timestamp are matched one by one. At the same time, the device association identifier corresponding to all data entries is completely retained through a device identifier binding mechanism. Finally, a processed associated dataset is formed that removes clutter interference and retains valid operating data and association relationships.

[0075] This application's embodiments achieve targeted detection of environmental electromagnetic interference by dynamically extracting representative signal segments and accurately identifying clutter signals; by dynamically adjusting the filter gain parameters based on the sampling point ratio and environmental interference intensity, it solves the problem that traditional static filtering strategies are difficult to adapt to complex electromagnetic environment changes, ensuring that the filter parameters match the interference level; and by using a threshold verification mechanism, it ensures the rationality of the filter gain parameters, avoiding over-filtering leading to loss of effective signals or under-filtering leading to residual interference.

[0076] Step 104: Based on the processed associated dataset, generate a device operation link, and use a graph neural network to identify abnormal devices in the device operation link whose energy consumption fluctuation data and wireless communication quality data deviate from the corresponding normal range, so as to generate a device anomaly determination result.

[0077] In this step, the device operation link refers to the network structure formed by connecting devices with synchronously related trends through logical links based on the changing trends of the operation data of each device in the processed associated dataset. The normal range refers to the historical normal operation data of each device in the process-layer device cluster, which includes a first normal range for energy consumption fluctuation data and a second normal range for wireless communication quality data. An abnormal device refers to a device in the device operation link whose energy consumption fluctuation data exceeds the first normal range or whose wireless communication quality data exceeds the second normal range. The device anomaly determination result refers to the result set that integrates all abnormal device information, including the abnormal device itself, the device identifier corresponding to the abnormal device, and the abnormal device location information queried through the device identifier.

[0078] In this embodiment of the application, step 104 specifically includes the following steps:

[0079] Step 401: Identify the changing trends of energy consumption fluctuation data and wireless communication quality data of each device under the same device association identifier from the processed associated dataset. Based on the physical connection frequency, connect devices with synchronously related changing trends to form a device operation link.

[0080] In this step, the trend refers to the regularity of the rise and fall, fluctuation frequency and amplitude of energy consumption fluctuation data or wireless communication quality data of each device in the process layer device cluster as the acquisition time progresses. The physical connection frequency refers to the total number of actual physical connections that occur between any two devices in the process layer device cluster within a preset time period.

[0081] In this embodiment of the application, firstly, all data entries carrying the same device association identifier are extracted from the processed associated dataset, and the energy consumption fluctuation data and wireless communication quality data corresponding to each device are separated; the energy consumption fluctuation data and wireless communication quality data of the same device are arranged in chronological order of collection time, and the rise and fall, fluctuation frequency and amplitude of the two types of data over time are observed to identify the change trend of energy consumption fluctuation data and wireless communication quality data of each device.

[0082] Simultaneously, based on the physical connection frequency between each device, it is determined whether there is an operational association between devices whose physical connection frequency is higher than a preset threshold. Devices with the same device association identifier, whose energy consumption fluctuation data change trend and wireless communication quality data change trend both show synchronous characteristics, and whose physical connection frequency meets the standard are connected in the visualization interface through logical link symbols, ultimately forming a device operation link with the device as the core, reflecting the operational linkage relationship between devices.

[0083] Step 402: Based on the historical normal operation data of each device in the process layer device cluster, set the first normal range of energy consumption fluctuation data and the second normal range of wireless communication quality data for each device.

[0084] In this step, historical normal operation data refers to the collection of energy consumption fluctuation data and wireless communication quality data collected from each device in the process layer equipment cluster over a relatively long period of time, when the devices were in a healthy operating state without faults or abnormalities. The first normal range refers to a reasonable numerical range set for the energy consumption fluctuation data of each device in the process layer equipment cluster. The second normal range refers to a reasonable numerical range set for the wireless communication quality data of each device in the process layer equipment cluster.

[0085] In this embodiment, energy consumption fluctuation data and wireless communication quality data of each device in the process layer device cluster during the past year under normal operating conditions are retrieved from the historical database pre-stored in the system. These data constitute the historical normal operating data of each device. Statistical analysis is performed on the energy consumption fluctuation data in the historical normal operating data of each device to calculate the maximum value, minimum value, and 95% confidence interval of this type of data. This confidence interval is set as the first normal range of the energy consumption fluctuation data of the device. At the same time, the same statistical analysis is performed on the wireless communication quality data in the historical normal operating data of each device to calculate its maximum value, minimum value, and 95% confidence interval. This confidence interval is set as the second normal range of the wireless communication quality data of the device, ensuring that both the first and second normal ranges can cover the vast majority of data fluctuations during normal device operation.

[0086] Step 403: Using the devices in the device operation link as nodes, and the energy consumption fluctuation data and wireless communication quality data corresponding to each node as node attributes, the node attributes of each node are compared with the first normal range and the second normal range respectively through a graph neural network to mark the nodes whose node attributes exceed the corresponding normal range as abnormal devices.

[0087] In this step, a node refers to the basic building block in a graph neural network, and in this application, it corresponds one-to-one with a device in the device operation link; that is, each device is mapped to a node in the graph neural network. Node attributes refer to the feature information carried by each node in the graph neural network that reflects the real-time operating status of the device. In this application, these specifically refer to the energy consumption fluctuation data and wireless communication quality data of the device corresponding to the node.

[0088] In this embodiment, each device in the device operation link is mapped to a node in a graph neural network. Simultaneously, the real-time energy consumption fluctuation data and real-time wireless communication quality data corresponding to each node are used as node attributes, constructing a graph structure data including nodes and node attributes. This graph structure data is input into the graph neural network, and the node attribute of each node is enhanced using the node feature extraction function of the graph neural network. Then, the enhanced energy consumption fluctuation data features are numerically compared with the first normal range of the device, and the enhanced wireless communication quality data features are numerically compared with the second normal range of the device. If the energy consumption fluctuation data features of a node exceed the first normal range, or the wireless communication quality data features exceed the second normal range, or both types of data features exceed their respective normal ranges, then the device corresponding to that node is an abnormal device.

[0089] Step 404: Based on the device identifier corresponding to the abnormal device, query the pre-stored correspondence between the device identifier and the device location information to determine the abnormal device location information corresponding to the abnormal device.

[0090] In this step, the correspondence between equipment identifiers and equipment location information refers to a structured data table that records a one-to-one correspondence between the unique equipment identifier of each device in the process-level equipment cluster and the actual installation location of that device. Abnormal equipment location information refers to the actual installation location information of devices marked as abnormal in the industrial field.

[0091] In this embodiment, a unique device identifier for each abnormal device is extracted from the marked abnormal device information; a correspondence table between device identifiers and device location information pre-stored in the database is retrieved, which records a one-to-one correspondence between each device identifier and the actual installation location of the device; and the device identifier matching query function is used to accurately match the device identifier of the abnormal device with the device identifier in the correspondence table to find the device location information associated with the abnormal device identifier, which is the abnormal device location information corresponding to the abnormal device.

[0092] Step 405: Collect all abnormal devices and their corresponding device identifiers and abnormal device location information to form an equipment abnormality determination result.

[0093] In the embodiments of the present application, the device identifiers of abnormal devices are classified and sorted to ensure that each abnormal device corresponds to a unique device identifier; the device identifier of each abnormal device, the corresponding abnormal device location information, and the abnormal device itself are bound to form a structured data entry including abnormal device - device identifier - abnormal device location information; all structured data entries are arranged in the order of the time when the abnormality is discovered, and integrated to form a complete device abnormality determination result that can be directly used for subsequent alarm processing. This result clearly presents the core information of all abnormal devices, facilitating the subsequent rapid positioning of the fault source.

[0094] In the embodiments of the present application, by constructing a device operation link, the scattered device operation data is transformed into an overall link reflecting the linkage relationship, realizing the modeling of the collaborative operation state between devices; based on historical normal operation data, targeted first and second normal ranges are set to avoid misjudgment caused by general thresholds; the graph neural network is used to compare the node attributes of devices in the link with the corresponding normal ranges, which can not only identify single - device abnormalities but also capture multi - device linkage abnormalities through link association, and can avoid missing early signs; the abnormal device location information is quickly matched through the device identifier, providing an accurate positioning basis for subsequent targeted alarms; the overall steps form a complete abnormal identification link, improving the accuracy and comprehensiveness of abnormal identification, and providing support for quickly responding to faults and ensuring the stable operation of the system.

[0095] Step 105: According to the device abnormality determination result, determine the alarm level to generate an alarm audio signal, an optical signal, and an ultrasonic carrier corresponding to the alarm level.

[0096] In this step, the alarm level refers to a hierarchical identifier based on the severity of abnormalities in the process - layer device cluster, used to distinguish the urgency and influence range of alarms. The alarm audio signal refers to an audio signal generated according to the alarm level determined by the device abnormality determination result and the audio parameters set in combination with the acoustic field environment characteristics of the process - layer device cluster. The optical signal refers to a visual signal generated according to the alarm level determined by the device abnormality determination result and the optical signal parameters set in combination with the human eye's perception characteristics of optical signals. The ultrasonic carrier refers to a high - frequency ultrasonic signal generated based on the physical layout of the process - layer device cluster and the central frequency within a preset frequency band assigned to the physical area where the abnormal device is located.

[0097] In the embodiments of the present application, step 105 specifically includes the following steps:

[0098] Step 501: Calculate the parameter deviation values of each abnormal device in the device abnormality determination result, and determine the alarm level in combination with the pre - stored alarm level rules.

[0099] In this step, the parameter deviation value refers to the absolute value of the difference between the real-time operating parameters of the abnormal device and the corresponding normal range boundary value, reflecting the severity of the deviation of the device parameters from the normal state.

[0100] In this embodiment, the device identifiers of all abnormal devices are first extracted from the device anomaly determination results. At the same time, the pre-stored process layer device cluster device file is retrieved, which records the correspondence between the device identifier and functional attributes of each device. The device identifiers of the abnormal devices are matched with the device identifiers in the process layer device cluster device file to obtain the functional attributes of each abnormal device. Then, the abnormal devices are classified according to their functional attributes: devices whose functional attributes are responsible for issuing system commands and controlling core parameters are classified as core control devices, and devices whose functional attributes are only responsible for providing auxiliary data acquisition and non-critical status monitoring are classified as auxiliary devices.

[0101] Next, real-time energy consumption fluctuation data and real-time wireless communication quality data of each abnormal device are extracted from the processed associated dataset. The real-time energy consumption fluctuation data is compared with the first normal range of the device. If the real-time data is higher than the maximum value of the first normal range, the maximum value is subtracted from the real-time data; if it is lower than the minimum value, the real-time data is subtracted from the minimum value to obtain the deviation value of the energy consumption parameter. Similarly, the real-time wireless communication quality data is compared with the second normal range to obtain the deviation value of the communication parameter. The maximum value of the two deviation values ​​is taken as the parameter deviation value of the abnormal device.

[0102] The system retrieves pre-stored alarm level rules, which are set based on the safety operation manual of the process layer device cluster. The device type and parameter deviation values ​​of each abnormal device are then matched against the alarm level rules. Specifically: if the abnormal device is a core control device and the parameter deviation value is greater than a preset high threshold, the alarm level is determined to be an emergency alarm; if the abnormal device is a core control device and the parameter deviation value is between a preset medium threshold and a preset high threshold, or if it is an auxiliary device and the parameter deviation value is greater than a preset high threshold, the alarm level is determined to be a critical alarm; if the abnormal device is an auxiliary device and the parameter deviation value is between a preset low threshold and a preset medium threshold, the alarm level is determined to be a general alarm. This process ultimately yields the alarm level corresponding to each abnormal device.

[0103] Step 502: Based on the human eye's perception characteristics of light signals, a first correspondence between the alarm level and the light signal parameters is established to generate a light signal corresponding to the alarm level.

[0104] In this step, the optical signal parameters refer to the quantitative indicators describing the characteristics of the optical signal, reflecting its visual presentation effect. Based on the human eye's perception of light, these parameters include the flashing frequency, luminous intensity, and luminous color. The first correspondence refers to the fixed matching rule between the alarm level and the optical signal parameters, reflecting the visual cue features corresponding to different degrees of anomaly severity, and is set based on the human eye's sensitivity to optical signals. The optical signal refers to the visual signal used to convey anomaly alerts, reflecting the urgency level corresponding to the alarm level. It is emitted through light sources such as warning lights and can be intuitively perceived by maintenance personnel to determine the severity of the anomaly. The flashing frequency of the optical signal refers to the number of times the optical signal alternates between bright and dark within a unit of time, set based on the human eye's perception threshold for dynamic light. The luminous intensity refers to the luminous flux emitted per unit solid angle of the optical signal, set based on ambient light intensity and the human eye's perception sensitivity. The luminous color refers to the color presented by the optical signal, set based on the human eye's psychological association with color.

[0105] In this embodiment, the sensitivity of the human eye to different light signal parameters is obtained, and a first correspondence between alarm levels and light signal parameters is set based on this perception characteristic. For example, one correspondence could be: an emergency alarm corresponds to a flashing frequency of 5 times / second, a luminous intensity of 800 candela, and a red light color; an important alarm corresponds to a flashing frequency of 3 times / second, a luminous intensity of 500 candela, and a yellow light color; and a general alarm corresponds to a flashing frequency of 1 time / second, a luminous intensity of 300 candela, and a blue light color. After determining the alarm level of each abnormal device, the light signal generation module is called according to the first correspondence to output an electrical signal according to the corresponding light signal parameters, driving the light source device to emit a light signal that meets the parameter requirements and corresponds to the alarm level.

[0106] Step 503: Based on the acoustic environment characteristics of the process layer device cluster, set a second correspondence between the alarm level and the audio parameters, and generate an alarm audio signal corresponding to the alarm level in combination with the alarm level.

[0107] In this step, the acoustic environment characteristics of the process layer equipment cluster refer to the acoustic environment features of the area where the process layer equipment cluster is located, obtained based on on-site measurement data from acoustic monitoring instruments. These characteristics include the background noise frequency range during equipment operation and the regional acoustic transmission loss in different areas. Audio parameters refer to quantitative indicators describing the characteristics of alarm audio signals, including playback duration, signal frequency, and pitch parameters. The second correspondence refers to the fixed matching rules between alarm levels and audio parameters. Alarm audio signals are audible signals used to convey abnormality alerts, played through audio devices such as speakers, and can be perceived by maintenance personnel in noisy industrial environments to determine the severity of the abnormality.

[0108] In this embodiment, the acoustic field environmental characteristics of the process layer equipment cluster are collected by an acoustic monitoring instrument: the background noise frequency range during equipment operation is monitored, and the regional acoustic transmission loss of different physical areas is recorded; based on the acoustic field environmental characteristics, a second correspondence between alarm levels and audio parameters is set. For example, one set second correspondence may be: to avoid the alarm audio signal being masked by background noise, the signal frequency of the audio parameters must avoid the background noise frequency range, and areas with high regional acoustic transmission loss correspond to higher signal strength. At the same time, emergency alarms correspond to a playback duration of 10 seconds, a signal frequency of 1000 Hz, and a high pitch parameter; important alarms correspond to a playback duration of 8 seconds, a signal frequency of 900 Hz, and a medium pitch parameter; general alarms correspond to a playback duration of 5 seconds, a signal frequency of 800 Hz, and a low pitch parameter; after determining the alarm level, the audio signal generation module is called according to the second correspondence to generate an electrical signal according to the corresponding audio parameters, which is then processed by the audio amplification circuit to form an alarm audio signal corresponding to the alarm level.

[0109] Step 504: Based on the physical layout of the process layer equipment cluster, divide the process layer equipment cluster into multiple physical regions and assign different preset frequency bands to each physical region.

[0110] In this step, the physical layout of the process-level equipment cluster refers to the spatial installation distribution of all equipment in the process-level equipment cluster. A physical region refers to a non-overlapping spatial unit defined based on the physical layout of the process-level equipment cluster, obtained through logical division based on equipment installation density and functional zoning. A preset frequency band refers to a dedicated ultrasonic frequency range allocated to each physical region.

[0111] In this embodiment, a physical layout diagram of the process layer equipment cluster is obtained, which indicates the installation location of all equipment, the boundary of the installation area, and the spatial dimensions. According to the spatial division logic of the physical layout diagram, the process layer equipment cluster is divided into multiple non-overlapping physical regions. Then, different preset frequency bands are assigned to each physical region. The preset frequency bands are selected as ultrasonic bands, and the preset frequency bands of each region do not overlap, so as to avoid mutual interference of ultrasonic signals from different regions.

[0112] Step 505: Based on the abnormal device location information in the device abnormality determination result, determine the target frequency band of the physical area to which each abnormal device belongs, select the center frequency within the target frequency band, and generate an ultrasonic carrier with the center frequency as its frequency.

[0113] In this step, the target frequency band refers to the preset frequency band corresponding to the physical area where the abnormal device belongs. It reflects the frequency range for generating ultrasonic carriers and is used to lock the frequency interval of the ultrasonic carriers to achieve regional orientation. The center frequency refers to the middle frequency value within the target frequency band, reflecting the core frequency of the ultrasonic carrier, and is used to generate an ultrasonic carrier with a region-specific identifier.

[0114] In this embodiment, the abnormal device location information of each abnormal device is extracted from the device abnormality determination result; the abnormal device location information is matched with each physical area to determine the physical area to which the abnormal device belongs, and the preset frequency band corresponding to the area is the target frequency band; the center frequency within the frequency band is obtained by calculating the average of the upper limit and lower limit of the target frequency band; the ultrasonic signal generation module is called to output an electrical signal according to the center frequency to drive the ultrasonic transmitter to generate an ultrasonic carrier with the center frequency as the frequency.

[0115] The embodiments of this application solve the problems of weak anti-interference ability and poor directionality of traditional alarms, improve the effectiveness and accuracy of alarm signals, and provide support for operation and maintenance personnel to quickly identify the severity of anomalies and locate abnormal areas.

[0116] Step 106: Modulate the characteristic parameters of the alarm audio signal onto the ultrasonic carrier to form a directional ultrasonic carrier signal, and combine it with the optical signal to complete the directional audio-visual prompt in the specific area where the abnormal device is located.

[0117] In this step, the characteristic parameters refer to the key parameters extracted from the alarm audio signal and loaded onto the ultrasonic carrier. These parameters include the signal frequency, playback duration, and electrical signal amplitude corresponding to the pitch of the alarm audio signal. The directional ultrasonic carrier signal refers to the ultrasonic signal that, after the characteristic parameters of the alarm audio signal are modulated onto the ultrasonic carrier, is controlled by adjusting the transmission angle and power of the ultrasonic transmitter to create a focused sound field in the specific area where the abnormal device is located, thus possessing spatial directional propagation properties. The specific area refers to the physical area including the abnormal device, determined based on the abnormal device location information in the device anomaly judgment result; it is the target coverage area for the directional audio-visual cues. The directional audio-visual cues combine auditory cues with visual cues, ensuring that the propagation range of both signals covers the specific area where the abnormal device is located, and that the propagation time is synchronized. Through the synergistic effect of hearing and vision, this method accurately conveys the location and anomaly level information of the abnormal device to maintenance personnel.

[0118] In the embodiments of this application, such as Figure 2 As shown, step 106 specifically includes the following steps:

[0119] Step 601: Extract the feature parameters for modulation onto the ultrasonic carrier from the alarm audio signal. The feature parameters include the signal frequency, playback duration, and electrical signal amplitude corresponding to the pitch.

[0120] In this step, the characteristic parameters refer to the set of key parameters extracted from the alarm audio signal that can be loaded onto the ultrasonic carrier to transmit audio information. These parameters include the signal frequency, playback duration, and the electrical signal amplitude corresponding to the pitch.

[0121] In this embodiment, the voltage waveform of the alarm audio signal is continuously sampled at preset time intervals. For the electrical signal amplitude corresponding to the signal frequency, the signal frequency is determined by counting the number of voltage waveform cycles sampled per second, and then the maximum peak value of all voltage waveforms at that frequency is extracted as the electrical signal amplitude corresponding to the signal frequency. For the electrical signal amplitude corresponding to the playback duration, the average voltage amplitude matching the duration is calculated by counting the total number of samplings of the alarm audio signal from the start of output to the stop of output, combined with the voltage maintenance state of each sampling, and is used as the electrical signal amplitude corresponding to the playback duration. For the electrical signal amplitude corresponding to the pitch, the voltage intensity difference when the pitch changes is extracted by comparing the peak differences of the voltage waveforms in different time periods, and is used as the electrical signal amplitude corresponding to the pitch. Finally, these three types of electrical signal amplitudes are integrated to form characteristic parameters for modulation onto an ultrasonic carrier.

[0122] Step 602: Modulate the characteristic parameters onto the ultrasonic carrier wave to form a modulated ultrasonic signal.

[0123] In this step, audio features refer to the perceptible sound attributes inherent in the alarm audio signal itself, including pitch, duration, and frequency characteristics. Modulated ultrasonic signal refers to the ultrasonic signal containing audio features formed after the characteristic parameters are loaded onto the ultrasonic carrier wave.

[0124] In this embodiment, an amplitude superposition modulation method is used to fuse the characteristic parameters with the ultrasonic carrier. Specifically: the amplitude of the electrical signal corresponding to the signal frequency in the characteristic parameters is input to the frequency modulation unit of the ultrasonic carrier, so that the instantaneous frequency of the ultrasonic carrier fluctuates synchronously with the amplitude value, ensuring that the carrier carries frequency characteristics; the amplitude of the electrical signal corresponding to the playback duration is input to the time control unit of the ultrasonic carrier, so that the ultrasonic carrier continuously outputs according to the duration corresponding to the amplitude value, ensuring that the carrier carries duration characteristics; the amplitude of the electrical signal corresponding to the pitch is input to the power amplification unit of the ultrasonic carrier, so that the output intensity of the ultrasonic carrier changes with the amplitude value, ensuring that the carrier carries pitch characteristics; through the above modulation, the ultrasonic carrier completely carries the core characteristic information of the alarm audio, forming a modulated ultrasonic signal.

[0125] Step 603: Based on the abnormal devices and their corresponding location information in the device anomaly determination results, the modulated ultrasonic signal is directionally transmitted through an ultrasonic transmitting device to form a focused sound field in the specific physical area where each abnormal device is located. Combined with the modulated ultrasonic signal, a directional ultrasonic carrier signal is formed.

[0126] In this step, the specific physical region refers to the area with clearly defined spatial boundaries where the abnormal device is located. The focused sound field refers to the sound field formed within the specific physical region when the modulated ultrasonic signal is emitted at a specific angle and power, where the signal intensity is concentrated and the coverage area matches the boundary of the region.

[0127] In this embodiment of the application, step 603 specifically includes the following steps:

[0128] Step 611: Based on the abnormal device location information in the device abnormality determination result, determine the relative orientation of the abnormal device and the ultrasonic transmitting device, and convert the relative orientation into a directional transmission angle.

[0129] In this step, relative orientation refers to the spatial position of the malfunctioning device relative to the ultrasonic transmitter, obtained by comparing the coordinates of the malfunctioning device's location with the coordinates of the ultrasonic transmitter's installation location. Directional emission angle refers to the horizontal and vertical angles set by the ultrasonic transmitter to align with the specific physical area where the malfunctioning device is located, calculated based on the coordinates of the relative orientation.

[0130] In this embodiment, the location information of the abnormal device is extracted from the device anomaly determination result, and the installation coordinates of the ultrasonic transmitter pre-stored in the system are retrieved. The horizontal and vertical differences are calculated by the coordinate difference, and then the relative position of the abnormal device with respect to the ultrasonic transmitter is determined by trigonometric function conversion. The relative position is converted into the horizontal turning angle and vertical pitch angle of the ultrasonic transmitter as the directional transmission angle to ensure that the transmission port is accurately aligned with the specific physical area where the abnormal device is located.

[0131] Step 612: Determine the directional transmission power required to maintain the signal strength based on the spatial distance between the specific physical area where the abnormal device is located and the ultrasonic transmitting device.

[0132] In this step, directional transmission power refers to the output power of the ultrasonic transmitter to ensure that the modulated ultrasonic signal can maintain sufficient intensity to form a focused sound field when it propagates to a specific physical area, reflecting the strength guarantee requirements of the ultrasonic signal.

[0133] In this embodiment, a laser rangefinder is used to measure the spatial distance between the specific physical area where the abnormal device is located and the ultrasonic transmitting device; combined with the propagation loss law of ultrasonic signals in the air, the loss value after the signal propagates over this distance is calculated; then, based on the effective signal strength required for the specific physical area, the power value that the ultrasonic transmitting device needs to output is deduced as the directional transmission power required to maintain the signal strength, ensuring that the signal still has sufficient strength when it reaches the specific physical area.

[0134] Step 613: Based on the directional emission angle and the directional emission power, the modulated ultrasonic signal is emitted using an ultrasonic emission device to form a focused sound field with a coverage area matching the specific physical area;

[0135] In this step, the coverage area refers to the size of the spatial region that the focused acoustic field can cover, reflecting the boundary of the ultrasonic signal's effect.

[0136] In this embodiment, the directional emission angle and directional emission power are input into the control module of the ultrasonic transmitting device. The control module first drives the horizontal steering motor and the vertical pitch motor to adjust the emission port to the set directional emission angle. Then, the power adjustment circuit adjusts the emission power to the set directional emission power. After the parameters stabilize, the ultrasonic transmitting device is controlled to continuously emit the modulated ultrasonic signal at the set angle and power. During the propagation process, the modulated ultrasonic signal converges towards a specific physical area along the directional emission angle, maintains a stable signal strength within the area, and rapidly attenuates outside the area, forming a focused sound field whose coverage area perfectly matches the specific physical area.

[0137] Step 614: Based on the spatial constraint effect and directional propagation properties of the modulated ultrasonic signal by the focused sound field, a directional ultrasonic carrier signal is formed.

[0138] In this step, spatial constraint refers to the limiting effect of the focused sound field on the propagation range of the modulated ultrasonic signal, ensuring that the ultrasonic signal maintains effective intensity only within a specific physical region, based on the signal convergence characteristics of the focused sound field. Directional propagation property refers to the characteristic of the modulated ultrasonic signal propagating along a specific direction under the influence of the focused sound field.

[0139] In this embodiment, the focused sound field creates a spatial constraint on the modulated ultrasonic signal, limiting the signal to maintain effective intensity only within a specific physical region and preventing the signal from spreading to non-target areas. Simultaneously, under the action of the focused sound field, the modulated ultrasonic signal propagates directionally in the direction pointing to the specific physical region, possessing a clear directional propagation attribute. Combining the spatial constraint effect of the focused sound field and the directional propagation attribute of the signal, the modulated ultrasonic signal becomes a signal that acts only on a specific physical region and has precise spatial directivity, thereby forming a directional ultrasonic carrier signal.

[0140] Step 604: Based on the optical signal parameters and the optical signal, generate an optimized optical signal whose transmission time is synchronized with the directional ultrasonic carrier signal and whose light coverage range matches the range of the focused sound field.

[0141] In this step, the light coverage area refers to the spatial region that the light signal can illuminate. The optimized light signal refers to the light signal after adjusting the transmission time and coverage area, reflecting the synergistic characteristics of the light signal and the directional ultrasonic carrier signal.

[0142] In this embodiment, optical signal parameters and an initial optical signal are first retrieved. The optical signal parameters include flash frequency, luminous intensity, and luminous color. Then, the propagation time of the directional ultrasonic carrier signal is calculated. The propagation time is obtained by calculating the quotient of the spatial distance between the specific physical area and the ultrasonic transmitting device and the speed of ultrasonic propagation in air. At the same time, the range of the focused sound field is measured to determine its length, width, and height. Based on these data, the optical signal is adjusted: the emission trigger time of the optical signal is delayed by a duration equal to the ultrasonic propagation time to ensure that the optical signal and the directional ultrasonic carrier signal arrive at the specific physical area simultaneously. The steering mechanism of the optical transmitting device is driven to adjust the illumination angle to match the range of the focused sound field. At the same time, the luminous intensity is adjusted so that the light coverage range completely overlaps with the range of the focused sound field, and finally, the optimized optical signal is generated.

[0143] Step 605: When the directional ultrasonic carrier signal is received within the focused sound field, an auditory cues corresponding to the alarm audio signal and a visual cues corresponding to the optimized optical signal and the specific physical area where each abnormal device is located are generated, so as to complete the directional audio-visual cues for the specific area where the abnormal device is located.

[0144] In this step, auditory cues refer to the sound cues corresponding to the alarm audio signal, which are demodulated and restored after the ultrasonic carrier signal is received. Visual cues refer to the light effects formed by the optimized light signal within a specific physical area, which are perceptible to the human eye and are used to convey visual anomalies to maintenance personnel.

[0145] In this embodiment, when the receiving device carried by maintenance personnel enters the focused sound field, the built-in ultrasonic demodulation module first captures the directional ultrasonic carrier signal, restores the frequency fluctuations and intensity changes of the signal to the corresponding voltage signal, and then converts the voltage signal into a sound signal through the audio decoding module, restoring the sound consistent with the alarm audio signal to generate an auditory prompt; at the same time, the optimized light signal works in a specific physical area according to the set flashing frequency, light intensity and light color to form a visual mark and generate a visual prompt corresponding to that area; through the synchronous action of auditory and visual prompts, maintenance personnel can accurately locate the specific area where the abnormal device is located by sound and light, completing the directional audio-visual prompt for the specific area where the abnormal device is located.

[0146] This application's embodiments achieve effective fusion of audio information and ultrasonic signals, solving the problems of traditional audio signals being easily interfered with and having difficulty controlling their propagation range in industrial environments. The formation of directional ultrasonic carrier signals enables precise directional propagation of ultrasonic signals, avoiding false alerts in non-target areas caused by signal diffusion. By adjusting the emission time and coverage of optical signals to generate optimized optical signals, the synchronization of audio and visual alerts and the matching of coverage ranges are ensured, enhancing the intuitiveness and synergy of the alerts. Through synchronized auditory and visual alerts, maintenance personnel can quickly locate the specific area where abnormal equipment is located and perceive the alarm level, improving the accuracy of abnormal alerts and the efficiency of maintenance response.

[0147] For example, one specific implementation of this application may involve using energy consumption sensors and long-range radio communication monitoring terminals to collect energy consumption fluctuation data and wireless communication quality data from the process layer equipment cluster. This data is used to construct raw operation datasets and associated datasets to support the identification of abnormal equipment and the generation of alarm audio signals, optical signals, and ultrasonic carriers. The data acquisition hardware uses RS-485 energy consumption sensors and long-range radio communication monitoring terminals; the signal processing hardware is equipped with AD8232 adaptive filtering modules and UT-100 ultrasonic transmitters; the interaction hardware includes LTE-507 RGB warning lights and VS1053 voice broadcast terminals; and the AI ​​interaction layer deploys EC-1000 edge computing terminals. Through the implantation of these hardware components, an artificial intelligence-based dialogue mechanism is constructed, classifying sound, light, and ultrasonic carrier signals into different levels: prompt, dialogue, warning, and elimination. This fully adapts to the operation and maintenance needs of the process layer equipment. The hardware implantation map is deployed according to a hierarchical logic of acquisition terminals, processing modules, and interaction devices, closely conforming to the physical layout of the equipment cluster, ultimately achieving safe and efficient operation and maintenance management.

[0148] Figure 3 This is a schematic diagram of a specific implementation of the anomaly detection and audio-visual interaction system for multimodal perception of process layer devices provided in this application embodiment, with reference to... Figure 3 The system may include:

[0149] The acquisition module 21 is used to collect energy consumption fluctuation data and wireless communication quality data generated by each device in the process layer device cluster during operation, and combine them with the device identifier of each device to form the original operation dataset of the process layer device cluster.

[0150] The annotation module 22 is used to count the frequency of physical connections and the amount of data interaction between each device, so as to annotate the original running dataset with device association identifiers and form an association dataset including device association identifiers;

[0151] The filtering module 23 is used to adjust and verify the initial filtering gain parameters of the adaptive filtering circuit according to the associated dataset, obtain the target filtering gain parameters, and perform signal filtering operation on the associated dataset to generate the processed associated dataset.

[0152] The identification module 24 is used to generate a device operation link based on the processed associated dataset, and to identify abnormal devices in the device operation link whose energy consumption fluctuation data and wireless communication quality data deviate from the corresponding normal range through a graph neural network, so as to generate a device abnormality judgment result.

[0153] The determination module 25 is used to determine the alarm level based on the device anomaly determination result, so as to generate an alarm audio signal, an optical signal, and an ultrasonic carrier corresponding to the alarm level;

[0154] The prompting module 26 is used to modulate the characteristic parameters of the alarm audio signal onto the ultrasonic carrier to form a directional ultrasonic carrier signal, which, combined with the optical signal, completes a directional audio-visual prompt for the specific area where the abnormal device is located.

[0155] The process layer device multimodal sensing anomaly detection and audio-visual interaction system of this application embodiment is used to implement the aforementioned process layer device multimodal sensing anomaly detection and audio-visual interaction method. Therefore, the specific implementation of the process layer device multimodal sensing anomaly detection and audio-visual interaction system can be found in the embodiment section of the process layer device multimodal sensing anomaly detection and audio-visual interaction method above. The specific implementation can be referred to the description of the corresponding embodiment, and will not be repeated here.

[0156] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the above-described process layer device multimodal perception anomaly detection and audio-visual interaction method.

[0157] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described process layer device multimodal perception anomaly detection and audio-visual interaction method.

[0158] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0159] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the embodiments of the above-described multimodal perception anomaly detection and audio-visual interaction method for any process layer device.

[0160] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0161] The above provides a detailed description of the anomaly detection and audio-visual interaction method and system for multimodal perception of process-level devices provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for anomaly detection and audio-visual interaction in multimodal sensing of process-level equipment, characterized in that, include: Collect energy consumption fluctuation data and wireless communication quality data generated by each device in the process layer device cluster during operation, and combine them with the device identifier of each device to form the original operation dataset of the process layer device cluster; The frequency of physical connections and the amount of data interaction between each device are counted to label the original running dataset with device association identifiers, forming an associated dataset including device association identifiers; Based on the associated dataset, the initial filter gain parameters of the adaptive filter circuit are adjusted and verified to obtain the target filter gain parameters, so as to perform signal filtering operation on the associated dataset and generate the processed associated dataset. Based on the processed associated dataset, a device operation link is generated. A graph neural network is used to identify abnormal devices in the device operation link whose energy consumption fluctuation data and wireless communication quality data deviate from the corresponding normal range, so as to generate a device abnormality judgment result. Based on the device anomaly determination result, an alarm level is determined to generate an alarm audio signal, an optical signal, and an ultrasonic carrier wave corresponding to the alarm level. The characteristic parameters of the alarm audio signal are modulated onto the ultrasonic carrier to form a directional ultrasonic carrier signal, which, combined with the optical signal, completes a directional audio-visual alert for the specific area where the abnormal device is located. The feature parameters used to modulate the ultrasonic carrier are extracted from the alarm audio signal. The feature parameters include the signal frequency, playback duration, and electrical signal amplitude corresponding to the pitch. The characteristic parameters are modulated onto the ultrasonic carrier wave to form a modulated ultrasonic signal; Based on the abnormal devices and their corresponding location information in the device anomaly determination results, the modulated ultrasonic signal is directionally transmitted through an ultrasonic transmitting device to form a focused sound field in the specific physical area where each abnormal device is located. Combined with the modulated ultrasonic signal, a directional ultrasonic carrier signal is formed. Based on the optical signal parameters and the optical signal, an optimized optical signal is generated with the transmission time synchronized with the directional ultrasonic carrier signal and the optical coverage range matching the range of the focused sound field. When the directional ultrasonic carrier signal is received within the focused sound field, an auditory cues corresponding to the alarm audio signal and a visual cues corresponding to the optimized optical signal and the specific physical area where each abnormal device is located are generated, so as to complete the directional audio-visual cues for the specific area where the abnormal device is located.

2. The method according to claim 1, characterized in that, Based on the abnormal devices and their corresponding location information in the device anomaly determination results, the modulated ultrasonic signal is directionally transmitted through an ultrasonic transmitting device to form a focused sound field in a specific physical area where each abnormal device is located. Combined with the modulated ultrasonic signal, a directional ultrasonic carrier signal is formed, including: Based on the abnormal equipment location information in the equipment abnormality determination result, the relative orientation of the abnormal equipment and the ultrasonic transmitting device is determined, and the relative orientation is converted into a directional transmission angle; The directional transmission power required to maintain signal strength is determined based on the spatial distance between the specific physical area where the abnormal device is located and the ultrasonic transmitting device. Based on the directional emission angle and the directional emission power, the modulated ultrasonic signal is emitted using an ultrasonic emission device to form a focused sound field with a coverage area matching the specific physical area; Based on the spatial constraint effect and directional propagation properties of the focused sound field on the modulated ultrasonic signal, a directional ultrasonic carrier signal is formed.

3. The method according to claim 1, characterized in that, Based on the associated dataset, the initial filter gain parameters of the adaptive filter circuit are adjusted and verified to obtain the target filter gain parameters, which are then used to perform signal filtering on the associated dataset to generate a processed associated dataset, including: Select signal data within a preset time period from the associated dataset; The signal detection unit of the adaptive filtering circuit identifies target signal components in the signal data whose signal amplitude exceeds a preset amplitude range. The target signal components are characteristic components corresponding to clutter signals generated by environmental electromagnetic interference. The proportion of the target signal components to the sampling points of the signal data is calculated, and the initial filter gain parameters of the adaptive filter circuit are adjusted and verified in combination with the parameter configuration data corresponding to the process layer equipment cluster to obtain the target filter gain parameters. Based on the target filter gain parameter, the signal processing unit of the adaptive filter circuit performs signal filtering operation on the associated dataset to form a processed associated dataset. In the filtering operation, the correspondence between the power consumption fluctuation data and wireless communication quality data of each device and the device association identifier are retained.

4. The method according to claim 3, characterized in that, Calculate the ratio of the target signal component to the sampling points of the signal data, and, in conjunction with the parameter configuration data corresponding to the process layer device cluster, adjust and verify the initial filter gain parameters of the adaptive filter circuit to obtain the target filter gain parameters, including: The total number of signal sampling points of the signal data and the number of signal sampling points corresponding to the target signal component are counted to calculate the sampling point ratio; Based on the electromagnetic interference environment of the process layer equipment cluster, the corresponding parameter configuration data is retrieved. The parameter configuration data includes a preset first proportion value, a second proportion value, and the gain adjustment rules of the adaptive filter circuit. Based on the gain adjustment rule and the comparison results of the sampling point ratio with the first ratio value and the second ratio value, the initial filter gain parameter of the adaptive filter circuit is adjusted to obtain the adjusted gain parameter. The adjusted gain parameters are verified to obtain the target filter gain parameters.

5. The method according to claim 1, characterized in that, Based on the processed associated dataset, a device operation link is generated. A graph neural network is used to identify abnormal devices in the device operation link whose energy consumption fluctuation data and wireless communication quality data deviate from the corresponding normal range, thereby generating a device anomaly determination result, including: Identify the changing trends of energy consumption fluctuation data and wireless communication quality data of each device under the same device association identifier from the processed associated dataset, and connect devices with synchronously associated changing trends according to the physical connection frequency to form a device operation link; Based on the historical normal operation data of each device in the process layer device cluster, a first normal range for energy consumption fluctuation data and a second normal range for wireless communication quality data of each device are set. Using the devices in the device operation link as nodes, and the energy consumption fluctuation data and wireless communication quality data corresponding to each node as node attributes, the node attributes of each node are compared with the first normal range and the second normal range respectively through a graph neural network, so as to mark the nodes whose node attributes exceed the corresponding normal range as abnormal devices. Based on the device identifier corresponding to the abnormal device, query the pre-stored correspondence between the device identifier and the device location information to determine the abnormal device location information corresponding to the abnormal device. All abnormal devices, along with their corresponding device identifiers and location information, are used to form an equipment anomaly determination result.

6. The method according to claim 1, characterized in that, Based on the device anomaly determination result, an alarm level is determined to generate an alarm audio signal, an optical signal, and an ultrasonic carrier wave corresponding to the alarm level, including: Calculate the parameter deviation values ​​of each abnormal device in the device anomaly determination results, and determine the alarm level by combining them with the pre-stored alarm level rules; Based on the characteristics of human eye perception of light signals, a first correspondence between the alarm level and light signal parameters is established to generate a light signal corresponding to the alarm level. Based on the acoustic environment characteristics of the process layer device cluster, a second correspondence between the alarm level and the audio parameters is set, and an alarm audio signal corresponding to the alarm level is generated in combination with the alarm level. Based on the physical layout of the process layer equipment cluster, the process layer equipment cluster is divided into multiple physical regions, and different preset frequency bands are assigned to each physical region. Based on the abnormal device location information in the device anomaly determination result, the target frequency band of the physical area to which each abnormal device belongs is determined, the center frequency within the target frequency band is selected, and an ultrasonic carrier with the center frequency as its frequency is generated.

7. A process-level device multimodal sensing anomaly detection and audio-visual interaction system, used to execute the process-level device multimodal sensing anomaly detection and audio-visual interaction method according to any one of claims 1 to 6, characterized in that, include: The data acquisition module is used to collect energy consumption fluctuation data and wireless communication quality data generated by each device in the process layer device cluster during operation, and combine them with the device identifier of each device to form the original operation dataset of the process layer device cluster. The annotation module is used to count the frequency of physical connections and the amount of data interaction between devices, so as to annotate the original running dataset with device association identifiers and form an association dataset including device association identifiers; The filtering module is used to adjust and verify the initial filter gain parameters of the adaptive filtering circuit based on the associated dataset to obtain the target filter gain parameters, so as to perform signal filtering operation on the associated dataset and generate the processed associated dataset. The identification module is used to generate a device operation link based on the processed associated dataset, and to identify abnormal devices in the device operation link whose energy consumption fluctuation data and wireless communication quality data deviate from the corresponding normal range through a graph neural network, so as to generate a device abnormality judgment result. The determination module is used to determine the alarm level based on the device anomaly determination result, so as to generate an alarm audio signal, an optical signal, and an ultrasonic carrier wave corresponding to the alarm level; The prompting module is used to modulate the characteristic parameters of the alarm audio signal onto the ultrasonic carrier to form a directional ultrasonic carrier signal, which, combined with the optical signal, completes directional audio-visual prompting for the specific area where the abnormal device is located.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the anomaly detection and audio-visual interaction method for process layer device multimodal perception as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the anomaly detection and audio-visual interaction method for multimodal perception of process layer devices as described in any one of claims 1 to 6.

Images