Method, apparatus, electronic device, storage medium, and program for detecting equipment faults based on audio

Abstract

An audio-based equipment fault detection method, apparatus, electronic device, storage medium, and program are provided. The method includes: obtaining initial audio data collected by a drone for a target equipment; pre-processing the initial audio data to obtain audio data to be detected; performing feature extraction on the audio data to be detected to obtain audio features of the audio data to be detected; constructing an information graph based on the audio features; and obtaining a fault detection result for the target equipment based on the information graph and a graph neural network model. According to the method provided in the embodiment of the present disclosure, the audio features of the audio data can be used to reflect the real-time situation of the equipment, thus realizing the automation of fault detection of the equipment and saving human resources.

Description

[Technical field]

[0001] The present disclosure relates to the data processing field, and in particular to technical fields such as deep learning and voice technology. [Background technology]

[0002] In the industrial scenario of the spinning process, because the flow of the spinning process is long and complicated, the number of related process equipment is huge, and the equipment may experience abnormalities during the operation of the process equipment. Summary of the Invention [Problem to be solved by the invention]

[0003] Since manual investigation methods are labor-intensive, the question of how to automate equipment fault detection has become an issue. [Means for solving the problem]

[0004] The present disclosure provides an audio-based equipment fault detection method, apparatus, electronic device, storage medium, and program to solve or alleviate one or more technical problems in the prior art.

[0005] In a first aspect, the present disclosure provides an audio-based equipment fault detection method, the method comprising: acquiring initial audio data collected by the drone relative to the target facility; pre-processing the initial audio data to obtain audio data awaiting detection; performing feature extraction on the audio data awaiting detection to obtain audio features of the audio data awaiting detection; constructing an information graph based on the audio features; and obtaining a fault detection result for the target equipment based on the information graph and the graph neural network model.

[0006] In a second aspect, the present disclosure provides an audio-based equipment fault detection apparatus, the apparatus comprising: a first collection module for acquiring initial audio data collected by the drone with respect to the target facility; a pre-processing module for pre-processing the initial audio data to obtain audio data awaiting detection; a feature extraction module for performing feature extraction on the audio data awaiting detection to obtain audio features of the audio data awaiting detection; a construction module for constructing an information graph based on audio features; and a detection module for obtaining a fault detection result for the target equipment based on the information graph and the graph neural network model.

[0007] In a third aspect, the present disclosure provides an electronic device, the device comprising: At least one processor; a memory in communication with the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the implementation of any one of the methods in the embodiments of the present disclosure.

[0008] In a fourth aspect, there is provided a non-transitory computer-readable storage medium having stored thereon computer instructions for causing a computer to perform any one of the methods in the embodiments of the present disclosure.

[0009] In a fifth aspect, a program is provided, which, when executed by a processor, implements any one of the methods in the embodiments of the present disclosure.

[0010] According to the method provided in the embodiments of the present disclosure, the audio features of the audio data can be used to reflect the real-time situation of the equipment, thus realizing the automation of equipment fault detection and saving human resources.

[0011] It should be understood that the contents described herein are not intended to describe key or important features of the embodiments of the present disclosure, nor are they used to limit the scope of the present disclosure. Other features of the present disclosure are facilitated by the following specification. [Brief description of the drawings]

[0012] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the several accompanying drawings indicate the same or similar components or elements. The accompanying drawings are not necessarily drawn to scale. It should be understood that the drawings depict only some examples provided by the present disclosure and should not be considered as limiting the scope of the present disclosure.

[0013] [Figure 1] 1 is a flowchart of an audio-based equipment fault detection method according to one embodiment of the present disclosure. [Diagram 2] FIG. 2 is a schematic diagram illustrating acquisition of first audio data according to an embodiment of the present disclosure. [Diagram 3] FIG. 2 is a schematic diagram illustrating an information graph according to an embodiment of the present disclosure. [Figure 4] FIG. 13 is a schematic diagram illustrating another information graph according to an embodiment of the present disclosure. [Diagram 5] FIG. 1 is a schematic diagram showing an overall audio-based equipment fault detection method according to one embodiment of the present disclosure. [Figure 6] FIG. 1 is a schematic diagram showing the configuration of an audio-based equipment fault detection device according to an embodiment of the present disclosure. [Figure 7] FIG. 2 is a block diagram of an electronic device for implementing an audio-based equipment fault detection method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The present disclosure will now be described in more detail with reference to the accompanying drawings, in which like reference numbers represent the same or similar elements, and in which various aspects of the embodiments are illustrated, and which, unless otherwise noted, are not necessarily drawn to scale.

[0015] Furthermore, in order to better explain the present disclosure, many specific details are described in the following specific embodiments. Those skilled in the art should understand that the present disclosure can be similarly implemented without some details. In some embodiments, methods, means, components, circuits, etc. that are well known to those skilled in the art are not described in detail so that the gist of the present disclosure is clear.

[0016] In the description herein, the orientations or positional relationships indicated by terms such as "center," "top," "bottom," "left," "right," "vertical," "horizontal," "inside," "outside," and the like, are based on the orientations or positional relationships shown in the drawings and are intended merely to facilitate and simplify the description of the present disclosure, and do not indicate or suggest that the devices or elements referenced have a particular orientation or must be configured and operated in a particular orientation, and therefore should not be construed as a limitation of the present disclosure.

[0017] The terms "first", "second", and the like in the present specification, claims, and the above-mentioned accompanying drawings are intended to distinguish between similar subjects without necessarily being used to describe a particular order or priority. It should be understood that the data used in this manner may be interchanged as necessary such that the present application described herein may be performed in an order other than that shown or described herein. The embodiments described in the following illustrative examples do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention as detailed in the appended claims.

[0018] In the industrial scenario of the spinning process, because the spinning process flow is long and complicated, the number of related process equipment is huge, and the equipment may have abnormalities during the operation of the process equipment. In order to ensure the normal operation of the spinning process flow, the embodiment of the present disclosure uses a drone to monitor the audio data of the process equipment in real time to ensure the normal operation of the spinning process flow. In the embodiment of the present disclosure, an audio-based equipment fault detection method is proposed, which can include the following, as shown in FIG.

[0019] At S101, initial audio data collected by the drone for a target facility is obtained.

[0020] Since there is flowing liquid in the piping in the process equipment, the microphone device of the drone can be used to collect data from detection points on the target equipment. There may be multiple detection points on one target equipment, and multiple sampling can be performed on each detection point to obtain initial audio data.

[0021] At S102, initial audio data is pre-processed to obtain audio data awaiting detection.

[0022] In S103, feature extraction is performed on the audio data waiting to be detected to obtain audio features of the audio data waiting to be detected.

[0023] At S104, an information graph is constructed based on the audio features.

[0024] In S105, a fault detection result for the target equipment is obtained based on the information graph and the graph neural network model.

[0025] The fault detection result of the target equipment includes a normal state and an abnormal state, where the abnormal state can include a first-class abnormality, a second-class abnormality, ..., an S-class abnormality. The specific abnormal state level included can be determined based on the actual situation, and the embodiment of the present disclosure is not limited thereto.

[0026] According to different levels of abnormality, different processing methods can be set for each level of abnormality. For example, Class 1 abnormality is when the audio data exceeds the normal range by a small amount, i.e., it is a minor abnormality, and the equipment can be idled and then inspected. Class S abnormality is when the audio data exceeds the normal range by the largest amount, i.e., it is the most serious abnormal situation, and the operator must be notified to immediately suspend the operation of the equipment, and the equipment must be immediately processed.

[0027] In an embodiment of the present disclosure, a drone is used to monitor audio data of an equipment and obtain initial audio data. The initial audio data is preprocessed to obtain audio data to be detected, and feature extraction is performed on the audio data to be detected to obtain audio features of the audio data to be detected. An information graph and a neural network model constructed based on the audio features provide a fault detection result for the target equipment. The audio features of the audio data can be used to reflect the real-time situation of the equipment, so that the fault detection of the equipment can be automated and manpower can be saved.

[0028] In some embodiments, the drone is controlled to execute a work plan to collect initial audio data for the reactor according to the method for acquiring the initial audio data, the work plan including a patrol inspection time, a position information of the reactor, and a request according to the collected initial audio data.

[0029] For example, the drone performs the first sampling of reactor 001 (representing the first esterification reactor in a certain factory, Factory A) at 15:45:06 on November 19th, i.e., acquires the first audio sequence, performs the second sampling at 15:45:08 on November 19th, i.e., acquires the second audio sequence, and performs the mth sampling at 16:35:05 on November 19th, i.e., acquires the mth audio sequence. In other words, the acquired initial audio data can be expressed as follows:

[0030]

number

[0031] In an embodiment of the present disclosure, in the process of using a drone to collect audio data, the patrol inspection time and the position information of the reaction vessel are simultaneously collected, and based on this, it is easier to identify which equipment is faulty.

[0032] In some embodiments, pre-processing the initial audio data to obtain audio data awaiting detection includes:

[0033] In step A1, initial audio data is processed based on noise suppression and silence detection methods to obtain first audio data.

[0034] In one embodiment, a noise suppression algorithm may be applied to the initial audio data. For example, methods such as average filtering, median filtering, etc. may be used to reduce noise in the initial audio data. Further, the initial audio data is analyzed based on a silence detection method to identify silence segments. Common silence detection methods include energy-based, spectral entropy-based, etc. The recognized silence segments are filtered. The processed audio data are fused to obtain the first audio data.

[0035] In another embodiment, the initial audio data may be subjected to noise reduction processing using a Hidden Markov Model (HMM) to obtain the first audio data. The HMM is a method used in speech signal processing, which can model the speech signal to map the speech signal from a high-dimensional space to a low-dimensional space. Furthermore, it can achieve dimensionality reduction of the speech signal while maintaining the speech features of the speech signal.

[0036] In implementation, the initial audio data may be obtained by sampling multiple times for a detection point, or may be obtained by sampling multiple detection points. The initial audio data consists of a set of audio sequences (each audio sequence can be understood as a continuous audio segment). As shown in Figure 2,

number

[0037] A plurality of sets of audio sequences are input to the HMM model, which processes them. The HMM model performs noise reduction and dimensionality reduction on the frequency features of each data time in the audio data to obtain a normal distribution of each data time, selects a valid audio range from the normal distribution of each data time, constructs an audio sequence, and obtains the first audio data.

[0038] In addition to the aforementioned HMM model, the initial audio data can be subjected to noise reduction processing using Principal Component Analysis (PCA), Independent Component Correlation Algorithm (ICA), and Linear Discriminant Analysis (LDA) to obtain the first audio data.

[0039] Here, PCA is a commonly used method for reducing the dimension of data, which can find the main components in the data, map the data from a high-dimensional space to a low-dimensional space, and maintain the variance information of the main parts in the data. For audio data, PCA can be used to reduce the dimension of the time domain and frequency domain data of the audio signal.

[0040] Here, ICA is a method that can be applied to signal processing and image processing, which can find independent components in a signal and map the signal from a high-dimensional space to a low-dimensional space. In the case of audio data, ICA can be used to reduce the dimensionality of the audio signal while maintaining the main features of the audio signal.

[0041] Here, LDA is a method that can project high-dimensional data into a low-dimensional space, and by finding the optimal projection direction, data can be mapped from the original space to a low-dimensional space while maintaining the inter-class variance in the data. In the case of audio data, LDA can be used to reduce the dimensionality of the speech signal while maintaining the categorical information of the speech signal.

[0042] The method of denoising the initial audio data in the embodiments of the present disclosure can be flexibly selected according to actual usage needs.

[0043] In step A2, the first audio data is divided into frames to obtain second audio data.

[0044] In implementation, the first audio data may be divided into short time frames to facilitate subsequent processing. Typically, the frame division is performed using windows of 10-30 milliseconds in length, and there may be overlap between adjacent frames. Thus, the resulting second audio data includes audio data for multiple frames.

[0045] In step A3, the second audio data is processed using a window function to obtain third audio data.

[0046] The audio data for each frame is processed using a window function to obtain third audio data, such as a Hamming window, a Henning window, etc., to reduce amplitude jitter at both ends of the frame.

[0047] In step A4, the third audio data is transformed into the frequency domain to obtain audio data awaiting detection.

[0048] The third audio data may be transformed into the frequency domain using a Fast Fourier Transform to obtain audio data awaiting detection.

[0049] In converting the third audio data into the frequency domain, processing such as filtering, spectrum subtraction, equalization and the like can be performed on the frequency domain signal based on the actual situation, so as to enhance correlation features and suppress noise, and further obtain audio data waiting to be detected.

[0050] In the embodiment of the present disclosure, pre-processing is performed on the data collected by the drone to achieve the purpose of removing noise and silent audio from the initial audio data, enhancing the useful data portion in the audio data, and providing useful audio data for subsequent equipment fault detection.

[0051] In some embodiments, the audio features include at least one of a frequency feature, a duration feature, an intensity feature, and the like.

[0052] In implementation, since the audio data to be detected is spectrum data, features can be directly extracted from the audio data to be detected to obtain the frequency features of the audio data to be detected. The frequency features may be information such as spectrum peaks, spectrum centroids, frequency entropy, Mel-frequency cepstral coefficients (MFCC), Mel-frequency cepstrum (MFC), etc.

[0053] The time length feature can be determined based on the time length of the preset frequency bands of the frequency feature. For example, the same audio segment includes preset frequency bands a to b, and the duration of each of the frequency bands a to b in the audio segment is determined, and the feature of the time length is extracted as the time length feature.

[0054] The intensity feature is determined based on the intensity of the preset frequency band in the frequency feature. Following the previous example, the frequency band is still preset as a to b, and the amplitude of the frequency band a to b is determined based on the spectrum diagram to obtain the intensity feature.

[0055] The audio features may be any of the features, or may be a combination of multiple sets of features, and the embodiments of the present disclosure are not limited thereto.

[0056] In the embodiment of the present disclosure, audio data in the equipment is collected and its features are extracted. The frequency feature, duration feature and intensity feature can all represent the status of the audio data, so that the goal of accurately identifying the fault status of the equipment can be achieved by collecting these three features.

[0057] In some embodiments, since the initial audio data includes multiple audio segments, audio features of the audio data can be obtained, and then an information graph can be constructed based on the audio features, which can be performed as follows:

[0058] In step B1, a similarity between audio segments is determined based on audio sub-features of each audio segment in the audio feature.

[0059] In step B2, an information graph is constructed by treating audio segments whose similarity satisfies a preset threshold as correlated nodes.

[0060] For example, the initial audio data includes audio segment 1, audio segment 2, ..., audio segment k, and an audio sub-feature in each audio segment is extracted. If the audio sub-feature is any of a frequency feature, a time length feature, and an intensity feature, the similarity may be directly calculated based on the audio sub-feature in each audio segment. The method of calculating the similarity may be calculating the cosine similarity. After obtaining the similarity between the audio segments, the audio segments that satisfy a preset threshold are associated to obtain an information graph. As shown in FIG. 3, the information graph is such that, when the preset threshold is 60%, if the similarity between the audio segment 1 and the audio segment 3 is 72%, the audio segment 1 is associated with the audio segment 3, but if the similarity between the audio segment 1 and the audio segment 2 is 32%, the audio segment 1 is not associated with the audio segment 2, and if the similarity between the audio segment 3 and the audio segment k is 87%, the audio segment 3 is associated with the audio segment k.

[0061] When the audio sub-features are at least two of frequency features, duration features, and intensity features, weighted summation can be performed to obtain an integrated audio feature for each audio segment, and the similarity can be further calculated.Also, the audio sub-features can be regarded as vectors containing multiple elements, and the cosine similarity can be used to determine the similarity between different audios.

[0062] In the embodiment of the present disclosure, a method for calculating similarity is used to construct an information graph to clarify the associated relationship of each audio segment in the initial audio data, which is helpful for subsequent fault detection.

[0063] In some embodiments, a fault detection result for a target equipment can be obtained based on the information graph and the graph neural network model, and can be implemented as follows.

[0064] In step C1, an adjacency matrix is ​​constructed based on the information graph.

[0065] Since the information graph is an adjacency matrix constructed based on the similarity between audio segments, an adjacency matrix (O) can be constructed based on the similarity between audio segments in the information graph, and the adjacency matrix can be shown in Equation (1).

[0066]

number

[0067] Here, aa represents the similarity between audio segment 1 and audio segment 1, and ab represents the similarity between audio segment 1 and audio segment 2, and thus, an analogy is made.

[0068] Since there are a large number of node pairs with zero similarity in the information graph, these zero elements can be removed and this adjacency matrix can be converted into a sparse adjacency matrix to facilitate subsequent processing by the graph neural network model.

[0069] In step C2, the adjacency matrix is ​​input into the graph neural network model to obtain the fault detection result for the target equipment.

[0070] Here, the graph neural network may be a graph convolutional network (GCN), a graph attention network (GAT), etc. A graph neural network is usually composed of multiple graph convolutional layers, and each convolutional layer can perform feature extraction and dimensionality reduction on graph data.

[0071] In some embodiments, the graph neural network is a GCN, and the method for obtaining the graph neural network by pre-training can be performed as follows:

[0072] In step D1, the GCN model is trained using the training set to optimize the parameters and structure of the GCN model.

[0073] Typically, a training set consists of audio segments with known labels, where the known labels include audio segments for normal conditions and audio segments for abnormal conditions.

[0074] In the implementation, when there are many equipments in the spinning process scenario and the process is complicated, training samples can be flexibly collected in an environment where multiple similar equipments are gathered. For example, when multiple reactors are gathered, first start one reactor equipment, collect a sample when one reactor equipment is started, then start the second reactor equipment, collect a training sample when two reactor equipments are started simultaneously, then start the third reactor equipment, collect a training sample when three reactor equipments are started simultaneously. In this way, when there are n equipments, equipments 1 to n are started in sequence, and training samples are collected each time one equipment is added.

[0075] Of course, the amount of samples collected can be increased for different weather conditions. For example, to deal with harsh environments such as rain and wind, audio data under bad weather such as rain and wind is collected as sample data in the training set.

[0076] During training, the parameters of the model can be updated using a backpropagation algorithm to minimize the prediction error.

[0077] Here, updating the model parameters using the backpropagation algorithm may be implemented as follows: initialize the parameters of the GCN model, input the training set into the GCN model to calculate the output of each layer of the GCN model, calculate the loss function (such as the mean squared error) and its gradient (partial derivative), calculate the gradient of the weights and biases of each layer from the output layer based on the chain rule, and update the parameters of the GCN model based on the method of multiplying the learning rate by the gradient using the gradient descent method.

[0078] In step D2, if the GCN model satisfies the convergence condition, i.e., a graph neural network is obtained.

[0079] In the embodiments of the present disclosure, by using a graph neural network model, feature information in the information graph can be better extracted, thereby improving the performance of the fault detection task of the target equipment.

[0080] In some embodiments, the polyester reactor is an important link in the flow of the spinning process. According to the four-kettle process, the polyester reactor can be divided into a preliminary polycondensation reactor, a first esterification reactor, a second esterification reactor, and a final polycondensation reactor, and according to the five-kettle process, it can be divided into a first preliminary polycondensation reactor, a second preliminary polycondensation reactor, a first esterification reactor, a second esterification reactor, and a final polycondensation reactor. The reactor is usually large, and the surrounding noise is large when it is operated. In order to detect the fault condition of the reactor, when the fault detection result shows that the reactor is faulty, that is, when the fault detection result is in an abnormal state, the following can be carried out.

[0081] In step E1, based on the collection time of the initial audio data, the operation parameters of the reaction vessel within a preset time range based on the collection time are obtained, the operation parameters including at least one of voltage, current, rotation speed and viscosity.

[0082] For example, if the collection time is T, the operating parameters of the reaction vessel s / 2 period before and s / 2 period after time T, i.e., s operating parameters, are collected, and the collected operating parameters of the reaction vessel may be {voltage A, voltage B, ..., voltage t, ..., voltage s}, {current A, current B, ..., current t, ..., current s}.

[0083] In step E2, a fault condition of the reaction vessel is determined based on the operating parameters.

[0084] The data range during normal operation of the reactor is obtained, and for example, if the collected voltage operating parameters of the reactor fluctuate on the boundary of the normal voltage data range, it can be determined that there is an abnormality in the voltage-related equipment in the reactor, and if the collected current operating parameters of the reactor fluctuate on the boundary of the normal current data range, it can be determined that there is an abnormality in the current-related equipment in the reactor.

[0085] In step E3, if it is determined based on the operating parameters that an abnormality has occurred in the reaction vessel, an emergency alarm is issued.

[0086] In an embodiment of the present disclosure, when the operating parameters of the reaction vessel are on the boundary of the normal operating state, the operating parameters of the reaction vessel are collected based on the abnormality of the audio data, and the operating parameters of the reaction vessel are compared with the parameters during normal operation to further determine the fault status of the reaction vessel.

[0087] In another embodiment, when the target equipment is a reactor required for a spinning process, collecting the equipment parameters of the reactor and the audio features of the audio data to construct an information graph can be implemented as follows:

[0088] In step F1, based on the collection time of the initial audio data, the operation parameters of the reaction vessel within a preset time range based on the collection time are obtained, and the operation parameters include at least one of voltage, current, rotation speed, and viscosity.

[0089] In step F2, an information graph is constructed based on the audio features and the driving parameters.

[0090] The audio features and driving parameters are the attribute information of the audio segments, the audio segments are the nodes, and edges are constructed based on the similarity between the audio features and driving parameters of each audio segment. The obtained information is as shown in FIG. 4, in which when the preset threshold is 60%, if the similarity between audio segment 1 and audio segment 3 is 42%, audio segment 1 is not associated with audio segment 3, but if the similarity between audio segment 1 and audio segment k is 67%, audio segment 1 is associated with audio segment k, if the similarity between audio segment 2 and audio segment k is 72%, audio segment 2 is associated with audio segment k, and if the similarity between audio segment 3 and audio segment k is 87%, audio segment 3 is associated with audio segment k.

[0091] In some embodiments, in order to easily determine a specific abnormality location of the reaction vessel equipment, a failure analysis graph for a pre-established failure analysis can be obtained and implemented as follows:

[0092] In step G1, historical fault records related to the reactor are input into the large-scale model to extract entities at the detection point locations of historical anomalies.

[0093] The historical fault records may include the cause of the fault, the type of fault, the audio characteristics when the fault occurred, and how the fault was resolved.

[0094] In step G2, based on the entity of the detection point location of the history anomaly, obtain element triples corresponding to the detection point locations of multiple history anomalies, the element triples include a fault cause and an audio feature, and the fault type is the type of the edge between them, and the element triples include a fault cause and a fault solution, and the type of the edge between them is fault repair.

[0095] In step G3, a failure analysis graph is constructed based on the multiple element triples.

[0096] Since there is an association relationship between the detection point positions of each history anomaly, specifically, the server expresses the entity relationship between the entity of the detection point position of the history anomaly extracted in step G2 and the entity of the detection point position of each history anomaly in a format conforming to the RDF (Resource Description Framework) expression standard, that is, in the element triple (subject-predicate-object) format. Here, the element triple is composed of three parts: entity-relation-entity.

[0097] In the embodiments of the present disclosure, based on the degree of association between entities at the detection points of different historical anomalies in the failure analysis graph, a large-scale language model is invoked to infer the entities at the detection points of different historical anomalies, and the inference properties of the large-scale language model are used to mine potential entity relationships to obtain more element triples, thereby improving the failure analysis graph and improving the accuracy of failure analysis.

[0098] In the embodiment of the present disclosure, the large-scale language model uses natural language processing technology to perform entity recognition, entity relationship extraction, and knowledge representation of the detection point location of historical anomalies, providing more accurate and precise semantic understanding, and thus improving the effect of generating fault analysis graphs. In addition, the large-scale model also plays the role of knowledge inference, which can infer existing knowledge, discover potential relationships and rules therein, and further expand the fault analysis graph.

[0099] In addition, if the fault analysis graph is not updated, it is not necessary to extract features from the fault analysis graph every time a voice segment is determined, and the graph features are stored for use in subsequent analyses. In addition, in order to apply a more complete fault analysis graph, recent fault information is stored in a key-value table, and the fault analysis graph is periodically updated while the fault information stored in the key-value table is deleted, thereby saving computation resources. Of course, the fault analysis graph may be updated when the fault information stored in the key-value table reaches a preset number.

[0100] As shown in FIG. 5, the failure analysis graph is input into the first graph neural network, and feature extraction is performed to obtain graph features. When an abnormality is detected, the failure analysis graph is used as a reference to quickly determine how to resolve the abnormality.

[0101] A second graph neural network is used to perform feature extraction on the information graph to obtain audio features.

[0102] Based on the collection time of the operating parameters, an operating parameter timing graph is constructed, and feature extraction is performed on the operating parameter graph using the third graph neural network to obtain the parameter features.

[0103] The graph features, parameter features and audio features are input to a knowledge feature extraction network to obtain knowledge features, and the parameter features and audio features are input to a timing feature extraction network to obtain timing features.

[0104] Here, the graph feature, the parameter feature and the audio feature are inputted into the knowledge feature extraction network to obtain the knowledge feature, which can be specifically implemented as shown in Fig. 5. That is, the graph feature is inputted into at least one first convolution layer and a first pooling layer to obtain the first feature, the audio feature is inputted into at least one second convolution layer and a second pooling layer to obtain the second feature, the parameter feature is inputted into at least one third convolution layer and a third pooling layer to obtain the third feature, and the second feature and the third feature are processed using the first cross-attention mechanism network to obtain the first intermediate feature, and the first intermediate feature and the first feature are processed using the third cross-attention mechanism network to obtain the knowledge feature.

[0105] Here, inputting the parameter feature and the audio feature into the timing feature extraction network to obtain the timing feature can be specifically implemented as shown in Fig. 5. That is, the audio feature is input into at least one fourth convolution layer and a fourth pooling layer to obtain the fourth feature, the parameter feature is input into at least one fifth convolution layer and a fifth pooling layer to obtain the fifth feature, the fourth feature and the fifth feature are processed using a second cross-attention mechanism network to obtain a second intermediate feature, and the second intermediate feature and the first intermediate feature are processed based on the fourth cross-attention mechanism network to obtain a timing feature.

[0106] The knowledge feature and the timing feature are fused to obtain a fused feature, and the fused feature is converted into a sparse adjacency matrix form and input into the graph neural network model to obtain the fault detection result of the target equipment. At the same time, if the fault detection result has appeared in the fault analysis graph, the solution method corresponding to the anomaly can be obtained.

[0107] In another embodiment, the information graph constructed based on the similarity of the audio features is obtained, and then converted into a first adjacency matrix; and based on the collection time of the initial audio data, the operation parameters of the reaction vessel within a preset time range based on the collection time are obtained, the operation parameters including at least one of voltage, current, rotation speed, and viscosity. A second adjacency matrix is ​​constructed based on the operation parameters of the reaction vessel. The second adjacency matrix is ​​constructed based on the similarity between the operation parameters.

[0108] The first and second adjacency matrices are fused to obtain an information graph. The fusion method may be multiplication, splicing, or addition. Since there may be a large number of node pairs with zero similarity in the information graph, the target adjacency matrix can be converted into a sparse adjacency matrix by removing these zero elements to facilitate subsequent processing by the graph neural network model. The sparse adjacency matrix is ​​input into the neural network model to obtain fault detection results for the target equipment.

[0109] In the embodiment of the present disclosure, the operating parameters of the reaction vessel are taken into account, so that the detection result of the target equipment can be accurately determined.

[0110] Based on the same technical idea, an embodiment of the present disclosure provides an audio-based equipment fault detection apparatus 600, as shown in FIG. 6 , which includes: A first collection module 601 for acquiring initial audio data collected by the drone with respect to the target facility; a pre-processing module 602 for pre-processing the initial audio data to obtain audio data awaiting detection; a feature extraction module 603 for performing feature extraction on the audio data awaiting detection to obtain audio features of the audio data awaiting detection; a construction module 604 for constructing an information graph based on the audio features; and a detection module 605 for obtaining a fault detection result for the target equipment based on the information graph and the graph neural network model.

[0111] In some embodiments, the pre-processing module 602 includes: processing the initial audio data based on the noise suppression and silence detection method to obtain first audio data; splitting the first audio data into frames to obtain second audio data; processing the second audio data with a window function to obtain third audio data; and third transforming the audio data into the frequency domain to obtain audio data awaiting detection.

[0112] In some embodiments, the detection module 605 includes: Constructing an adjacency matrix based on the information graph; The adjacency matrix is ​​input into a graph neural network model to obtain fault detection results for the target equipment.

[0113] In some embodiments, the initial audio data includes a plurality of audio segments, and the construction module 604 determining a similarity between the audio segments based on audio sub-features of each audio segment in the audio features; An information graph is constructed by treating audio segments whose similarity satisfies a preset threshold as nodes having a correlation with each other.

[0114] In some embodiments, the audio features include at least one of a frequency feature, a duration feature, and an intensity feature.

[0115] In some embodiments, the target equipment is a reactor required for a spinning process, and the construction module 604 Acquiring operation parameters of the reaction vessel within a preset time range based on the collection time of the initial audio data, the operation parameters including at least one of voltage, current, rotation speed, and viscosity; and constructing an information graph based on the audio features and driving parameters.

[0116] In some embodiments, the target equipment is a reactor required for a spinning process, and the equipment fault detection device includes: When the fault detection result indicates that a fault occurs in the reactor, based on the collection time of the initial audio data, acquiring operation parameters of the reactor within a preset time range based on the collection time, the operation parameters including at least one of voltage, current, rotation speed, and viscosity; determining a fault condition of the reactor based on the operating parameters; The apparatus further includes an abnormality processing module for issuing emergency alarm information when it is determined that an abnormality has occurred in the reaction vessel based on the operating parameters.

[0117] In some embodiments, the equipment fault detection device comprises: and a second collection module for controlling the drone to execute the task plan and acquiring initial audio data; The work plan includes inspection round times, reactor location information, and requirements based on initial audio data collected.

[0118] The specific functions and exemplary descriptions of each module and sub-module of the apparatus according to the embodiments of the present disclosure may refer to the relevant descriptions of the corresponding steps in the above-mentioned method embodiments, and will not be repeated here.

[0119] In the technical solution disclosed herein, the acquisition, storage, and application of users' personal information complies with the provisions of relevant laws and regulations and does not violate public order and morals.

[0120] FIG. 7 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 7, the electronic device includes a memory 710 and a processor 720, and the memory 710 stores a computer program executable by the processor 720. The number of the memory 710 and the processor 720 may be one or more. The memory 710 may store one or more computer programs, which, when executed by the electronic device, cause the electronic device to perform the method provided by the above method embodiment. The electronic device may further include: The communication interface 730 is used for communicating with an external device and performing data interaction and transmission.

[0121] When the memory 710, the processor 720, and the communication interface 730 are implemented independently, the memory 710, the processor 720, and the communication interface 730 are connected to each other via a bus and can communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be classified into an address bus, a data bus, a control bus, etc. For ease of explanation, only one thick line is shown in FIG. 7, but this does not indicate only one bus or only one type of bus.

[0122] Optionally, in a specific implementation, when the memory 710, the processor 720, and the communication interface 730 are integrated on one chip, the memory 710, the processor 720, and the communication interface 730 can communicate with each other via an internal interface.

[0123] It should be understood that the processor may be a Central Processing Unit (CPU) or may be other general purpose processors, Digital Signal Processing (DSP), Application Specific Integrated Circuits (ASIC), Field Programmable Gate Arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware assemblies, etc. The general purpose processor may be a microprocessor or any conventional processor, etc. Additionally, the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0124] Additionally, optionally, the memory may include read-only memory and random access memory, and may further include non-volatile random access memory. The memory may be either volatile or non-volatile memory, or may include both volatile and non-volatile memory. Here, non-volatile memory may include Read-Only Memory (ROM), Programmable ROM (PROM), Erasable PROM (EPROM), Electrically EPROM (EEPROM), or flash memory. Volatile memory may include Random Access Memory (RAM), which acts as an external cache. By way of example and not limitation, many forms of RAM are available. For example, static random access memory (Static RAM, SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (Enhanced SDRAM, ESDRAM), synchlink DRAM (Synchlink DRAM, SLDRAM), and direct RAMBUS RAM (DR RAM).

[0125] The above-mentioned embodiments may be implemented in whole or in part in software, hardware, firmware, or any combination thereof. When implemented in software, the whole or in part may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, a process or function according to the embodiments of the present disclosure is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center via wire (e.g., coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, Bluetooth, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device including a server, a data center, etc. integrated with one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a Digital Versatile Disc (DVD)), or a semiconductor medium (e.g., a Solid State Disk (SSD)). Note that the computer-readable storage medium referred to in this disclosure may be a non-volatile storage medium, in other words, a non-transitory storage medium.

[0126] Those skilled in the art can understand that all or part of the steps for realizing the above embodiments may be implemented by hardware, or may be implemented by instructing relevant hardware by a program, and the program may be stored in a computer-readable storage medium, and the storage medium may be a read-only memory, a magnetic disk, an optical disk, etc.

[0127] In the description of the embodiments of the present disclosure, the reference words "one embodiment", "several embodiments", "examples", "specific examples", or "several examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. And, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. Furthermore, a person skilled in the art may combine different embodiments or examples described in the present disclosure and features of different embodiments or examples to the extent that they are not mutually inconsistent.

[0128] In the description of the embodiments of the present disclosure, unless otherwise stated, " / " means "or", for example, A / B can mean either A or B. "And / or" in the present disclosure is only for describing the related relationship of related objects, and indicates that three types of relationships may exist, for example, A and / or B can indicate the following three types of situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0129] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying a relative importance, nor should they be construed as implying a number of technical features shown. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present disclosure, "plurality" means two or more, unless otherwise specified.

[0130] The above are merely illustrative examples of the present disclosure, and are not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure should be included within the scope of protection of the present disclosure.

Claims

1. 1. A method for audio-based equipment fault detection, comprising: obtaining initial audio data collected by sampling a plurality of times by the drone at each of a plurality of detection points of the target facility, the initial audio data being comprised of a plurality of sets of audio sequences, each audio sequence being a contiguous audio segment; processing the initial audio data according to a noise suppression algorithm to remove noise from the initial audio data, and according to a silence detection method to identify and remove silence segments from the initial audio data; fusing the processed audio data to obtain first audio data; dividing the first audio data into frames to obtain second audio data; processing the second audio data using a window function to obtain third audio data; and transforming the third audio data into a frequency domain to obtain audio data to be detected; performing feature extraction on the audio data to be detected to obtain audio features of the audio data to be detected; constructing an information graph based on the audio features; and and obtaining a fault detection result for the target equipment based on the information graph and a graph neural network model. An audio-based method for equipment fault detection.

2. Obtaining a fault detection result for the target equipment based on the information graph and a graph neural network model, constructing an adjacency matrix based on the information graph; inputting the adjacency matrix into the graph neural network model to obtain a fault detection result for the target equipment.

2. The method of claim 1, wherein the method comprises:

3. the initial audio data includes a plurality of audio segments; Constructing an information graph based on the audio features includes: determining a similarity between the audio segments based on audio sub-features of each audio segment in the audio features; and constructing the information graph by treating the audio segments whose similarity satisfies a preset threshold as nodes having a correlation with each other.

2. The method of claim 1, wherein the method comprises:

4. The audio-based equipment fault detection method according to claim 1 , wherein the audio features include at least one of a frequency feature, a time duration feature, and an intensity feature.

5. The target equipment is a reaction vessel required for the spinning process; Constructing an information graph based on the audio features includes: Acquiring operation parameters of the reaction vessel within a preset time range based on the collection time of the initial audio data, the operation parameters including at least one of voltage, current, rotation speed, and viscosity; and constructing the information graph based on the audio features and the driving parameters.

2. The method of claim 1, wherein the method comprises:

6. The target equipment is a reaction vessel required for the spinning process; The method for detecting a fault in the equipment comprises: When the fault detection result indicates that a fault has occurred in the reactor, based on a collection time of the initial audio data, acquiring an operation parameter of the reactor within a preset time range based on the collection time, the operation parameter including at least one of a voltage, a current, a rotation speed, and a viscosity; determining a fault condition of the reactor based on the operating parameters; and and issuing emergency alert information when it is determined that an abnormality has occurred in the reaction vessel based on the operating parameters.

2. The method of claim 1, wherein the method comprises:

7. The method for detecting a fault in the equipment comprises: Further comprising: controlling the unmanned aerial vehicle to execute a work plan and collecting the initial audio data for the reaction vessel according to a method for acquiring the initial audio data; The work plan includes inspection time, location information of the reactor, and requirements based on collected initial audio data; 6. The audio-based equipment fault detection method of claim 5.

8. 1. An audio-based equipment fault detection apparatus, comprising: a first collection module for acquiring initial audio data collected by sampling a plurality of times by the drone for each of a plurality of detection points of the target facility, the initial audio data being comprised of a plurality of sets of audio sequences, each audio sequence being a consecutive audio segment; a pre-processing module for processing the initial audio data according to a noise suppression algorithm to remove noise from the initial audio data, and according to a silence detection method to identify and remove silence segments from the initial audio data, fusing the processed audio data to obtain first audio data, dividing the first audio data into frames to obtain second audio data, processing the second audio data using a window function to obtain third audio data, and transforming the third audio data into a frequency domain to obtain audio data to be detected; a feature extraction module for performing feature extraction on the audio data to be detected to obtain audio features of the audio data to be detected; a construction module for constructing an information graph based on the audio features; a detection module for obtaining a fault detection result for the target equipment based on the information graph and a graph neural network model; Audio-based equipment fault detection device.

9. At least one processor; a memory in communication with the at least one processor; The memory stores instructions executable by the at least one processor, the instructions, when executed by the at least one processor, causing the at least one processor to perform a method according to any one of claims 1 to 7. Electronic devices.

10. A non-transitory computer readable storage medium for storing instructions that cause a computer to perform the method of any one of claims 1 to 7.

11. A program for implementing the method according to any one of claims 1 to 7 when executed by a processor in a computer.

Images