An industrial equipment anomaly detection method, an electronic device, and a storage medium

Abstract

Embodiments of the present application relate to an industrial equipment anomaly detection method, an electronic device and a storage medium, which obtain time series data, preprocess the time series data to obtain standard data, process the standard data using a dynamic anomaly processing module and a static anomaly processing module to obtain a time-frequency image and a state transition image, realize double detection of dynamic anomalies and static anomalies, reduce the possibility of missing reports of static anomalies or dynamic anomalies, reconstruct the time-frequency image and the state transition image using a double-branch coding and decoding model to obtain a first reconstructed image and a second reconstructed image, calculate a reconstruction quality score matrix according to the first reconstructed image, the second reconstructed image, the time-frequency image and the state transition image, and determine whether the industrial equipment is abnormal according to the reconstruction quality score matrix. In this way, the detected dynamic anomalies and static anomalies can be accurately and efficiently processed, so that it can be accurately judged whether the industrial equipment is abnormal, and the accuracy of industrial equipment anomaly detection is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of device detection, in particular to an industrial device anomaly detection method, an electronic device and a storage medium. BACKGROUND

[0002] Anomaly detection has evolved from classical statistics and machine learning algorithms to deep learning models. These deep models work based on the assumption that normal data can be reconstructed with low error, while the reconstruction error of abnormal data is high. However, when dealing with complex industrial device vibration time series data, the device has both dynamic running state and static running state. Single data representation and single branch architecture cannot simultaneously consider dynamic anomaly (which refers to the anomaly related to the transient and non-stationary characteristics of time series signals, usually manifested as a sudden change in a short period of time, such as a sudden impact or high-frequency vibration during the operation of an industrial device or a pattern mutation) and static anomaly (which refers to the anomaly related to the macro-stable state of time series signals, the residence of working conditions or the transition rule of state, usually manifested as the stable working state of an industrial device in a certain abnormal state for a long time, or the switching mode deviates from the normal range, such as from normal operation to abnormal static state). There is a situation of "trade-off", which is not sensitive to static anomaly or dynamic anomaly, resulting in false negatives. There is a lack of clear distinction and targeted processing of dynamic anomaly and static anomaly, resulting in inaccurate device anomaly detection. SUMMARY

[0003] Therefore, an object of embodiments of the present application is to provide an industrial device anomaly detection method, an electronic device and a storage medium, aiming to solve the technical problem of inaccurate device anomaly detection due to the inability of single data representation and single branch architecture to consider dynamic anomaly and static anomaly when detecting device anomaly in related technologies.

[0004] To solve the above technical problems, embodiments of the present application provide the following technical solutions:

[0005] In a first aspect, embodiments of the present application provide an industrial device anomaly detection method applied to an electronic device, the electronic device comprising a dual-branch coding and decoding model, the dual-branch coding and decoding model comprising a dynamic anomaly processing module and a static anomaly processing module, and the method comprising:

[0006] obtaining target data, the target data comprising a plurality of time series data, the time series data being sensor data detected by a sensor equipped on an industrial device;

[0007] preprocessing the time series data to obtain standard data;

[0008] inputting the standard data into the dynamic anomaly processing module for processing to obtain a time-frequency image;

[0009] The standard data is input into the static exception processing module for processing to obtain a state transition image;

[0010] The time-frequency image and the state transition image are input into the dual-branch coding and decoding model for reconstruction to obtain a first reconstructed image and a second reconstructed image;

[0011] Based on the first reconstructed image, the second reconstructed image, the time-frequency image and the state transition image, a reconstructed quality score matrix corresponding to the time series data is obtained.

[0012] Based on the reconstructed quality score matrices corresponding to all the time series data, it is determined whether the industrial equipment is abnormal.

[0013] In a second aspect, an electronic device is provided, comprising:

[0014] a controller and a dual-branch coding and decoding model in communication connection with the controller;

[0015] The dual-branch coding and decoding model comprises a dynamic exception processing module, a static exception processing module, a first reconstruction network and a second reconstruction network, wherein the dynamic exception processing module, the static exception processing module, the first reconstruction network and the second reconstruction network are in communication connection with the controller.

[0016] The first reconstruction network comprises a first encoding module, a first decoding module and a first connection layer, the first encoding module comprises a first encoder and a first feature mapping layer, and the first decoding module comprises a first feature transformation layer and a first decoder.

[0017] The second reconstruction network comprises a second encoding module and a second decoding module, the second encoding module comprises a second encoder and a second feature mapping layer, and the second decoding module comprises a second feature transformation layer and a second decoder.

[0018] The controller comprises:

[0019] a processor and a memory in communication connection with the processor;

[0020] The memory stores computer program instructions executable by the processor, and the computer program instructions, when executed by the processor, cause the controller to execute any one of the industrial equipment abnormality detection methods proposed in the first aspect.

[0021] In a third aspect, a computer readable storage medium is provided, and the computer readable storage medium stores computer program instructions executable by a processor, and the computer program instructions, when executed by the processor, cause the computer to execute any one of the industrial equipment abnormality detection methods proposed in the first aspect.

[0022] The embodiment of the present application has the following beneficial effects: Different from the prior art, the industrial equipment anomaly detection method provided by the embodiment of the present application is applied to an electronic device, and the electronic device comprises a double-branch coding and decoding model, and the double-branch coding and decoding model comprises a dynamic anomaly processing module and a static anomaly processing module. In the embodiment of the present application, a plurality of time series data sensed by a sensor equipped in an industrial equipment is acquired, and the time series data is preprocessed to obtain standard data, the dynamic anomaly processing module and the static anomaly processing module are used to process the standard data to obtain a time-frequency image and a state transition image, the double detection of dynamic anomalies and static anomalies is realized, the possibility of missing reports of static anomalies or dynamic anomalies is reduced, then the double-branch coding and decoding model is used to reconstruct the time-frequency image and the state transition image to obtain a first reconstructed image and a second reconstructed image, a reconstruction quality score matrix corresponding to the time series data is calculated according to the first reconstructed image, the second reconstructed image, the time-frequency image and the state transition image, and whether the industrial equipment is abnormal is determined according to the reconstruction quality score matrix corresponding to all the time series data. In this way, the detected dynamic anomalies and static anomalies can be accurately and efficiently processed, so that whether the industrial equipment is abnormal can be accurately judged, and the accuracy of the industrial equipment anomaly detection is improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the prior art or the embodiments. Obviously, the drawings described below only show some of the embodiments of the present application, and should not be regarded as a limitation to the protection scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0024] Figure 1 is an application scenario diagram of the industrial equipment anomaly detection method provided by some embodiments of the present application;

[0025] Figure 2 is a structural diagram of an electronic device provided by some embodiments of the present application;

[0026] Figure 3 is Figure 2 is a structural diagram of a controller in the electronic device shown in the embodiment;

[0027] Figure 4 is a flow diagram of the industrial equipment anomaly detection method provided by some embodiments of the present application;

[0028] Figure 5 is a process diagram of processing time series data to obtain a time-frequency image and a state transition image in some embodiments of the present application;

[0029] Figure 6is a schematic diagram of a plurality of time-frequency images and a plurality of state transition images in some embodiments of the present application;

[0030] Figure 7 is a process schematic diagram of reconstructing a time-frequency image by a first reconstruction network to obtain a first reconstructed image in some embodiments of the present application;

[0031] Figure 8 is a process schematic diagram of reconstructing a state transition image by a second reconstruction network to obtain a second reconstructed image in some embodiments of the present application. DETAILED DESCRIPTION

[0032] In order to make the objects and advantages of the embodiments of the present application more easily understood, the technical solutions of the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. The following detailed description of the embodiments of the present application in the drawings is not intended to limit the scope of the present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present application.

[0033] It should be noted that, if there is no conflict, each technical feature involved in the embodiments of the present application described below can be combined with each other, and all within the scope of the present application. In addition, although the functional modules are divided in the device or structure schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the expressions "first", "second", "third" and other similar expressions used herein do not limit the data and execution order, but are only for the purpose of explanation and to distinguish the same or similar items with basically the same function and effect, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features.

[0034] Unless otherwise defined, the technical terms and scientific terms used in the present specification have the same meanings as those commonly understood by those of ordinary skill in the art to which the present application belongs. The terms used in the present specification are only for the purpose of describing the specific embodiments and are not intended to limit the present application. It should be understood that the term "and / or" used in the present specification includes any and all combinations of one or more listed items.

[0035] Please refer to Figure 1 , Figure 1 The application scenario schematic diagram of the industrial equipment anomaly detection method provided by some embodiments of the present application is schematically shown.

[0036] As Figure 1As shown, the application scenario includes an electronic device 100 and an industrial device 200, the electronic device 100 and the industrial device 200 are connected through a network, and embodiments of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. It should be understood that the industrial device 200 includes any suitable type of device or apparatus, etc., for example, the industrial device 200 includes a robot 210, a hoisting apparatus 220, and a fan 230. Of course, the industrial device 200 can also be a component or element in a device or apparatus, for example, the industrial device 200 can be a mechanical arm, a turntable, or a telescopic cylinder, an oil cylinder, etc. component or element in the hoisting apparatus 220.

[0037] For example, the electronic device 100 is configured to detect device abnormalities, and the electronic device 100 is configured with a controller 110 (not shown in the figure) and a dual-branch coding and decoding model 120, wherein the dual-branch coding and decoding model 120 includes a dynamic abnormality processing module and a static abnormality processing module (neither of which is shown in the figure). The controller 110 is configured to work in cooperation with the dual-branch coding and decoding model 120 to execute corresponding business logic and perform abnormality detection on the industrial device 200. Figure 1 Figure 1 For example, the electronic device 100 is configured to detect device abnormalities, and the electronic device 100 is configured with a controller 110 (not shown in the figure) and a dual-branch coding and decoding model 120, wherein the dual-branch coding and decoding model 120 includes a dynamic abnormality processing module and a static abnormality processing module (neither of which is shown in the figure). The controller 110 is configured to work in cooperation with the dual-branch coding and decoding model 120 to execute corresponding business logic and perform abnormality detection on the industrial device 200.

[0038] For example, the electronic device 100 is configured to detect device abnormalities, and the electronic device 100 is configured with a controller 110 (not shown in the figure) and a dual-branch coding and decoding model 120, wherein the dual-branch coding and decoding model 120 includes a dynamic abnormality processing module and a static abnormality processing module (neither of which is shown in the figure). The controller 110 is configured to work in cooperation with the dual-branch coding and decoding model 120 to execute corresponding business logic and perform abnormality detection on the industrial device 200.

[0039] For example, the electronic device 100 is configured to detect device abnormalities, and the electronic device 100 is configured with a controller 110 (not shown in the figure) and a dual-branch coding and decoding model 120, wherein the dual-branch coding and decoding model 120 includes a dynamic abnormality processing module and a static abnormality processing module (neither of which is shown in the figure). The controller 110 is configured to work in cooperation with the dual-branch coding and decoding model 120 to execute corresponding business logic and perform abnormality detection on the industrial device 200.

[0040] ​After obtaining the standard data by preprocessing the time series data, the standard data is input to the dynamic anomaly processing module for processing to obtain a time-frequency image. For example, the dynamic anomaly processing module is used to perform sampling, scaling, data conversion and other operations on the standard data to process the standard data into a time-frequency image. Similarly, the electronic device 100 inputs all the standard data to the dynamic anomaly processing module for processing to obtain the time-frequency images corresponding to all the standard data.

[0041] After obtaining the standard data by preprocessing the time series data, the standard data is input to the static anomaly processing module for processing to obtain a state transition image. For example, the static anomaly processing module is used to perform sampling, scaling, data conversion and other operations on the standard data to process the standard data into a state transition image. Similarly, the electronic device 100 inputs all the standard data to the static anomaly processing module for processing to obtain the state transition images corresponding to all the standard data.

[0042] After obtaining the time-frequency image and the state transition image by processing the standard data, the time-frequency image and the state transition image are input to the dual-branch coding and decoding model for reconstruction to obtain a first reconstructed image and a second reconstructed image. For example, the time-frequency image is input to the first branch network of the dual-branch coding and decoding model, and the first branch network is used to perform reconstruction operation on the time-frequency image to obtain the first reconstructed image. The state transition image is input to the second branch network in the dual-branch coding and decoding model, and the second branch network is used to perform reconstruction operation on the state transition image to obtain the second reconstructed image.

[0043] After obtaining the first reconstructed image and the second reconstructed image, the reconstruction quality score matrix corresponding to the time series data is obtained based on the first reconstructed image, the second reconstructed image, the time-frequency image and the state transition image. That is, the similarity between the feature maps of each channel in the first reconstructed image and the time-frequency image is calculated according to the first reconstructed image and the time-frequency image to obtain the time-frequency reconstruction score of each channel in the first reconstructed image and the time-frequency image. The similarity between the feature maps of each channel in the second reconstructed image and the state transition image is calculated according to the second reconstructed image and the state transition image to obtain the state transition reconstruction score of each channel in the second reconstructed image and the state transition image. The time-frequency reconstruction scores and the state transition reconstruction scores of all channels are constructed into the reconstruction quality score matrix corresponding to the time series data, and finally the reconstruction quality score matrix corresponding to all the time series data is obtained.

[0044] Finally, whether the industrial equipment is abnormal is determined based on the reconstruction quality score matrix corresponding to all time series data, for example, summing all elements in the reconstruction quality score matrix corresponding to all time series data to obtain a target sum value, comparing the target sum value with a preset threshold, if the target sum value is greater than or equal to the preset threshold, it is determined that the industrial equipment is normal / there is no abnormality, and if the target sum value is less than the preset threshold, it is determined that the industrial equipment is abnormal.

[0045] In the above manner, the multi-segment time series data sensed by the sensor equipped in the industrial equipment is acquired and preprocessed to obtain standard data, the dynamic and static abnormality processing modules are used to process the standard data to obtain the time-frequency image and the state transition image, the dual detection of dynamic and static abnormalities is realized, the possibility of missing reports of static or dynamic abnormalities is reduced, then the dual-branch coding and decoding model is used to reconstruct the time-frequency image and the state transition image to obtain the first reconstruction image and the second reconstruction image, the reconstruction quality score matrix corresponding to the time series data is calculated according to the first reconstruction image, the second reconstruction image, the time-frequency image and the state transition image, whether the industrial equipment is abnormal is determined according to the reconstruction quality score matrix corresponding to all time series data, in this way, the detected dynamic and static abnormalities can be accurately and efficiently processed, so that whether the industrial equipment is abnormal can be accurately judged, and the accuracy of the industrial equipment abnormality detection is improved.

[0046] It should be understood that, Figure 1 Only one case of detecting whether the industrial equipment 200 is abnormal by using the electronic device 100 in some embodiments of the present application is schematically shown, and in other application scenarios or embodiments, Figure 1 In the application scenario shown, the electronic device 100 is a desktop computer, which does not cause any limitation on the structure, type and number of the electronic device, the structure, type and number of the industrial equipment and any other conditions in other application scenarios or embodiments. For example, in other application scenarios or embodiments, the electronic device 100 can also be a single-chip microcomputer, a microcontroller, a notebook computer, a tablet computer, an FPGA chip or other suitable types of devices, apparatuses or components, etc.

[0047] In order to facilitate understanding of the industrial equipment abnormality detection method provided by the embodiments of the present application, the electronic device provided by the embodiments of the present application is first described in detail.

[0048] Please refer to Figure 2 , Figure 2 The structural schematic diagram of the electronic device provided by some embodiments of the present application is schematically shown.

[0049] Specifically, as Figure 2As shown, the electronic device 100 includes a controller 110 and a double-branch coding and decoding model 120, the controller 110 is communicatively connected with the double-branch coding and decoding model 120 through a network, the implementation of the network includes a wired communication network (such as a fiber, a USB, a CAN bus, etc.) and a wireless communication network (such as a 4G, a 5G, a Wi-Fi, a Bluetooth, etc.). The controller 110 is used to work cooperatively with the double-branch coding and decoding model 120 to execute corresponding business logic to detect whether the industrial equipment 200 exists an abnormality.

[0050] For example, the double-branch coding and decoding model 120 includes a dynamic abnormality processing module 121, a static abnormality processing module 122, a first reconstruction network 123, and a second reconstruction network 124, wherein the dynamic abnormality processing module 121, the static abnormality processing module 122, the first reconstruction network 123, and the second reconstruction network 124 are all communicatively connected with the controller 110. Among them, the dynamic abnormality processing module 121 is used to process the preprocessed time series data (i.e. standard data) to obtain a time-frequency image, the static abnormality processing module 122 is used to process the preprocessed time series data (i.e. standard data) to obtain a state transition image, the first reconstruction network 123 is used to reconstruct the time-frequency image to obtain a first reconstruction image, and the second reconstruction network 124 is used to reconstruct the state transition image to obtain a second reconstruction image.

[0051] Specifically, the first reconstruction network 123 includes a first encoding module 1231, a first decoding module 1232, and a first connection layer 1233. The first encoding module 1231 includes a first encoder 12311 and a first feature mapping layer 12312, wherein the first encoder 12311 and the first feature mapping layer 12312 are used to down-sample and feature map the time-frequency image to obtain a first latent vector. The first decoding module 1232 includes a first feature transformation layer 12321 and a first decoder 12322, and the first feature transformation layer 12321 and the first decoder 12322 are used to feature transform and up-sample the first latent vector to obtain the first reconstruction image.

[0052] It should be understood that the first connection layer 1233 is used to filter out a first target feature map with a resolution meeting the target resolution requirement in the down-sampling process of the time-frequency image by the first encoder 12311, and fuse the first target feature map with the feature map obtained by the first decoder 12322 in the up-sampling process to obtain a first fusion feature map, and finally decode the first fusion feature map by the first decoder 12322 to obtain the first reconstruction image.

[0053] It is easy to understand that the first encoder 12311 includes one or more first down-sampling layers, the first decoder 12322 includes one or more first up-sampling layers, and the number of the first down-sampling layers is the same as that of the first up-sampling layers. When the number of the first down-sampling layers and the first up-sampling layers is multiple, the first connection layer 1233 fuses the first target feature map obtained by the first down-sampling layer at a corresponding layer and the first reference feature map obtained by the first up-sampling layer to obtain a first fusion feature map, where the first target feature map and the reference feature map have the same resolution. In the embodiment of the present application, the first feature mapping layer 12312 and the first feature transformation layer 12321 can be a MLP (Multilayer Perceptron) layer or any other suitable network layer.

[0054] In the embodiment of the present application, the second reconstruction network 124 includes a second encoding module 1241 and a second decoding module 1242. The second encoding module 1241 includes a second encoder 12411 and a second feature mapping layer 12412, and the second encoder 12411 and the second feature mapping layer 12412 are used for down-sampling and feature mapping of the state transition image to obtain a second latent vector. The second decoding module 1242 includes a second feature transformation layer 12421 and a second decoder 12422, and the second feature transformation layer 12421 and the second decoder 12422 are used for feature transformation and up-sampling of the second latent vector to obtain a second reconstruction image.

[0055] It can be understood that the second encoder 12411 includes one or more second down-sampling layers, the second decoder 12422 includes one or more second up-sampling layers, and the number of the second down-sampling layers is the same as that of the second up-sampling layers. When the number of the second down-sampling layers and the second up-sampling layers is multiple, the resolution of the second target feature map obtained by the second down-sampling layer at a corresponding layer is the same as that of the second reference feature map obtained by the second up-sampling layer. In the embodiment of the present application, the second feature mapping layer 12412 and the second feature transformation layer 12421 can be a MLP (Multilayer Perceptron) layer or any other suitable network layer.

[0056] Please refer to Figure 3 , Figure 3 The structure of the controller in the electronic device provided by some embodiments of the present application is schematically shown.

[0057] As shown in Figure 3 , the controller 110 includes at least one processor 111 and a memory 112 connected in communication, Figure 3The controller 110 can include a bus system 113, a processor 111, a memory 112, an input device 114, and an output device 115, which are communicatively coupled via the bus system 113. It is to be understood that various bus systems 113, as well as the components of the controller 110, are merely provided for the purpose of illustration and are in no way intended to limit the scope of the disclosure. For example, the controller 110 can include more or fewer components than those shown, have a different configuration of components, or have additional components not shown. Figure 3 The various buses are collectively referred to as the bus system 113. It is to be understood that the bus system 113 can include not only a data bus, but also a power bus, a control bus, a status signal bus, and the like. For the sake of clarity and brevity, however, the various buses are collectively referred to as the bus system 113. Figure 3 The structure shown in the embodiments is merely illustrative and does not impose any limitation on the structure of the controller. For example, the controller can include more or fewer components than those shown, or have a different configuration of components. Figure 3 The structure shown in the embodiments is merely illustrative and does not impose any limitation on the structure of the controller. For example, the controller can include more or fewer components than those shown, or have a different configuration of components. Figure 3 The structure shown in the embodiments is merely illustrative and does not impose any limitation on the structure of the controller. For example, the controller can include more or fewer components than those shown, or have a different configuration of components.

[0058] The processor 111 is configured to provide computing and control capabilities to support the controller 110 in performing corresponding business logic and functions, such as supporting the controller 110 in performing any of the industrial equipment anomaly detection methods provided by the embodiments of the disclosure, or performing the steps in any of the possible implementation manners of the industrial equipment anomaly detection methods provided by the embodiments of the disclosure. Those skilled in the art can understand that the processor 111 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), and the like; or a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0059] The memory 112, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs, instructions and modules, such as programs, instructions and modules corresponding to the industrial equipment anomaly detection method in the embodiments of the present application. In some embodiments, the memory 112 can include a program storage area and a data storage area, the program storage area can store an operating system and application programs required by at least one function, and the data storage area can store data created according to the use of the processor 111, etc. The processor 111 executes various functions and data processing of the controller 110 by running the non-transitory software programs, instructions and modules stored in the memory 112, thereby implementing any one of the industrial equipment anomaly detection methods provided by the embodiments of the present application, or executing the steps in any one of the possible implementation manners of any one of the industrial equipment anomaly detection methods provided by the embodiments of the present application. In some embodiments, the memory 112 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device or other non-transitory solid-state memory device. In some embodiments, the memory 112 can also include a memory remotely arranged with respect to the processor 111, which can be connected to the processor 111 through a network. It can be understood that the embodiments of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0060] According to the above, it can be understood that the implementation execution subject of any one of the industrial equipment anomaly detection methods provided by the embodiments of the present application can be any suitable type of electronic device with certain computing and control capabilities, which can be implemented by the above-mentioned electronic device 100. In some possible implementation manners, the industrial equipment anomaly detection method provided by the embodiments of the present application can be implemented by executing computer program instructions stored in the memory by the processor.

[0061] The industrial equipment anomaly detection method provided by the embodiments of the present application will be described in detail below in combination with an exemplary application and implementation of the electronic device provided by the embodiments of the present application.

[0062] Please refer to Figure 4 , Figure 4 The flowchart schematically shows the industrial equipment anomaly detection method provided by some embodiments of the present application.

[0063] As can be understood by those skilled in the art, the industrial equipment anomaly detection method provided by the embodiments of the present application can be applied to the above-mentioned electronic device (such as the electronic device 100). Specifically, the execution subject of the industrial equipment anomaly detection method is the controller of the electronic device.

[0064] Specifically, as Figure 4As shown, the industrial equipment anomaly detection method includes but is not limited to the following steps S100-S700:

[0065] S100: Obtain target data.

[0066] In the embodiment of the present application, the target data includes multiple time series data, and the time series data is the sensing data detected by the sensor equipped in the industrial equipment, that is, the sensing data collected by the sensor, and the sensing data includes working data and state data, etc. The time series data can be PWM signal, vibration signal and other sensing signal data.

[0067] Specifically, the sensor forms the sensing data collected in time sequence into time series data, and transmits the time series data to the industrial equipment. The embodiment of the present application obtains the target data (i.e. multiple time series data) from the industrial equipment, or the industrial equipment sends the target data to the electronic equipment through the communication network, so as to obtain the target time.

[0068] S200: Preprocess the time series data to obtain standard data.

[0069] For example, after obtaining the target data, for each time series data, the embodiment of the present application preprocesses each time series data to obtain standard data. The preprocessing includes but is not limited to normalization, noise removal, missing value filling, time alignment and sampling, data enhancement and the like. For example, each time series data is normalized, noise is removed and other preprocessing operations are performed to obtain the preprocessed time series data (i.e. standard data). It should be understood that all time series data are preprocessed in the embodiment of the present application, and after performing the preprocessing operation on all time series data, the standard data corresponding to all time series data is obtained, and each time series data corresponds to a standard data.

[0070] S300: Input the standard data into the dynamic anomaly processing module for processing to obtain a time-frequency image.

[0071] In this step, the standard data is processed into a time-frequency image by using the dynamic anomaly processing module. The purpose is to convert the time series data into a two-dimensional representation that can depict the "time-frequency change characteristics". The dynamic anomaly processing module converts the simple standard data (i.e. preprocessed time series data) into an image feature containing time and frequency information, which facilitates the learning of complex anomaly patterns by the double-branch coding and decoding model.

[0072] For example, the standard data is input into the dynamic anomaly processing module, and the dynamic anomaly processing module processes the standard data by using a time-frequency analysis algorithm such as a short-time Fourier transform, a mel spectrogram, or a discrete cosine transform built in the dynamic anomaly processing module. For example, in some embodiments of the present application, the time series data is divided into small segments by using a short-time Fourier transform algorithm, a Fourier transform is performed on each segment to obtain a two-dimensional distribution of time-frequency, and finally a frequency spectrum amplitude matrix with time on the horizontal axis and frequency on the vertical axis is output, and the value of the frequency is the amplitude or power intensity. Then, the frequency spectrum amplitude matrix obtained by performing time-frequency transformation on the standard data by using the time-frequency analysis algorithm is mapped into a gray image or a pseudo-color image to obtain a time-frequency image. In the time-frequency image, the horizontal coordinate represents time, the vertical coordinate represents a frequency component, and the pixel value represents the amplitude or energy (i.e., power intensity).

[0073] In some embodiments, the standard data can also be processed by using any other suitable algorithm to obtain a time-frequency image according to actual needs. For example, a person skilled in the art can also use a short-time autocorrelation analysis, an empirical mode decomposition (EMD), a Hilbert-Huang transform, a continuous wavelet transform (CWT), or the like.

[0074] S400: The standard data is input into the static anomaly processing module for processing to obtain a state transition image.

[0075] In this step, the standard data is processed into a state transition image by using the static anomaly processing module. The purpose is to convert the time series data into a two-dimensional representation that can depict the working condition residence, state co-occurrence, and switching rule. The static anomaly processing module first discretizes the continuous signal (i.e., the standard data / the preprocessed time series data) into "states", and then spreads it on a "time-time" two-dimensional plane through the transition probability between the states to obtain a state transition image.

[0076] In embodiments of the present application, the standard data is input into the static anomaly processing module, and the static anomaly processing module processes the standard data by using a state transition analysis algorithm such as a hidden Markov model (HMM), a Gramian angular field (GAF), or a recurrence plot (RP) built in the static anomaly processing module. For example, in some embodiments, the standard data / the preprocessed time series data is fitted by using a hidden Markov model to obtain a hidden state number K, a transition matrix A, and an emission parameter, and a hidden state sequence is decoded by using a Viterbi / posterior decoding according to the hidden state number K, the transition matrix A, and the emission parameter. The time-time transition image is generated or constructed, so as to obtain a state transition image.

[0077] In some embodiments, the standard data can also be processed according to actual needs by using other any suitable algorithm to obtain the state transition image, for example, a neighboring point phase diagram histogram (phase diagram density map), a finite state machine mapping algorithm, and the like can also be used.

[0078] S500: inputting the time-frequency image and the state transition image into a dual-branch coding and decoding model for reconstruction to obtain a first reconstructed image and a second reconstructed image.

[0079] Specifically, after the time-frequency image and the state transition image are obtained by processing the standard data, the time-frequency image and the state transition image are input into the dual-branch coding and decoding model for reconstruction to obtain the first reconstructed image and the second reconstructed image.

[0080] In this embodiment, the pixel value of the pixel in the time-frequency image is normalized to a unified pixel range (for example, [0, 1]), and the time-frequency image is cropped to a unified size, and then the time-frequency image with the normalized pixel value and the cropped size is input into the first branch network of the dual-branch coding and decoding model, and the first branch network is used to perform a reconstruction operation on the time-frequency image to obtain the first reconstructed image. The first branch network includes an encoder and a decoder, the encoder uses a convolutional neural network (CNN), ResNet, Transformer or the like to extract features in the time-frequency image, and outputs a corresponding latent feature vector, i.e., a dynamic feature vector. The decoder is responsible for upsampling / deconvolution of the latent feature vector to reconstruct the original time-frequency image to obtain the first reconstructed image. The decoder can use a deconvolution or upsampling convolution structure.

[0081] In this embodiment, the pixel value of the pixel in the state transition image is normalized to a unified pixel range (for example, [0, 1]), and the state transition image is cropped to a unified size, and then the state transition image with the normalized pixel value and the cropped size is input into the second branch network of the dual-branch coding and decoding model, and the second branch network is used to perform a reconstruction operation on the state transition image to obtain the second reconstructed image. The second branch network includes an encoder and a decoder, the encoder uses a convolutional neural network (CNN), ResNet or Transformer or the like to extract features in the state transition image, and outputs a corresponding latent feature vector, i.e., a static feature vector. The decoder is responsible for upsampling / deconvolution of the latent feature vector to reconstruct the original state transition image to obtain the second reconstructed image. The decoder can use a deconvolution or upsampling convolution structure.

[0082] It can be understood that the first branch network and the second branch network run in parallel and do not interfere with each other, and reconstruct the time-frequency image and the state transition image respectively to obtain the first reconstructed image and the second reconstructed image.

[0083] S600: Obtain a reconstructed quality score matrix corresponding to the time series data based on the first reconstructed image, the second reconstructed image, the time-frequency image, and the state transition image.

[0084] It can be understood that the purpose of obtaining the reconstructed quality score matrix is to quantify the reconstruction errors between the first reconstructed image and the time-frequency image and the reconstruction errors between the second reconstructed image and the state transition image as a score matrix for subsequent industrial equipment abnormality determination.

[0085] In some embodiments, the first reconstructed image and the time-frequency image are compared pixel by pixel to obtain a first pixel error map, the second reconstructed image and the state transition image are compared pixel by pixel to obtain a second pixel error map, the errors at corresponding positions in the first pixel error map and the second pixel error map are weighted and summed to obtain weighted and summed reconstruction error values at all positions, and the reconstructed quality score matrix corresponding to the time series data is constructed according to the weighted and summed reconstruction error values at all positions, that is, the weighted and summed reconstruction error values at corresponding positions in the first pixel error map or the second pixel error map are used to construct the reconstructed quality score matrix corresponding to the time series data.

[0086] In other embodiments, after obtaining the first pixel error map and the second pixel error map, the errors at corresponding positions in the first pixel error map and the second pixel error map can also be averaged, maximized, or the like to obtain weighted and summed reconstruction error values at all positions, and the reconstructed quality score matrix corresponding to the time series data is constructed according to the weighted and summed reconstruction error values at all positions.

[0087] Of course, other methods or ways can also be used to calculate the first pixel error map and the second pixel error map, and when the time series data is multi-channel data, a first pixel error map and a second pixel error map can be obtained for each channel of data, and multiple first pixel error maps and multiple second pixel error maps can be obtained, or a first pixel error map and a second pixel error map can be obtained for multiple channels of data, and the embodiments of the present application do not make any limitation in this regard.

[0088] It is easy to understand that for each time series data, a reconstructed quality score matrix corresponding to each time series data is obtained based on the first reconstructed image, the second reconstructed image, the time-frequency image, and the state transition image corresponding to each time series data, so that reconstructed quality score matrices corresponding to all time series data are obtained.

[0089] S700: Determine whether the industrial equipment is abnormal based on the reconstruction quality score matrix corresponding to all time series data.

[0090] For example, based on the reconstruction quality score matrix corresponding to all time series data, it is determined whether the industrial equipment is abnormal, for example, the sum of all elements in the reconstruction quality score matrix corresponding to all time series data is obtained, that is, the sum of all elements in the reconstruction quality score matrix corresponding to each piece of time series data is obtained, the candidate sum value corresponding to the time series data is obtained, and the target sum value is obtained by adding the candidate sum values corresponding to all time series data. Then obtain the score threshold, compare the target sum value with the score threshold, if the target sum value is greater than or equal to the score threshold, it is determined that the industrial equipment is abnormal, and if the target sum value is less than the score threshold, it is determined that the industrial equipment is normal / abnormal.

[0091] It should be understood that the score threshold can be set by the engineer according to the actual demand and the characteristics of the equipment, and the embodiments of the present application do not make any limitation thereto.

[0092] The embodiments of the present application obtain the multiple pieces of time series data sensed by the sensors equipped in the industrial equipment, and preprocess the time series data to obtain standard data, use the dynamic anomaly processing module and the static anomaly processing module to process the standard data to obtain the time-frequency image and the state transition image, realize double detection of dynamic anomaly and static anomaly, reduce the possibility of missing report of static anomaly or dynamic anomaly, and then use the double-branch coding and decoding model to reconstruct the time-frequency image and the state transition image to obtain the first reconstruction image and the second reconstruction image. According to the first reconstruction image, the second reconstruction image, the time-frequency image and the state transition image, the reconstruction quality score matrix corresponding to the time series data is calculated, and whether the industrial equipment is abnormal is determined according to the reconstruction quality score matrix corresponding to all time series data. In this way, the detected dynamic anomaly and static anomaly can be accurately and efficiently processed, so as to accurately judge whether the industrial equipment is abnormal, and improve the accuracy of the industrial equipment anomaly detection.

[0093] In some embodiments, the time series data is preprocessed to obtain standard data, specifically including but not limited to the following steps S210-S220:

[0094] S210: The original data of each acquisition channel in the time series data is subjected to mean-variance normalization processing to obtain the initial data of each acquisition channel.

[0095] S220: The initial data of each acquisition channel is subjected to error rounding processing to obtain the reference data of each acquisition channel.

[0096] In this embodiment, the time series data includes raw data of multiple acquisition channels, i.e., each piece of time series data includes raw data of multiple acquisition channels. The standard data includes reference data of all acquisition channels, i.e., the standard data corresponding to each piece of time series data includes reference data of all acquisition channels in the piece of time series data.

[0097] Specifically, for each piece of time series data, the raw data of each acquisition channel in the piece of time series data is extracted, and the raw data of each acquisition channel is processed by mean-variance normalization to obtain initial data of each acquisition channel.

[0098] It should be understood that mean-variance normalization is more commonly referred to as standardization, which converts data into a standard normal distribution with a mean of 0 and a standard deviation of 1 by subtracting the mean of the data and dividing by the standard deviation of the data.

[0099] Specifically, for each piece of time series data, the initial data of each acquisition channel in the piece of time series data is processed by error rounding to obtain reference data of each acquisition channel, and the reference data of all acquisition channels constitutes standard data. For example, in some embodiments, an error rounding function is applied to the initial data of each acquisition channel to remove noise, uncertain data, abnormal data points, etc. of the initial data to obtain reference data of each acquisition channel, and the reference data of all acquisition channels is combined to obtain standard data.

[0100] In some embodiments, the error rounding function is wherein, is the initial data, is an adjustment factor, As a key hyperparameter, it is used to adjust the sensitivity of the dual-branch coding model to different physical systems / industrial equipment. Wherein, for signals with strong regularity (such as fans), a larger value is taken to retain accuracy, and for signals with high noise (such as robotic arms), a smaller value is taken to enhance robustness. is the reference data, is a floor operation, and the error rounding function processes the initial data to obtain the reference data.

[0101] For example, please refer to Figure 5The time series data (including data of C acquisition channels) are input into a preprocessing module, the preprocessing module performs mean-variance normalization processing and error rounding processing on the original data of each acquisition channel in the time series data (including data of C acquisition channels) to obtain reference data of each acquisition channel, and finally obtains reference data (i.e., standard data) of all acquisition channels.

[0102] In some embodiments, the standard data are input into a dynamic anomaly processing module for processing to obtain a time-frequency image, specifically including but not limited to the following steps S310-S320:

[0103] S310: performing linear interpolation scaling on the first candidate data to obtain first reference data.

[0104] In this embodiment, the first candidate data is the reference data of any one acquisition channel in the standard data input into the dynamic anomaly processing module.

[0105] For example, the reference data of any one acquisition channel in the standard data input into the dynamic anomaly processing module is selected as the first candidate data, and one-dimensional linear interpolation scaling processing is performed on the first candidate data to obtain first reference data with a smaller fixed length corresponding to the acquisition channel. After processing the reference data of all acquisition channels as the first reference data, finally, the first reference data corresponding to all acquisition channels is obtained.

[0106] It can be understood that the linear interpolation scaling processing is performed on the reference data of any one acquisition channel in the standard data input into the dynamic anomaly processing module, in order to obtain data with a suitable size, so as to be suitable for input into a time-frequency analysis algorithm for efficient and accurate processing, and to improve the data processing speed.

[0107] S320: processing all first reference data by using a time-frequency analysis algorithm to obtain a time-frequency image.

[0108] In this embodiment, the number of channels of the time-frequency image is the same as the number of acquisition channels, and the features in the time-frequency image can effectively represent the non-stationary dynamic characteristics such as impact and vibration in the time series signal.

[0109] In the embodiments of the present application, for each acquisition channel, the first reference data corresponding to the acquisition channel is processed by using a time-frequency analysis algorithm (such as a short-time Fourier transform, a continuous wavelet transform (CWT), a mel spectrogram, or a discrete cosine transform algorithm) to obtain a time-frequency image. For example, in the embodiments of the present application, a continuous wavelet transform algorithm is used to process all first reference data corresponding to each acquisition channel, i.e., a wavelet cluster is used as The continuous wavelet transform (such as Morlet Wavelet) converts the first reference data corresponding to each acquisition channel into a time-frequency image, and finally stacks the time-frequency images corresponding to each acquisition channel according to the channel to obtain a time-frequency image of multiple channels, the time-frequency image includes multiple channels (i.e. data channels), and the number of channels of the time-frequency image is the same as the number of acquisition channels of the time series data.

[0110] For example, the continuous wavelet transform (CWT, Continuous Wavelet Transform) generates a two-dimensional time-frequency image by convolving the time series data with a set of basis functions called "wavelets". The time-frequency image can simultaneously show the energy distribution of the time series signal at different times and frequencies. The continuous wavelet transform is good at capturing transient and non-stationary characteristics in time series signals, such as sudden impacts, vibrations or pattern changes in industrial equipment motion. In some embodiments, Morlet wavelet is preferred as the mother wavelet type in CWT to more accurately capture transient and non-stationary characteristics in time series signals. The continuous wavelet transform algorithm has high time resolution in the high frequency region and high frequency resolution in the low frequency region. This adaptive time-frequency window feature makes the continuous wavelet transform very suitable for accurately locating "short duration anomalies", such as impact signals. This adaptive time-frequency window feature is difficult to achieve by traditional short-time Fourier transform (STFT) algorithms, so the continuous wavelet transform has better advantages and can more accurately capture transient and non-stationary characteristics in time series signals.

[0111] For example, please refer to Figure 5 The standard data (including reference data of C acquisition channels) is input into the dynamic anomaly processing module 121, the dynamic anomaly processing module 121 performs linear interpolation scaling processing on the reference data of each acquisition channel in the standard data and processes the scaled reference data using the continuous wavelet transform (CWT) algorithm to obtain a time-frequency image corresponding to each acquisition channel, and finally obtains time-frequency images corresponding to all acquisition channels. The time-frequency images corresponding to all acquisition channels are stacked according to the channel to obtain a time-frequency image 41 with multiple data channels.

[0112] It can be understood that Figure 5 Only a time series data is shown in some embodiments of the present application to obtain a corresponding time-frequency image 41. In actual application, multiple time series data need to be processed to obtain multiple time-frequency images, for example, please refer to Figure 6 , Figure 6It is shown that eight time sequence data are processed respectively to obtain eight time-frequency images, respectively CWT 0, CWT 1, CWT 2, CWT 3, CWT 4, CWT 5, CWT 6 and CWT 7, in some embodiments of the application. The 3 in (3, 32, 32) of the time-frequency image indicates that the time-frequency image includes three channels / data channels, and 32, 32 indicates that the size of the time-frequency image is 32*32.

[0113] In some embodiments, the standard data is input into the static anomaly processing module for processing to obtain a state transition image, specifically including but not limited to the following steps S410-S420:

[0114] S410: filtering and down-sampling the second candidate data to obtain second reference data.

[0115] In this embodiment, the second candidate data is the reference data of any one acquisition channel in the standard data input into the static anomaly processing module.

[0116] For example, the reference data of any one acquisition channel in the standard data input into the static anomaly processing module is selected as the second candidate data, and the second candidate data is filtered and down-sampled to obtain the second reference data corresponding to the acquisition channel. After processing the reference data of all acquisition channels as the second reference data, the second reference data corresponding to all acquisition channels is finally obtained. In some embodiments, a FIR filter (Finite Impulse Response Filter) is used to filter the second candidate data (sensing data with excessive external noise) to smooth the second candidate data to obtain the second reference data corresponding thereto, and then the PAA (Piecewise Aggregate Approximation) technology is used to down-sample the second reference data to a data with a smaller fixed length to obtain the second reference data, so as to reduce the computational complexity.

[0117] It can be understood that the reference data of any one acquisition channel in the standard data input into the static anomaly processing module is filtered and down-sampled, in order to obtain data with appropriate size, so as to be suitable for input into the state transition analysis algorithm for efficient and accurate processing, and to improve the data processing speed.

[0118] S420: processing all second reference data by using a state transition analysis algorithm to obtain a state transition image.

[0119] In this embodiment, the number of channels of the state transition image is the same as the number of acquisition channels, and the features in the state transition image effectively represent the state transition probability of the time series signal between different quantization intervals, and are extremely sensitive to macro state changes (e.g., from movement to abnormal stillness).

[0120] For example, for each acquisition channel, the second reference data corresponding to the acquisition channel is processed by using a state transition analysis algorithm (such as a Markov Transition Field (MTF), a Recurrence Plot (RP), or a Gramian Angular Field (GAF) algorithm) to obtain a state transition image. For example, in the embodiment of the present application, the Markov Transition Field algorithm is used to process the second reference data corresponding to all acquisition channels, that is, the second reference data corresponding to each acquisition channel is discretized to obtain a state sequence, the state sequence includes the state at each time, and the state is obtained by mapping the second reference data at each time to a state label. The transition count of adjacent time is counted, the first-order state transition probability is calculated, the Markov Transition Field matrix is generated or constructed according to the state sequence, and thus the state transition image corresponding to each acquisition channel is obtained. Finally, the state transition images corresponding to the acquisition channels are stacked according to the channels to obtain the state transition images of multiple channels, the state transition image includes multiple channels (i.e., data channels), and the number of channels of the state transition image is the same as the number of acquisition channels of the time series data.

[0121] For example, the Markov Transition Field (MTF) is a technology for encoding one-dimensional time series data into a two-dimensional image, which focuses on capturing the statistical structure and time evolution law of the time series signal. The Markov Transition Field algorithm first divides the value range of the time series signal into several quantization intervals, then constructs a Markov transition matrix, each element in the matrix represents the probability of the signal transition from one interval to another, and finally arranges the Markov transition matrix into a state transition image, which reveals the macro state and pattern of the signal, and is suitable for analyzing the state transition characteristics of systems / industrial equipment.

[0122] For example, please refer to Figure 5The standard data (including reference data of C acquisition channels) is input into the static anomaly processing module 122, the static anomaly processing module 122 filters the reference data of each acquisition channel in the standard data by using a FIR filter, performs downsampling on the filtered reference data by using a piecewise aggregate approximation (PAA) technology, and processes the filtered and downsampled reference data by using a Markov transition field (MTF) algorithm to obtain a state transition image corresponding to each acquisition channel, and finally obtains state transition images corresponding to all acquisition channels. The state transition images corresponding to all acquisition channels are stacked according to the channels to obtain a state transition image 42 with multiple data channels.

[0123] It can be understood that Figure 5 Only processing of a piece of time sequence data to obtain a corresponding state transition image 42 in some embodiments of the present application is schematically shown, and in actual application, multiple pieces of time sequence data need to be processed to obtain a plurality of corresponding state transition images, for example, please refer to Figure 6 , Figure 6 In some embodiments of the present application, eight pieces of time sequence data are respectively processed to obtain eight corresponding state transition images, which are state transition images MTF 0, MTF 1, MTF 2, MTF 3, MTF 4, MTF 5, MTF 6 and MTF 7 respectively. The 3 in (3, 32, 32) of the state transition image indicates that the state transition image includes three channels / data channels, and 32, 32 indicates that the size of the state transition image is 32*32.

[0124] The embodiments of the present application ingeniously combine the signal processing tool (CWT algorithm) for physical transient analysis and the statistical probability tool (MTF algorithm) for logical state change analysis, thereby realizing more comprehensive coverage of industrial equipment anomaly detection, more accurately capturing dynamic anomaly features and static anomaly features in time sequence signals, thereby realizing double detection of dynamic anomalies and static anomalies, reducing the possibility of missing reports of static anomalies or dynamic anomalies, and finally accurately and efficiently processing the detected dynamic anomalies and static anomalies, thereby accurately judging whether the industrial equipment is abnormal and improving the accuracy of industrial equipment anomaly detection.

[0125] For example, in some embodiments, the time-frequency image and the state transition image are input into a double-branch coding and decoding model for reconstruction to obtain a first reconstructed image and a second reconstructed image, specifically including but not limited to the following steps S510-S540:

[0126] S510: inputting the time-frequency image into a first encoding module for downsampling and feature mapping to obtain a first candidate vector.

[0127] S520: input the first candidate vector into the first decoding module for feature transformation and up-sampling to obtain a first reconstructed image.

[0128] S530: input the state transition image into the second encoding module for down-sampling and feature mapping to obtain a second candidate vector.

[0129] S540: input the second candidate vector into the second decoding module for feature transformation and up-sampling to obtain a second reconstructed image.

[0130] In step S510, the time-frequency image is input into the first encoding module of the first reconstruction network, the first encoding module is used to down-sample the time-frequency image to obtain a deep feature map after multiple convolutions and down-sampling, then the first encoding module is used to perform flattening processing on the deep feature map, so as to further compress, fuse and abstract the information of the deep feature map, obtain a high-dimensional feature vector, and then the first encoding module is used to map the high-dimensional feature vector to a normal distribution space to predict a mean vector of the normal distribution and a logarithmic variance vector , a preset standard feature vector is obtained, and a first candidate vector is generated according to the standard feature vector, the mean vector and the logarithmic variance vector .

[0131] In step S520, the first candidate vector is input into the first decoding module of the first reconstruction network, the first decoding module is used to perform feature transformation on the first candidate vector, that is, to perform nonlinear transformation on the first candidate vector, to expand the first candidate vector into a high-dimensional feature vector, then to reshape the high-dimensional feature vector to obtain a structured first feature map with spatial dimensions, then to up-sample the first feature map with spatial dimensions to obtain a first up-sampled feature map, and finally to decode the first up-sampled feature map to obtain a first reconstructed image. The first reconstructed image corresponds to the time-frequency image.

[0132] In step S530, the state transition image is input into the second encoding module of the second reconstruction network, the second encoding module is used to down-sample the state transition image to obtain a deep feature map after multiple convolutions and down-sampling, then the second encoding module is used to perform flattening processing on the deep feature map, so as to further compress, fuse and abstract the information of the deep feature map, obtain a high-dimensional feature vector, and then the second encoding module is used to map the high-dimensional feature vector to a normal distribution space to predict a mean vector of the normal distribution and a logarithmic variance vector , a preset standard feature vector is obtained, and a second candidate vector is generated according to the standard feature vector, the mean vector and the logarithmic variance vector .

[0133] In step S540, the second candidate vector is input to the second decoding module of the second reconstruction network. The second decoding module performs feature transformation on the second candidate vector, that is, it performs a nonlinear transformation on the second candidate vector to expand it into a high-dimensional feature vector. Then, the high-dimensional feature vector is reshaped to obtain a structured second feature map with spatial dimension. Next, the second feature map with spatial dimension is upsampled to obtain a second upsampled feature map. Finally, the second upsampled feature map is decoded to obtain the second reconstructed image. The second reconstructed image corresponds to the state transition image.

[0134] For example, in some embodiments, the time-frequency image is input into the first encoding module for downsampling and feature mapping to obtain a first candidate vector, specifically including but not limited to the following steps S511-S515:

[0135] S511: Input the time-frequency image into the first encoder for downsampling to obtain the first feature map set.

[0136] In this embodiment, the first feature map set includes a first reference feature map. The first reference feature map is the feature map with the smallest resolution during the downsampling process of the first encoder on the time-frequency image. That is, when the first encoder performs downsampling, the feature map with the smallest resolution is selected as the first reference feature map.

[0137] In this embodiment of the invention, a time-frequency image is input into a first encoding module. The first encoder of the first encoding module performs multiple convolutions and multiple downsampling operations on the time-frequency image to obtain first downsampled feature maps of multiple resolutions. The first downsampled feature maps of multiple resolutions constitute the first feature map set.

[0138] In this embodiment of the invention, a pixel shuffle technique is used for downsampling. Pixel shuffle rearranges the pixels of a high-resolution, low-channel time-frequency image, converting it into one or more low-resolution, high-channel feature maps, thereby achieving downsampling. Pixel shuffle is more computationally efficient, requires fewer parameters, and effectively avoids the checkerboard artifacts that may be caused by traditional upsampling methods (such as transposed convolution and deconvolution).

[0139] See Figure 7 As shown, the first encoder uses the convolutional layer Conv2D, the convolutional layer SAConv2D, and the first downsampling layer PSDS (PixelShuffle Down Sample) to perform multiple convolution and downsampling operations on the time-frequency image to obtain first downsampled feature maps of multiple resolutions.

[0140] The convolutional layer SAConv2D introduces a self-attention mechanism to capture long-range dependencies in the feature map, and by performing weighted averaging on the feature map output by the convolutional layer Conv2D or the feature map obtained by downsampling in the first downsampling layer PSDS, the convolutional layer SAConv2D can enhance important features and suppress irrelevant features, thereby improving the expression ability of the model.

[0141] In the embodiment of the application, when the pixel reorganization (PixelShuffle) technology is applied to the PSDS downsampling, the advantage is that the PSDS module combining the inverse process of PixelShuffle and convolution can retain as many fine details in the original image as possible during downsampling (or dimension reduction), avoid losing key texture information, and facilitate high-speed inference on edge devices.

[0142] S512: flattening the first reference feature map using the first encoder to obtain a first flattened vector.

[0143] Referring to Figure 7 , after obtaining a plurality of first down-sampling feature maps (i.e., a first feature map set) of different resolutions by downsampling, the embodiment of the application performs flattening processing on the first reference feature map using the first encoder (i.e., using the MLP layer in the first encoder) to obtain a first flattened vector of high dimension. The first flattened vector is output from the MLP layer in the first encoder to the first feature mapping layer.

[0144] S513: inputting the first flattened vector into the first feature mapping layer, and mapping the first flattened vector to a normal distribution using the first feature mapping layer to obtain a first parameter.

[0145] In this embodiment, the first parameter includes the mean and standard deviation of the first normal distribution.

[0146] For example, the embodiment of the application inputs the first flattened vector into the first feature mapping layer, and maps the first flattened vector to a normal distribution space using the first feature mapping layer, i.e., through independent fully connected layers, activation functions (such as ReLU) and the like in the first feature mapping layer to perform nonlinear transformation, to predict the mean and standard deviation of the first normal distribution.

[0147] For example, referring to Figure 7 , the embodiment of the application inputs the first flattened vector into the first feature mapping layer FMLP (i.e., a variational forward MLP), and uses the first feature mapping layer FMLP to predict the mean and standard deviation of the first normal distribution.

[0148] S514: obtaining a first target vector.

[0149] S515: generating a first candidate vector based on the first target vector and the first parameter.

[0150] It can be understood that, to ensure deterministic behavior and reduce computational overhead when deploying a model on an edge device, a pre-generated and saved fixed random vector (i.e., the first target vector) of a dual-branch codec model can be conditionally used as a latent feature representation when the dual-branch codec model is inferred, combined with the first parameter (i.e., the mean and standard deviation of the first normal distribution and standard deviation ) to generate a first candidate vector, thereby achieving effective regularization while ensuring the reproducibility and stability of the results and simplifying the quantization deployment process of the dual-branch codec model. The elements of the first target vector follow a standard normal distribution, ensuring the regularization effect.

[0151] The embodiment of the present application obtains the pre-generated and saved first target vector from the storage module or unit of the dual-branch codec model, multiplies the standard deviation of the first normal distribution by the first target vector to obtain a first intermediate result, adds the first intermediate result to the mean of the first normal distribution to obtain the first candidate vector. The specific calculation formula of the first candidate vector is as follows:

[0152]

[0153] In the above formula, is the first candidate vector, is the mean of the first normal distribution, is the standard deviation of the first normal distribution, is the first target vector.

[0154] For example, referring to Figure 7 , the embodiment of the present application multiplies the standard deviation of the first normal distribution by the first target vector to obtain a first intermediate result , adds the first intermediate result to the mean of the first normal distribution to obtain the first candidate vector .

[0155] For example, in some embodiments, the first candidate vector is input into the first decoding module for feature transformation and upsampling to obtain a first reconstructed image, specifically including but not limited to the following steps S521-S524:

[0156] S521: input the first candidate vector into the first feature transformation layer for transformation to obtain a first initial feature map.

[0157] S522: input the first initial feature map into the first decoder for upsampling to obtain a first upsampled feature map.

[0158] S523: fuse the first reference feature map and the first upsampled feature map by using the first connection layer to obtain a first fused feature map.

[0159] S524: decode the first fused feature map by using the first decoder to obtain a first reconstructed image.

[0160] For example, the first candidate vector is input into the first feature transformation layer, the first candidate vector is nonlinearly transformed by using the first feature transformation layer, the first candidate vector is expanded into a high-dimensional feature vector, and the high-dimensional feature vector is reshaped according to the input feature dimension of the first decoder to obtain a first initial feature map, the size of the first initial feature map is the same as the input feature dimension of the first decoder, so as to adapt to input the first decoder. The first initial feature map is input into the first decoder for upsampling to obtain a first upsampled feature map.

[0161] In this embodiment, the first feature map set further includes a first reference feature map, wherein the first reference feature map is a feature map with a resolution greater than the first resolution threshold in the down-sampling process of the time-frequency image by the first encoder, that is, in the down-sampling process of the first encoder, the feature map with a resolution greater than the first resolution threshold is selected as the first reference feature map. The designer can set the first resolution threshold according to actual requirements and task characteristics. The resolution of the first reference feature map is greater than the resolution of the first reference feature map.

[0162] Specifically, the first reference feature map is superimposed on the first upsampled feature map by using the first connection layer, that is, the first reference feature map is fused with the first upsampled feature map to obtain a first fused feature map. Finally, the first fused feature map is decoded by using the first decoder to reconstruct the original time-frequency image to obtain a first reconstructed image. In this embodiment, the bottom layer detail information (such as edge and texture) of the first reference feature map in the encoding stage is supplemented to the decoding stage by using the first connection layer, which greatly improves the clarity and accuracy of the reconstructed image and prevents the loss of key details at the encoding “bottleneck”.

[0163] It can be understood that when the first encoder comprises a plurality of down-sampling layers and the first decoder comprises a plurality of up-sampling layers, the first encoder obtains a plurality of first down-sampling feature maps (i.e. a first feature map set, comprising a first reference feature map and a first reference feature map) of different resolutions by down-sampling through each layer of down-sampling layers, each first down-sampling feature map of different resolutions being obtained by down-sampling through one layer of down-sampling layers of the first encoder. The first decoder obtains a plurality of first up-sampling feature maps of different resolutions by up-sampling through each layer of up-sampling layers, each first up-sampling feature map of different resolutions being obtained by up-sampling through one layer of up-sampling layers of the first decoder. The process of up-sampling and feature map fusion performed by the first decoder is as follows: the first layer of up-sampling layers of the first decoder up-samples the first initial feature map to obtain a first up-sampling feature map of a first resolution, and the first connection layer is used to fuse the first reference feature map of the first resolution with the first up-sampling feature map of the first resolution to obtain a first fusion feature map of the first resolution, and then the first fusion feature map of the first resolution is input into the second layer of up-sampling layers in the first decoder for up-sampling to obtain a first up-sampling feature map of a second resolution, and the first connection layer is used to fuse the first reference feature map of the second resolution with the first up-sampling feature map of the second resolution to obtain a first fusion feature map of the second resolution, and so on. The up-sampling layers of the first encoder continuously up-sample the first fusion feature map, and the first connection layer continuously fuses the first reference feature map and the first up-sampling feature map of the same resolution, until the first connection layer fuses the first reference feature map of the maximum resolution with the first up-sampling feature map of the maximum resolution to obtain a first fusion feature map of the maximum resolution, and the decoding layer of the first decoder decodes the first fusion feature map of the maximum resolution to obtain a first reconstructed image. The size of the first reconstructed image is the same as that of the time-frequency image, i.e. the size of the first fusion feature map is the same as that of the time-frequency image.

[0164] For example, please refer to Figure 7The first candidate vector is input to the MLP layer (i.e., the first feature transformation layer) of the first decoding module for feature transformation to obtain a first initial feature map. The first initial feature map is input to the first decoder for up-sampling, i.e., after the first initial feature map is input to the first decoder, the MLP layer of the first decoder reshapes the features in the first initial feature map to obtain structured features with spatial dimensions, thereby providing structured features for the subsequent up-sampling process. The first decoder uses the convolution layer Conv2D, the convolution layer SAConv2D, and the first up-sampling layer PSUS (PixelShuffleUpSample) to perform multiple convolution and multiple up-sampling operations on the reshaped features to obtain multiple first up-sampled feature maps with different resolutions. In the up-sampling process of the first decoder, the first connection layer is used to fuse the first reference feature map with the same resolution and the first up-sampled feature map, and then the fused feature map is convolved and up-sampled. In this way, finally, the first decoder decodes the fused feature map with the maximum resolution to obtain the first reconstructed image.

[0165] In some embodiments, the state transition image is input to the second encoding module for down-sampling and feature mapping to obtain a second candidate vector, specifically including but not limited to the following steps S531-S535:

[0166] S531: The state transition image is input to the second encoder for down-sampling to obtain a second reference feature map.

[0167] In this embodiment, the second reference feature map is the feature map with the smallest resolution in the down-sampling process of the second encoder on the state transition image, i.e., when the second encoder is down-sampling, the feature map with the smallest resolution is selected as the second reference feature map.

[0168] In this embodiment, the second reference feature map is the feature map with the smallest resolution in the down-sampling process of the second encoder on the state transition image, i.e., when the second encoder is down-sampling, the feature map with the smallest resolution is selected as the second reference feature map.

[0169] Referring to Figure 8As shown, the second encoder utilizes the convolutional layer Conv2D, the convolutional layer SAConv2D, and the first down-sampling layer PSDS (PixelShuffle Down Sample) to perform multiple convolutional and down-sampling operations on the state transition image, to obtain a plurality of first down-sampling feature maps of different resolutions. In the down-sampling process of the state transition image by the second encoder, an addition operation is included, in which the output of a convolutional layer (SAConv2D) is added to the output of a previous down-sampling layer (PSDS), to obtain a feature map after addition. The addition operation constitutes a residual connection (Residual Connection), which helps to alleviate the gradient vanishing problem in a deep network.

[0170] S532: performing flattening processing on the second reference feature map by using the second encoder to obtain a second flattened vector.

[0171] Referring to Figure 8 , after obtaining a plurality of second down-sampling feature maps of different resolutions by sampling, the embodiment of the present application performs flattening processing on the second reference feature map by using the second encoder (i.e., by using the MLP layer in the second encoder), to obtain a second flattened vector of high dimension. The second flattened vector is output from the MLP layer in the first encoder to the second feature mapping layer.

[0172] S533: inputting the second flattened vector into the second feature mapping layer, and mapping the second flattened vector to a normal distribution by using the second feature mapping layer, to obtain a second parameter.

[0173] In this embodiment, the second parameter includes the mean and the standard deviation of the second normal distribution.

[0174] For example, the embodiment of the present application inputs the second flattened vector into the second feature mapping layer, and maps the second flattened vector to a normal distribution space by using the second feature mapping layer, i.e., by performing non-linear transformation through independent fully connected layers, activation functions (such as ReLU), etc. in the second feature mapping layer, to predict the mean and the standard deviation of the second normal distribution.

[0175] For example, referring to Figure 8 , the embodiment of the present application inputs the second flattened vector into the second feature mapping layer FMLP (i.e., the variational forward MLP), and predicts the mean and the standard deviation of the second normal distribution by using the second feature mapping layer FMLP.

[0176] S534: obtaining a second target vector.

[0177] S535: generating a second candidate vector based on the second target vector and the second parameter.

[0178] Understandably, to ensure deterministic behavior and reduce computational overhead when deploying models on edge devices, embodiments of the present invention may conditionally use a pre-generated and stored fixed random vector (i.e., the second target vector) as a latent feature representation during inference of the two-branch codec model, combined with a second parameter (i.e., the mean of the second normal distribution). and standard deviation This process generates a second candidate vector, thereby achieving effective regularization while ensuring the reproducibility and stability of the results, simplifying the quantization deployment process of the dual-branch encoding / decoding model. The elements of the second target vector follow a standard normal distribution to guarantee the regularization effect. Obviously, the second target vector can be the same as or different from the first target vector.

[0179] In this embodiment of the invention, a pre-generated and saved second target vector is obtained from the storage module or unit of the dual-branch encoding / decoding model, and the standard deviation of the second normal distribution is... Multiply by the second target vector to obtain the second intermediate result, and then compare the second intermediate result with the mean of the second normal distribution. The two vectors are added together to obtain the second candidate vector. The specific formula for calculating the second candidate vector is as follows:

[0180]

[0181] In the above formula, As the second candidate vector, The mean of the second normal distribution is . The standard deviation of the second normal distribution. This is the second target vector.

[0182] For example, please see Figure 8 In this embodiment of the invention, the standard deviation of the second normal distribution is... With the second target vector Multiply to obtain the second intermediate result. , the second intermediate result The mean of the second normal distribution Add them together to obtain the second candidate vector. .

[0183] For example, in some embodiments, the second candidate vector is input into the second decoding module for feature transformation and upsampling to obtain the second reconstructed image, specifically including but not limited to the following steps S541-S543:

[0184] S541: Input the second candidate vector into the second feature transformation layer for transformation to obtain the second initial feature map.

[0185] S542: inputting the second initial feature map into the second decoder for up-sampling to obtain a second up-sampled feature map.

[0186] S543: decoding the second up-sampled feature map by using the second decoder to obtain a second reconstructed image.

[0187] For example, the second candidate vector is input into the second feature transformation layer, the second feature transformation layer is used for nonlinear transformation of the second candidate vector, the second candidate vector is expanded into a high-dimensional feature vector, and the high-dimensional feature vector is reshaped according to the input feature dimension of the second decoder to obtain a second initial feature map, the size of the second initial feature map is the same as the input feature dimension of the second decoder, so as to adapt to inputting the second decoder. The second initial feature map is input into the second decoder for up-sampling to obtain a second up-sampled feature map. Finally, the second up-sampled feature map is decoded by using the second decoder to reconstruct the original state transition image to obtain a second reconstructed image.

[0188] For example, please refer to Figure 8 The second candidate vector is input into the MLP layer (i.e., the second feature transformation layer) of the second decoding module for feature transformation to obtain a second initial feature map. The second initial feature map is input into the second decoder for up-sampling, that is, after the second initial feature map is input into the second decoder, the MLP layer of the second decoder reshapes the features in the second initial feature map to obtain structured features with spatial dimensions, thereby providing structured features for the subsequent up-sampling process. The second decoder uses the convolution layer SAConv2D, the convolution layer Conv2D and the second up-sampling layer PSUS (PixelShuffleUpSample) to perform multiple convolution and multiple up-sampling operations on the reshaped features to obtain multiple second up-sampled feature maps with different resolutions. In the up-sampling process of the second decoder, an addition operation is included, which adds the output of a convolution layer (SAConv2D) to the output of the previous MLP layer to obtain a feature map after addition, and the addition operation constitutes a residual connection (Residual Connection), which helps to optimize the information flow and gradient propagation in the deep network. Finally, the second reconstructed image is obtained by decoding the second up-sampled feature map with the maximum resolution by using the second decoder.

[0189] It can be understood that when the multi-dimensional features are sequentially passed between the convolutional layers, the dimension of the features can change (often the channel dimension), and the size change of the feature maps between adjacent convolutional layers or convolutional modules is not directly related to the encoding and decoding process. In the encoder, the spatial size of the feature map is gradually reduced through the downsampling operation (such as PSDS), and the number of channels can increase to capture more feature information. When the feature map is encoded and downsampled through the complete encoder, its spatial size will be significantly reduced. In the decoder, the spatial size of the feature map is gradually increased through the upsampling operation (such as PSUS), and the number of channels can decrease to restore the dimension of the original input. When the hidden feature / feature vector is upsampled and decoded through the complete decoder, its spatial size will recover to the size of the original input (i.e. the image input to the first encoder or the second encoder).

[0190] In some embodiments, the reconstruction quality score matrix corresponding to the time series data is obtained based on the first reconstructed image, the second reconstructed image, the time-frequency image, and the state transition image, specifically including but not limited to the following steps S610-S640:

[0191] S610: Extract the feature map of each channel in the first reconstructed image, the second reconstructed image, the time-frequency image, and the state transition image to obtain the first reconstructed feature map, the second reconstructed feature map, the state transition feature map, and the time-frequency feature map of each channel.

[0192] In this embodiment, the number of channels / data channels of the first reconstructed image and the number of channels / data channels of the time-frequency image are the same, and the number of channels / data channels of the second reconstructed image and the number of channels / data channels of the state transition image are the same.

[0193] Specifically, the first reconstructed image, the second reconstructed image, the time-frequency image, and the state transition image are processed respectively, and the feature map of each channel / data channel in each image is extracted to obtain the first reconstructed feature map, the second reconstructed feature map, the state transition feature map, and the time-frequency feature map of each channel / data channel corresponding to the first reconstructed image, the second reconstructed image, the time-frequency image, and the state transition image. For example, the number of channels / data channels of the first reconstructed image and the time-frequency image is three, and the first reconstructed feature map of the three channels / data channels in the first reconstructed image and the time-frequency feature map of the three channels / data channels in the time-frequency image are extracted. For another example, the number of channels / data channels of the second reconstructed image and the state transition image is three, and the second reconstructed feature map of the three channels / data channels in the second reconstructed image and the state transition feature map of the three channels / data channels in the state transition image are extracted.

[0194] S620: Calculate the similarity of the first reconstructed feature map and the time-frequency feature map of each channel to obtain the time-frequency reconstruction score of each channel.

[0195] Referring to Figure 7 For example, the similarity of the first reconstruction feature map and the time-frequency feature map of each channel / data channel is calculated, and the similarity of the first reconstruction feature map and the time-frequency feature map of each channel / data channel is taken as the time-frequency reconstruction score of the channel / data channel, and finally the time-frequency reconstruction scores of all channels / data channels are obtained. For example, if there are three channels / data channels, the time-frequency reconstruction scores of the three channels / data channels are calculated correspondingly.

[0196] In some embodiments, the similarity of the first reconstruction feature map and the time-frequency feature map of each channel / data channel can be calculated by using the following first formula:

[0197]

[0198] In the above formula, is the maximum value of the pixel values of all pixels of the first reconstruction image and the time-frequency image, is the mean square error of all elements of the first reconstruction feature map and the time-frequency feature map, is the similarity of the first reconstruction feature map and the time-frequency feature map.

[0199] S630: The similarity of the second reconstruction feature map and the state transition feature map of each channel is calculated to obtain the state transition reconstruction score of each channel.

[0200] Referring to Figure 8 For example, the similarity of the second reconstruction feature map and the state transition feature map of each channel / data channel is calculated, and the similarity of the second reconstruction feature map and the state transition feature map of each channel / data channel is taken as the state transition reconstruction score of the channel / data channel, and finally the state transition reconstruction scores of all channels / data channels are obtained. For example, if there are three channels / data channels, the state transition reconstruction scores of the three channels / data channels are calculated correspondingly.

[0201] In some embodiments, the similarity of the second reconstruction feature map and the state transition feature map of each channel / data channel can be calculated by using the following second formula:

[0202]

[0203] In the above formula, a maximum value among pixel values of all pixels of the second reconstructed image and the state transition image, a mean square error of all elements of the second reconstructed feature map and the state transition feature map, a similarity of the second reconstructed feature map and the state transition feature map.

[0204] S640: Based on the time-frequency reconstruction scores of all channels and the state transition reconstruction scores, a reconstruction quality score matrix corresponding to the time series data is constructed.

[0205] Specifically, the time-frequency reconstruction scores of all channels / data channels in the first reconstructed image and the time-frequency image and the state transition reconstruction scores of all channels / data channels in the second reconstructed image and the state transition image are constructed into a reconstruction quality score matrix, that is, a reconstruction quality score matrix corresponding to the time series data is obtained.

[0206] For example, a blank quality score matrix is constructed, and all time-frequency reconstruction scores and all state transition reconstruction scores are filled into the corresponding positions of the blank quality score matrix in the order of the reconstruction network (including the first reconstruction network and the second reconstruction network) and the order of the data channels, to obtain a reconstruction quality score matrix corresponding to the time series data.

[0207] For example, the reconstruction quality score matrix is:

[0208]

[0209] The reconstruction quality score matrix , , and are respectively the time-frequency reconstruction scores of the first channel / data channel, the second channel / data channel and the third channel / data channel in the first reconstructed image and the time-frequency image, , and are respectively the state transition reconstruction scores of the first channel / data channel, the second channel / data channel and the third channel / data channel in the second reconstructed image and the state transition image.

[0210] In some embodiments, based on the reconstruction quality score matrices corresponding to all time series data, it is determined whether the industrial equipment is abnormal, specifically including but not limited to the following steps S710-S760:

[0211] S710: Based on the reconstruction quality score matrices corresponding to all time series data, a threshold score matrix is obtained.

[0212] In this embodiment, the size of the threshold score matrix is the same as the size of the reconstruction quality score matrix.

[0213] For example, a blank original matrix is constructed, and the size of the original matrix is the same as the size of the reconstruction quality score matrix. The elements in the corresponding positions of the reconstruction quality score matrix corresponding to all time series data are subjected to corresponding mathematical operations to obtain an operation value corresponding to the position, and the operation value corresponding to the position is taken as the element in the corresponding position of the original matrix. After the operation values corresponding to all positions are obtained by subjecting the elements in all positions of the reconstruction quality score matrix corresponding to all time series data to corresponding mathematical operations, the operation values corresponding to the positions are filled into the corresponding positions of the original matrix to obtain the threshold score matrix. The mathematical operations performed on the elements in all positions of the reconstruction quality score matrix corresponding to all time series data are the same, and the mathematical operations are any one of taking the mean value, taking the median value, taking the maximum value, and taking the minimum value, or the mathematical operations can also be other suitable operations.

[0214] For example, the original matrix is as follows:

[0215]

[0216] Specifically, the elements in the corresponding positions of the reconstruction quality score matrix corresponding to all time series data are subjected to corresponding mathematical operations to obtain an operation value corresponding to the position, and the operation value corresponding to the position is filled into the corresponding position of the original matrix, for example, the elements in the first row and the first column (the first position) of the reconstruction quality score matrix corresponding to all time series data are averaged to obtain an operation value corresponding to the first row and the first column, and the operation value corresponding to the first row and the first column is filled into the first row and the first column of the constructed original matrix, that is, the elements in the first row and the first column of the original matrix are assigned to the operation value corresponding to the first row and the first column.

[0217] In some embodiments, based on the reconstruction quality score matrix corresponding to all time series data, a threshold score matrix is obtained, specifically including but not limited to the following steps S711-S713:

[0218] S711: Construct an original matrix.

[0219] S712: Perform a target calculation operation on the elements in the corresponding positions of the reconstruction quality score matrix corresponding to all time series data to obtain a target value corresponding to the position.

[0220] S713: Fill the target values corresponding to all positions into the corresponding positions of the original matrix to obtain a threshold score matrix.

[0221] ​In this embodiment, the size of the original matrix is the same as that of the reconstruction quality score matrix. The target calculation operation is any one of taking the mean, taking the median, taking the maximum, and taking the minimum, or the target calculation operation can also be any other suitable calculation operation.

[0222] For example, a blank original matrix is constructed, and a target value corresponding to each position in the original matrix is obtained by performing a target calculation operation on the element in the corresponding position in the reconstruction quality score matrix corresponding to all time series data, and the target value is filled into the corresponding position in the original matrix. Obviously, after performing a target calculation operation on the elements in all positions in the reconstruction quality score matrix corresponding to all time series data to obtain target values corresponding to all positions, the target values corresponding to all positions are filled into the corresponding positions in the original matrix to obtain a threshold score matrix.

[0223] S720: Obtain a binary matrix corresponding to the time series data based on the threshold score matrix and the reconstruction quality score matrix corresponding to the time series data.

[0224] For example, the reconstruction quality score matrix corresponding to each piece of time series data is compared with the threshold score matrix element by element to obtain error values between the elements in all positions in the reconstruction quality score matrix and the threshold score matrix, and it is determined whether the element (i.e., the time-frequency reconstruction score or the state transition reconstruction score) in the position in the reconstruction quality score matrix is abnormal according to the error values between the elements in each position. A blank initial matrix is constructed, and the element in the corresponding position in the initial matrix is assigned a value of 1 or 0 according to whether the element (i.e., the time-frequency reconstruction score or the state transition reconstruction score) in the position in the reconstruction quality score matrix is abnormal, to obtain a final binary matrix corresponding to the time series data. Obviously, each element of the binary matrix is 1 or 0.

[0225] In some embodiments, when the error value between the elements in the corresponding positions in the reconstruction quality score matrix and the threshold score matrix is greater than or equal to a preset error threshold, it is determined that the element in the position in the reconstruction quality score matrix is abnormal, and the element in the position in the initial matrix is assigned a value of 1, and when the error value between the elements in the corresponding positions in the reconstruction quality score matrix and the threshold score matrix is less than the preset error threshold, it is determined that the element in the position in the reconstruction quality score matrix is normal, and the element in the position in the initial matrix is assigned a value of 0.

[0226] For example, the binary matrix corresponding to the time series data is:

[0227]

[0228] The above binary matrix The element "1" in the first row and the first column in the matrix indicates that the element in the first row and the first column in the reconstruction quality score matrix (i.e., the time-frequency reconstruction score or the state transition reconstruction score) is abnormal. The binary matrix The element "0" in the first row and the second column in the matrix indicates that the element in the first row and the second column in the reconstruction quality score matrix (i.e., the time-frequency reconstruction score or the state transition reconstruction score) is normal. Similarly, the binary matrix The element "0" in the second row and the third column in the matrix indicates that the element in the second row and the third column in the reconstruction quality score matrix (i.e., the time-frequency reconstruction score or the state transition reconstruction score) is normal.

[0229] In some embodiments, the binary matrix corresponding to the time series data is obtained based on the threshold score matrix and the reconstruction quality score matrix corresponding to the time series data, specifically including but not limited to the following steps S721-S723:

[0230] S721: Construct an initial matrix.

[0231] S722: Compare the two elements in the first position in the reconstruction quality score matrix corresponding to the time series data and the threshold score matrix to obtain a comparison result.

[0232] S723: According to the comparison result, assign the element in the first position in the initial matrix to be 1 or 0 to obtain the binary matrix corresponding to the time series data.

[0233] In this embodiment, the size of the initial matrix is the same as the size of the reconstruction quality score matrix. The first position is the position corresponding to the ith row and the jth column in the reconstruction quality score matrix or the threshold score matrix.

[0234] Specifically, a blank initial matrix is constructed, and the two elements in the first position in the reconstruction quality score matrix corresponding to the time series data and the threshold score matrix are compared, that is, the element in the first position in the reconstruction quality score matrix corresponding to the time series data and the element in the first position in the threshold score matrix are compared in size to obtain a comparison result. Obviously, the comparison result includes a first result and a second result. The first result is that the element in the first position in the reconstruction quality score matrix corresponding to the time series data is greater than or equal to the element in the first position in the threshold score matrix, and the second result is that the element in the first position in the reconstruction quality score matrix corresponding to the time series data is less than the element in the first position in the threshold score matrix.

[0235] Exemplarily, when the comparison result is the first result, the element at the first position in the initial matrix is assigned a value of 0, and when the comparison result is the second result, the element at the first position in the initial matrix is assigned a value of 1. After comparing all the elements in the positions of the reconstructed quality score matrix and the threshold score matrix corresponding to the time series data, a binary matrix corresponding to the time series data is finally obtained.

[0236] S730: Obtain the first mask matrix and the second mask matrix.

[0237] Exemplarily, the embodiment of the present application obtains the first mask matrix and the second mask matrix from the storage module or unit of the double-branch coding and decoding model, wherein each element of the first mask matrix and the second mask matrix is 1 or 0, the size of the first mask matrix is the same as that of the reconstructed quality score matrix, and the size of the second mask matrix is the same as that of the reconstructed quality score matrix. The first mask matrix and the second mask matrix are matrices that are set in advance by engineers according to device characteristics and actual requirements and pre-stored in the storage module or unit.

[0238] In the embodiment of the present application, the first mask matrix is:

[0239]

[0240] The second mask matrix is:

[0241]

[0242] S740: Determine whether the time series data is abnormal based on the first mask matrix, the second mask matrix, and the binary matrix.

[0243] Exemplarily, the first logical calculation is performed on the binary matrix corresponding to the time series data and the first mask matrix to obtain a first calculation result, and then the second logical calculation is performed on the first calculation result and the second mask matrix to obtain a second calculation result. Whether the time series data is abnormal is determined according to the second calculation result.

[0244] In some embodiments of the present application, logical and calculation is performed on the binary matrix corresponding to the time series data and the first mask matrix to obtain a logical and calculation result, which is in the form of a matrix. Then, logical or calculation is performed on the logical and calculation result and the second mask matrix to obtain a logical or calculation result, which is in the form of a matrix. Finally, all elements in the logical or calculation result are added to obtain a reference sum, and if the reference sum is greater than or equal to a preset threshold value, it is determined that the time series data is abnormal, and if the reference sum is less than the preset threshold value, it is determined that the time series data is normal.

[0245] In some embodiments, determining whether the time series data is abnormal based on the first mask matrix, the second mask matrix and the binary matrix specifically includes but is not limited to the following steps S741-S746:

[0246] S741: performing a logical and calculation operation on the first mask matrix and the binary matrix to obtain a first logical and matrix.

[0247] S742: summing all elements in the first logical and matrix to obtain a first sum value.

[0248] S743: performing a logical and calculation operation on the second mask matrix and the binary matrix to obtain a second logical and matrix.

[0249] S744: summing all elements in the second logical and matrix to obtain a second sum value.

[0250] S745: determining that the time series data is abnormal in response to the first sum value being greater than or equal to a first preset threshold value or the second sum value being greater than or equal to a second preset threshold value.

[0251] S746: determining that the time series data is normal in response to the first sum value being less than the first preset threshold value and the second sum value being less than the second preset threshold value.

[0252] Specifically, performing a logical and calculation operation on the first mask matrix and the binary matrix to obtain a first logical and matrix, that is, performing a logical and calculation operation on elements at corresponding positions in the first mask matrix and the binary matrix to obtain the first logical and matrix. Summing all elements in the first logical and matrix to obtain a first sum value.

[0253] Specifically, performing a logical and calculation operation on the second mask matrix and the binary matrix to obtain a second logical and matrix, that is, performing a logical and calculation operation on elements at corresponding positions in the second mask matrix and the binary matrix to obtain the second logical and matrix. Summing all elements in the second logical and matrix to obtain a second sum value. Comparing the first sum value with the first preset threshold value and comparing the second sum value with the second preset threshold value, if the first sum value is greater than or equal to the first preset threshold value or the second sum value is greater than or equal to the second preset threshold value, determining that the time series data is abnormal, if the first sum value is less than the first preset threshold value and the second sum value is less than the second preset threshold value, determining that the time series data is normal.

[0254] For example, the binary matrix corresponding to the time series data is , the first mask matrix is , the second mask matrix is , and the first preset threshold value and the second preset threshold value are 1 and 2 respectively.

[0255] Thus, the logical and calculation operation is performed on the first mask matrix and the binary matrix to obtain a first logical and matrix as follows: The sum of all elements in the first logical and matrix is 2.

[0256] Thus, the logical and calculation operation is performed on the second mask matrix and the binary matrix to obtain a second logical and matrix as follows: The sum of all elements in the second logical and matrix is 2.

[0257] Obviously, through comparison, it can be known that the first sum value 2 is greater than the first preset threshold value 1, and the second sum value 2 is equal to the second preset threshold value 2, and it is determined that the time series data is abnormal.

[0258] S750: In response to the fact that any one of all the time series data is abnormal, it is determined that the industrial equipment is abnormal.

[0259] S760: In response to the fact that all the time series data is normal, it is determined that the industrial equipment is normal.

[0260] Exemplarily, after it is determined whether all the time series data is abnormal, it is determined whether the industrial equipment is abnormal according to the result of whether the time series data is abnormal, that is, if any one of all the time series data is abnormal, it is determined that the industrial equipment is abnormal, and if all the time series data is normal / does not exist abnormal, it is determined that the industrial equipment is normal.

[0261] As described above, it can be known that the industrial equipment abnormality detection method provided by the embodiment of the present application is applied to an electronic device, the electronic device includes a double-branch coding and decoding model, the double-branch coding and decoding model includes a dynamic abnormality processing module and a static abnormality processing module, the embodiment of the present application acquires the multi-segment time series data sensed by the sensor equipped in the industrial equipment and pre-processes the time series data to obtain standard data, and processes the standard data by using the dynamic abnormality processing module and the static abnormality processing module to obtain a time-frequency image and a state transition image, realizes double detection of dynamic abnormality and static abnormality of the equipment, reduces the possibility of missing report of static abnormality or dynamic abnormality, reconstructs the time-frequency image and the state transition image by using the double-branch coding and decoding model to obtain a first reconstructed image and a second reconstructed image, calculates a reconstructed quality score matrix corresponding to the time series data according to the first reconstructed image, the second reconstructed image, the time-frequency image and the state transition image, and determines whether the industrial equipment is abnormal according to the reconstructed quality score matrix corresponding to all the time series data, which can accurately and efficiently process the detected dynamic abnormality and static abnormality, thereby accurately judging whether the industrial equipment is abnormal and improving the accuracy of industrial equipment abnormality detection.

[0262] The embodiment of the present application provides a computer readable storage medium, and computer readable storage medium has computer program instructions executable by a processor stored thereon, and the computer program instructions are executed by the processor to make the computer execute the industrial equipment anomaly detection method provided by the embodiment of the present application, or execute the steps in any one of possible implementation manners of the industrial equipment anomaly detection method provided by the embodiment of the present application.

[0263] Those skilled in the art can understand that the embodiments provided by the present application are only illustrative, and the writing order of each step in the method of the embodiments does not mean a strict execution order and constitute any limitation on the implementation process, and can be adjusted, combined and deleted according to actual needs. The modules or sub-modules, units or sub-units and the like in the device or system of the embodiments can be combined, divided and deleted according to actual needs. For example, the division of the unit is only a logical function division, and another division mode can also be used in actual implementation. For another example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0264] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and a general hardware platform, and of course, can also be realized by hardware. Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiment of each method can be included. It should be understood that the storage medium can be a flash memory, a hard disk, an optical disk, a register, a magnetic surface memory, a removable disk, a CD-ROM, a random access memory (RAM), a read-only memory (ROM), an electrically programmable ROM and an electrically erasable programmable ROM, etc.

[0265] It should be noted that the above embodiments are used to illustrate the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and implement it accordingly, and cannot limit the protection scope of the present application. Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments are implemented, and the technical solutions described in the embodiments of the present application are modified, or some technical features are replaced. It can be understood that these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and should be regarded as equivalent changes and modifications based on the embodiments of the present application, and should belong to the scope covered by the claims of the present application.

Claims

1. An industrial equipment anomaly detection method applied to an electronic device, characterized by, The electronic device includes a double-branch coding and decoding model, the double-branch coding and decoding model includes a dynamic exception processing module and a static exception processing module, and the method includes: Obtaining target data, the target data including multiple time series data, the time series data being sensor data detected by a sensor equipped on the industrial equipment; Pretreating the time series data to obtain standard data; Inputting the standard data into the dynamic exception processing module for processing to obtain a time-frequency image; Inputting the standard data into the static exception processing module for processing to obtain a state transition image; Inputting the time-frequency image and the state transition image into the double-branch coding and decoding model for reconstruction to obtain a first reconstructed image and a second reconstructed image; Based on the first reconstructed image, the second reconstructed image, the time-frequency image, and the state transition image, obtaining a reconstruction quality score matrix corresponding to the time series data; Based on the reconstruction quality score matrices corresponding to all the time series data, determining whether the industrial equipment is abnormal.

2. The method of claim 1, wherein, The time series data includes raw data of multiple acquisition channels, and the pretreating the time series data to obtain standard data includes: Performing mean-variance normalization processing on the raw data of each acquisition channel in the time series data to obtain initial data of each acquisition channel; Performing error rounding processing on the initial data of each acquisition channel to obtain reference data of each acquisition channel, and the standard data includes the reference data of all the acquisition channels.

3. The method of claim 2, wherein, The inputting the standard data into the dynamic exception processing module for processing to obtain a time-frequency image includes: Performing linear interpolation scaling on first candidate data to obtain first reference data, wherein the first candidate data is reference data of any acquisition channel in the standard data input into the dynamic exception processing module; Performing processing on all the first reference data by using a time-frequency analysis algorithm to obtain the time-frequency image, and the number of channels of the time-frequency image is the same as the number of the acquisition channels.

4. The method of claim 2, wherein, The inputting the standard data into the static exception processing module for processing to obtain a state transition image includes: Performing filtering and down-sampling on second candidate data to obtain second reference data, wherein the second candidate data is reference data of any acquisition channel in the standard data input into the static exception processing module; Performing processing on all the second reference data by using a state transition analysis algorithm to obtain the state transition image, and the number of channels of the state transition image is the same as the number of the acquisition channels.

5. The method of claim 1, wherein, The double-branch coding and decoding model includes a first reconstruction network and a second reconstruction network, the first reconstruction network includes a first encoding module and a first decoding module, the second reconstruction network includes a second encoding module and a second decoding module, and the inputting the time-frequency image and the state transition image into the double-branch coding and decoding model for reconstruction to obtain a first reconstructed image and a second reconstructed image includes: Inputting the time-frequency image into the first encoding module for down-sampling and feature mapping to obtain a first candidate vector; The first candidate vector is input into the first decoding module for feature transformation and up-sampling to obtain a first reconstructed image; The state transition image is input into the second encoding module for down-sampling and feature mapping to obtain a second candidate vector; The second candidate vector is input into the second decoding module for feature transformation and up-sampling to obtain a second reconstructed image.

6. The method of claim 5, wherein, The first encoding module comprises a first encoder and a first feature mapping layer, and the inputting of the time-frequency image into the first encoding module for down-sampling and feature mapping to obtain a first candidate vector comprises: The time-frequency image is input into the first encoder for down-sampling to obtain a first feature map set, wherein the first feature map set comprises a first reference feature map, and the first reference feature map is a feature map with the smallest resolution in the down-sampling process of the time-frequency image by the first encoder; The first reference feature map is flattened by the first encoder to obtain a first flattened vector; The first flattened vector is input into the first feature mapping layer, and the first flattened vector is mapped to a normal distribution by the first feature mapping layer to obtain a first parameter, wherein the first parameter comprises a mean value and a standard deviation of the first normal distribution; A first target vector is obtained; A first candidate vector is generated based on the first target vector and the first parameter.

7. The method of claim 6, wherein, The first decoding module comprises a first feature transformation layer and a first decoder, and the first reconstructed network further comprises a first connection layer, the first feature map set further comprises a first reference feature map, the first reference feature map is a feature map with a resolution greater than a first resolution threshold in the down-sampling process of the time-frequency image by the first encoder, and the resolution of the first reference feature map is greater than that of the first reference feature map; The inputting of the first candidate vector into the first decoding module for feature transformation and up-sampling to obtain a first reconstructed image comprises: The first candidate vector is input into the first feature transformation layer for transformation to obtain a first initial feature map; The first initial feature map is input into the first decoder for up-sampling to obtain a first up-sampled feature map; The first reference feature map and the first up-sampled feature map are fused by the first connection layer to obtain a first fused feature map; The first fused feature map is decoded by the first decoder to obtain the first reconstructed image.

8. The method of claim 5, wherein, The second encoding module comprises a second encoder and a second feature mapping layer, and the inputting of the state transition image into the second encoding module for down-sampling and feature mapping to obtain a second candidate vector comprises: The state transition image is input into the second encoder for down-sampling to obtain a second reference feature map, and the second reference feature map is a feature map with the smallest resolution in the down-sampling process of the state transition image by the second encoder; The second reference feature map is flattened by the second encoder to obtain a second flattened vector; inputting the second flattened vector into the second feature mapping layer, mapping the second flattened vector to a normal distribution by using the second feature mapping layer to obtain a second parameter, the second parameter comprising a mean and a standard deviation of the second normal distribution; obtaining a second target vector; generating a second candidate vector based on the second target vector and the second parameter.

9. The method of claim 8, wherein, The second decoding module comprises a second feature transformation layer and a second decoder, and the second candidate vector is inputted into the second decoding module for feature transformation and up-sampling to obtain a second reconstructed image, comprising: inputting the second candidate vector into the second feature transformation layer for transformation to obtain a second initial feature map; inputting the second initial feature map into the second decoder for up-sampling to obtain a second up-sampled feature map; decoding the second up-sampled feature map by using the second decoder to obtain the second reconstructed image.

10. The method according to any one of claims 1 to 9, characterized in that, The number of channels of the first reconstructed image is the same as that of the time-frequency image, and the number of channels of the second reconstructed image is the same as that of the state transition image, and the reconstructed quality score matrix corresponding to the time series data is obtained based on the first reconstructed image, the second reconstructed image, the time-frequency image and the state transition image, comprising: extracting the feature map of each channel in the first reconstructed image, the second reconstructed image, the time-frequency image and the state transition image to obtain the first reconstructed feature map, the second reconstructed feature map, the state transition feature map and the time-frequency feature map of each channel; calculating the similarity of the first reconstructed feature map and the time-frequency feature map of each channel to obtain the time-frequency reconstruction score of each channel; calculating the similarity of the second reconstructed feature map and the state transition feature map of each channel to obtain the state transition reconstruction score of each channel; based on the time-frequency reconstruction scores and the state transition reconstruction scores of all channels, the reconstructed quality score matrix corresponding to the time series data is constructed.

11. The method of claim 10, wherein, The reconstructed quality score matrix comprises the time-frequency reconstruction scores and the state transition reconstruction scores of all channels, and based on the reconstructed quality score matrices corresponding to all the time series data, it is determined whether the industrial equipment is abnormal, comprising: based on the reconstructed quality score matrices corresponding to all the time series data, a threshold score matrix is obtained, the size of the threshold score matrix being the same as that of the reconstructed quality score matrix; based on the threshold score matrix and the reconstructed quality score matrix corresponding to the time series data, a binary matrix corresponding to the time series data is obtained, each element of the binary matrix being 1 or 0; obtaining a first mask matrix and a second mask matrix, wherein each element of the first mask matrix and the second mask matrix is 1 or 0, the size of the first mask matrix and the size of the second mask matrix being the same as the size of the reconstructed quality score matrix; based on the first mask matrix, the second mask matrix and the binary matrix, it is determined whether the time series data is abnormal; in response to the fact that any one of all the time series data is abnormal, it is determined that the industrial equipment is abnormal; In response to all the time series data being normal, it is determined that the industrial equipment is normal.

12. The method of claim 11, wherein, The threshold score matrix is obtained based on the reconstruction quality score matrix corresponding to all the time series data, including: An original matrix is constructed, and the size of the original matrix is the same as that of the reconstruction quality score matrix; A target calculation operation is performed on the elements in the corresponding positions of the reconstruction quality score matrix corresponding to all the time series data to obtain target values corresponding to the positions, and the target calculation operation is any one of taking the mean value, taking the median, taking the maximum value, and taking the minimum value; The target values corresponding to all positions are filled into the corresponding positions of the original matrix to obtain the threshold score matrix.

13. The method of claim 11, wherein, The binary matrix corresponding to the time series data is obtained based on the threshold score matrix and the reconstruction quality score matrix corresponding to the time series data, including: An initial matrix is constructed, and the size of the initial matrix is the same as that of the reconstruction quality score matrix; The two elements in the first position of the reconstruction quality score matrix corresponding to the time series data and the threshold score matrix are compared to obtain a comparison result, and the first position is the position corresponding to the i-th row and the j-th column in the reconstruction quality score matrix or the threshold score matrix; According to the comparison result, the element in the first position of the initial matrix is assigned as 1 or 0 to obtain the binary matrix corresponding to the time series data.

14. The method of claim 11, wherein, The time series data is determined to be abnormal based on the first mask matrix, the second mask matrix, and the binary matrix, including: A first logical and matrix is obtained by performing a logical and calculation operation on the first mask matrix and the binary matrix; A first sum value is obtained by summing all elements in the first logical and matrix; A second logical and matrix is obtained by performing a logical and calculation operation on the second mask matrix and the binary matrix; A second sum value is obtained by summing all elements in the second logical and matrix; In response to the first sum value being greater than or equal to a first preset threshold value, or the second sum value being greater than or equal to a second preset threshold value, it is determined that the time series data is abnormal; In response to the first sum value being less than the first preset threshold value and the second sum value being less than the second preset threshold value, it is determined that the time series data is normal.

15. An electronic device, comprising: It includes: A controller and a double-branch coding and decoding model in communication connection with the controller; The double-branch coding and decoding model includes a dynamic anomaly processing module, a static anomaly processing module, a first reconstruction network, and a second reconstruction network, wherein the dynamic anomaly processing module, the static anomaly processing module, the first reconstruction network, and the second reconstruction network are all in communication connection with the controller; The first reconstruction network includes a first encoding module, a first decoding module, and a first connection layer, the first encoding module includes a first encoder and a first feature mapping layer, and the first decoding module includes a first feature transformation layer and a first decoder; The second reconstruction network comprises a second encoding module and a second decoding module, the second encoding module comprises a second encoder and a second feature mapping layer, and the second decoding module comprises a second feature transformation layer and a second decoder; The controller comprises: a processor and a memory connected to the processor in communication; The memory stores computer program instructions executable by the processor, and the computer program instructions, when executed by the processor, cause the controller to perform the industrial equipment anomaly detection method according to any one of claims 1-14.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions executable by the processor, and the computer program instructions, when executed by the processor, cause the computer to perform the industrial equipment anomaly detection method according to any one of claims 1-14.

Images