Industrial equipment state monitoring and anomaly detection method and system

Abstract

The invention discloses an industrial equipment state monitoring and anomaly detection method and system, and belongs to the technical field of computers. The method comprises the following steps: an edge end performs feature extraction processing on a real-time monitoring image of target industrial equipment to obtain a feature map of a first granularity, and sends the feature map of the first granularity to a cloud end; the cloud side judges whether the target industrial equipment is normal or not based on the feature map of the first granularity, and sends a granularity adjustment instruction to the edge side when the judgment result is that the target industrial equipment is abnormal; and the edge end responds to the granularity adjustment instruction, performs feature extraction processing on the real-time monitoring image of the target industrial equipment, obtains a feature map of a second granularity, and sends the feature map of the second granularity to the cloud end. According to the industrial equipment state monitoring and anomaly detection method and system provided by the invention, data transmission is optimized and resource consumption is calculated by adopting hierarchical sampling, and the sampling precision of the image is dynamically adjusted, so that the response speed and real-time performance of monitoring and anomaly detection can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer technology, and particularly relates to an industrial equipment state monitoring and abnormality detection method and system. BACKGROUND

[0002] At present, for the state monitoring and abnormality detection of industrial equipment, an artificial intelligence model is generally used to analyze a monitoring image to determine whether the industrial equipment is abnormal. However, in an industrial environment, the existing method has a delay when processing real-time data, and cannot immediately respond to sudden equipment failure or changes. Therefore, the existing industrial equipment state monitoring and abnormality detection method has the disadvantage of poor real-time performance. SUMMARY

[0003] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application provides an industrial equipment state monitoring and abnormality detection method and system, which can improve the response speed and real-time performance of industrial equipment state monitoring and abnormality detection.

[0004] In a first aspect, the present application provides an industrial equipment state monitoring and abnormality detection method, which comprises:

[0005] The edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment, obtains a first-granularity feature map, and sends the first-granularity feature map to the cloud end;

[0006] The cloud end determines whether the target industrial equipment is normal based on the first-granularity feature map, and sends a granularity adjustment instruction to the edge end in the case where the determination result is that the target industrial equipment is abnormal;

[0007] The edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment in response to the granularity adjustment instruction, obtains a second-granularity feature map, and sends the second-granularity feature map to the cloud end; the second granularity is a finer granularity than the first granularity, and the first granularity is not the finest granularity.

[0008] According to the industrial equipment state monitoring and anomaly detection method, the edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment, obtains a feature map of a first granularity, and sends the feature map of the first granularity to the cloud end. The cloud end determines whether the target industrial equipment is normal based on the feature map of the first granularity, and sends a granularity adjustment instruction to the edge end in a case where the determination result is that the target industrial equipment is abnormal. The edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment in response to the granularity adjustment instruction, obtains a feature map of a second granularity that is finer than the first granularity, and sends the feature map of the second granularity to the cloud end. The hierarchical sampling optimization is used to optimize data transmission and computing resource consumption, and the sampling precision and the amount of calculation of the image are dynamically adjusted between the edge end and the cloud end, so that the operation efficiency, the monitoring quality and the real-time performance of the entire system are improved.

[0009] According to an embodiment of the present application, after the edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment in response to the granularity adjustment instruction, obtains a feature map of a second granularity and sends the feature map to the cloud end, the method further comprises:

[0010] The cloud end sends an alarm information in a case where the results of determining whether the target industrial equipment is normal based on the feature maps of all granularities are all abnormal.

[0011] According to an embodiment of the present application, before the edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment and obtains a first granularity, the method further comprises:

[0012] The edge end receives an EfficientNetV2-s model and converts the EfficientNetV2-s model into an IR format based on OpenVINO.

[0013] According to an embodiment of the present application, the edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment and obtains a feature map of a first granularity, comprising:

[0014] The edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment based on the EfficientNetV2-s model and obtains a feature map of a first granularity.

[0015] According to an embodiment of the present application, the cloud end determines whether the target industrial equipment is normal based on the feature map of the first granularity, comprising:

[0016] The cloud end performs feature extraction processing on the feature maps of all granularities that have been obtained based on a flow model and obtains a fusion feature of the first granularity;

[0017] The cloud judges whether the target industrial equipment is normal based on the first-granularity fused features.

[0018] According to an embodiment of the present application, before the cloud performs feature extraction processing on the acquired feature maps of each granularity based on the flow model to acquire the first-granularity features, the method further comprises:

[0019] The flow model is trained based on the sample feature maps of each granularity uploaded by the edge terminal under the normal running state of the target industrial equipment, and a fitter is generated; the fitter is used to convert the image feature distribution represented by the sample feature maps into a standard normal distribution.

[0020] According to an embodiment of the present application, the cloud judges whether the target industrial equipment is normal based on the first-granularity features, comprising:

[0021] The cloud acquires the confidence of each position in the first-granularity feature map based on the first-granularity feature map.

[0022] The cloud determines that the target industrial equipment is abnormal if there is a position with a confidence greater than a threshold corresponding to the first granularity.

[0023] According to an embodiment of the present application, the cloud issues an alarm information if the results of judging whether the target industrial equipment is normal based on the feature maps of all granularities are all abnormal, comprising:

[0024] The cloud acquires the confidence of each position in the first-granularity feature map based on the first-granularity feature map.

[0025] The cloud determines that the position with a confidence greater than a threshold corresponding to the first granularity in the first-granularity feature map is abnormal if there is a position with a confidence greater than a threshold corresponding to the first granularity.

[0026] The cloud issues the alarm information; the alarm information carries information of the abnormal position.

[0027] According to an embodiment of the present application, the cloud and the edge terminal communicate with each other by using a control protocol based on HTTPS protocol, and the cloud and the edge terminal transmit data in JSON format.

[0028] In a second aspect, the present application provides an industrial equipment state monitoring and abnormality detection system, which comprises a cloud and at least one edge terminal.

[0029] The edge end is configured to perform feature extraction processing on the real-time monitoring image of the target industrial equipment, obtain a first-granularity feature map, and send the first-granularity feature map to the cloud end.

[0030] The cloud end is configured to determine whether the target industrial equipment is normal based on the first-granularity feature map, and send a granularity adjustment instruction to the edge end in a case where the determination result is that the target industrial equipment is abnormal.

[0031] The edge end is further configured to perform feature extraction processing on the real-time monitoring image of the target industrial equipment in response to the granularity adjustment instruction, obtain a second-granularity feature map, and send the second-granularity feature map to the cloud end, where the second granularity is finer than the first granularity and the first granularity is not the finest granularity.

[0032] According to the industrial equipment state monitoring and abnormality detection system provided in the present application, the edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment, obtains a first-granularity feature map, and sends the first-granularity feature map to the cloud end, the cloud end determines whether the target industrial equipment is normal based on the first-granularity feature map, and sends a granularity adjustment instruction to the edge end in a case where the determination result is that the target industrial equipment is abnormal, the edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment in response to the granularity adjustment instruction, obtains a second-granularity feature map that is finer than the first-granularity feature map, and sends the second-granularity feature map to the cloud end, the hierarchical sampling optimization is used to optimize data transmission and computing resource consumption, and the sampling precision and the amount of calculation of the image are dynamically adjusted between the edge end and the cloud end, so that the operation efficiency, the monitoring quality, and the real-time performance of the entire system are improved.

[0033] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the industrial equipment state monitoring and abnormality detection method according to the first aspect.

[0034] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the industrial equipment state monitoring and abnormality detection method according to the first aspect.

[0035] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or an instruction to implement the industrial equipment state monitoring and abnormality detection method according to the first aspect.

[0036] In a sixth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the industrial equipment state monitoring and anomaly detection method according to the first aspect above.

[0037] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0039] Figure 1 Fig. 1 is one of flow diagrams of the industrial equipment state monitoring and anomaly detection method according to an embodiment of the present application;

[0040] Figure 2 Fig. 2 is another of flow diagrams of the industrial equipment state monitoring and anomaly detection method according to an embodiment of the present application;

[0041] Figure 3 Fig. 3 is a diagram of the data processing process of the flow model in the industrial equipment state monitoring and anomaly detection method according to an embodiment of the present application;

[0042] Figure 4 Fig. 4 is a diagram of the feature fusion process in the industrial equipment state monitoring and anomaly detection method according to an embodiment of the present application;

[0043] Figure 5 Fig. 5 is a diagram of the structure of the industrial equipment state monitoring and anomaly detection apparatus according to an embodiment of the present application;

[0044] Figure 6 Fig. 6 is a diagram of the structure of the electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0046] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of a kind and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates that the objects before and after are in an "or" relationship.

[0047] In related technologies, an industrial visual monitoring system is a system for monitoring the state and detecting abnormalities of industrial equipment. The industrial visual monitoring system can capture images of the industrial equipment using a monocular camera and / or a multi-camera (such as a binocular camera), identify the images through a visual algorithm, and determine whether the industrial equipment is normal.

[0048] The industrial visual monitoring system mainly uses supervised learning technology, and needs a large number of labeled defect pictures (i.e., the industrial equipment in the picture has an abnormality) as training samples to train an AI (Artificial Intelligence) model. However, the acquisition of a large number of labeled defect pictures is a major problem in actual industrial applications. On the one hand, for new equipment or rare defect types, it is often difficult to obtain enough training samples; on the other hand, even if there are enough training samples, the training and updating of the AI model also require a large amount of time and computing resources.

[0049] The traditional industrial visual monitoring system is highly dependent on a large amount of labeled data, and has strong data dependency. The disadvantages of strong data dependency are a major challenge for variable industrial environments, especially in scenarios where the model of industrial equipment is rapidly updated or the production types are diversified.

[0050] The traditional industrial visual monitoring system has a large delay in processing real-time data, and cannot respond to sudden equipment failures or changes in real time, which is insufficient in real-time performance and flexibility, and the application effect in emergency situations is limited.

[0051] In addition, due to the dependence on a large number of training samples, not only is the initial investment large, but also the maintenance and updating cost is high. In the case of handling new field industrial equipment, new type industrial equipment or special type of fault, the AI model often needs to be retrained, which is inefficient.

[0052] The industrial equipment state monitoring and abnormality detection method, the industrial equipment state monitoring and abnormality detection device, the electronic device and the readable storage medium provided by the embodiments of the present application will be described in detail in combination with the drawings and specific embodiments and their application scenarios.

[0053] Among them, the industrial equipment status monitoring and anomaly detection method can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0054] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0055] In the following embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse, and a joystick.

[0056] The industrial equipment status monitoring and anomaly detection method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the industrial equipment status monitoring and anomaly detection method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The industrial equipment status monitoring and anomaly detection method provided in the embodiment of the present application is described below using an electronic device as an example of the execution subject.

[0057] like Figure 1 As shown, the industrial equipment status monitoring and anomaly detection method includes: step 110, step 120 and step 130.

[0058] In practice, the industrial equipment status monitoring and anomaly detection method provided in the embodiments of this application involves both the edge and the cloud. The edge monitors the status of industrial equipment, collects real-time monitoring images, and extracts features from these images. The cloud detects whether the industrial equipment is abnormal based on the features extracted by the edge.

[0059] Step 110: The edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment, obtains a feature map of the first granularity, and sends the feature map of the first granularity to the cloud.

[0060] In actual implementation, the edge end can be provided with an image sensor. The edge end can collect image data of the target industrial equipment in real time through the image sensor (e.g., a monocular camera, etc.) to obtain a real-time monitoring image of the target industrial equipment. For the obtained real-time monitoring image, the edge end can perform feature extraction based on a visual algorithm to extract image features of a current granularity (hereinafter referred to as “features”) to obtain a feature map of the current granularity. The current granularity can be denoted as a first granularity, and accordingly, the edge end extracts features of the first granularity to obtain a feature map of the first granularity.

[0061] It should be noted that which granularity of features the edge end extracts can be controlled by the cloud end. In some embodiments, in the case where the cloud end does not send any granularity adjustment instruction, the edge end can extract features of the coarsest granularity by default.

[0062] After the edge end obtains the feature map of the first granularity, the edge end can send the feature map of the first granularity to the cloud end. It should be noted that the edge end does not send the real-time monitoring image to the cloud end, but sends the features extracted from the real-time monitoring image to the cloud end in the form of a feature map. The data amount of the feature map is smaller than that of the real-time monitoring image from which the feature map is derived, which can reduce the bandwidth requirement and delay of transmission and is more real-time.

[0063] In some embodiments, before the edge end performs feature extraction on the real-time monitoring image of the target industrial equipment, the edge end can also perform instantaneous object false alarm exclusion based on a sliding time window. In the edge end, the appearance of instantaneous objects (such as passing pedestrians or flying insects, etc.) can cause the cloud end to issue unexpected and false abnormal alarms, which can reduce the detection accuracy. The instantaneous object false alarm exclusion based on the sliding time window in the edge end can effectively reduce or even avoid the unexpected and false abnormal alarms, thereby ensuring the detection accuracy and improving the robustness and performance of the system.

[0064] In some embodiments, the edge end can use a dynamic image analysis method based on a sliding time window and a peak signal-to-noise ratio (PSNR) to perform instantaneous object false alarm exclusion, which can improve the accuracy of instantaneous object detection and reduce false alarms. The most recent k frames of images can be tracked and stored based on a preset sliding time window. Every fixed T seconds, the real-time monitoring image at the current time is collected and added to the time window, and the oldest frame of image is removed. For the real-time monitoring image at the current time, the average peak signal-to-noise ratio PSNR(I, K i ) of the real-time monitoring image at the current time and the frames of images in the sliding time window is obtained. The calculation formula can be as follows:

[0065]

[0066] wherein, MSE(I, K i) is the mean square error of the real-time monitoring image I at the current moment and the reference frame K i MAX I is the maximum pixel value of the real-time monitoring image I at the current moment and the reference frame K i is the i-th frame image in the current time window except the real-time monitoring image I at the current moment.

[0067] In some embodiments, in the case that the average peak signal-to-noise ratio PSNR(I, K i ) is greater than a preset threshold, it indicates that the real-time monitoring image at the current moment does not contain transient objects, and subsequent feature extraction operations can be performed and then transmitted to the cloud; otherwise, it indicates that the real-time monitoring image at the current moment contains transient objects, and the real-time monitoring image at the current moment is discarded. The specific value of the preset threshold is not limited in the embodiments of the present application. For example, the preset threshold can be 25 to 30, etc.

[0068] Step 120, the cloud judges whether the target industrial equipment is normal based on the first granularity feature map, and sends a granularity adjustment instruction to the edge terminal in the case that the judgment result is that the target industrial equipment is abnormal.

[0069] In actual execution, after the cloud receives the first granularity feature map sent by the edge terminal, it can detect whether the target industrial equipment is normal based on the first granularity feature map. The cloud can use any artificial intelligence technology to detect whether the target industrial equipment is normal based on the first granularity feature map.

[0070] In the case that the target industrial equipment is determined to be abnormal based on the first granularity feature map, the cloud can send a granularity adjustment instruction to the edge terminal. The granularity adjustment instruction is used to instruct the edge terminal to perform feature extraction with finer granularity.

[0071] Step 130, the edge terminal performs feature extraction processing on the real-time monitoring image of the target industrial equipment in response to the granularity adjustment instruction, obtains a second granularity feature map, and sends the second granularity feature map to the cloud; the second granularity is finer than the first granularity and the first granularity is not the finest granularity.

[0072] In actual execution, after the edge terminal receives the granularity adjustment instruction, it can adjust the current granularity to a finer granularity. The current granularity at this time can be denoted as the second granularity. It can be understood that the second granularity is finer than the first granularity.

[0073] In some embodiments, the second granularity can be one level finer than the first granularity.

[0074] After the edge terminal receives the granularity adjustment instruction, it adjusts the current granularity to a finer granularity, which is a process of state switching. The current granularity can correspond to the detection state one by one. For example, as shown in FIG. 1, the first granularity corresponds to the first detection state, the second granularity corresponds to the second detection state, and the third granularity corresponds to the third detection state.Figure 2 As shown, in the case that feature extraction of three granularities (coarse granularity, medium granularity and fine granularity respectively) can be performed at the edge end, the edge end is in a coarse granularity detection state, a medium granularity detection state and a fine granularity detection state, and the current granularity is coarse granularity, medium granularity and fine granularity respectively. Correspondingly, the edge end sends a feature map of coarse granularity, a feature map of medium granularity and a feature map of fine granularity to the cloud end respectively.

[0075] After the edge end adjusts the current granularity to the second granularity, the edge end can collect a real-time monitoring image of a scale corresponding to the second granularity, and then perform feature extraction processing on the real-time monitoring image to extract features of the second granularity and obtain a feature map of the second granularity.

[0076] It should be noted that the embodiments of the present application adopt hierarchical sampling optimization. The hierarchical sampling optimization can include two aspects: edge end sampling optimization and cloud end computing optimization.

[0077] In the edge end sampling optimization of the hierarchical sampling optimization, the edge end can collect image data of different scales through hardware with different precision parameters, or collect fine scale images and then apply a bilinear interpolation technique for down-sampling to other scales. Through the above process, a real-time monitoring image of a scale corresponding to the second granularity can be obtained.

[0078] As shown in FIG. 6, in the case that feature extraction of three granularities (coarse granularity, medium granularity and fine granularity respectively) can be performed at the edge end, the edge end is in a coarse granularity detection state, a medium granularity detection state and a fine granularity detection state, and the current granularity is coarse granularity, medium granularity and fine granularity respectively. Correspondingly, the edge end sends a feature map of coarse granularity, a feature map of medium granularity and a feature map of fine granularity to the cloud end respectively. Figure 2 As shown, in the conventional monitoring state, the edge end first collects a real-time monitoring image of coarse scale and extracts features of coarse granularity, and then the features of coarse granularity are transmitted to the cloud end for preliminary abnormality analysis. If the cloud end detects an abnormality based on the features of coarse granularity, the edge end is sent a granularity adjustment instruction for coarse granularity abnormality prompt. The edge end can switch states according to the granularity adjustment instruction and switch to collect a real-time monitoring image of medium scale and extract features of medium granularity, and then transmit the features of medium granularity to the cloud end for more in-depth analysis. If the cloud end still detects an abnormality based on the features of medium granularity, the edge end is sent a granularity adjustment instruction for medium granularity abnormality prompt. The edge end can switch states according to the granularity adjustment instruction and further collect a real-time monitoring image of fine scale and extract features of fine granularity, and then transmit the features of fine granularity to the cloud end for final judgment. The above strategy not only improves the efficiency of data transmission and calculation, but also optimizes resource utilization and response speed in the monitoring process by seamlessly switching between different scales.

[0079] In some embodiments, the edge end can use a finite state machine to determine a specific sampling strategy to effectively adapt to different monitoring requirements.

[0080] In the cloud computing optimization of hierarchical sampling optimization, under normal operation state, the cloud mainly processes coarse-grained features to determine whether an anomaly occurs. In the case that potential anomalies are captured based on coarse-grained features (i.e., coarse-grained detection of anomalies), state switching can be performed, and further introduction of medium-grained features for analysis can be performed. Since the detection results based on coarse-grained features in the AI model are not affected by the medium-grained features, the detection results based on coarse-grained features can be frozen, and only the medium-grained features are processed to determine whether the feature distribution exceeds the anomaly threshold. If the medium-grained features also indicate anomalies (i.e., medium-grained detection of anomalies), state switching can be performed, and further introduction of fine-grained features for analysis can be performed. Similarly, the AI model can analyze only the fine-grained features based on the detection results based on coarse-grained features and medium-grained features to determine whether the fine-grained features show an abnormal distribution exceeding the threshold. Only in the case that the fine-grained features also show an abnormal state (i.e., fine-grained detection of anomalies), the anomaly can be formally reported and subsequent anomaly capturing steps can be performed. This hierarchical progressive detection method can achieve optimized use of network resources of the AI model, improve processing efficiency and accuracy.

[0081] It should be noted that, although Figure 2 The device state monitoring and anomaly detection process in the case of three granularities is shown, but those skilled in the art can understand the device state monitoring and anomaly detection process in the case of two granularities, four granularities, or more than four granularities on the basis of Figure 2 It should be noted that, although

[0082] It should be noted that an intelligent image sampling and feature transmission strategy is adopted in the embodiments of the present application. The strategy intelligently selects the amount and type of data to be transmitted (determined according to the granularity, and the feature map is transmitted instead of real-time monitoring images), which can not only ensure the efficiency of data transmission, but also significantly reduce the consumption of transmission bandwidth and computing resources. Based on this strategy, not only the response speed and accuracy of the monitoring system can be maintained, but also the performance of the entire monitoring system can be optimized.

[0083] It should be noted that the deployment of target industrial equipment is very flexible, and it can be in a relatively remote place. The data processing capability of the edge end is generally limited. The edge end only performs image acquisition and feature extraction, and can fully utilize the data processing capability of the edge end. More complex detection processes are performed by the cloud with stronger data processing capability, and the edge end transmits feature maps to the cloud instead of real-time monitoring images, which can effectively reduce the delay of processing real-time data and achieve immediate reaction to sudden equipment failures or changes.

[0084] According to the industrial equipment state monitoring and anomaly detection method provided in the embodiments of the present application, the real-time monitoring image of the target industrial equipment is subjected to feature extraction processing by the edge end, a first granularity feature map is obtained, and the first granularity feature map is sent to the cloud end. The cloud end determines whether the target industrial equipment is normal based on the first granularity feature map, and sends a granularity adjustment instruction to the edge end in the case that the determination result is that the target industrial equipment is abnormal. The edge end responds to the granularity adjustment instruction to perform feature extraction processing on the real-time monitoring image of the target industrial equipment, obtains a second granularity feature map finer than the first granularity, and sends the second granularity feature map to the cloud end. The hierarchical sampling optimization is used to optimize data transmission and computing resource consumption, and the sampling accuracy and the amount of calculation of the image are dynamically adjusted between the edge end and the cloud end, which can improve the operation efficiency, monitoring quality and real-time performance of the whole system.

[0085] In some embodiments, after the edge end responds to the granularity adjustment instruction to perform feature extraction processing on the real-time monitoring image of the target industrial equipment, obtains the second granularity feature map and sends it to the cloud end, the method further comprises: in the case that the results of determining whether the target industrial equipment is normal based on all granularity feature maps are all abnormal, the cloud end sends an alarm information.

[0086] In actual execution, the cloud end can send an alarm information in the case that the target industrial equipment is determined to be abnormal based on the feature map of the finest granularity.

[0087] It can be understood that the cloud end receives the feature map of the fine granularity, which indicates that the cloud end has determined the target industrial equipment to be abnormal based on the feature maps of other granularities.

[0088] In some embodiments, sending the alarm information can be in the form of sending the alarm information to the electronic device used by the operation and maintenance personnel to report the abnormality to the operation and maintenance personnel.

[0089] According to the industrial equipment state monitoring and anomaly detection method provided in the embodiments of the present application, the cloud end sends an alarm information in the case that the results of determining whether the target industrial equipment is normal based on all granularity feature maps are all abnormal, instead of sending an alarm information based on the feature map of any granularity determining the target industrial equipment to be abnormal, which can improve the accuracy of anomaly detection and improve the robustness and performance of the system.

[0090] In some embodiments, before the edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment to obtain the first granularity, the method further comprises: the edge end receives an EfficientNetV2-s model, and converts the EfficientNetV2-s model into an IR format based on OpenVINO.

[0091] In actual implementation, the edge end can perform fast and accurate image feature extraction based on the EfficientNetV2-s model. Efficient feature extraction at the edge end can improve the reliability and real-time performance of the system, meeting the dual requirements of speed and accuracy for industrial equipment state monitoring and anomaly detection.

[0092] In some embodiments, the model optimizer of OpenVINO can be used for model conversion to convert the trained EfficientNetV2-s model into an Intermediate Representation (IR) format. The EfficientNetV2-s model in IR format can include an XML description file and a binary weight file to adapt to the inference requirements of specific hardware.

[0093] In some embodiments, before converting the EfficientNetV2-s model into the IR format based on OpenVINO, the corresponding driver can be pre-installed according to the hardware type of the edge end (such as an embedded device using a VPU or a mobile device using a GPU, etc.) to ensure compatibility with the OpenVINO tool suite. Subsequently, the OpenVINO tool suite can be installed on the edge end, including all necessary drivers and library files to ensure that the software environment can support the running of the EfficientNetV2-s model and optimize hardware calls. After installation is complete, the IR format file of the EfficientNetV2-s model is loaded and run, and the network is activated for feature extraction operations.

[0094] According to the industrial equipment state monitoring and anomaly detection method provided in the embodiments of the present application, the EfficientNetV2-s model is converted into the IR format by the edge end and based on OpenVINO, the EfficientNetV2-S model is optimized and deployed, efficient image feature extraction is achieved, and the final obtained feature map is transmitted instead of the original resolution real-time monitoring image, which can improve detection efficiency and reduce bandwidth requirements for uploading data to the cloud.

[0095] In some embodiments, the edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment to obtain a first-granularity feature map, including: the edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment based on the EfficientNetV2-s model to obtain a first-granularity feature map.

[0096] In actual implementation, the open-source EfficientNetV2-S model can be used to perform feature extraction on the real-time monitoring image. The EfficientNetV2-S model has the advantages of fewer parameters, fast inference speed, low hardware memory requirement, and easy deployment on Internet of Things devices. In some embodiments, the edge side can be an Internet of Things device.

[0097] The EfficientNetV2-S model in the embodiments of the present application can start with a 3x3 convolutional layer with a step size of 2 and 24 channels for preliminary image feature extraction. Then, a series of fused MBConv blocks and MBConv blocks can be used. In the first few stages (Stage 1 and Stage 2), the network uses fused MBConv blocks with a step size from 1 to 2 and a channel number increasing from 24 to 48, gradually increasing the complexity and depth of processing images. In the subsequent stages (Stage 3 to Stage 6), Squeeze-and-Excitation modules are added, with the channel number increasing from 64 to 256 and the number of layers increasing from 4 to 15, to enhance the network's ability to process images. Finally, a 1x1 convolutional layer and a pooling layer are used to obtain the required feature map, and a fully connected layer is used to output the final feature vector.

[0098] In some embodiments, the parameters of the EfficientNetV2-S model in the embodiments of the present application can come from the open-source field. For example, these parameters can be trained on the ImageNet ILSVRC2012 dataset, which contains 1.28 million training images, enabling the EfficientNetV2-S model to effectively extract features from various images in the real world.

[0099] According to the industrial equipment state monitoring and anomaly detection method provided in the embodiments of the present application, the edge side extracts image features based on the EfficientNetV2-S model, which can efficiently, quickly and accurately extract image features on the edge side, improve the reliability and real-time performance of the system, and meet the dual requirements of speed and accuracy for industrial equipment state monitoring and anomaly detection.

[0100] In some embodiments, the cloud side determines whether the target industrial equipment is normal based on the first-granularity feature map, including: the cloud side performs feature extraction processing on the acquired feature maps of each granularity based on the flow model to obtain the first-granularity fused feature.

[0101] In actual implementation, the cloud side can further perform feature extraction processing and fusion on the acquired feature map of each granularity based on the flow model to obtain the fused feature of each granularity. It can be understood that the fused feature of each granularity can include the first-granularity fused feature.

[0102] Flow models are a class of powerful generative models that learn an exact representation of complex data distributions. Flow models work by constructing a reversible mapping from a multi-dimensional Gaussian distribution to the image data. Through the unique network structure design, flow models guarantee that the transformation is reversible, i.e. each step of transformation from data space to latent space can be exactly inverted and the Jacobian determinant can be calculated. This property can guarantee that flow models can calculate the true data probability density, and can judge whether a new sample belongs to the same distribution through this density, thereby performing semi-supervised OOD (Out-Of-Distribution) detection.

[0103] In some embodiments, a novel multi-scale flow model improved on the basis of a conventional flow model can be used for semi-supervised anomaly judgment, and has multiple performance scales to balance the performance and computational load of the system.

[0104] As shown in Figure 3 , the flow model in the embodiments of the present application can include N (N can be a positive integer, Figure 3 indicated by ×N) multi-size flow units. The flow model is a multi-scale flow model. Through the multi-scale flow model, a complex mapping of image feature distribution to normal distribution can be fitted.

[0105] As shown in Figure 2 , taking a scenario including three granularities as an example, the multi-scale flow model can receive inputs of three scales of feature maps y1, y2, y3. The feature maps y1, y2, y3 can be a coarse-grained feature map, a medium-grained feature map, and a fine-grained feature map, respectively. It should be noted that if the edge end does not acquire a finer-grained feature map, the finer-grained feature map in the input of the multi-scale flow model can be regarded as 0.

[0106] In some embodiments, each input feature map can first undergo a fixed random channel permutation to disrupt the original channel order and enhance the generalization ability of the model. The permuted feature map can be divided into at least two parts Figure 3 , two parts in this embodiment, but can be divided into more parts) through uniform channel segmentation, to provide input for subsequent cross-scale convolution.

[0107] Taking division into two parts as an example, in the cross-scale convolution module, the uniformly divided part Figure 3 , the upper half structure in this embodiment, i.e. coarse-grained feature 1, medium-grained feature 1 and fine-grained feature 1) can be processed through feature fusion, and then dot product addition, and merged into another part of the feature Figure 3 , the lower half structure in this embodiment, i.e. coarse-grained feature 2, medium-grained feature 2 and fine-grained feature 2). Then the direction is reversed to perform feature fusion again (i.e. the upper half structure in Figure 3After the upper half structure is processed by feature fusion, it is multiplied, added and merged into Figure 3 Through the above process, the coarse-grained features can be merged into the medium-grained features after passing through the convolution network, and the medium-grained features can also be merged into the fine-grained features through the convolution network. Then the two parts segmented are recombined.

[0108] The multi-size flow unit can be repeatedly applied, and finally the multi-scale flow model can obtain three kinds of feature outputs: fine-grained fusion features, medium-grained fusion features and coarse-grained fusion features.

[0109] In some embodiments, the feature fusion module in the multi-size flow unit can fuse multi-scale information and support calculation amount adjustment under different requirements. As Figure 4 shown, the coarse-grained feature map can be merged into the medium-grained and fine-grained feature maps after passing through the upper convolution network, and the medium-grained feature map can be merged into the fine-grained feature map after passing through the upper convolution network. Then, each is further extracted through the convolution network. And after channel segmentation, two affine coupling vectors are obtained through two MLP networks s and b, and the two affine coupling vectors are element-wise multiplied and added to obtain the final fusion vector. The corresponding mathematical expression is y' = s(yx) ⊙ y b +b(y a ).

[0110] In some embodiments, the multi-scale flow model can use the FrEIA framework. According to Figure 3 and Figure 4 the structure shown, the corresponding network module can be constructed in the FrEIA framework. The FrEIA framework supports automatic calculation of the corresponding determinant value and the corresponding Jacobian matrix, which is crucial for the calculation in the training and inference process of the network.

[0111] The cloud judges whether the target industrial equipment is normal based on the first-grained fusion features.

[0112] In actual execution, after the normal distribution training is completed, the parameters of the multi-dimensional flow model can be fixed. In actual production lines, the evaluation function value of the output vector of the multi-scale flow model can be calculated, and it is compared with the preset threshold to determine whether an abnormality occurs.

[0113] Taking the scene shown in Figure 2 including three granularities as an example, comprehensive abnormality judgment can be performed in combination with the three-grained fusion features. In some embodiments, the evaluation function values of the three-grained fusion features can be compared with the respective thresholds, which can effectively evaluate whether the target industrial equipment to be detected has deviated from the normal operating state.

[0114] In some embodiments, the expression of the evaluation function corresponding to each granularity of features can be

[0115] Through the above process, the abnormal state can be judged and recognized in real time.

[0116] According to the industrial equipment state monitoring and abnormality detection method provided in the embodiments of the present application, the mapping and abnormality detection of multi-scale image features are performed through the multi-scale flow model, and in particular, the semi-supervised OOD detection of new samples can be realized through the reversibility of the flow model and the calculation of the Jacobian matrix, so as to improve the judgment accuracy and reliability of the flow model and reduce the difficulty of industrial deployment.

[0117] In some embodiments, the cloud performs feature extraction processing on the acquired feature maps of each granularity based on the flow model, and before acquiring the features of the first granularity, the method further comprises: training the flow model based on the sample feature maps of each granularity uploaded by the edge terminal under the normal operation state of the target industrial equipment and generating a fitter; the fitter is used to convert the image feature distribution represented by the sample feature map into a standard normal distribution.

[0118] In actual execution, the flow model can be trained first.

[0119] In the embodiments of the present application, the images of different scales of the target industrial equipment in normal operation can be collected first on the edge terminal, and image enhancement operations such as rotation and cropping can be applied; then the EfficientNetV2-s model can be used for feature extraction and deformation processing, so as to obtain normal image features for training the flow model.

[0120] Then the multi-scale flow model network can be trained based on the above normal image features, and after a preset number of iterations, a fitter can be generated. The fitter can be used to convert the image feature distribution into a standard normal distribution. The specific value of the preset number of times is not limited in the embodiments of the present application. For example, the preset number of times can be 200 to 500.

[0121] When training the multi-scale flow model network, a mathematical relationship between the vector y extracted from the features uploaded by the edge terminal and a target standard normal distribution vector z can be established wherein, In some embodiments, the loss function used can be

[0122] This normal distribution training method only requires collecting image data from devices in normal conditions and then building a normal distribution model through precise mathematical derivation, thereby achieving semi-supervised anomaly detection. Compared to traditional deep learning methods, this semi-supervised approach eliminates the need to collect difficult-to-obtain image data from abnormal conditions, enhancing the stability of model deployment and reducing deployment costs.

[0123] According to the industrial equipment status monitoring and anomaly detection method provided in the embodiment of the present application, by training the flow model based on the sample feature graphs of each granularity under the normal operation state of the target industrial equipment uploaded by the edge end, and generating a fitter for converting the image feature distribution represented by the sample feature graph into a standard normal distribution, semi-supervised anomaly detection can be achieved without the need to collect image data of abnormal states that are difficult to obtain, which can enhance the stability of model deployment and reduce deployment costs.

[0124] In some embodiments, the cloud determines whether the target industrial equipment is normal based on the feature map of the first granularity, including: the cloud obtains the confidence level of the abnormality at each position in the feature map of the first granularity based on the feature map of the first granularity.

[0125] In actual execution, let the image of the input flow model be I, and after passing through the flow model, back propagation is performed to calculate the gradient The norm of this gradient can be used to indicate the local spatial location of the anomaly. Then, Gaussian convolution and operation G are introduced to the gradient map Perform Gaussian smoothing, that is, perform The non-zero area in the result of this operation can identify the specific spatial location of the potential fault, that is, the confidence level of the abnormal location.

[0126] When there is a position where the confidence level is greater than the threshold corresponding to the first granularity, the result of determining whether the target industrial equipment is normal is determined as the target industrial equipment being abnormal.

[0127] In actual execution, if there is a location with a confidence level greater than a threshold corresponding to the first granularity, it can be determined that the target industrial equipment is abnormal.

[0128] According to the industrial equipment status monitoring and anomaly detection method provided in the embodiment of the present application, the confidence level of an anomaly occurring at each position in the feature map of the first granularity is obtained through the cloud. When there is a position with a confidence level greater than a threshold value corresponding to the first granularity, the judgment result of whether the target industrial equipment is normal is determined as an anomaly of the target industrial equipment, which can improve the accuracy of detection.

[0129] In some embodiments, the cloud sends an alarm information in a case where all the results of determining whether the target industrial equipment is normal based on the feature maps of all the granularities are abnormal, including: the cloud obtains the confidence of each position in the feature map of the finest granularity based on the feature map of the finest granularity; in a case where there is a position with a confidence greater than a threshold corresponding to the finest granularity, determining the position with the confidence greater than the threshold corresponding to the finest granularity in the feature map of the finest granularity as an abnormal position; sending an alarm information; and the alarm information carrying information of the abnormal position.

[0130] In actual implementation, the confidence map of the abnormal position can be obtained through the confidence of the abnormal position. Through the foregoing steps, the confidence of each position in the feature map of the finest granularity can be obtained. In a case where there is a position with a confidence greater than a threshold corresponding to the finest granularity, the position with the confidence greater than the threshold corresponding to the finest granularity in the feature map of the finest granularity is determined as an abnormal position, and then the abnormal position is visualized.

[0131] In a case where an abnormality is captured, visualization can be achieved by calculating the confidence map of the abnormal position, and the result of visualization is pushed to an operation and maintenance personnel. It can be understood that the confidence map of the abnormal position can be used to indicate information of the abnormal position. In some embodiments, the alarm information can carry the confidence map of the abnormal position, so as to inform the operation and maintenance personnel of a specific position of a fault of the target industrial equipment.

[0132] According to the industrial equipment state monitoring and abnormality detection method provided in the embodiments of the present application, the confidence of each position in the feature map of the finest granularity is obtained, and the abnormal position is determined based on the confidence, the alarm information carrying information of the abnormal position is sent, the probability distribution function and the gradient map technology are used to capture and locate the fault, not only the possible abnormal area can be displayed specifically, but also compared with other abnormality capturing schemes only providing an abnormality occurrence prompt, the operation and maintenance personnel can be more effectively helped to determine the specific abnormality of the equipment, and the positioning accuracy of the system to the abnormal state and the friendliness to the operation and maintenance personnel can be improved.

[0133] In some embodiments, the cloud and the edge end communicate with each other through a control protocol based on an HTTPS protocol, and data transmission between the cloud and the edge end adopts a JSON format.

[0134] In actual implementation, the foregoing hierarchical sampling optimization can also involve a control protocol and a protocol message description.

[0135] In the hierarchical sampling optimization control protocol, communication between edge devices and the cloud uses an HTTPS-based control protocol to ensure the security and integrity of data transmission. All communication packets are organized in JSON format to facilitate cross-platform processing and parsing. Each JSON message can contain necessary fields such as image_scale (indicating the image scale level, such as coarse, medium, and fine), feature_data (encoded image feature data), and status (current monitoring status or anomaly detection results).

[0136] In the protocol message description of hierarchical sampling optimization, the protocol content may include: messages from the edge to the cloud and messages from the cloud to the edge.

[0137] The messages from the edge to the cloud may mainly include image feature data (i.e., feature map), sampling size, and current monitoring status.

[0138] Each message from the edge to the cloud is a JSON object that can contain the following fields:

[0139] image_scale: indicates the sampling scale of the image. The value can be "coarse", "medium" or "fine", indicating that the granularity corresponding to the currently transmitted image features is coarse, medium or fine, respectively.

[0140] feature_data: represents the encoded image feature data, i.e., feature map;

[0141] encoding: Used to describe the encoding format of image data. When the value is "H.265", it indicates that the H.265 encoding standard is used; when the value is "TIFF", it indicates that the TIFF lossless compression encoding is used.

[0142] Both encoding methods above use Base64 encoding as a string for network transmission. Using the same granularity as the previous moment, using the H.265 encoding standard to compress the feature image data in time series before transmitting the compressed data and decompressing it in the cloud can optimize data transmission efficiency and reduce network bandwidth usage.

[0143] When the cloud detects an anomaly based on the received feature graph or needs to adjust the sampling accuracy, it will send a granularity adjustment instruction to the edge. The message from the cloud to the edge can carry the granularity adjustment instruction. The message from the cloud to the edge can also use JSON format and contain the following fields:

[0144] command: command type; e.g., "increase_scale" indicates to increase the image sampling precision, i.e., to use finer granularity;

[0145] new_scale: new image sampling scale, e.g., "medium" or "fine", specifically indicates the scale of the image to be collected next at the edge end, and can indicate the granularity of feature extraction;

[0146] reason: reason for sending the command; e.g., "detected_anomaly" indicates that an anomaly is detected.

[0147] In some embodiments, the edge end can use a heartbeat detection and reconnection strategy. In order to ensure the normal progress of hierarchical sampling optimization, the automaton state of the edge end needs to be maintained. When communicating, the cloud end will perform a heartbeat test, periodically detecting the connection state of the edge end by sending a heartbeat signal. Each edge end needs to respond to the corresponding confirmation signal after receiving the heartbeat signal. If the cloud end does not receive a response from a certain edge end for several consecutive times, it can be determined that the edge end has been disconnected.

[0148] In some embodiments, in the case where the edge end is determined to be in a disconnected state, the cloud end can automatically attempt to reestablish a connection with the edge end. Attempting to reestablish a connection with the edge end can include sending a reconnection request to the last known IP address of the edge end.

[0149] In some embodiments, the edge end can also attempt to actively reconnect to the cloud end and send the current IP address in the case of starting or detecting network recovery.

[0150] In some embodiments, the cloud end can use an exponential backoff strategy to space reconnection attempts, avoiding overloading the network when the network condition is poor. After each failed reconnection attempt, the waiting time will be increased exponentially until a successful connection is made or the maximum number of attempts is reached. Once the connection is successfully reestablished, the edge end needs to synchronize its latest state with the cloud end, including any data collected during the disconnection period. Through the above mechanism, the reconnection process can be optimized to ensure the continuity and integrity of data collection and transmission, minimize the impact of disconnection on the overall performance of the monitoring system, and ensure the real-time performance and reliability of the aforementioned optimization algorithm.

[0151] According to the industrial equipment state monitoring and anomaly detection method provided in the embodiments of the present application, the control protocol based on the HTTPS protocol is used for communication between the cloud and the edge, and the JSON format is used for data transmission between the cloud and the edge. The use of the HTTPS protocol and the JSON format data packet can improve the operation efficiency, monitoring quality and real-time performance of the system, improve the data transmission security and integrity between the edge and the cloud, increase the robustness and performance of the system, and enhance the data processing capability and cross-platform compatibility of the system.

[0152] The industrial equipment state monitoring and anomaly detection system provided in the embodiments of the present application can execute the industrial equipment state monitoring and anomaly detection method. In the embodiments of the present application, the industrial equipment state monitoring and anomaly detection system is taken as an example to illustrate the industrial equipment state monitoring and anomaly detection system provided in the embodiments of the present application.

[0153] The embodiments of the present application further provide an industrial equipment state monitoring and anomaly detection system.

[0154] As shown in Figure 5 The industrial equipment state monitoring and anomaly detection system comprises a cloud 510 and at least one edge 520.

[0155] The edge 520 is configured to perform feature extraction processing on the real-time monitoring image of the target industrial equipment, obtain a feature map of a first granularity, and send the feature map of the first granularity to the cloud 510.

[0156] The cloud is configured to determine whether the target industrial equipment is normal based on the feature map of the first granularity, and send a granularity adjustment instruction to the edge 520 in the case that the determination result is that the target industrial equipment is abnormal.

[0157] The edge 520 is further configured to perform feature extraction processing on the real-time monitoring image of the target industrial equipment in response to the granularity adjustment instruction, obtain a feature map of a second granularity, and send the feature map of the second granularity to the cloud 510. The second granularity is finer than the first granularity, and the first granularity is not the finest granularity.

[0158] According to the industrial equipment state monitoring and anomaly detection system provided in the embodiments of the present application, the edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment, obtains a first granularity feature map, and sends the first granularity feature map to the cloud end. The cloud end determines whether the target industrial equipment is normal based on the first granularity feature map, and in the case that the determination result is that the target industrial equipment is abnormal, sends a granularity adjustment instruction to the edge end. The edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment in response to the granularity adjustment instruction, obtains a second granularity feature map finer than the first granularity, and sends the second granularity feature map to the cloud end. The hierarchical sampling optimization is used to optimize data transmission and computing resource consumption, and the sampling accuracy and computational complexity of the image are dynamically adjusted between the edge end and the cloud end, which can improve the operation efficiency, monitoring quality and real-time performance of the entire system.

[0159] In some embodiments, the cloud end 510 can also be configured to send an alarm information in the case that the results of determining whether the target industrial equipment is normal based on all the granularity feature maps are abnormal.

[0160] In some embodiments, the edge end 520 can also be configured to receive the EfficientNetV2-s model and convert the EfficientNetV2-s model into an IR format based on OpenVINO.

[0161] In some embodiments, the edge end 520 can also be configured to perform feature extraction processing on the real-time monitoring image of the target industrial equipment based on the EfficientNetV2-s model, and obtain a first granularity feature map.

[0162] In some embodiments, the cloud end 510 can also be configured to perform feature extraction processing on the obtained feature maps of each granularity based on the stream model, and obtain a first granularity fusion feature; and determine whether the target industrial equipment is normal based on the first granularity fusion feature.

[0163] In some embodiments, the cloud end 510 can also be configured to train the stream model based on the sample feature maps of each granularity of the target industrial equipment in a normal operating state uploaded by the edge end 520 and generate a fitter; and the fitter is configured to convert the image feature distribution represented by the sample feature maps into a standard normal distribution.

[0164] In some embodiments, the cloud end 510 can also be configured to obtain a confidence degree of each position in the first granularity feature map based on the first granularity feature map; and in the case that there is a position with a confidence degree greater than a threshold corresponding to the first granularity, determine that the target industrial equipment is abnormal.

[0165] In some embodiments, the cloud 510 can be specifically configured to obtain a confidence degree of each position in the most fine-grained feature map in which an anomaly occurs based on the most fine-grained feature map; in a case where there is a position in which the confidence degree is greater than a threshold corresponding to the most fine-grained feature map, the position in the most fine-grained feature map in which the confidence degree is greater than the threshold corresponding to the most fine-grained feature map is determined as an abnormal position.

[0166] An alarm information is sent out, and the alarm information carries information of the abnormal position.

[0167] In some embodiments, the cloud 510 and the edge 520 communicate with each other by using a control protocol based on an HTTPS protocol, and data transmission between the cloud 510 and the edge 520 adopts a JSON format.

[0168] The industrial equipment state monitoring and anomaly detection apparatus in the embodiments of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and the like. The electronic device can also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like. The embodiments of the present application are not limited in this regard.

[0169] The industrial equipment state monitoring and anomaly detection apparatus in the embodiments of the present application can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems. The embodiments of the present application are not limited in this regard.

[0170] The industrial equipment state monitoring and anomaly detection apparatus provided in the embodiments of the present application can implement the method embodiments Figures 1 to 4 The processes implemented by the method embodiments are not repeated here to avoid repetition.

[0171] In some embodiments, as Figure 6As shown, the electronic device 600 according to the embodiment of the present application further includes a processor 610, a memory 620, and a computer program stored in the memory 620 and executable on the processor 610. The computer program is executed by the processor 610 to implement the processes of the above-mentioned industrial equipment state monitoring and anomaly detection method embodiments, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0172] It should be noted that the electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0173] The embodiment of the present application further provides a non-transitory computer readable storage medium having a computer program stored thereon. The computer program is executed by a processor to implement the processes of the above-mentioned industrial equipment state monitoring and anomaly detection method embodiments, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0174] The processor is the processor in the electronic device described in the above-mentioned embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0175] The embodiment of the present application further provides a computer program product including a computer program. The computer program is executed by a processor to implement the above-mentioned industrial equipment state monitoring and anomaly detection method.

[0176] The processor is the processor in the electronic device described in the above-mentioned embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0177] The embodiment of the present application further provides a chip including a processor and a communication interface. The communication interface is coupled to the processor. The processor is configured to run a program or an instruction to implement the processes of the above-mentioned industrial equipment state monitoring and anomaly detection method embodiments, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0178] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system-level chip, a system chip, a chip system, or a system-on-chip, etc.

[0179] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0180] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0181] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0182] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0183] While the embodiments of the application have been shown and described, it is to be understood that the embodiments can be varied, modified, substituted and changed by those skilled in the art without departing from the principles and spirit of the application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for industrial equipment status monitoring and anomaly detection, characterized in that: include: The edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment, obtains a feature map of a first granularity, and sends the feature map of the first granularity to the cloud; The cloud determines, based on the feature graph of the first granularity, whether the target industrial equipment is normal, and sends a granularity adjustment instruction to the edge end if the determination result shows that the target industrial equipment is abnormal; The edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment in response to the granularity adjustment instruction, obtains a feature map of a second granularity, and sends the feature map of the second granularity to the cloud; the second granularity is finer than the first granularity and the first granularity is not the finest granularity.

2. The industrial equipment status monitoring and anomaly detection method according to claim 1, characterized in that: After the edge terminal performs feature extraction processing on the real-time monitoring image of the target industrial equipment in response to the granularity adjustment instruction, obtains a feature map of the second granularity, and sends it to the cloud, the method further includes: The cloud sends an alarm message when the results of judging whether the target industrial equipment is normal based on the feature graphs of all granularities are all abnormal.

3. The industrial equipment status monitoring and anomaly detection method according to claim 1, characterized in that: Before performing feature extraction processing on the real-time monitoring image of the target industrial equipment at the edge end to obtain the first granularity, the method further includes: The edge end receives the EfficientNetV2-s model and converts the EfficientNetV2-s model into IR format based on OpenVINO.

4. The industrial equipment status monitoring and anomaly detection method according to claim 3, characterized in that: The edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment to obtain a feature map of a first granularity, including: The edge end performs feature extraction processing on the real-time monitoring image of the target industrial equipment based on the EfficientNetV2-s model to obtain a feature map of the first granularity.

5. The industrial equipment status monitoring and anomaly detection method according to claim 1, characterized in that: The cloud determines whether the target industrial equipment is normal based on the feature graph of the first granularity, including: The cloud performs feature extraction processing on the acquired feature maps of each granularity based on the flow model to obtain a fusion feature of the first granularity; The cloud determines whether the target industrial equipment is normal based on the fusion features of the first granularity.

6. The method for industrial equipment status monitoring and anomaly detection according to claim 5, characterized in that: Before the cloud performs feature extraction processing on the acquired feature graphs of each granularity based on the flow model to obtain the features of the first granularity, the method further includes: Based on the sample feature graphs of each granularity under the normal operation state of the target industrial equipment uploaded by the edge end, the flow model is trained and a fitter is generated; the fitter is used to convert the image feature distribution represented by the sample feature graph into a standard normal distribution.

7. The industrial equipment status monitoring and anomaly detection method according to claim 2, characterized in that: The cloud determines whether the target industrial equipment is normal based on the feature graph of the first granularity, including: The cloud obtains, based on the feature map of the first granularity, a confidence level of anomaly occurrence at each location in the feature map of the first granularity; When there is a location with a confidence level greater than a threshold corresponding to the first granularity, the cloud determines the result of the judgment of whether the target industrial equipment is normal as the target industrial equipment being abnormal.

8. The method for industrial equipment status monitoring and anomaly detection according to claim 7, characterized in that: When the cloud determines whether the target industrial equipment is normal based on the feature graphs of all granularities and the results are all abnormal, an alarm message is issued, including: The cloud obtains the confidence level of anomaly occurrence at each location in the finest-grained feature map based on the finest-grained feature map; When there is a location with a confidence level greater than a threshold corresponding to the finest granularity, the cloud determines the location in the finest granularity feature map with a confidence level greater than the threshold corresponding to the finest granularity as an abnormal location; The cloud sends the alarm information; the alarm information carries information about the abnormal location.

9. The method for industrial equipment status monitoring and anomaly detection according to any one of claims 1 to 8, characterized in that: The cloud and the edge communicate with each other using a control protocol based on the HTTPS protocol, and data transmitted between the cloud and the edge is in JSON format.

10. An industrial equipment status monitoring and anomaly detection system, characterized in that: include: The cloud and at least one edge; The edge end is used to perform feature extraction processing on the real-time monitoring image of the target industrial equipment, obtain a feature map of a first granularity, and send the feature map of the first granularity to the cloud; The cloud is configured to determine whether the target industrial equipment is normal based on the feature graph of the first granularity, and send a granularity adjustment instruction to the edge end if the target industrial equipment is determined to be abnormal; The edge end is also used to respond to the granularity adjustment instruction, perform feature extraction processing on the real-time monitoring image of the target industrial equipment, obtain a feature map of a second granularity, and send the feature map of the second granularity to the cloud; the second granularity is finer than the first granularity and the first granularity is not the finest granularity.

Images