A method and apparatus for detecting the operating status of equipment
By performing translation and interpolation processing on multiple frames of the device's operating images, combined with feature image analysis, the problems of high hardware resource consumption and high latency in existing technologies are solved, achieving low latency and low false alarm rate device operating status detection.
Patent Information
- Application Number
- CN202111646001.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-29
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-12-29
AI Technical Summary
In existing technologies, video processing networks for detecting the operating status of equipment consume a lot of hardware resources and increase latency, affecting the timeliness of detecting abnormal equipment operation and leading to an increase in product defect rate.
By acquiring multiple frames of operating images of the target device, each frame is translated and compared to generate a difference image and perform edge processing. Combined with feature image analysis, the operating status of the device is determined and alarm information is output.
It achieves low latency and low false alarm rate in equipment operation status monitoring, reduces resource consumption and system uptime, and improves the timeliness and accuracy of detection.
Smart Images

Figure CN114490247B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of industrial vision technology, and in particular to a method and apparatus for detecting the operating status of equipment. Background Technology
[0002] In the field of industrial vision, it is often necessary to perform tasks to check whether equipment is operating normally. In these tasks, if an abnormality is detected, an alarm should be triggered to alert the operator. This anomaly detection task falls under the category of video behavior recognition. However, current technologies commonly use video processing networks such as 3D convolutional networks. While consuming significant hardware resources, the increased inference time also increases the latency of the corresponding processing network. This affects the timeliness of detecting abnormal equipment operation and increases the probability of defective products produced by the equipment. Summary of the Invention
[0003] In view of the above problems, the present invention is proposed to provide a method and apparatus for detecting the operating status of a device to overcome or at least partially solve the above problems.
[0004] According to a first aspect of the present invention, a method for detecting the operating status of a device is provided, the method comprising:
[0005] Acquire multiple frames of running images of the target device within a specified time period;
[0006] Each frame of the running image is translated to obtain the translation image of each frame. The pixels of the running image and the pixels of the corresponding frame of the translation image are corresponding. The running image and the comparison image are the same size. The comparison image is the image when the device is running normally.
[0007] Each frame of the running image is compared with its corresponding translation image to determine the difference image for each frame;
[0008] Edge processing is performed on each difference image to determine multi-frame feature images;
[0009] Image analysis is performed on each feature image to obtain the state analysis results of each frame of the feature image;
[0010] Based on the results of multiple status analyses, it is determined whether the target device is operating abnormally.
[0011] According to a second aspect of the present invention, an apparatus for detecting the operating status of a device is provided, the apparatus comprising:
[0012] The image acquisition module is used to acquire multiple frames of running images of the target device within a specified time period;
[0013] The image translation module is used to translate the comparison image corresponding to each frame of running image to obtain the translated image of each frame. The pixels of the running image and the pixels of the corresponding frame of translated image are corresponding. The running image and the comparison image are the same size. The comparison image is the image when the device is running normally.
[0014] The first processing module is used to compare each frame of running image with the corresponding translation image to determine the difference image corresponding to each frame.
[0015] The second processing module is used to perform edge processing on each difference image to determine multi-frame feature images;
[0016] The image analysis module is used to perform image analysis on each feature image to obtain the state analysis results of each frame of the feature image;
[0017] The status determination module is used to determine whether the target device is operating abnormally based on multiple status analysis results.
[0018] In this invention, multiple frames of operational images of a target device within a specified time period are acquired. Each frame includes spatial dimension information. Then, the corresponding comparison image for each frame is translated to obtain translated images for each frame. These comparison images represent the device when it is operating normally. Next, each frame is compared with its corresponding translated image to determine a difference image, which includes temporal dimension information. Edge processing is performed on each difference image to determine multiple feature images. Finally, image analysis is performed on each feature image, and based on the results of multiple state analyses, it is determined whether the target device is operating abnormally. If the target device is operating abnormally, alarm information can be output promptly. This invention effectively solves the problem of high latency in current algorithms. Furthermore, by combining two dimensions of information, it can accurately monitor the device's operating status with less resource consumption and system runtime, exhibiting low latency and a low false alarm rate.
[0019] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description
[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings.
[0021] In the attached diagram:
[0022] Figure 1 This is a flowchart illustrating the steps of a method for detecting the operating status of a device according to an embodiment of the present invention;
[0023] Figure 2 This is a flowchart illustrating the steps of another method for detecting the operating status of a device provided in an embodiment of the present invention;
[0024] Figure 3 This is a structural diagram of a feature fusion model provided in an embodiment of the present invention;
[0025] Figure 4 This is a structural diagram of a multi-scale feature fusion network provided in an embodiment of the present invention;
[0026] Figure 5 This is a schematic diagram of the steps for data processing using a multi-scale feature fusion network, provided by an embodiment of the present invention.
[0027] Figure 6 This is a structural diagram of a first multi-scale residual fusion subnetwork provided in an embodiment of the present invention;
[0028] Figure 7 This is a schematic diagram of a dual-scale dense connection layer provided in an embodiment of the present invention;
[0029] Figure 8 This is a schematic diagram of the steps for data processing via a dual-stream feature fusion subnetwork provided in an embodiment of the present invention;
[0030] Figure 9 This is a block diagram of a device for detecting the operating status of an equipment, provided in an embodiment of the present invention. Detailed Implementation
[0031] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.
[0032] Reference Figure 1 The diagram illustrates a step-by-step flowchart of a method for detecting the operating status of a device according to an embodiment of the present invention. The method may include:
[0033] Step 101: Obtain multiple frames of running images of the target device within a specified time period.
[0034] Step 102: Translate the comparison image corresponding to each frame of the running image to obtain the translated image of each frame.
[0035] In this embodiment of the invention, during the operation of the target device, the camera can take cyclical photos of the target device within a specified time period to obtain multiple frames of operational images corresponding to the target device. The operational images and comparison images are of the same size. Those skilled in the art can also select multiple frames of operational images according to actual needs; for example, 10 frames of operational images of the target device can be continuously acquired within a specified time period. State prediction is performed by combining the state analysis results predicted from each frame of operational images to ensure the accuracy of the predicted device operating state. In one example, the operational image can be processed in real time after it is acquired. In another example, since the time interval between acquiring multiple frames of images is relatively short, multiple frames of operational images can be acquired continuously and then processed sequentially.
[0036] The operating status of the target equipment includes: abnormal operation and normal operation. When the target equipment is operating abnormally, an alarm needs to be triggered to prompt relevant operators to handle the abnormal situation in a timely manner and ensure the normal operation of the target equipment. The target equipment may include textile equipment, and the abnormal operating conditions corresponding to the target equipment during abnormal operation may include: yarn breakage, yarn tangling, and blocked thread holes, etc.
[0037] When textile equipment is operating normally, only slight disturbances occur in the textile thread, resulting in minimal changes in the corresponding multi-frame operation images. However, when the equipment is operating abnormally, the corresponding multi-frame operation images show significant differences compared to those during normal operation. Therefore, assuming the equipment is operating normally, a pre-captured normal operation image can be stored as a comparison image. Consequently, image analysis can be performed by comparing this comparison image with the real-time acquired operation images to determine the operating status of the target equipment.
[0038] To avoid irrelevant image information caused by camera shake, a comparison image can be shifted based on the running image to obtain a shifted image corresponding to the running image. Specifically, a comparison image is acquired for each frame of the running image for image analysis. However, lens shake is unavoidable during device operation, and irrelevant information caused by lens displacement may be captured during the comparison process. To filter out this irrelevant information, the comparison image can be shifted based on the corresponding frame of the running image when it is acquired, resulting in a shifted image. The pixels of the shifted image correspond to the pixels of the corresponding frame of the running image. This process is repeated for other frames of the running image to obtain multiple shifted images.
[0039] Step 103: Compare each frame of the running image with its corresponding translation image to determine the difference image for each frame.
[0040] Step 104: Perform edge processing on each difference image to determine the feature images of multiple frames.
[0041] In this embodiment of the invention, comparison refers to subtracting the pixel value of a pixel in the running image of each frame from the pixel value of the corresponding pixel in the translated image of the corresponding frame. The absolute value of the difference is taken as the comparison difference. Based on each comparison difference, a difference image for the corresponding frame can be generated. The image region with a larger comparison difference represents the region where the running image has changed significantly compared to the comparison image. Similarly, the running images of other frames are compared with their corresponding images to determine the corresponding difference images.
[0042] To eliminate edge errors caused by translation and comparison operations in the comparison images, edge processing can be performed on the obtained multi-frame difference images. This edge processing can include edge cropping and edge masking. In one example, zero-masking can be used to mask the edges of each difference image. Zero-masking refers to a mask with a pixel value of zero. In another example, the edges of each difference image can be cropped according to a preset edge cropping length. For example, the masked edge or the preset edge cropping length can be one-twentieth of a pixel. Simultaneously, the edge-processed difference images are used as feature images of the corresponding frames, thereby obtaining multi-frame feature images.
[0043] Step 105: Perform image analysis on each feature image to obtain the state analysis results of each frame of the feature image.
[0044] Step 106: Based on the results of multiple status analyses, determine whether the target device is operating abnormally.
[0045] In this embodiment of the invention, image analysis can be performed on feature images individually, or by combining feature images and running images. When performing image analysis on feature images individually, the average or maximum pixel value in each feature image can be determined. In one example, an average threshold can be preset. If the average pixel value in the feature images is greater than the average threshold, it indicates that the running image of the corresponding frame has a larger area of significant change compared to the comparison image. This can preliminarily predict that the target device is malfunctioning, and the malfunction is taken as the state analysis result. If the average pixel value in the feature images is less than or equal to the average threshold, it indicates that the running image of the corresponding frame has not changed significantly compared to the comparison image. This can preliminarily predict that the target device is operating normally, and normal operation is taken as the state analysis result. Where the running image has a larger area of significant change compared to the comparison image, based on the characteristics of the textile equipment, it can be preliminarily determined that the textile thread is malfunctioning. Possible malfunctions include: thread breakage and thread entanglement.
[0046] In another example, a maximum threshold can be preset. If the maximum pixel value in the feature image is greater than the maximum threshold, it indicates that the pixel values of one or more pixels in the corresponding frame's running image have changed significantly compared to the comparison image. This can preliminarily predict that the target device is malfunctioning, and the malfunction can be used as a result of state analysis. If the maximum pixel value in the feature image is less than or equal to the maximum threshold, it indicates that the corresponding frame's running image has not changed significantly compared to the comparison image. Where the area where the running image shows a significant change compared to the comparison image is small, based on the characteristics of the textile equipment, it can be preliminarily determined that there is an abnormality in the thread hole. Possible abnormalities include thread hole blockage. For example, after determining that the equipment is malfunctioning, the specific location of the blocked thread hole in the target device can be determined based on the pixel value corresponding to the maximum value.
[0047] When performing image analysis by combining feature images and running images, a feature fusion model can be introduced. A single-frame running image and its corresponding feature image are input into this feature fusion model for state analysis, and the state analysis results are output. These state analysis results include normal and abnormal states. After obtaining multiple state analysis results, the corresponding results can be converted into corresponding arrays. These arrays are then used as analysis arrays and arranged sequentially according to the time order of the acquired running images.
[0048] In one example, a weight array corresponding to the analysis array can be pre-defined. Since the closer the time of acquiring the running image is to the current time, the more representative the corresponding state analysis result is to the current operating state of the target device, the larger the corresponding weight coefficient is for the state analysis result of the running image of the corresponding frame. Based on each weight coefficient, the weight array is generated sequentially. The elements in the analysis array and the corresponding elements in the weight array are multiplied and summed, and the sum is used as the anomaly prediction value. It is determined whether the corresponding anomaly prediction value is greater than an anomaly threshold, which is pre-set. If the anomaly prediction value is greater than the anomaly threshold, it can be determined that the target device is operating abnormally. If the anomaly prediction value is less than or equal to the anomaly threshold, it can be determined that the target device is operating normally. If the target device is determined to be operating abnormally, an alarm message is output to prompt the operator to handle the abnormal situation of the target device in a timely manner. After the anomaly situation is handled, the system re-performs anomaly detection.
[0049] In summary, the present invention provides a method for detecting the operating status of a device. The method includes acquiring multiple frames of operating images of a target device within a specified time period. Each frame of the operating image includes spatial dimension information. Then, a translation image corresponding to each frame of the operating image is performed to obtain translation images for each frame. The translation images are images of the device when it is operating normally. Next, each frame of the operating image is compared with its corresponding translation image to determine a difference image for each frame. The difference image includes temporal dimension information. Edge processing is performed on each difference image to determine multiple frame feature images. Finally, image analysis is performed on each feature image, and the operating status of the target device is determined based on the corresponding state analysis results. If the target device is operating abnormally, an alarm message is output. This invention effectively solves the problem of high latency in current algorithms. Furthermore, by combining change information in two dimensions, it can accurately monitor the operating status of the device with less resource consumption and system runtime, exhibiting low latency and a low false alarm rate.
[0050] Reference Figure 2 This diagram illustrates a step-by-step flowchart of another method for detecting the operating status of a device provided by an embodiment of the present invention. The method may include:
[0051] Step 201: Obtain multiple frames of running images of the target device within a specified time period.
[0052] Step 202: Translate the comparison image corresponding to each frame of the running image to obtain the translated image of each frame.
[0053] In this embodiment of the invention, the description of steps 201 and 202 is the same as that of steps 101 and 102 described above.
[0054] For example, the phase displacement between pixels in each frame of the running image and the corresponding pixels in the comparison image can be determined using the phase correlated method. Then, based on the phase displacement, the corresponding comparison images are translated to obtain a multi-frame translated image corresponding to the multi-frame running images. The phase correlated method is used to detect the relative displacement between two images with similar content. It is based on the Fourier transform displacement theorem: the Fourier transform of a translated function is the product of the Fourier transform of the untranslated function and an exponential factor with linear phase; that is, translation in the spatial domain causes a phase shift in the frequency spectrum. Assuming the Fourier transform of image f(x,y) is F(u,v), i.e., DFT[f(x,y)]=F(u,v), if f(x,y) is translated by a vector...<a,b> The Fourier transform after translation is:
[0055] DFT[f(xa,yb)]=e -j2π(ua+vb) F(U,V) (Formula 1)
[0056] Therefore, the boundary effects can be removed by performing window function processing on the running image f2(x,y) and the corresponding comparison image f1(x,y). The Hanning window can be used as the window function. A Fourier transform is performed on the running image f2(x,y) to obtain F2(u,v), and a Fourier transform is performed on the comparison image f1(x,y) to obtain F1(u,v). According to Equation 1, F2(u,v) can be expressed as the comparison image F1(u,v) before translation multiplied by a coefficient:
[0057] F2(U,V)=e -j2π(ua+vb) F1(U,V) (Formula 2)
[0058] The cross-power spectrum of the running image f2(x,y) and the comparison image f1(x,y) can be easily obtained from Equation 2:
[0059]
[0060] Next, the inverse Fourier transform of the cross-power spectrum is performed to determine the maximum value of the cross-power spectrum. At the position corresponding to the maximum value of the cross-power spectrum, a 5x5 window is applied, and the sub-pixel position is obtained by applying the following formula 3:
[0061]
[0062] Where 'a' is the displacement vector along the X-axis and 'b' is the displacement vector along the Y-axis. Using 'a' and 'b' as phase displacements, and with the target reference point of the comparison image as the origin of the coordinate system, the comparison image is translated to obtain a translated image. The target reference point can be the lower left corner of the comparison image. In one example, to reduce the amount of data for subsequent image analysis, the translated image and the running image can be scaled down to a suitable size. This further improves the timeliness of detecting abnormal operation of the target device.
[0063] Step 203: Perform mean smoothing on each frame of the running image and the corresponding translation image.
[0064] In this embodiment of the invention, since the textile thread on the target device is constantly moving during normal operation, it will experience slight disturbances. To reduce the impact of these slight disturbances during image analysis, mean smoothing can be applied to both the running image and the translational image. Mean smoothing refers to using the average pixel value of pixels within the target region as the pixel value of each pixel within that region. For example, the target region can be a square region with a side length of 3 pixels. This improves the robustness of the algorithm.
[0065] Step 204: Compare the running image of each frame after mean smoothing with the corresponding translation image to determine the difference image corresponding to each frame.
[0066] Step 205: Perform edge processing on each difference image to determine the multi-frame feature images.
[0067] In this embodiment of the invention, the description of steps 204 and 205 refers to the description of steps 103 and 104 above.
[0068] Step 206: Input the running image and the corresponding feature image of each frame into the feature fusion model for state analysis to determine the state analysis result of each frame.
[0069] In this embodiment of the invention, image analysis is performed by combining the feature image and the running image. A feature fusion model can be introduced, in which a single frame of running image and the corresponding feature image are input into the feature fusion model for state analysis, and the state analysis result corresponding to each frame is output. The state analysis result includes: normal and abnormal.
[0070] An optional embodiment of the invention, referring to... Figure 3The feature fusion model may include a spatial flow feature extraction network 301, a temporal flow feature extraction network 302, and a multi-scale feature fusion network 303 (MSFF-net, Multi-scale feature fusion network).
[0071] The running image and corresponding feature image of each frame are input into the feature fusion model for state analysis to determine the state analysis result of each frame, which may include the following steps:
[0072] Each frame of the running image is input into the spatial flow feature extraction network 301 for processing, and multiple running feature images of different resolutions are obtained respectively;
[0073] The feature image corresponding to the running image is input into the time-stream feature extraction network 302 for processing, and multiple target feature images of different resolutions are obtained respectively;
[0074] Multiple running feature images and multiple target feature images of the corresponding feature images of each frame are input into the multi-scale feature fusion network 303 for processing to obtain the state analysis results corresponding to each frame.
[0075] In this embodiment of the invention, after inputting a single-frame running image into the feature fusion model, the spatial flow feature extraction network 301 operates on the running image to obtain multiple running feature images with different image resolutions. For example, four running feature images with different image resolutions are obtained, which can be running feature images with the original image resolution reduced by a factor of 2, 4, 8, and 16, respectively. After inputting the feature image of the corresponding frame into the temporal flow feature extraction network 302, the temporal flow feature extraction network 302 operates on the feature image to obtain multiple target feature images with different image resolutions. For example, four target feature images with different image resolutions are obtained, which can be target feature maps with the original image resolution reduced by a factor of 2, 4, 8, and 16, respectively. The running feature images and the corresponding target feature images with the same image resolution are used as image pairs to obtain multiple image pairs, which are then processed by the multi-scale feature fusion network 303 to determine the state analysis results.
[0076] An optional embodiment of the invention, referring to... Figure 4The multi-scale feature fusion network includes a first multi-scale residual fusion sub-network 401 (MSRF-net, Multi-Scale Residual Fusion Network), a second multi-scale residual fusion sub-network 402, and a two-stream feature fusion sub-network 403. (Refer to...) Figure 5 The process involves inputting multiple running feature images of each frame and multiple target feature images of the corresponding feature images into the multi-scale feature fusion network for processing, thereby obtaining the state analysis result corresponding to each frame. This process may include the following steps: Step S51-Step S54.
[0077] Step S51: Input multiple running feature images of each frame of running image into the first multi-scale residual fusion sub-network for processing to obtain multiple running fused images.
[0078] Step S52: Input multiple target feature images of the feature image corresponding to each frame of running image into the second multi-scale residual fusion sub-network for processing to obtain multiple feature fusion images.
[0079] In this embodiment of the invention, the first multi-scale residual fusion subnetwork and the second multi-scale residual fusion subnetwork can have the same structure, and MSRF-net is used to fuse high-dimensional features and low-dimensional features. For example, the structure of MSRF-net can be as follows: Figure 6 As shown, it can be composed of eight dual-scale dense fusion (DSDF) layers. Specifically, for the first multi-scale residual fusion sub-network, its input information consists of multiple running feature images at different image resolutions, and each DSDF requires two running feature images at different image resolutions as input. For the second multi-scale residual fusion sub-network, its input information consists of multiple target feature images at different image resolutions, and each DSDF requires two target feature images at different image resolutions as input. Furthermore, the high-dimensional and low-dimensional features of the DSDF are relative; for example, when two images at different resolutions are input into the DSDF, the image with the higher resolution is used as the high-dimensional feature, and the image with the lower resolution is used as the low-dimensional feature. The structure of the DSDF can be as follows: Figure 7 As shown, the image feature fusion employs a dense stitching method. When there is at least one input corresponding to the DSDF, that is, after multiple input images are stitched together, convolution and activation operations are sequentially performed in the corresponding DSDF. The activation function corresponding to the activation operation can be the SiLU activation function, which avoids the problem of neuron deactivation. The SiLU activation function S(X) is shown below:
[0080]
[0081] Furthermore, when the DSDF receives two image inputs with different resolutions, it pre-sampling the image with the lower resolution or downsampling the image with the higher resolution before performing the image stitching operation. The upsampling method can include bilinear interpolation. The downsampling uses the sum of average pooling and max pooling. Thus, based on the first multi-scale residual fusion subnetwork, multiple running fused images with different resolutions are output, and based on the second multi-scale residual fusion subnetwork, multiple feature fused images with different resolutions are output. The image resolution of the running fused image and the feature fused image corresponding to the same image pair is the same.
[0082] Step S53: Stitch together the running fusion image and feature fusion image corresponding to the same image resolution for each frame to obtain multiple stitched images.
[0083] Step S54: Input multiple stitched images into the dual-stream feature fusion sub-network and determine the state analysis result corresponding to each frame.
[0084] In this embodiment of the invention, image stitching refers to overlapping two edges of the same size corresponding to the running fused image and the feature fused image to form a single overall feature image, which is then used as the stitched image. Multiple stitched images are input into a corresponding dual-stream feature fusion sub-network. After processing by the dual-stream feature fusion sub-network, the state analysis result is output. The dual-stream feature fusion sub-network deeply fuses spatial and temporal information, thereby improving the overall accuracy of the state analysis result. Furthermore, compared to using a 3D convolutional model to predict the device's operating state, using a feature fusion model can reduce hardware resource consumption and shorten the result inference time. Additionally, compared to using a single-frame running image to predict the device's operating state, using a feature fusion model can reduce latency and improve prediction accuracy.
[0085] In one optional embodiment of the invention, the dual-stream feature fusion subnetwork includes: multiple convolutional layers, pooling layers, linear layers, and activation function layers. The step of inputting the multiple stitched images into the dual-stream feature fusion subnetwork and determining the state analysis result corresponding to each frame may include the steps: steps S81-S84.
[0086] Step S81: Perform convolution operation on the multiple stitched images through the multi-layer convolutional layers to obtain the convolution processing result.
[0087] In this embodiment of the invention, the number of convolutional layers matches the number of pairs of the first image pairs. For example, when there are 4 pairs of the first image pairs, the number of convolutional layers is 4. Specifically, the stitched image corresponding to the first image pair with the highest image resolution is input into the first layer of the convolutional layer for convolution. The stitched image corresponding to the second highest image pair with the highest image resolution is added to the convolution result of the first convolutional layer, and then input into the second convolutional layer for convolution. The image obtained by the first convolutional layer has the same image resolution as the stitched image corresponding to the second highest image pair with the highest image resolution. This process continues until multiple convolutional layers are completed. Feature addition refers to adding the values at two corresponding positions. The convolution result of the last convolutional layer is used as the result of the multi-layer convolutional processing and input into the pooling layer.
[0088] Step S82: Pool the convolution processing result through the pooling layer to obtain the pooling processing result.
[0089] In this embodiment of the invention, pooling is an operation that reduces the amount of feature information. For example, global average pooling can be used to reduce the amount of feature information. Pooling is performed on the image corresponding to the convolution processing result to obtain the pooling result. This allows for a reduction in the network parameters without affecting the state recognition result, thereby reducing computational cost and space usage, minimizing overfitting, and improving the network's fault tolerance.
[0090] Step S83: Perform a full connection on the pooling result through the linear layer to obtain the connection processing result.
[0091] Step S84: Perform nonlinear processing on the connection processing result through the activation function layer to obtain the state analysis result corresponding to each frame.
[0092] In this embodiment of the invention, the linear layer can be understood as a simple multi-class sub-network. The convolutional layer flattens the input pooling result, which means flattening the corresponding two-dimensional matrix into a one-dimensional vector, wherein the one-dimensional vector is a 1*2 vector. The connection result maps to the probability corresponding to different operating states of the target device, and the one-dimensional vector is denoted as [G1, G2]. THere, G1 represents the probability of predicting the target device's normal operation, and G2 represents the probability of predicting the target device's abnormal operation. This one-dimensional vector is then input as the connection processing result into the activation function layer. Nonlinear processing refers to mapping the connection processing result to the range (0,1) using an activation function, such as the softmax function. The operating state type corresponding to the maximum value in the one-dimensional vector is determined as the MSFF-net state analysis result. For example, the one-dimensional vector obtained after nonlinear processing could be [0.1, 0.9]. T Therefore, the probability that the target device is operating abnormally is 0.9. The abnormality is output as the state analysis result.
[0093] Step 207: Based on the results of multiple status analyses, determine whether the target device is operating abnormally.
[0094] In this embodiment of the invention, after obtaining multiple state analysis results, the corresponding state analysis results can be converted into corresponding arrays, which are then used as analysis arrays and arranged sequentially according to the time order of the acquired running images. To avoid false alarms from the state analysis results output by the feature fusion model, each state analysis result can be stored sequentially through a stack. Correspondingly, the length of the stack can match the number of frames of the acquired running images. For example, the length of the stack can be 10. When a state analysis result is determined to be abnormal, 0 is pushed onto the stack; if the state analysis result is normal, 1 is pushed onto the stack. When determining the operating state of the target device, the state analysis results in the stack are retrieved sequentially, and the corresponding analysis array is determined. For example, abnormal can be represented by 1, and normal can be represented by 0, resulting in an analysis array of (0, 0, 0, 0, 1, 1, 1, 1, 1, 1).
[0095] In one example, a weight array corresponding to the analysis array can be pre-defined. Since the closer the time of acquiring the running image is to the current time, the more representative the corresponding state analysis result is to the current operating state of the target device, the larger the weight coefficient is for the state analysis result of the corresponding frame's running image, the closer it is to the current time. The weight array is generated sequentially based on each weight coefficient. For example, the weight array can be (0, 0, 0, 0.05, 0.05, 0.05, 0.1, 0.15, 0.25, 0.35). The elements in the analysis array and the corresponding elements in the weight array are multiplied and summed, and the sum is used as the anomaly prediction value. For example, based on the analysis array and weight array listed above, the corresponding anomaly prediction value can be 0.95. It is then determined whether the corresponding anomaly prediction value is greater than an anomaly threshold. The anomaly threshold is pre-set; if the anomaly prediction value is greater than the anomaly threshold, it can be determined that the target device is operating abnormally. For example, the anomaly threshold can be 0.4. An anomaly prediction value of 0.95 is much greater than 0.4. Therefore, it is determined that the target device is operating abnormally. If the predicted anomaly value is less than or equal to the anomaly threshold, the target device can be determined to be operating normally. If the target device is determined to be operating abnormally, an alarm message is output to alert the operator to promptly address the anomaly. Once the anomaly has been resolved, the system re-detects the anomaly.
[0096] In an optional embodiment of the invention, the method further includes the step of training the feature fusion model:
[0097] Obtain the running image and feature image corresponding to the target device when it is in the target running state.
[0098] The corresponding running image and feature image are input into the feature fusion model to be trained for processing to determine the state analysis result.
[0099] Determine the loss function between the target operating state and the state analysis results.
[0100] Based on the loss function, the parameters of the feature fusion model to be trained are adjusted.
[0101] If the loss function of the feature fusion model to be trained obtained by adjusting the parameters meets the preset loss conditions, the corresponding feature fusion model to be trained is determined as the trained feature fusion model.
[0102] In this embodiment of the invention, the operator can acquire the corresponding running image and feature image when it is determined that the target device is operating abnormally or operating normally. The feature image is obtained according to the above-described method steps. Next, the running image and the corresponding feature image are input into a feature fusion model to be trained for processing to determine the state analysis result. Based on the obtained state analysis result, a loss function between the target state and the state analysis result is determined, and this function is used as the fusion loss function.
[0103] To prevent the feature fusion model network from relying solely on either spatial or temporal information, during model training, the network can be fed only running images to obtain state analysis results and determine the corresponding spatial loss function. Then, the network can be fed only feature images to obtain state analysis results and determine the corresponding temporal loss function. A cross-entropy loss function is used for deep supervision of the feature fusion model. Therefore, the overall loss function of the feature fusion model is a weighted sum of the fusion loss function, spatial loss function, and temporal loss function. The weights of these functions can be pre-set; for example, the weights of the fusion loss function and the temporal loss function are both 0.2, and the weight of the fusion loss function is 1. This yields the corresponding overall loss function.
[0104] After determining the overall loss function, the relevant parameters of the feature fusion model can be adjusted based on it. Pre-set conditions can be established to determine when the model training is complete. For example, these conditions could be that after several thousand state analyses, the matching degree between the target running state and the state analysis results reaches 97%. After adjusting the parameters, if the overall loss function of the feature fusion model to be trained satisfies the pre-set loss conditions, then the corresponding feature fusion model to be trained can be determined as a successfully trained feature fusion model.
[0105] In summary, the present invention provides a method for detecting the operating status of a device. The method includes acquiring multiple frames of operating images of a target device within a specified time period. Each frame of the operating image includes spatial dimension information. Then, a translation image corresponding to each frame of the operating image is performed to obtain translation images for each frame. The translation images are images of the device when it is operating normally. Next, each frame of the operating image is compared with its corresponding translation image to determine a difference image for each frame. The difference image includes temporal dimension information. Edge processing is performed on each difference image to determine multiple frame feature images. Finally, image analysis is performed on each feature image, and the operating status of the target device is determined based on the corresponding state analysis results. If the target device is operating abnormally, an alarm message is output. This invention effectively solves the problem of high latency in current algorithms. Furthermore, by combining change information in two dimensions, it can accurately monitor the operating status of the device with less resource consumption and system runtime, exhibiting low latency and a low false alarm rate.
[0106] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.
[0107] Reference Figure 9 This invention illustrates an apparatus for detecting the operating status of a device, according to an embodiment of the present invention. The apparatus may include:
[0108] The image acquisition module 901 is used to acquire multiple frames of running images of the target device within a specified time period.
[0109] The image translation module 902 is used to translate the comparison image corresponding to each frame of running image to obtain the translation image of each frame. The pixels of the running image and the pixels of the corresponding frame of translation image correspond to each other. The running image and the comparison image are the same size. The comparison image is the image when the device is running normally.
[0110] The first processing module 903 is used to compare each frame of running image with the corresponding translation image to determine the difference image corresponding to each frame.
[0111] The second processing module 904 is used to perform edge processing on each difference image to determine multi-frame feature images.
[0112] The image analysis module 905 is used to perform image analysis on each feature image to obtain the state analysis result of each frame of the feature image.
[0113] The status determination module 906 is used to determine whether the target device is in abnormal operation based on multiple status analysis results.
[0114] In an optional embodiment of the invention, the image analysis module can also be used for:
[0115] The running image and corresponding feature image of each frame are input into the feature fusion model for state analysis to determine the state analysis result of each frame. The state analysis result includes: normal and abnormal.
[0116] In one optional embodiment of the invention, the feature fusion model includes a spatial flow feature extraction network, a temporal flow feature extraction network, and a multi-scale feature fusion network.
[0117] The image analysis module may further include:
[0118] The running feature image determination submodule is used to input each frame of running image into the spatial flow feature extraction network for processing, thereby obtaining multiple running feature images of different resolutions.
[0119] The target feature image determination submodule is used to input the feature image corresponding to the running image into the time-stream feature extraction network for processing, so as to obtain multiple target feature images of different resolutions.
[0120] The result determination submodule is used to input multiple running feature images of each frame of running image and multiple target feature images of the corresponding feature images into the multi-scale feature fusion network for processing, so as to obtain the state analysis result corresponding to each frame.
[0121] In one optional embodiment of the invention, the multi-scale feature fusion network may include a first multi-scale residual fusion sub-network, a second multi-scale residual fusion sub-network, and a dual-stream feature fusion sub-network.
[0122] The result determination submodule may include:
[0123] The first image fusion unit is used to input multiple running feature images of each frame of running image into the first multi-scale residual fusion sub-network for processing to obtain multiple running fused images.
[0124] The second image fusion unit is used to input multiple target feature images corresponding to the feature images of each frame of running image into the second multi-scale residual fusion sub-network for processing to obtain multiple feature fusion images.
[0125] The image stitching unit is used to stitch together the running fused image and the feature fused image corresponding to the same image resolution in each frame to obtain multiple stitched images.
[0126] The result determination unit is used to input multiple stitched images into the dual-stream feature fusion sub-network and determine the state analysis result corresponding to each frame.
[0127] In one optional embodiment of the invention, the dual-stream feature fusion subnetwork includes: multiple convolutional layers, pooling layers, linear layers, and activation function layers.
[0128] In an optional embodiment of the invention, the result determining unit may further be used for:
[0129] The multiple stitched images are convolutionally processed using the multi-layer convolutional layers to obtain the convolutional processing result.
[0130] The convolution result is pooled using the pooling layer to obtain the pooling result.
[0131] The pooling result is fully connected through the linear layer to obtain the connection processing result.
[0132] The connection processing results are nonlinearly processed by the activation function layer to obtain the state analysis results for each frame.
[0133] In an optional embodiment of the invention, the apparatus further includes a training module for training the feature fusion model, the training module being used for:
[0134] Obtain the running image and feature image corresponding to the target device when it is in the target running state.
[0135] The corresponding running image and feature image are input into the feature fusion model to be trained for processing to determine the state analysis result.
[0136] Based on the target operating state and the state analysis results, the parameters of the feature fusion model to be trained are adjusted to determine the trained feature fusion model.
[0137] In one optional embodiment of the invention, the training module can also be used for:
[0138] Determine the loss function between the target operating state and the state analysis results.
[0139] Based on the loss function, the parameters of the feature fusion model to be trained are adjusted.
[0140] If the loss function of the feature fusion model to be trained obtained by adjusting the parameters meets the preset loss conditions, the corresponding feature fusion model to be trained is determined as the trained feature fusion model.
[0141] In an optional embodiment of the invention, the image analysis module may further include an average value determination submodule, the average value determination submodule being used for:
[0142] Determine the average pixel value of each feature image.
[0143] If the average value is greater than the average threshold, the state analysis result is determined to be abnormal.
[0144] If the average value is less than or equal to the average threshold, the state analysis result is determined to be normal.
[0145] In an optional embodiment of the invention, the image analysis module may further include a maximum value determination submodule, the maximum value determination submodule being used for:
[0146] Determine the maximum value of the pixel values in the feature image.
[0147] If the maximum value is greater than the maximum threshold, the state analysis result is determined to be abnormal.
[0148] If the maximum value is less than or equal to the maximum threshold, the state analysis result is determined to be normal.
[0149] In an optional embodiment of the invention, the state determination module is further configured to:
[0150] The analysis array corresponding to multiple state analysis results is determined according to the time sequence.
[0151] Based on the analysis array and weight array, the anomaly prediction value is determined.
[0152] If the predicted abnormal value is greater than the abnormal threshold, the target device is determined to be operating abnormally.
[0153] If the predicted abnormal value is less than or equal to the abnormal threshold, the target device is determined to be in normal operation.
[0154] In an optional embodiment of the invention, the apparatus may further include:
[0155] The result storage module is used to store the analysis results of each state sequentially through a stack.
[0156] In one optional embodiment of the invention, the first processing module may include:
[0157] The image smoothing submodule is used to perform mean smoothing on each running image and its corresponding translation image.
[0158] The first processing submodule is used to compare each frame of the running image after mean smoothing with the corresponding translation image to determine the difference image corresponding to each frame.
[0159] In one optional embodiment of the invention, the image translation module may include:
[0160] The displacement determination submodule is used to determine the phase displacement between the pixels of each frame of the running image and the corresponding pixels of the comparison image, based on the phase correlation method.
[0161] The image translation submodule is used to translate the corresponding comparison images according to the phase displacement amount to obtain multiple frames of translated images.
[0162] In summary, the present invention provides an apparatus for detecting the operating status of a device. The apparatus includes acquiring multiple frames of operating images of a target device within a specified time period. Each frame of the operating image includes spatial dimension information. Then, a translation image corresponding to each frame of the operating image is performed to obtain translation images for each frame. The translation images are images of the device when it is operating normally. Next, each frame of the operating image is compared with its corresponding translation image to determine a difference image for each frame. The difference image includes temporal dimension information. Edge processing is performed on each difference image to determine multiple frame feature images. Finally, image analysis is performed on each feature image, and the operating status of the target device is determined based on the corresponding state analysis results. If the target device is operating abnormally, an alarm message is output. The present invention effectively solves the problem of high latency in current algorithms. Furthermore, by combining change information in two dimensions, it can accurately monitor the operating status of the device with less resource consumption and system runtime, exhibiting low latency and a low false alarm rate.
[0163] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0164] It will be readily apparent to those skilled in the art that any combination of the above embodiments is feasible, and therefore any combination of the above embodiments is an implementation scheme of the present invention. However, due to space limitations, this specification will not describe them in detail here.
[0165] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.
[0166] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in fewer than all features of the single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.
[0167] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.
[0168] An electronic device, comprising:
[0169] One or more processors;
[0170] Memory;
[0171] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the methods described in the above embodiments.
[0172] A computer-readable storage medium stores a computer program for use in conjunction with an electronic device, the computer program being executable by a processor to perform the methods described in the embodiments above.
[0173] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0174] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0175] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0177] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.
[0178] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.
[0179] The above provides a detailed description of a method and apparatus for detecting the operating status of a device. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the invention. At the same time, those skilled in the art will recognize that, based on the ideas of the invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the invention.
Claims
1. A method for detecting the operating status of equipment, characterized in that, The method includes: Acquire multiple frames of running images of the target device within a specified time period; Each frame of the running image is translated to obtain the translation image of each frame. The pixels of the running image and the pixels of the corresponding frame of the translation image are corresponding. The running image and the comparison image are the same size. The comparison image is the image when the device is running normally. Each frame of the running image is compared with its corresponding translation image to determine the difference image for each frame; Edge processing is performed on each difference image to determine multi-frame feature images; Image analysis is performed on each feature image to obtain the state analysis results of each frame of the feature image; Based on the results of multiple status analyses, determine whether the target device is operating abnormally; The step of performing image analysis on each feature image to obtain the state analysis result of each frame of the feature image includes: The running image and corresponding feature image of each frame are input into the feature fusion model for state analysis to determine the state analysis result of each frame. The state analysis result includes: normal and abnormal. The feature fusion model includes a spatial flow feature extraction network, a temporal flow feature extraction network, and a multi-scale feature fusion network; The step of inputting the running image and corresponding feature image of each frame into the feature fusion model for state analysis and determining the state analysis result of each frame includes: Each frame of the running image is input into the spatial flow feature extraction network for processing, resulting in multiple running feature images at different resolutions. The feature image corresponding to the running image is input into the time-stream feature extraction network for processing, and multiple target feature images of different resolutions are obtained respectively; Multiple running feature images and multiple target feature images of the corresponding feature images of each frame are input into the multi-scale feature fusion network for processing to obtain the state analysis results corresponding to each frame.
2. The method for detecting the operating status of equipment according to claim 1, characterized in that, The multi-scale feature fusion network includes a first multi-scale residual fusion sub-network, a second multi-scale residual fusion sub-network, and a two-stream feature fusion sub-network; The process involves inputting multiple running feature images of each frame and multiple target feature images of the corresponding feature images into the multi-scale feature fusion network for processing, thereby obtaining the state analysis results for each frame, including: Multiple running feature images of each frame of running image are input into the first multi-scale residual fusion sub-network for processing to obtain multiple running fused images; Multiple target feature images corresponding to the feature images of each frame of running image are input into the second multi-scale residual fusion sub-network for processing to obtain multiple feature fusion images; Each frame of the running fused image and feature fused image with the same image resolution is stitched together to obtain multiple stitched images; Multiple stitched images are input into the dual-stream feature fusion subnetwork to determine the state analysis result corresponding to each frame.
3. The method for detecting the operating status of equipment according to claim 2, characterized in that, The dual-stream feature fusion subnetwork includes: multiple convolutional layers, pooling layers, linear layers, and activation function layers; The step of inputting the multiple stitched images into the dual-stream feature fusion subnetwork and determining the state analysis result corresponding to each frame includes: The multiple convolutional layers are used to perform convolution operations on the multiple stitched images to obtain the convolution processing result. The convolution processing result is pooled through the pooling layer to obtain the pooling processing result; The pooling result is fully connected through the linear layer to obtain the connection processing result; The connection processing results are nonlinearly processed by the activation function layer to obtain the state analysis results for each frame.
4. The method for detecting the operating status of an equipment according to any one of claims 1-3, characterized in that, The method also includes the step of training the feature fusion model: Obtain the running image and feature image corresponding to the target device when it is in the target running state; The corresponding running image and feature image are input into the feature fusion model to be trained for processing to determine the state analysis result; Based on the target operating state and the state analysis results, the parameters of the feature fusion model to be trained are adjusted to determine the trained feature fusion model.
5. The method for detecting the operating status of equipment according to claim 1, characterized in that, The step of performing image analysis on each feature image to obtain the state analysis result of each frame of the feature image includes: Determine the average pixel value of each feature image; If the average value is greater than the average threshold, the state analysis result is determined to be abnormal; If the average value is less than or equal to the average threshold, the state analysis result is determined to be normal. Alternatively, the step of performing image analysis on each feature image to obtain the state analysis result of each frame of the feature image includes: Determine the maximum value of the pixel values in the feature image; If the maximum value is greater than the maximum threshold, the state analysis result is determined to be abnormal; If the maximum value is less than or equal to the maximum threshold, the state analysis result is determined to be normal.
6. The method for detecting the operating status of an equipment according to any one of claims 1-3 or 5, characterized in that, The step of determining whether the target device is operating abnormally based on multiple state analysis results includes: Determine the analysis array corresponding to multiple state analysis results according to the time sequence; Based on the analysis array and weight array, determine the anomaly prediction value; If the predicted abnormal value is greater than the abnormal threshold, the target device is determined to be operating abnormally. If the predicted abnormal value is less than or equal to the abnormal threshold, the target device is determined to be in normal operation.
7. The method for detecting the operating status of equipment according to claim 1, characterized in that, The method further includes: The results of each state analysis are stored sequentially using a stack.
8. The method for detecting the operating status of equipment according to claim 1, characterized in that, The step of comparing each frame of the running image with its corresponding translation image to determine the difference image for each frame includes: Perform mean smoothing on each running image and its corresponding translational image; Each frame of the running image after mean smoothing is compared with its corresponding translation image to determine the difference image for each frame.
9. The method for detecting the operating status of equipment according to claim 1, characterized in that, The step of translating the comparison image corresponding to each frame of the running image to obtain the translated image of each frame includes: Based on the phase correlation method, the phase shift between the pixels of each frame of the running image and the corresponding pixels of the comparison image is determined; Based on the phase displacement, the comparison image corresponding to each frame of the running image is translated to obtain the translated image of each frame.
10. A device for detecting the operating status of equipment, characterized in that, The device includes: The image acquisition module is used to acquire multiple frames of running images of the target device within a specified time period; The image translation module is used to translate the comparison image corresponding to each frame of running image to obtain the translated image of each frame. The pixels of the running image and the pixels of the corresponding frame of translated image are corresponding. The running image and the comparison image are the same size. The comparison image is the image when the device is running normally. The first processing module is used to compare each frame of running image with the corresponding translation image to determine the difference image corresponding to each frame. The second processing module is used to perform edge processing on each difference image to determine multi-frame feature images; The image analysis module is used to perform image analysis on each feature image to obtain the state analysis results of each frame of the feature image; The status determination module is used to determine whether the target device is operating abnormally based on multiple status analysis results. The image analysis module can be used for: The running image and corresponding feature image of each frame are input into the feature fusion model for state analysis to determine the state analysis result of each frame. The state analysis result includes: normal and abnormal. The feature fusion model includes a spatial flow feature extraction network, a temporal flow feature extraction network, and a multi-scale feature fusion network. The running feature image determination submodule is used to input each frame of running image into the spatial flow feature extraction network for processing, and obtain multiple running feature images of different resolutions respectively; The target feature image determination submodule is used to input the feature image corresponding to the running image into the time-stream feature extraction network for processing, so as to obtain multiple target feature images of different resolutions respectively; The result determination submodule is used to input multiple running feature images of each frame of running image and multiple target feature images of the corresponding feature images into the multi-scale feature fusion network for processing, so as to obtain the state analysis result corresponding to each frame.
11. An electronic device, comprising: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method of any one of claims 1-9.
12. A computer-readable storage medium storing a computer program for use in conjunction with an electronic device, said computer program being executable by a processor to perform the method of any one of claims 1-9.
