An image processing-based automated equipment anomaly monitoring method and system

Abstract

The application provides an image processing-based automatic equipment anomaly monitoring method and system, and relates to the technical field of automatic equipment anomaly monitoring.The application converts equipment operation video into multi-dimensional physical field characteristics: in the motion field dimension, based on the light flow field analysis, a full-cycle displacement statistical histogram and a spatial heat map are extracted to accurately quantify the motion instability characteristics of a mechanical transmission system; in the vibration field dimension, the energy spectrum and vibration heat map are generated through the time-frequency transformation of the spectrum monitoring point displacement signal to realize the frequency domain feature visualization of the implicit vibration fault; in the structure field dimension, the edge gradient analysis and texture feature extraction technology are fused to construct a time sequence structure heat map sequence to capture the gradual damage evolution law; the three-field anomaly index dynamic weighted fusion mechanism overcomes the limitations of the traditional single-point monitoring; the connected domain analysis of the heat map is combined with the LBP texture and morphological feature decision tree to realize the image processing-based automatic equipment anomaly monitoring.

Description

Technical Field

[0001] This invention relates to the field of automated equipment anomaly monitoring technology, and in particular to an automated equipment anomaly monitoring method and system based on image processing. Background Technology

[0002] With the rapid development of industry and intelligent manufacturing, health monitoring technology for automated equipment has increasingly become a core element in ensuring production safety and efficiency. Traditional equipment anomaly monitoring methods mainly rely on single sensors, such as accelerometers and acoustic emission sensors, to collect vibration or acoustic signals, and then use spectral analysis or machine learning models to diagnose faults. However, these methods have significant limitations, such as insufficient spatial resolution making it difficult to locate minute damages, invasive installation limiting their application in complex structural equipment, and a lack of multi-physics collaborative analysis capabilities making it difficult to comprehensively capture the coupled fault characteristics of motion anomalies, mechanical vibrations, and structural deformations. Traditional vibration sensors cannot accurately locate microcracks at the tooth root, and the motion instability of high-speed rotating components and structural damage often exhibit spatiotemporal correlations, making single signal source analysis prone to misjudgment or missed detection. In addition, diagnostic schemes based on fixed thresholds or static models are difficult to adapt to dynamic changes in equipment operating conditions, resulting in a high false alarm rate. In recent years, although machine vision technology has been introduced into the field of equipment monitoring, existing solutions mostly focus on a single dimension: either analyzing motion trajectory anomalies solely through optical flow field analysis, or relying on thermal maps for static detection of structural defects, failing to address the core challenge of fusing multi-field data from motion, vibration, and structure.

[0003] Therefore, it is necessary to provide an automated equipment anomaly monitoring method and system based on image processing to solve the above-mentioned technical problems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an automated equipment anomaly monitoring method and system based on image processing, achieving the beneficial effect of fusing data from multiple fields to monitor equipment anomalies.

[0005] This invention provides an automated device anomaly monitoring method based on image processing, comprising:

[0006] S1: Decode the operating video stream of the device to be monitored into a continuous frame sequence, and divide the continuous frame sequence into single-period frame sequence segments based on the trajectory period of the preset device motion reference point.

[0007] S2: Based on the registration keyframes in the single-cycle frame sequence segment determined by the preset registration phase points, the single-cycle frame sequence segment is registered to a unified coordinate system through image affine transformation to obtain a set of registration frame sequence segments.

[0008] S3: Calculate the continuous optical flow field of all registered frame sequence segments in the set of registered frame sequence segments, generate motion histograms and motion anomaly heatmaps based on the continuous optical flow field, determine the spectrum monitoring points based on the continuous optical flow field, extract the displacement signals of the spectrum monitoring points, and generate vibration energy spectrum through windowed Fourier transform;

[0009] S4: Extract the LBP histogram and edge gradient map of each frame image of the registration frame sequence segment, normalize the amplitude of the edge gradient map and perform thermal color level mapping to generate structural anomaly heat map, and combine the structural anomaly heat map into a structural anomaly heat map sequence in chronological order.

[0010] S5: Generate motion anomaly index based on motion histogram, generate vibration anomaly heatmap and vibration anomaly index based on vibration energy spectrum, and generate structural anomaly index through spatial self-reference analysis of structural anomaly heatmap sequence;

[0011] S6: Based on the motion anomaly index, vibration anomaly index, and structural anomaly index, the motion anomaly heatmap, vibration anomaly heatmap, and structural anomaly heatmap are weighted and fused to generate a fused heatmap, and the connected domains of the fused heatmap are analyzed to locate anomaly regions.

[0012] Preferably, in step S1, perspective projection calculation is performed using the CAD model of the device to be monitored and the camera calibration parameters, and the three-dimensional trajectory feature points of the preset device motion reference points are mapped to the image coordinate system.

[0013] Preferably, in step S3, the step of determining the spectrum monitoring point includes:

[0014] The optical flow fields of the first and second frames of the registered frame sequence segment are obtained based on the continuous inter-frame optical flow fields;

[0015] Select a pre-selected pixel block whose optical flow displacement amplitude is greater than a preset displacement amplitude and whose SIFT feature of the first frame image is greater than a preset number of feature points;

[0016] Morphological clustering is performed on the pre-selected pixel blocks, and the centroid of the largest connected region is taken as the spectrum monitoring point.

[0017] Preferably, in step S3, the step of generating the vibrational energy spectrum includes:

[0018] At the coordinates of the spectrum monitoring point, the displacement amplitude of the optical flow field between consecutive frames is extracted to form a time-series signal;

[0019] Apply a Hanning window function to the time-series signal to obtain a windowed signal, and perform a fast Fourier transform on the windowed signal to obtain a complex spectrum;

[0020] By taking the square of the complex spectrum modulus and normalizing it, a one-sided energy distribution spectrum, i.e., the vibration energy spectrum, is generated.

[0021] Preferably, in step S4, the LBP histogram extraction step includes:

[0022] Dynamically select the analysis scale based on the local gradient variance of each frame of the registered frame sequence segment;

[0023] Small-radius operators are used to capture fine texture features in regions with high gradient variance;

[0024] Large radius operators are used to extract macroscopic texture features in regions with low gradient variance;

[0025] Fine-grained texture features and macro-grained texture features are fused to generate a unified texture histogram, namely the LBP histogram.

[0026] Preferably, in step S4, the edge gradient map extraction step includes:

[0027] Each frame of the registration frame sequence segment is converted into a registration grayscale image;

[0028] The initial gradient map is obtained by applying the Laplacian second-order differential operator to the registered grayscale image;

[0029] The local curvature field is calculated based on the registered grayscale image. The initial gradient map is then subjected to exponential gain modulation using the local curvature field and normalized to obtain the edge gradient map.

[0030] Preferably, in step S5, the calculation of the vibration anomaly index includes:

[0031] Identify the frequency corresponding to the maximum energy peak in the vibration energy spectrum, and determine the fundamental frequency of the device through parabolic interpolation;

[0032] The sensitive frequency band range is determined based on the equipment type and the equipment fundamental frequency. The proportion of sensitive frequency band energy falling within the sensitive frequency band range to the total frequency band energy is calculated to generate the vibration anomaly index.

[0033] Preferably, in step S5, the step of generating the structural anomaly index includes:

[0034] Each frame of the heatmap sequence is divided into multiple blocks, and the gradient values ​​of all pixels in each block are sorted, with the median value taken as the median value of that block.

[0035] For each block, the median of the median values ​​of the neighboring blocks is used as the health reference benchmark;

[0036] Calculate the absolute difference between the median value of each block and the health reference baseline, and divide it by the standard deviation of the median value of the neighborhood blocks to obtain the standardized deviation.

[0037] Blocks whose standardized deviation exceeds a preset deviation threshold are marked as abnormal blocks, and the proportion of abnormal blocks is counted to generate a structural anomaly index.

[0038] Preferably, step S6 further includes determining the fault type by extracting LBP texture features and morphological features within the located abnormal region.

[0039] This invention also provides an automated equipment anomaly monitoring system based on image processing, applied to an automated equipment anomaly monitoring method based on image processing, comprising:

[0040] The data segmentation module is used to decode the operating video stream of the device under monitoring into a continuous frame sequence, and to segment the continuous frame sequence into single-period frame sequence segments based on the trajectory period of the preset device motion reference point.

[0041] The image registration module is used to register the single-cycle frame sequence segments to a unified coordinate system through image affine transformation based on the registration phase points determined by the preset registration phase points, thereby obtaining a set of registered frame sequence segments.

[0042] The motion and vibration field generation module is used to calculate the continuous optical flow field of all registered frame sequence segments in the registered frame sequence segment set, generate motion histograms and motion anomaly heatmaps based on the continuous optical flow field, determine the spectrum monitoring points based on the continuous optical flow field, extract the displacement signals of the spectrum monitoring points, and generate the vibration energy spectrum through windowed Fourier transform.

[0043] The structure field analysis module is used to extract the LBP histogram and edge gradient map of each frame image of the registration frame sequence segment, perform amplitude normalization and thermal color level mapping on the edge gradient map to generate a structural anomaly heat map, and combine the structural anomaly heat maps in chronological order to form a structural anomaly heat map sequence.

[0044] The multi-domain anomaly quantification module is used to generate motion anomaly indices based on motion histograms, generate vibration anomaly heatmaps and vibration anomaly indices based on vibration energy spectra, and generate structural anomaly indices through spatial self-reference analysis of structural anomaly heatmap sequences.

[0045] The anomaly fusion diagnosis module is used to perform weighted fusion of motion anomaly heatmaps, vibration anomaly heatmaps, and structural anomaly heatmaps based on motion anomaly index, vibration anomaly index, and structural anomaly index to generate a fused heatmap, and analyze the connected domains of the fused heatmap to locate anomaly regions.

[0046] Compared with related technologies, the automated equipment anomaly monitoring method and system based on image processing provided by this invention has the following beneficial effects:

[0047] This invention constructs a collaborative diagnostic system encompassing motion, vibration, and structure. Through video stream analysis, it achieves non-contact, multi-dimensional perception of equipment health status. First, it transforms equipment operation videos into multi-dimensional physical field features: In the motion field dimension, it extracts full-cycle displacement statistical histograms and spatial heatmaps based on optical flow field analysis, accurately quantifying the motion instability characteristics of the mechanical transmission system; in the vibration field dimension, it innovatively generates energy spectra and vibration heatmaps through time-frequency transformation of displacement signals from spectral monitoring points, visualizing the frequency domain characteristics of latent vibration faults; in the structural field dimension, it integrates edge gradient analysis and texture feature extraction techniques to construct a temporal structural heatmap sequence to capture the evolution of progressive damage. The dynamic weighted fusion mechanism of the three-field anomaly indices effectively overcomes the limitations of traditional single-point monitoring. The fusion of connected component analysis from the heatmaps, combined with LBP texture and morphological feature decision trees, enables the system to possess full-chain diagnostic capabilities from early anomaly sign identification to fault type determination, achieving automated equipment anomaly monitoring based on image processing-based data fusion analysis. Attached Figure Description

[0048] Figure 1 This is a flowchart of an automated equipment anomaly monitoring method based on image processing according to the present invention;

[0049] Figure 2 This is a module structure diagram of an automated equipment anomaly monitoring system based on image processing according to the present invention. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the drawings, not all structures. Moreover, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0051] It should also be noted that, for ease of description, the accompanying drawings show only the parts relevant to the invention and not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as being processed sequentially, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it may also have additional steps not included in the drawings. The process may correspond to a method, function, procedure, subroutine, subprogram, etc.

[0052] Example 1

[0053] An automated equipment anomaly monitoring method based on image processing, in its specific implementation, such as... Figure 1 As shown, it includes:

[0054] Step S1: Decode the operating video stream of the device to be monitored into a continuous frame sequence, and divide the continuous frame sequence into single-period frame sequence segments based on the trajectory period of the preset device motion reference point.

[0055] Specifically, in step S1, perspective projection calculation is performed using the CAD model of the device to be monitored and the camera calibration parameters, and the three-dimensional trajectory feature points of the preset device motion reference points are mapped to the image coordinate system.

[0056] In the specific implementation process, the operating video stream of the device to be monitored is first decoded into a continuous image frame sequence in real time using a hardware decoder or software decoding library. During decoding, the original resolution and frame rate are maintained to ensure the integrity of the motion information. Then, based on the motion trajectory of a preset device motion reference point in the image (for example, selecting, but not limited to, the axis marker point of rotating machinery or the feature corner points of linear motion components as the device motion reference point), the pixel coordinates of the device motion reference point in each frame are dynamically calculated using a trajectory tracking algorithm. When a complete cycle of its motion trajectory is detected (for example, a closed path from the highest point to the lowest point and back to the highest point in a rotating device constitutes a complete cycle), the continuous frame sequence is divided into segments containing... A single-cycle frame sequence segment of the complete motion cycle is generated, with each segment containing all frame images of the device from the start phase to the end phase of motion. To improve segmentation accuracy, a perspective projection relationship from the world coordinate system to the image coordinate system is established using the CAD 3D model of the device under test and camera calibration parameters, including but not limited to the camera intrinsic matrix, distortion coefficients, and extrinsic rotation and translation matrices. The 3D trajectory feature points of the preset device motion reference points are mapped to the image pixel coordinate system. This mapping relationship is used to dynamically correct the trajectory drift error of the reference points in the actual image, ensuring that the phase alignment accuracy of the cycle segmentation is controlled within ±0.5 pixels. This eliminates the problem of cycle missegmentation caused by perspective distortion or mechanical vibration, providing a basis for subsequent calculations.

[0057] Step S2: Based on the registration keyframes in the single-cycle frame sequence segment determined by the preset registration phase points, register the single-cycle frame sequence segment to a unified coordinate system through image affine transformation to obtain a set of registration frame sequence segments.

[0058] In the specific implementation process, firstly, registration keyframes in a single-cycle frame sequence segment are determined based on preset registration phase points. Registration phase points are essentially physical reference points with stable spatial positional characteristics during the device's motion cycle, such as the highest point of a rotating device or the extreme position point of reciprocating motion, which are pre-calibrated using the device's kinematic model. Then, the perspective projection matrix is ​​calculated using the CAD model of the device to be monitored and the camera calibration parameters. The three-dimensional trajectory feature points of the registration phase points are mapped to the image coordinate system, obtaining the image frame closest to the registration phase point within each motion cycle as the registration keyframe. Subsequently, using this keyframe as a spatial reference, image affine transformation is performed. All frames of the entire single-cycle frame sequence segment are registered to a unified coordinate system. For example, at least three pre-defined non-collinear registration markers are extracted from the keyframe, such as the rigid features of the bolt hole center and the weld intersection. The spatial offset of the corresponding markers in the remaining frames is calculated, and the optimal affine transformation matrix is ​​solved using the least squares method. This affine transformation matrix includes rotation, translation, and scaling parameters. Through matrix operations, the pixel coordinates of each frame are mapped to a coordinate system based on the keyframe, eliminating image jitter caused by equipment vibration or viewpoint shift. Finally, a set of spatiotemporally aligned registered frame sequence segments is output, providing a geometric consistency basis for subsequent motion, vibration, and structural analysis.

[0059] Step S3: Calculate the continuous optical flow field of all registered frame sequence segments in the registered frame sequence segment set, generate motion histogram and motion anomaly heat map based on the continuous optical flow field, determine the spectrum monitoring point based on the continuous optical flow field, extract the displacement signal of the spectrum monitoring point, and generate vibration energy spectrum through windowed Fourier transform.

[0060] Specifically, in step S3, the steps for determining the spectrum monitoring point include:

[0061] The optical flow fields of the first and second frames of the registered frame sequence segment are obtained based on the continuous inter-frame optical flow fields;

[0062] Select a pre-selected pixel block whose optical flow displacement amplitude is greater than a preset displacement amplitude and whose SIFT feature of the first frame image is greater than a preset number of feature points;

[0063] Morphological clustering is performed on the pre-selected pixel blocks, and the centroid of the largest connected region is taken as the spectrum monitoring point.

[0064] Specifically, in step S3, the generation of the vibrational energy spectrum includes:

[0065] At the coordinates of the spectrum monitoring point, the displacement amplitude of the optical flow field between consecutive frames is extracted to form a time-series signal;

[0066] Apply a Hanning window function to the time-series signal to obtain a windowed signal, and perform a fast Fourier transform on the windowed signal to obtain a complex spectrum;

[0067] By taking the square of the complex spectrum modulus and normalizing it, a one-sided energy distribution spectrum, i.e., the vibration energy spectrum, is generated.

[0068] In the specific implementation process, the continuous inter-frame optical flow field is first calculated for the registered frame sequence fragment set. For example, the motion vector between adjacent frames is calculated pixel by pixel using the Farneback dense optical flow algorithm to generate a vector field containing horizontal and vertical components. Based on this continuous inter-frame optical flow field, three core processes are performed: 1. Extract the optical flow displacement amplitude of all frames in the entire cycle, and statistically analyze its distribution histogram as a motion histogram. The histogram is divided into 16 intervals to quantify the displacement intensity distribution; 2. Take the optical flow displacement amplitude of the first frame to generate a motion anomaly heatmap, map the displacement value to a gray level of 0-255, and mark the motion anomaly area with a bright color for high displacement areas; 3. The location of the spectrum monitoring point needs to combine the optical flow field and image features: select pixel blocks in the optical flow field of the first and second frames with displacement amplitude greater than a preset displacement amplitude (for example, the preset displacement amplitude is 0.5 pixels) and the number of SIFT feature points in the first frame image is greater than the preset number of feature points (for example, the preset number of feature points is 50) as pre-selected areas. After performing morphological closure operations on these blocks to eliminate holes, calculate the centroid coordinates of the largest connected region as the spectrum monitoring point. At a defined spectral monitoring point, the displacement amplitude of that point in a continuous frame throughout the entire period is extracted to form a time-series signal. After applying a Hanning window function to the signal to suppress spectral leakage, a fast Fourier transform is performed to obtain a complex spectrum. The square of the complex modulus is taken and divided by the signal length to generate a one-sided vibration energy spectrum. This vibration energy spectrum can characterize the vibration energy distribution at the monitoring point. In the entire process, the optical flow field data is reused three times: displacement amplitude statistics are used for motion histograms, the spatial mapping of the first frame is used for motion heatmaps, and the time-series signal of the monitoring point is used for vibration spectrum. Multi-threaded processing is achieved through memory caching and parallel computing to provide a basis for subsequent calculations.

[0069] Step S4: Extract the LBP histogram and edge gradient map of each frame image of the registration frame sequence segment, normalize the amplitude of the edge gradient map and perform thermal color level mapping to generate structural anomaly heat map, and combine the structural anomaly heat map into a structural anomaly heat map sequence in chronological order.

[0070] Specifically, in step S4, the LBP histogram extraction steps include:

[0071] Dynamically select the analysis scale based on the local gradient variance of each frame of the registered frame sequence segment;

[0072] Small-radius operators are used to capture fine texture features in regions with high gradient variance;

[0073] Large radius operators are used to extract macroscopic texture features in regions with low gradient variance;

[0074] Fine-grained texture features and macro-grained texture features are fused to generate a unified texture histogram, namely the LBP histogram.

[0075] Specifically, in step S4, the edge gradient map extraction step includes:

[0076] Each frame of the registration frame sequence segment is converted into a registration grayscale image;

[0077] The initial gradient map is obtained by applying the Laplacian second-order differential operator to the registered grayscale image;

[0078] The local curvature field is calculated based on the registered grayscale image. The initial gradient map is then subjected to exponential gain modulation using the local curvature field and normalized to obtain the edge gradient map.

[0079] In the specific implementation process, the LBP histogram and edge gradient map of each frame of the registration frame sequence segment are first extracted: For LBP histogram extraction, the system dynamically selects the analysis scale based on the local gradient variance of each frame image. For example, in areas with high gradient variance such as bolt connections, a small operator with a radius of 1 pixel is used to capture fine textures such as microcracks, while in areas with low gradient variance such as equipment shells, a large operator with a radius of 7 pixels is used to extract macroscopic wear textures. Through weighted fusion, a 256-dimensional unified texture histogram is generated. For example, the weight of the small operator is 0.6 and the weight of the large operator is 0.4. For edge gradient map extraction, the RGB image is first converted to grayscale, and the initial gradient is calculated by applying the second-order Laplacian differential operator. At the same time, the local curvature field is calculated, and the initial gradient is modulated by the exponential gain function. For example, the gradient value is amplified by 2.8 times in areas with high curvature such as tooth root fillets, while the original value is maintained in planar areas. Finally, it is normalized to [0,255] to generate the edge gradient map. Then, a thermal color mapping was performed on the edge gradient map: gradient values ​​0-85 were mapped to dark blue, 86-170 to yellow, and 171-255 to bright red, generating a single-frame structural anomaly heatmap, which was then combined into a structural anomaly heatmap sequence in chronological order.

[0080] Step S5: Generate motion anomaly index based on motion histogram, generate vibration anomaly heatmap and vibration anomaly index based on vibration energy spectrum, and generate structural anomaly index through spatial self-reference analysis of structural anomaly heatmap sequence.

[0081] Specifically, in step S5, the calculation steps for the vibration anomaly index include:

[0082] Identify the frequency corresponding to the maximum energy peak in the vibration energy spectrum, and determine the fundamental frequency of the device through parabolic interpolation;

[0083] The sensitive frequency band range is determined based on the equipment type and the equipment fundamental frequency. The proportion of sensitive frequency band energy falling within the sensitive frequency band range to the total frequency band energy is calculated to generate the vibration anomaly index.

[0084] Specifically, in step S5, the generation of the structural anomaly index includes:

[0085] Each frame of the heatmap sequence is divided into multiple blocks, and the gradient values ​​of all pixels in each block are sorted, with the median value taken as the median value of that block.

[0086] For each block, the median of the median values ​​of the neighboring blocks is used as the health reference benchmark;

[0087] Calculate the absolute difference between the median value of each block and the health reference baseline, and divide it by the standard deviation of the median value of the neighborhood blocks to obtain the standardized deviation.

[0088] Blocks whose standardized deviation exceeds a preset deviation threshold are marked as abnormal blocks, and the proportion of abnormal blocks is counted to generate a structural anomaly index.

[0089] In the specific implementation process, firstly, a motion anomaly index is generated based on the motion histogram. This motion histogram is formed by statistically analyzing the displacement amplitudes of the continuous optical flow field in the registered frame sequence. The degree of motion anomaly is quantified by calculating the KL divergence between the actual displacement distribution and the theoretical uniform distribution. This KL divergence value is directly used as the motion anomaly index. The theoretical uniform distribution is constructed by dividing the entire optical flow displacement amplitude domain into 16 equally wide intervals and assigning them the same probability density. Next, a vibration anomaly index and a vibration anomaly heatmap are generated based on the vibration energy spectrum. The calculation of the vibration anomaly index requires the following details: identifying the frequency corresponding to the maximum energy peak in the vibration energy spectrum, accurately calculating the device's fundamental frequency using parabolic interpolation, and dynamically defining the sensitive frequency band range according to the device type. For example, for bearings, the sensitive frequency band range is defined as 3-8 times the device's fundamental frequency. The proportion of energy within this sensitive frequency band to the total energy of the entire frequency band is calculated as the vibration anomaly index. The vibration anomaly heatmap is generated by extracting the sensitive frequency band at the coordinates of the spectrum monitoring points. The energy value is used to diffuse the energy distribution through bilinear interpolation over its 5×5 pixel neighborhood, and then normalized and mapped to a grayscale value of 0-255 to generate a spatial heatmap. Finally, a structural anomaly index is generated through spatial self-reference analysis of the structural anomaly heatmap sequence. For example, each frame of the structural anomaly heatmap sequence is divided into 10×10 pixel blocks. The median value of all pixel gradient values ​​in each block is sorted and taken as the representative value of the block. The median value of the median value set of the surrounding 3×3 neighborhood blocks is used as the health reference benchmark. The absolute difference between the median value of the current block and the health reference benchmark is calculated and divided by the neighborhood standard deviation to obtain the standardized deviation. When the deviation exceeds the preset deviation threshold, it is marked as an abnormal block. For example, the preset deviation threshold is 3.0. Finally, the percentage of abnormal blocks in the total number of blocks is used as the structural anomaly index. This step realizes the comprehensive quantification of anomalies in three fields: motion, vibration, and structure, providing a basis for subsequent anomaly monitoring and identification.

[0090] Step S6: Based on the motion anomaly index, vibration anomaly index, and structural anomaly index, the motion anomaly heatmap, vibration anomaly heatmap, and structural anomaly heatmap are weighted and fused to generate a fused heatmap, and the connected domains of the fused heatmap are analyzed to locate the anomaly region.

[0091] Specifically, step S6 also includes determining the fault type by extracting LBP texture features and morphological features within the located abnormal region.

[0092] In the specific implementation process, firstly, weight coefficients are dynamically determined based on the motion anomaly index, vibration anomaly index, and structural anomaly index. For example, the sum of the motion anomaly index, vibration anomaly index, and structural anomaly index is calculated to obtain the total index. Then, the ratio of each anomaly index to the total index is calculated, and each ratio is multiplied by a preset coefficient. For example, the motion anomaly coefficient is 0.5, the vibration anomaly coefficient is 0.3, and the structural anomaly coefficient is 0.2. Finally, the fusion weights for weighted fusion of the motion anomaly heatmap, vibration anomaly heatmap, and structural anomaly heatmap are obtained. Subsequently, pixel-level weighted fusion of the three heatmaps is performed to generate a fused heatmap. Next, the fused heatmap is segmented using a fixed segmentation threshold. For example, the fixed segmentation threshold is 0.7 times the maximum value of all pixels in the fused heatmap. Connected components are extracted as candidate anomaly regions. Refined analysis is performed within each connected component. First, LBP texture features are extracted, and the texture entropy of the region is calculated using the LBP histogram. Then, morphological features, including but not limited to roundness and aspect ratio, are calculated. Finally, the fault type is determined by combining the preset decision rules. For example, if the circularity is greater than 0.7 and the texture entropy is less than 3.0, it is determined as pitting; if the aspect ratio is greater than 3.0 and the average value of the motion anomaly heatmap of the candidate anomaly region is greater than 150, it is determined as tearing. Here, the average value of the motion anomaly heatmap of the candidate anomaly region refers to the arithmetic mean of the displacement amplitude grayscale values ​​of all pixels extracted from the corresponding region of the motion anomaly heatmap of the candidate anomaly region located by the fused heatmap, which quantifies the overall intensity of motion in the region; if the texture entropy is greater than 5.0 and the average value of the structural heatmap of the candidate anomaly region is greater than 180, it is determined as wear. Here, the average value of the structural heatmap of the candidate anomaly region refers to the arithmetic mean of the enhanced edge gradient grayscale values ​​of all pixels extracted from the corresponding region of the structural anomaly heatmap of the same candidate anomaly region, which reflects the average intensity of structural abrupt changes in the region; other cases are determined as compound faults. Through the above, the purpose of detecting and identifying anomalies in automated equipment is achieved by using image processing.

[0093] The working principle of the automated equipment anomaly monitoring method based on image processing provided by this invention is as follows:

[0094] This invention establishes a framework for analyzing the motion cycle of automated equipment by performing temporal segmentation and spatial registration on the video stream of the equipment's operation. First, the video stream is decoded into a continuous frame sequence and then segmented into single-cycle segments based on the trajectory cycle of preset equipment motion reference points. Next, spatial alignment of the entire cycle image is achieved through affine transformation of keyframes. Dynamic and static features are extracted synchronously from the registered frame sequence: a motion histogram quantifying the overall motion distribution and a motion anomaly heatmap reflecting the spatial distribution of instantaneous displacement anomalies are generated using the optical flow field between continuous frames. Simultaneously, high-frequency vibrations are located and displacement signals are extracted based on the optical flow field, and a spectrum representing vibration energy is generated through windowed Fourier transform. Furthermore, table data is extracted frame by frame. The system utilizes surface texture features and structural edge features. The latter generates a single-frame structural anomaly heatmap through gradient map normalization and thermal mapping, and combines them chronologically to form a structural anomaly heatmap sequence to capture the structural evolution process. Based on these features, three anomaly indices are generated: the statistical distribution anomaly degree of the motion histogram, the proportion of sensitive frequency band energy in the vibration energy spectrum, and the proportion of spatial self-referenced anomaly blocks in the structural heatmap sequence. Finally, a comprehensive anomaly distribution map is generated by dynamically weighting and fusing the motion, vibration, and structural heatmaps using the three indices. After locating the anomaly region through connected component analysis, fault classification decisions are made by combining the LBP texture entropy value and morphological features within the region, achieving intelligent monitoring, identification, and diagnosis from video stream to fault type.

[0095] Example 2

[0096] An image processing-based automated equipment anomaly monitoring system is applied to an image processing-based automated equipment anomaly monitoring method. In specific implementation processes, such as... Figure 2 As shown, it includes:

[0097] The data segmentation module 100 is used to decode the operating video stream of the device to be monitored into a continuous frame sequence, and to segment the continuous frame sequence into single-period frame sequence segments based on the trajectory period of the preset device motion reference point.

[0098] The image registration module 200 is used to register the single-cycle frame sequence segments to a unified coordinate system through image affine transformation based on the registration key frames determined by the preset registration phase points, thereby obtaining a set of registration frame sequence segments.

[0099] The motion and vibration field generation module 300 is used to calculate the continuous optical flow field of all registered frame sequence segments in the set of registered frame sequence segments, generate motion histograms and motion anomaly heatmaps based on the continuous optical flow field, determine the spectrum monitoring points based on the continuous optical flow field, extract the displacement signals of the spectrum monitoring points, and generate vibration energy spectrum through windowed Fourier transform.

[0100] The structure field analysis module 400 is used to extract the LBP histogram and edge gradient map of each frame image of the registration frame sequence segment, perform amplitude normalization and thermal color level mapping on the edge gradient map, generate a structural anomaly heat map, and combine the structural anomaly heat maps in chronological order into a structural anomaly heat map sequence.

[0101] The multi-domain anomaly quantification module 500 is used to generate motion anomaly indices based on motion histograms, generate vibration anomaly heatmaps and vibration anomaly indices based on vibration energy spectra, and generate structural anomaly indices through spatial self-reference analysis of structural anomaly heatmap sequences.

[0102] The anomaly fusion diagnosis module 600 is used to perform weighted fusion of motion anomaly heatmaps, vibration anomaly heatmaps, and structural anomaly heatmaps based on motion anomaly index, vibration anomaly index, and structural anomaly index to generate a fused heatmap, and analyze the connected domains of the fused heatmap to locate anomaly regions.

[0103] The working principle of the automated equipment anomaly monitoring system based on image processing provided by this invention is as follows:

[0104] This invention first decodes the device's operating video stream into a continuous frame sequence using a data segmentation module 100, and then segments it into single-period frame segments based on the trajectory period of a preset motion reference point. The image registration module 200 uses keyframes determined by preset registration phase points as a reference, and registers all single-period segments to a unified coordinate system through affine transformation to form a registration frame set. The motion and vibration field generation module 300 calculates the continuous optical flow field of the registration frame set, simultaneously generating a motion histogram and a motion anomaly heatmap. It also locates spectral monitoring points based on the optical flow field and extracts displacement signals, generating a vibration energy spectrum through windowed Fourier transform. The structural field analysis module 400 extracts the LBP histogram and edge gradient map of each registered image frame. Gradient maps are normalized and thermally mapped to generate single-frame structural anomaly heatmaps, which are then combined into a heatmap sequence in chronological order. The multi-domain anomaly quantization module 500 calculates motion anomaly indices based on motion histograms, generates vibration anomaly indices and vibration anomaly heatmaps based on vibration energy spectra, and generates structural anomaly indices through spatial self-reference analysis of the heatmap sequence. The anomaly fusion diagnosis module 600 finally generates a fused heatmap by dynamically weighting and fusing the motion, vibration, and structural heatmaps based on the three anomaly indices, analyzes its connected domains to locate anomaly regions, and determines the fault type within these regions by combining LBP texture features and morphological features, thus achieving fully automated monitoring from video input to fault diagnosis.

[0105] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0106] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0107] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. An automated equipment anomaly monitoring method based on image processing, characterized in that, The automated equipment anomaly monitoring method includes the following steps: S1: Decode the operating video stream of the device to be monitored into a continuous frame sequence, and divide the continuous frame sequence into single-period frame sequence segments based on the trajectory period of the preset device motion reference point. S2: Based on the registration keyframes in the single-cycle frame sequence segment determined by the preset registration phase points, the single-cycle frame sequence segment is registered to a unified coordinate system through image affine transformation to obtain a set of registration frame sequence segments. S3: Calculate the continuous optical flow field of all registered frame sequence segments in the set of registered frame sequence segments, generate motion histograms and motion anomaly heatmaps based on the continuous optical flow field, determine the spectrum monitoring points based on the continuous optical flow field, extract the displacement signals of the spectrum monitoring points, and generate vibration energy spectrum through windowed Fourier transform; S4: Extract the LBP histogram and edge gradient map of each frame image of the registration frame sequence segment, normalize the amplitude of the edge gradient map and perform thermal color level mapping to generate structural anomaly heat map, and combine the structural anomaly heat map into a structural anomaly heat map sequence in chronological order. S5: Generate motion anomaly index based on motion histogram, generate vibration anomaly heatmap and vibration anomaly index based on vibration energy spectrum, and generate structural anomaly index through spatial self-reference analysis of structural anomaly heatmap sequence; S6: Based on the motion anomaly index, vibration anomaly index, and structural anomaly index, the motion anomaly heatmap, vibration anomaly heatmap, and structural anomaly heatmap are weighted and fused to generate a fused heatmap, and the connected domains of the fused heatmap are analyzed to locate anomaly regions; In step S5, the motion anomaly index is generated by calculating the KL divergence between the actual displacement distribution represented by the motion histogram and the theoretical uniform distribution. In step S5, the generation of the vibration anomaly index includes: Identify the frequency corresponding to the maximum energy peak in the vibration energy spectrum, and determine the fundamental frequency of the device through parabolic interpolation; Based on the equipment type and fundamental frequency, the sensitive frequency band range is determined, the proportion of energy falling within the sensitive frequency band range to the total frequency band energy is calculated, and a vibration anomaly index is generated. In step S5, the generation of the structural anomaly index includes: Each frame of the heatmap sequence is divided into multiple blocks, and the gradient values ​​of all pixels in each block are sorted, with the median value taken as the median value of that block. For each block, the median of the median values ​​of the neighboring blocks is used as the health reference benchmark; Calculate the absolute difference between the median value of each block and the health reference baseline, and divide it by the standard deviation of the median value of the neighborhood blocks to obtain the standardized deviation. Blocks whose standardized deviation exceeds a preset deviation threshold are marked as abnormal blocks, and the proportion of abnormal blocks is counted to generate a structural anomaly index.

2. The automated equipment anomaly monitoring method based on image processing according to claim 1, characterized in that, In step S1, perspective projection calculation is performed using the CAD model of the device to be monitored and the camera calibration parameters, and the three-dimensional trajectory feature points of the preset device motion reference points are mapped to the image coordinate system.

3. The automated equipment anomaly monitoring method based on image processing according to claim 2, characterized in that, In step S3, the steps for determining the spectrum monitoring points include: The optical flow fields of the first and second frames of the registered frame sequence segment are obtained based on the continuous inter-frame optical flow fields; Select a pre-selected pixel block whose optical flow displacement amplitude is greater than a preset displacement amplitude and whose SIFT feature of the first frame image is greater than a preset number of feature points; Morphological clustering is performed on the pre-selected pixel blocks, and the centroid of the largest connected region is taken as the spectrum monitoring point.

4. The automated equipment anomaly monitoring method based on image processing according to claim 3, characterized in that, In step S3, the generation of the vibrational energy spectrum includes: At the coordinates of the spectrum monitoring point, the displacement amplitude of the optical flow field between consecutive frames is extracted to form a time-series signal; Apply a Hanning window function to the time-series signal to obtain a windowed signal, and perform a fast Fourier transform on the windowed signal to obtain a complex spectrum; By taking the square of the complex spectrum modulus and normalizing it, a one-sided energy distribution spectrum, i.e., the vibration energy spectrum, is generated.

5. The automated equipment anomaly monitoring method based on image processing according to claim 4, characterized in that, In step S4, the LBP histogram extraction steps include: Dynamically select the analysis scale based on the local gradient variance of each frame of the registered frame sequence segment; Small-radius operators are used to capture fine texture features in regions with high gradient variance; Large radius operators are used to extract macroscopic texture features in regions with low gradient variance; Fine-grained texture features and macro-grained texture features are fused to generate a unified texture histogram, namely the LBP histogram.

6. The automated equipment anomaly monitoring method based on image processing according to claim 5, characterized in that, In step S4, the edge gradient map extraction step includes: Each frame of the registration frame sequence segment is converted into a registration grayscale image; The initial gradient map is obtained by applying the Laplacian second-order differential operator to the registered grayscale image; The local curvature field is calculated based on the registered grayscale image. The initial gradient map is then subjected to exponential gain modulation using the local curvature field and normalized to obtain the edge gradient map.

7. The automated equipment anomaly monitoring method based on image processing according to claim 6, characterized in that, In step S5, the calculation steps for the vibration anomaly index include: Identify the frequency corresponding to the maximum energy peak in the vibration energy spectrum, and determine the fundamental frequency of the device through parabolic interpolation; The sensitive frequency band range is determined based on the equipment type and the equipment fundamental frequency. The proportion of sensitive frequency band energy falling within the sensitive frequency band range to the total frequency band energy is calculated to generate the vibration anomaly index.

8. The automated equipment anomaly monitoring method based on image processing according to claim 7, characterized in that, Step S6 also includes determining the fault type by extracting LBP texture features and morphological features within the located abnormal region.

9. An automated equipment anomaly monitoring system based on image processing, characterized in that, An automated equipment anomaly monitoring system based on image processing, as described in any one of claims 1-8, comprises: The data segmentation module is used to decode the operating video stream of the device under monitoring into a continuous frame sequence, and to segment the continuous frame sequence into single-period frame sequence segments based on the trajectory period of the preset device motion reference point. The image registration module is used to register the single-cycle frame sequence segments to a unified coordinate system through image affine transformation based on the registration phase points determined by the preset registration phase points, thereby obtaining a set of registered frame sequence segments. The motion and vibration field generation module is used to calculate the continuous optical flow field of all registered frame sequence segments in the registered frame sequence segment set, generate motion histograms and motion anomaly heatmaps based on the continuous optical flow field, determine the spectrum monitoring points based on the continuous optical flow field, extract the displacement signals of the spectrum monitoring points, and generate the vibration energy spectrum through windowed Fourier transform. The structure field analysis module is used to extract the LBP histogram and edge gradient map of each frame image of the registration frame sequence segment, perform amplitude normalization and thermal color level mapping on the edge gradient map to generate a structural anomaly heat map, and combine the structural anomaly heat maps in chronological order to form a structural anomaly heat map sequence. The multi-domain anomaly quantification module is used to generate motion anomaly indices based on motion histograms, generate vibration anomaly heatmaps and vibration anomaly indices based on vibration energy spectra, and generate structural anomaly indices through spatial self-reference analysis of structural anomaly heatmap sequences. The anomaly fusion diagnosis module is used to perform weighted fusion of motion anomaly heatmaps, vibration anomaly heatmaps, and structural anomaly heatmaps based on motion anomaly index, vibration anomaly index, and structural anomaly index to generate a fused heatmap, and analyze the connected domains of the fused heatmap to locate anomaly regions.

Images