Elevator drum brake monitoring method integrating machine vision and infrared thermal imaging

Abstract

The invention belongs to the technical field of elevator drum brakes, and relates to an elevator drum brake monitoring method fusing machine vision and infrared thermal imaging, which comprises the following steps of: acquiring an original visible light image and an original infrared thermal imaging image of a drum brake in a running state; preprocessing the original visible light image and the original infrared thermal imaging image to obtain a preprocessed visible light image and a preprocessed infrared thermal imaging image; performing image registration on the pre-processed visible light image and the pre-processed infrared thermal imaging image to obtain a registered visible light image and a registered infrared thermal imaging image; performing fusion operation on the registered visible light image and the registered infrared thermal imaging image to obtain a fused image; and performing brake state monitoring and temperature monitoring on the fused image to obtain a monitoring result. According to the invention, the running state of the brake can be rapidly monitored, and the timeliness and accuracy of monitoring data are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of elevator drum brakes, and in particular to a monitoring method for elevator drum brakes integrating machine vision and infrared thermal imaging. Background Art

[0002] The brake is one of the indispensable important safety devices of the elevator. The performance and reliability of the brake are directly related to the safe operation of the elevator. Especially in emergencies such as power failure of the elevator and emergency braking, the brake bears the responsibility of ensuring the safety of passengers' lives. Therefore, the monitoring and maintenance of the brake are very important for elevator safety.

[0003] The monitoring of elevator drum brakes is to monitor the operation of the drum brakes in the elevator system to ensure their normal operation and guarantee elevator braking. Common fault types of elevator drum brakes include: jamming caused by mechanical wear, insufficient braking force, electrical faults, and abnormal brake temperature, etc. Since each elevator has differences in performance, operating status, and operating conditions, the same monitoring method cannot obtain the same monitoring effect on different elevators.

[0004] The traditional monitoring of drum brakes most commonly uses visual observation by humans. This method has low accuracy and is inconvenient for monitoring. With the development of technology, intelligent monitoring methods have gradually been applied. The prior art proposes a method for detecting the wear of elevator brakes based on deep learning object detection, but this method is only applicable to the fault types caused by jamming due to mechanical wear and is difficult to comprehensively monitor the safety status of elevator drum brakes.

[0005] In view of this, there is an urgent need to propose a monitoring method for elevator drum brakes that can monitor the operating status of drum brakes in real time, with accurate and stable monitoring results. Summary of the Invention

[0006] The purpose of the present invention is to propose a monitoring method for elevator drum brakes integrating machine vision and infrared thermal imaging, which can realize the rapid monitoring of the operating status of the brake, ensure the timeliness and accuracy of monitoring data. At the same time, it can accurately obtain parameters such as the rotation angle, speed, and acceleration of the brake, ensure the reliability of monitoring results, and can operate stably under different environmental conditions, maintaining the stability and consistency of monitoring data.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] The present invention provides a monitoring method for elevator drum brakes integrating machine vision and infrared thermal imaging, including:

[0009] S1. Obtain the original visible light image and the original infrared thermal imaging image of the drum brake in the operating state;

[0010] S2. Preprocess the original visible light image and the original infrared thermal imaging image to obtain the preprocessed visible light image and the preprocessed infrared thermal imaging image;

[0011] S3. Perform image registration on the preprocessed visible light image and the preprocessed infrared thermal imaging image to obtain the registered visible light image and the registered infrared thermal imaging image;

[0012] S4. Perform a fusion operation on the registered visible light image and the registered infrared thermal imaging image to obtain the fused image;

[0013] S5. Monitor the brake state and temperature of the fused image to obtain the monitoring results.

[0014] As a possible implementation, S3 includes:

[0015] S30. Detect the edges of the preprocessed visible light image and the preprocessed infrared thermal imaging image respectively, extract the points with the maximum curvature on the edge contour as corner points, and obtain multiple corner points;

[0016] S31. Obtain the contour point coordinates within the minimum value point intervals on both sides of each corner point, perform Gaussian weighted averaging on each corner point and the corresponding contour point coordinates, and obtain multiple feature points;

[0017] S32. Extract the feature point descriptors of the preprocessed visible light image and the preprocessed infrared thermal imaging image respectively;

[0018] S33. Match the feature point descriptors of the preprocessed visible light image and the feature point descriptors of the preprocessed infrared thermal imaging image to obtain multiple matching points;

[0019] S34. Scale and rotate the multiple matching points to align the preprocessed visible light image and the preprocessed infrared thermal imaging image in space, and obtain the registered visible light image and the registered infrared thermal imaging image.

[0020] As a possible implementation, the bilinear interpolation method based on affine transformation is used for the image fusion operation, including:

[0021] S401. Configure the fusion function, obtain the affine transformation matrix of the registered image based on the fusion function, and calculate the inverse matrix of the affine transformation matrix;

[0022] S402. Map the pixel coordinates in the registered infrared thermal imaging image to the new image based on the inverse matrix of the affine transformation matrix, and obtain the corresponding new image coordinate points;

[0023] S403. Perform bilinear interpolation at the new image coordinate points to obtain the interpolated pixel values;

[0024] S404. Place the registered visible light image in the new image to obtain the fused image.

[0025] As a possible implementation, the principal component analysis image fusion method is used for image fusion operation, including:

[0026] S411. Extract the first eigenvector of the registered visible light image and the second eigenvector of the registered infrared thermal imaging image, and combine the first eigenvector and the second eigenvector into an eigenmatrix;

[0027] S411. Perform principal component analysis on the eigenmatrix to obtain the principal components;

[0028] S412. Configure the number of principal components to be retained, extract the features of the original visible light image and the original infrared thermal imaging image, and project the extracted features onto the retained principal components.

[0029] As a possible implementation, the state of the brake is monitored for the fused image, including:

[0030] S501. Binarize the fused image to obtain the binarized image;

[0031] S502. Perform edge detection on the binarized image to obtain the edge image;

[0032] S503. Detect the straight lines in the edge image based on the Hough transform, and calculate the rotation speed and acceleration of the brake based on the change in the straight line angle between the current frame and the previous frame in the fused image;

[0033] S504. Configure the rotation speed threshold and acceleration threshold of the brake. When the calculated rotation speed exceeds the rotation speed threshold or the calculated acceleration exceeds the acceleration threshold, it is determined that the brake state is abnormal.

[0034] As a possible implementation, the temperature of the fused image is monitored, including:

[0035] S511. Map the gray value of the preprocessed infrared thermal imaging image to a temperature value;

[0036] S512. Configure the temperature threshold. When the temperature value exceeds the temperature threshold, it is determined that the brake temperature is abnormal.

[0037] As a possible implementation, the edge detection method based on gradient calculation is used to perform edge detection on the binarized image to obtain the edge image.

[0038] As a possible implementation, S401 includes:

[0039] S4010. Extract multiple pairs of matching points in the registered image, and eliminate the abnormal matching point pairs based on the Random Sample Consensus (RANSAC) algorithm;

[0040] S4011. Calculate the optimal affine transformation matrix, where the optimal affine transformation matrix is a 2×3 affine transformation matrix;

[0041] S4012. Complete the 2×3 affine transformation matrix into a 3×3 homogeneous matrix, and calculate the inverse matrix of the homogeneous matrix.

[0042] As a possible implementation, the preprocessing includes grayscale processing, image filtering, and histogram equalization.

[0043] As a possible implementation, the median filtering method is used for image filtering; and / or, the Contrast Limited Adaptive Histogram Equalization (CLAHE) method is used for histogram equalization.

[0044] Compared with the prior art, the beneficial effects produced by the present invention are as follows:

[0045] 1. The elevator drum brake monitoring method that fuses machine vision and infrared thermal imaging proposed by the present invention preprocesses the obtained original image, including grayscale conversion, image filtering, and histogram equalization, etc. Grayscale conversion converts the color image into a grayscale image, which speeds up the processing speed; filtering is used to remove noise in the image and improve the image quality; histogram equalization is used to enhance the contrast of the image, making the key features of the brake more prominent. This can ensure the accuracy and reliability of subsequent analysis.

[0046] 2. The elevator drum brake monitoring method that fuses machine vision and infrared thermal imaging proposed by the present invention uses the points with the maximum curvature on the edge contour as corner points. The maximum curvature can accurately capture the local sharp changes of the contour, improving the geometric accuracy of corner detection. By calculating the mathematical curvature, noise interference is reduced, making it more robust than the traditional gradient method. It directly correlates with geometric features and reduces the computational complexity of feature matching or recognition.

[0047] 3. The elevator drum brake monitoring method that fuses machine vision and infrared thermal imaging proposed by the present invention performs image registration on the preprocessed visible light image and the preprocessed infrared thermal imaging image respectively, aligning the images in space. This can eliminate the differences in resolution, viewing angle, image size, etc. between the two images, and make the two images achieve the best geometric match.

[0048] 4. The present invention proposes a monitoring method for elevator drum brakes that integrates machine vision and infrared thermal imaging. By performing a fusion operation on the registered visible light image and the registered infrared thermal imaging image, a fused image with richer information and higher quality can be generated, ensuring the accuracy and reliability of subsequent monitoring.

[0049] 5. The present invention proposes a monitoring method for elevator drum brakes that integrates machine vision and infrared thermal imaging. The state and temperature of the brake are monitored for the fused image. By analyzing the brake structure information in the fused image, it can be determined whether there is wear, deformation, or other abnormal conditions in the brake. At the same time, combined with the temperature information in the infrared image, we can monitor the temperature distribution and change trend of the brake in real time, detect temperature anomalies in a timely manner, and prevent failures caused by overheating of the brake. Brief Description of the Drawings

[0050] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0051] Figure 1 is a flowchart of the monitoring method for elevator drum brakes that integrates machine vision and infrared thermal imaging provided by an embodiment of the present invention;

[0052] Figure 2 are the feature point maps of the preprocessed visible light image and the preprocessed infrared thermal imaging image in an embodiment of the present invention;

[0053] Figure 3 is the connected line diagram of the matched feature points in an embodiment of the present invention;

[0054] Figure 4 is the connected line diagram of the matched feature points after affine transformation in an embodiment of the present invention;

[0055] Figure 5 is the effect diagram obtained by performing an image fusion operation using the bilinear interpolation method based on affine transformation in an embodiment of the present invention;

[0056] Figure 6 is the effect diagram obtained by performing an image fusion operation using the principal component analysis image fusion method in an embodiment of the present invention;

[0057] Figure 7 is the effect diagram of the binary image in an embodiment of the present invention;

[0058] Figure 8 is the edge image obtained by performing edge detection on the binary image using the Canny edge detection algorithm in an embodiment of the present invention;

[0059] Figure 9Screenshot of the program interface for detecting straight lines in an edge image using the Hough transform in an embodiment of the present invention;

[0060] Figure 10 Screenshot of the interface for monitoring the brake state of the fused image in an embodiment of the present invention. Detailed implementation manners

[0061] For the convenience of clearly describing the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, terms such as "first" and "second" are used to distinguish identical or similar items with basically the same functions and roles. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and do not limit their order. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and "first", "second", etc. do not necessarily mean different.

[0062] It should be noted that in the present invention, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the present invention should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly, using words such as "exemplary" or "for example" is intended to present related concepts in a specific manner.

[0063] In the present invention, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. The following at least one (item) or its similar expression refers to any combination of these items, including any combination of single (item) or plural items (items). For example, at least one (item) of a, b or c can represent: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b and c can be single or multiple.

[0064] The embodiments of the present invention aim to provide an elevator drum brake monitoring method that combines machine vision and infrared thermal imaging, which can achieve rapid monitoring of the brake operating state, ensure the timeliness and accuracy of monitoring data. At the same time, it can accurately obtain parameters such as the rotation angle, speed and acceleration of the brake, ensure the reliability of the monitoring results, and can operate stably under different environmental conditions, maintaining the stability and consistency of the monitoring data.

[0065] See Figure 1, a monitoring method for elevator drum brakes that integrates machine vision and infrared thermal imaging, includes:

[0066] S1. Obtain the original visible light image and the original infrared thermal imaging image of the drum brake under operating conditions;

[0067] As an example, an industrial camera is used to obtain the original visible light image, and an infrared camera is used to obtain the original infrared thermal imaging image.

[0068] S2. Preprocess the original visible light image and the original infrared thermal imaging image to obtain the preprocessed visible light image and the preprocessed infrared thermal imaging image;

[0069] As a possible implementation, the preprocessing includes grayscale processing, image filtering, and histogram equalization.

[0070] As an example, the weighted average method is used for grayscale processing to convert the original color image into an 8-bit grayscale image for subsequent processing.

[0071] As a possible implementation, the median filtering method is used for image filtering.

[0072] The purpose of image filtering is to control and eliminate noise by adjusting the frequency characteristics of the image, removing high-frequency components or enhancing low-frequency components. There are various methods for filtering, including: mean filtering, Gaussian filtering, and median filtering. In this embodiment, the median filtering method is used for image filtering. The gray values of all pixels inside the image are sorted to obtain a data set, and the value at the middle position is taken from the sorted data set as the filtering output. This method can effectively process sharp noise and maintain the detail features and edge information of the image.

[0073] As a possible implementation, the contrast-limited adaptive histogram equalization method is used for histogram equalization.

[0074] In the actual operating environment of an elevator brake, due to the illumination of the light source and the imaging characteristics of the industrial camera, the captured image may be affected by optical properties, manifesting as reflection and diffuse reflection. The image acquisition environment and imaging mode of the industrial camera may cause some defects in the image, such as short exposure time, slow camera sensor response, and suboptimal ambient light. These problems result in insufficient image brightness, dark grayscale values, and unclear contrast, affecting the accuracy and analysis effect of the image recognition method. Therefore, it is necessary to enhance image contrast through histogram equalization. Histogram equalization processing includes: global histogram equalization, local adaptive histogram equalization, and contrast-limited adaptive histogram equalization. This embodiment adopts the contrast-limited adaptive histogram equalization method for histogram equalization processing. The slope of the transformation function is used to amplify the contrast near the pixel value. This slope is proportional to the slope of the domain cumulative distribution function and is therefore also proportional to the pixel histogram of the region. Contrast-limited adaptive histogram equalization limits amplification by clipping the histogram to a predetermined value before calculating the cumulative distribution function. This limits the slope of the cumulative distribution function, and thus the slope of the transfer function. When contrast amplification is limited, the appearance of noise is also limited.

[0075] After acquiring the original image, this embodiment performs image preprocessing, including grayscale conversion, image filtering, and histogram equalization. Grayscale conversion converts color images into grayscale images, speeding up processing; filtering removes image noise and improves image quality; and histogram equalization enhances image contrast, making the key features of the brake more prominent. This ensures the accuracy and reliability of subsequent analysis.

[0076] S3. Perform image registration on the preprocessed visible light image and the preprocessed infrared thermal imaging image to obtain a registered visible light image and a registered infrared thermal imaging image;

[0077] Because visible light images and infrared thermal images differ in resolution, viewing angle, and image size, image registration is required between pre-processed visible light images and pre-processed infrared thermal images. This involves finding similar features between the two images, such as corners and edges, to establish a geometric transformation relationship between them. This transformation relationship can include translation, rotation, scaling, and affine transformations, achieving optimal geometric matching between the two images.

[0078] As a possible implementation, S3 includes:

[0079] S30. Detecting edges of the pre-processed visible light image and the pre-processed infrared thermal imaging image, extracting the maximum curvature on the edge contour as a corner point, and obtaining multiple corner points;

[0080] As an example, the Canny edge detection algorithm is used to detect the edges of an image. The Canny edge detection algorithm uses a Gaussian filter to smooth the preprocessed visible light image and the preprocessed infrared thermal imaging image to reduce the influence of noise on edge detection. The Gaussian filter can retain more image detail information after smoothing the image. The two-dimensional Gaussian function is as follows:

[0081]

[0082] where (x, y) are coordinates and σ is the standard deviation of the Gaussian function, which determines the width of the Gaussian filter. When σ is large, the Gaussian filter will perform stronger smoothing on the image, thus better suppressing noise, but it may also lose some image details. On the contrary, when σ is small, the smoothing effect of the filter on the image is weak, and it can better retain image details, but the noise suppression effect may not be ideal enough. Next, the Sobel operator is used to calculate the gradients of each pixel point in the image to accurately capture the intensity and direction of the edges. The gradient formula is as follows:

[0083]

[0084]

[0085] The above formula gives the angle between the edge direction and the horizontal axis. By knowing the gradient direction and amplitude of each pixel point, the edge information in the image can be more accurately located and analyzed.

[0086] Then non-maximum suppression is adopted to eliminate non-edge pixels based on these gradient information to ensure that only the real edges are retained. It is achieved by comparing the gradient amplitude of each pixel point with that of its adjacent pixel points in its gradient direction. If the gradient amplitude of a pixel point is locally maximum in its gradient direction, then this pixel point is considered to be part of the edge and is retained. On the contrary, if the gradient amplitude of a pixel point is not locally maximum, then this pixel point is suppressed or eliminated, and thus a more accurate and refined edge image is obtained.

[0087] As an example, a scale-space corner detector is used to extract the curvature maximum points on the edge contour as corner points. The curvature maximum can accurately capture the local sharp changes of the contour, improve the geometric accuracy of corner detection, reduce noise interference through mathematical curvature calculation, be more robust than the traditional gradient method, directly associate with geometric features, and reduce the computational complexity of feature matching or recognition.

[0088] S31. Obtain the contour point coordinates within the minimum value point intervals on both sides of each corner point, and perform Gaussian weighted averaging on each corner point and the corresponding contour point coordinates to obtain multiple feature points;

[0089] SeeFigure 2 , after the processing of S31, it is possible to remove the influence of contour burrs, smooth the edges, and make the contour line segments of the same feature point in the preprocessed visible light image and the preprocessed infrared thermal imaging image more similar.

[0090] S32. Respectively extract the feature point descriptors of the preprocessed visible light image and the preprocessed infrared thermal imaging image;

[0091] As an example, the Scale Invariant Feature Transform (SIFT) algorithm is used to detect the feature points in the image. The SIFT algorithm detects these key points by constructing a Difference of Gaussian (DOG) scale space. The DOG scale space is obtained by subtracting Gaussian filtered images of different scales, which can effectively highlight the local extreme points in the image. When the key points are detected, the SIFT algorithm selects a fixed-size region around each key point and calculates the gradient direction histogram of the pixels in this region. This histogram contains the shape and texture information of the region around the key point, forming a feature descriptor. The SIFT descriptor has rotational invariance and scale invariance. Even if the image undergoes rotation or scale change, the corresponding key points can be found by matching these descriptors.

[0092] S33. Match the feature point descriptors of the preprocessed visible light image and the preprocessed infrared thermal imaging image to obtain multiple matching points;

[0093] As an example, for a feature point A in the preprocessed visible light image, we find a feature point B in the preprocessed infrared thermal imaging image that is most similar to its descriptor; for the feature point B in the preprocessed infrared thermal imaging image, we also find a feature point C in the preprocessed visible light image that is most similar to its descriptor. If A and C are the same feature point, then A and B are considered a pair of correct matching points.

[0094] S34. Scale and rotate the multiple matching points to align the preprocessed visible light image and the preprocessed infrared thermal imaging image in space, obtaining the registered visible light image and the registered infrared thermal imaging image.

[0095] See Figures 3 to 4 , as an example, perform scaling and rotation adjustment on the matched feature points, select a set of matched feature points to calculate the scale factor and relative direction difference in the two images, scale the image according to the scale factor, and use affine transformation to match the angle of one image with that of the other according to the relative direction difference to adapt to the possible scale and direction differences between different images and align the images in space.

[0096] S4. Perform a fusion operation on the registered visible light image and the registered infrared thermal imaging image to obtain a fused image;

[0097] Image fusion is used to combine information from multiple image sources from different sensors, different time points, or different perspectives to generate an image with richer information and higher quality.

[0098] As a possible implementation, a bilinear interpolation method based on affine transformation is used for the image fusion operation, and the obtained effect diagram is as shown in Figure 5 shown. This method includes:

[0099] S401. Configure a fusion function, obtain an affine transformation matrix of the registered image based on the fusion function, and calculate the inverse matrix of the affine transformation matrix;

[0100] As a possible implementation, S401 includes:

[0101] S4010. Extract multiple matching point pairs in the registered image, and eliminate abnormal matching point pairs based on the random sample consensus algorithm;

[0102] S4011. Call the cv2.estimateAffine2D function to calculate the optimal affine transformation matrix, and the optimal affine transformation matrix is a 2×3 affine transformation matrix;

[0103] S4012. Complete the 2×3 affine transformation matrix into a 3×3 homogeneous matrix, and calculate the inverse matrix of the homogeneous matrix based on the np.linalg.inv function.

[0104] S402. Map the pixel coordinates in the registered infrared thermal imaging image to a new image based on the inverse matrix of the affine transformation matrix, and obtain the corresponding new image coordinate points;

[0105] S403. Perform a bilinear interpolation operation at the new image coordinate points to obtain the interpolated pixel values;

[0106] S404. Place the registered visible light image on the new image to obtain a fused image.

[0107] As an example, through the inverse matrix of the affine transformation matrix, the pixel coordinates (u, v) in the infrared image can be mapped to a new image to obtain the corresponding coordinates (ut, vt). At the coordinates (ut, vt) mapped to the new image, a bilinear interpolation operation is performed to obtain the interpolated pixel values. Placing the visible light image directly on the new image can obtain the fused image. The following is the script of this method:

[0108]

[0109]

[0110] As a possible implementation, the principal component analysis image fusion method is used for image fusion operation. The effect diagram obtained by this method is as shown in Figure 6 shown. This method includes:

[0111] S411. Extract the first eigenvector of the registered visible light image and the second eigenvector of the registered infrared thermal imaging image, and combine the first eigenvector and the second eigenvector into an eigenmatrix;

[0112] As an example, in MATLAB, extract the first eigenvector of the registered visible light image and the second eigenvector of the registered infrared thermal imaging image, and combine the first eigenvector and the second eigenvector into a large eigenmatrix. Each row of this matrix represents a pixel, and each column represents a feature.

[0113] S411. Perform principal component analysis on the eigenmatrix to obtain the principal components;

[0114] S412. Configure the number of principal components to be retained, extract the features of the original visible light image and the original infrared thermal imaging image, and project the extracted features onto the retained principal components.

[0115] As an example, after constructing the eigenmatrix, perform PCA analysis on the eigenmatrix, find the principal components, and select the number of principal components to be retained, and project the original features onto the principal components. The script of this fusion method is as follows:

[0116] % Assume that kejianGray and hongwaiGray are both image matrices that have been read and converted to grayscale

[0117] % Flatten all pixel values of kejianGray into a one-dimensional array

[0118] zkejianVector = flatten(visibleImageGray)

[0119] % Flatten all pixel values of hongwaiGray into a one-dimensional array

[0120] zhongwaiVector = flatten(infraredImageGray)

[0121] % Create an empty eigenmatrix matrix, which will contain two column vectors: one from the visible light image and one from the infrared image

[0122] matrix = initialize an empty matrix

[0123] % Merge the pixel vectors of the visible light image and the infrared image into the matrix

[0124] matrix = merge([zkejian, zhongwai])

[0125] % matrix is now a two-dimensional matrix, where each row represents a pixel, and each column represents the pixel values of the visible light image and the infrared image respectively

[0126] % Calculate the covariance matrix

[0127] covmatrix = calculateCovariance(matrix)

[0128] % Calculate the eigenvectors and eigenvalues of the covariance matrix

[0129] [vector, ~] = calculateEigenvaluesAndEigenvectors(covmatrix)

[0130] % Sort the eigenvectors according to the corresponding eigenvalue magnitudes

[0131] % First, obtain the diagonal elements of the covariance matrix (i.e., the eigenvalues)

[0132] values = getDiagonalElements(covmatrix)

[0133] % Sort the eigenvalues in descending order and obtain the sorting indices

[0134] [~, idx] = sortArray(values, 'descending')

[0135] % Rearrange the eigenvectors using the sorting indices

[0136] vector = rearrange(vector, idx)

[0137] % Select the number of principal components to retain (e.g., select the first N)

[0138] N = 2 % This is just an example, and you need to select the value of N according to the specific situation

[0139] % Extract the first N eigenvectors as the principal components

[0140] pc = extractColumns(vector, 1 to N)

[0141] % Project the original features onto the principal components

[0142] pf = matrixMultiplication(featureMatrix, pc).

[0143] S5. Monitor the brake state and temperature of the fused image to obtain the monitoring results.

[0144] Before the drum brake works, make a straight line mark on the brake drum. During the working process of the brake, the straight line mark rotates with the brake drum. The rotation state of the brake can be calculated through the rotation of the straight line mark on the surface of the brake drum.

[0145] As a possible implementation, monitoring the brake state of the fused image includes:

[0146] S501. Binarize the fused image to obtain the binarized image;

[0147] Binarization will set the gray value of the pixel points of the image to 0 or 255, that is, convert the image into a form with only black and white colors. The foreground and background of the image are segmented. In this monitoring process, the straight line mark is the foreground, and the rest is the background. Through binarization segmentation, the straight line mark can be made clearer and more conducive to straight line detection. The binarized image is as Figure 7 shown.

[0148] As an example, use the Ostu method for binarization. Calculate the gray histogram of each frame of the fused image, count the number of pixels at each gray level, traverse the possible gray levels, and calculate the within-class variance of the foreground and background for each gray level. Finally, select the optimal threshold according to the value of the within-class variance to minimize the within-class variance, and use this threshold to perform binarization segmentation on the image, classifying the pixels with gray values greater than the threshold as the foreground and the pixels with gray values less than or equal to the threshold as the background.

[0149] S502. Detect the edges of the binarized image to obtain the edge image;

[0150] See Figure 8 As a possible implementation, still use the Canny edge detection algorithm to detect the edges of the binarized image to obtain the edge image.

[0151] S503. Detect the straight lines in the edge image based on the Hough transform, and calculate the rotation speed and acceleration of the brake based on the change in the straight line angle between the current frame and the previous frame in the fused image;

[0152] As an example, the program interface for straight line detection is as Figure 9 shown. When the slope of the straight line is obtained, the arctangent function can be used to convert the slope into an angle. In each frame, compare the angle and time of the previous frame with the angle and time of the current frame to calculate the change rate of the rotation speed and acceleration.

[0153] S504. Configure the rotational speed threshold and acceleration threshold of the brake. When the calculated rotational speed exceeds the rotational speed threshold or the calculated acceleration exceeds the acceleration threshold, it is determined that the brake state is abnormal.

[0154] As an example, determine an operating threshold range before monitoring. When the rotational speed or rotational acceleration of the brake exceeds the threshold range, it is recognized as an abnormal state and the abnormal state is recorded. Display the operating parameters in the form of a line chart, and show the abnormal state stage as Figure 10 shown by the black solid line in

[0155] As a possible implementation, perform temperature monitoring on the fused image, including:

[0156] S511. Map the gray value of the preprocessed infrared thermal imaging image to a temperature value;

[0157] As an example, use a calibration function to map the gray value to a temperature value. Map the gray levels in the image to the actual temperature range. Generally, the calibration function is provided by the camera manufacturer to associate the pixel value with the corresponding temperature value to ensure an accurate mapping between the gray levels in the image and the actual temperature.

[0158] S512. Configure the temperature threshold. When the temperature value exceeds the temperature threshold, it is determined that the brake temperature is abnormal.

[0159] As an example, after completing the mapping from the gray value to the temperature value, determine the temperature change of the brake according to the change of the gray value. Set a temperature threshold that meets the regulations. When the temperature exceeds the set threshold, it can be defined as a temperature abnormality. The abnormal temperature will be recorded and trigger a temperature abnormality alarm to take corresponding measures in a timely manner.

[0160] In this embodiment, the brake state monitoring and temperature monitoring are performed on the fused image. By analyzing the brake structure information in the fused image, it can be judged whether there is wear, deformation or other abnormal conditions in the brake. At the same time, combined with the temperature information in the infrared image, we can monitor the temperature distribution and change trend of the brake in real time, discover temperature abnormalities in time, and prevent failures caused by overheating of the brake.

[0161] Although the present invention has been described in connection with various embodiments, those skilled in the art will recognize other variations of the disclosed embodiments while practicing the claimed invention, by viewing the drawings, the disclosure, and the accompanying description. In the specification, the word "comprising" does not exclude other components or steps, and the singular "a" or "one" does not exclude a plurality. A single processor or other unit may perform several functions recited in the specification. Certain measures are recited in mutually different embodiments, but this does not mean that these measures cannot be combined to produce favorable results.

[0162] Although the invention has been described in connection with specific features and embodiments thereof, it will be apparent that various modifications and combinations can be made without departing from the spirit and scope of the invention. Accordingly, the specification and drawings are merely exemplary of the invention and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Obviously, those skilled in the art can make various changes and modifications to the invention without departing from the spirit and scope of the invention. Thus, if these modifications and variations of the invention fall within the scope of the invention and its equivalent technologies, the invention is also intended to include these changes and modifications.

Claims

1. A monitoring method for elevator drum brakes integrating machine vision and infrared thermal imaging, characterized in that, Including: S1. Obtain the original visible light image and the original infrared thermal imaging image under the operating state of the drum brake; S2. Preprocess the original visible light image and the original infrared thermal imaging image to obtain the preprocessed visible light image and the preprocessed infrared thermal imaging image; S3. Perform image registration on the preprocessed visible light image and the preprocessed infrared thermal imaging image to obtain the registered visible light image and the registered infrared thermal imaging image; S4. Perform a fusion operation on the registered visible light image and the registered infrared thermal imaging image to obtain a fused image; S5. Monitor the brake state and temperature of the fused image to obtain a monitoring result.

2. The elevator drum brake monitoring method integrating machine vision and infrared thermal imaging according to claim 1, wherein The S3 includes: S30. Detect the edges of the preprocessed visible light image and the preprocessed infrared thermal imaging image respectively, extract the points with the maximum curvature on the edge contour as corner points, and obtain a plurality of corner points; S31. Obtain the contour point coordinates within the minimum value point interval on both sides of each corner point, perform Gaussian weighted averaging on each corner point and the corresponding contour point coordinates, and obtain a plurality of feature points; S32. Extract the feature point descriptors of the preprocessed visible light image and the preprocessed infrared thermal imaging image respectively; S33. Match the feature point descriptors of the preprocessed visible light image and the preprocessed infrared thermal imaging image to obtain a plurality of matching point pairs; S34. Scale and rotate the plurality of matching point pairs so that the preprocessed visible light image and the preprocessed infrared thermal imaging image are aligned in space, and obtain the registered visible light image and the registered infrared thermal imaging image.

3. The method for monitoring an elevator drum brake by integrating machine vision and infrared thermal imaging according to claim 2, wherein The image fusion operation is performed using the bilinear interpolation method based on affine transformation, including: S401. Configure a fusion function, obtain the affine transformation matrix of the registered image based on the fusion function, and calculate the inverse matrix of the affine transformation matrix; S402. Map the pixel coordinates in the registered infrared thermal imaging image to a new image based on the inverse matrix of the affine transformation matrix, and obtain the corresponding new image coordinate points; S403. Perform bilinear interpolation operation at the new image coordinate points to obtain the interpolated pixel values; S404. Place the registered visible light image on the new image to obtain a fused image.

4. The elevator drum brake monitoring method integrating machine vision and infrared thermal imaging according to claim 1, characterized in that The image fusion operation is performed using the principal component analysis image fusion method, including: S411. Extract the first eigenvector of the registered visible light image and the second eigenvector of the registered infrared thermal imaging image, and combine the first eigenvector and the second eigenvector into an eigenmatrix; S411. Perform principal component analysis on the eigenmatrix to obtain the principal components; S412. Configure the number of principal components to be retained, extract the features of the original visible light image and the original infrared thermal imaging image, and project the extracted features onto the retained principal components.

5. The elevator drum brake monitoring method integrating machine vision and infrared thermal imaging according to claim 1, characterized in that Monitoring the brake state of the fused image includes: S501. Binarize the fused image to obtain a binarized image; S502. Perform edge detection on the binarized image to obtain an edge image; S503. Detect straight lines in the edge image based on the Hough transform, and calculate the rotation speed and acceleration of the brake based on the change in the straight line angle between the current frame and the previous frame in the fused image; S504. Configure the rotation speed threshold and acceleration threshold of the brake. When the calculated rotation speed exceeds the rotation speed threshold or the calculated acceleration exceeds the acceleration threshold, determine that the brake state is abnormal.

6. The elevator drum brake monitoring method integrating machine vision and infrared thermal imaging according to claim 1, wherein Perform temperature monitoring on the fused image, including: S511. Map the gray value of the preprocessed infrared thermal imaging image to a temperature value; S512. Configure the temperature threshold. When the temperature value exceeds the temperature threshold, determine that the brake temperature is abnormal.

7. The method for monitoring an elevator drum brake by fusing machine vision and infrared thermal imaging according to claim 5, wherein Perform edge detection on the binarized image using an edge detection method based on gradient calculation to obtain an edge image.

8. The method for monitoring an elevator drum brake by integrating machine vision and infrared thermal imaging according to claim 3, wherein The S401 includes: S4010. Extract multiple pairs of matching points in the registered image, and eliminate abnormal pairs of matching points based on the random sample consensus algorithm; S4011. Calculate the optimal affine transformation matrix, where the optimal affine transformation matrix is a 2×3 affine transformation matrix; S4012. Complete the 2×3 affine transformation matrix to a 3×3 homogeneous matrix, and calculate the inverse matrix of the homogeneous matrix.

9. The method for monitoring an elevator drum brake by integrating machine vision and infrared thermal imaging according to claim 1, wherein The preprocessing includes grayscale processing, image filtering, and histogram equalization.

10. The elevator drum brake monitoring method integrating machine vision and infrared thermal imaging according to claim 9, wherein, Perform image filtering using the median filtering method; and / or, perform histogram equalization using the adaptive histogram equalization method with contrast limiting.

Images