Elevator drum brake monitoring method fusing machine vision and infrared thermography

Abstract

The present application belongs to the technical field of elevator drum brake, and relates to a kind of monitoring method of elevator drum brake that fuses machine vision and infrared thermal imaging, the method includes the following steps: obtaining original visible light image and original infrared thermal imaging image under the running state of drum brake;Original visible light image and original infrared thermal imaging image are preprocessed, and obtain preprocessed visible light image and preprocessed infrared thermal imaging image;Image registration is carried out to preprocessed visible light image and preprocessed infrared thermal imaging image, and obtain registered visible light image and registered infrared thermal imaging image;Fusion operation is carried out to registered visible light image and registered infrared thermal imaging image, and obtain fused image;Brake state monitoring and temperature monitoring are carried out to fused image to obtain monitoring result.The present application can realize the rapid monitoring of brake running state, ensure the timeliness and accuracy of monitoring data.

Description

Technical Field

[0001] This invention relates to the field of elevator drum brake technology, and in particular to a monitoring method for elevator drum brakes that integrates machine vision and infrared thermal imaging. Background Technology

[0002] The brake is an indispensable and crucial safety device in elevators. Its performance and reliability directly affect the elevator's operational safety. Especially in emergencies such as power outages or emergency braking, the brake bears the responsibility of protecting passenger lives. Therefore, monitoring and maintaining the brake is extremely important for elevator safety.

[0003] Elevator drum brake monitoring involves monitoring the operation of the drum brake in the elevator system to ensure its normal operation and guarantee elevator braking. Common fault types of elevator drum brakes include: mechanical wear leading to jamming, insufficient braking force, electrical faults, and abnormal brake temperature. Because each elevator differs in performance, operating status, and operating conditions, the same monitoring methods will not yield the same monitoring results on different elevators.

[0004] Traditionally, monitoring of drum brakes most commonly relies on visual observation, which is inaccurate and inconvenient. With technological advancements, intelligent monitoring methods are gradually being applied. Existing technology proposes a wear detection method for elevator brakes based on deep learning object detection; however, this method is only applicable to faults caused by mechanical wear leading to jamming, and cannot comprehensively monitor the safety status of elevator drum brakes.

[0005] Therefore, there is an urgent need to propose a monitoring method for elevator drum brakes that can monitor the operating status of the drum brakes in real time and provide accurate, stable and reliable monitoring results. Summary of the Invention

[0006] The purpose of this invention is to propose a monitoring method for elevator drum brakes that integrates machine vision and infrared thermal imaging. This method enables rapid monitoring of the brake's operating status, ensuring the timeliness and accuracy of the monitoring data. Simultaneously, it can accurately acquire parameters such as the brake's rotation angle, speed, and acceleration, guaranteeing the reliability of the monitoring results. Furthermore, it can operate stably under different environmental conditions, maintaining the stability and consistency of the monitoring data.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] This invention provides a method for monitoring elevator drum brakes that integrates machine vision and infrared thermal imaging, comprising:

[0009] S1. Acquire the raw visible light image and raw infrared thermal image of the drum brake in operation;

[0010] S2. Preprocess the original visible light image and the original infrared thermal imaging image to obtain the preprocessed visible light image and the preprocessed infrared thermal imaging image.

[0011] S3. Perform image registration on the preprocessed visible light image and the preprocessed infrared thermal imaging image to obtain the registered visible light image and the registered infrared thermal imaging image.

[0012] S4. Perform a fusion operation on the registered visible light image and the registered infrared thermal image to obtain the fused image;

[0013] S5. Perform brake status monitoring and temperature monitoring on the fused image to obtain monitoring results.

[0014] As one possible implementation, S3 includes:

[0015] S30. Detect the edges of the preprocessed visible light image and the preprocessed infrared thermal imaging image respectively, extract the curvature maxima on the edge contour as corner points, and obtain multiple corner points;

[0016] S31. Obtain the coordinates of the contour points within the minimum point interval on both sides of each corner point, and perform a Gaussian weighted average of the coordinates of each corner point and the corresponding contour points to obtain multiple feature points;

[0017] S32. Extract feature point descriptors from the preprocessed visible light image and the preprocessed infrared thermal imaging image, respectively;

[0018] S33. Match the feature point descriptors of the preprocessed visible light image and the feature point descriptors of the preprocessed infrared thermal imaging image to obtain multiple matching points;

[0019] S34. Scale and rotate multiple matching points to align the preprocessed visible light image and the preprocessed infrared thermal image in space, thereby obtaining the registered visible light image and the registered infrared thermal image.

[0020] As one possible implementation, image fusion is performed using bilinear interpolation based on affine transformation, including:

[0021] S401. Configure the fusion function, obtain the affine transformation matrix of the registered image based on the fusion function, and calculate the inverse matrix of the affine transformation matrix;

[0022] S402. Based on the inverse matrix of the affine transformation matrix, the pixel coordinates in the registered infrared thermal imaging image are mapped to the new image, and the corresponding new image coordinate points are obtained;

[0023] S403. Perform bilinear interpolation at the new image coordinates to obtain the interpolated pixel values;

[0024] S404. Place the registered visible light image onto the new image to obtain the fused image.

[0025] As one possible implementation, principal component analysis image fusion is used for image fusion operations, including:

[0026] S411. Extract the first feature vector of the registered visible light image and the second feature vector of the registered infrared thermal image, and combine the first feature vector and the second feature vector into a feature matrix;

[0027] S411. Perform principal component analysis on the feature matrix to obtain the principal components;

[0028] S412. Configure the number of principal components to be retained, extract features from the original visible light image and the original infrared thermal image, and project the extracted features onto the retained principal components.

[0029] As one possible implementation, brake state monitoring is performed on the fused image, including:

[0030] S501. Binarize the fused image to obtain a binarized image;

[0031] S502. Perform edge detection on the binarized image to obtain the edge image;

[0032] S503. Detect straight lines in the edge image based on Hough transform, and calculate the rotational speed and acceleration of the brake based on the change in the angle between the straight lines in the current frame and the previous frame in the fused image;

[0033] S504. Configure the rotational speed threshold and acceleration threshold of the brake. When the calculated rotational speed exceeds the rotational speed threshold or the calculated acceleration exceeds the acceleration threshold, the brake status is determined to be abnormal.

[0034] As one possible implementation, temperature monitoring of the fused image includes:

[0035] S511. Map the grayscale values ​​of the preprocessed infrared thermal imaging image to temperature values;

[0036] S512. Configure a temperature threshold. When the temperature value exceeds the temperature threshold, the brake temperature is determined to be abnormal.

[0037] As one possible implementation, an edge detection method based on gradient calculation is used to perform edge detection on the binarized image to obtain an edge image.

[0038] As one possible implementation, S401 includes:

[0039] S4010. Extract multiple matching point pairs from the registered image and remove abnormal matching point pairs based on the random sample consensus algorithm;

[0040] S4011. Calculate the optimal affine transformation matrix, which is a 2×3 affine transformation matrix;

[0041] S4012. Complete the 2×3 affine transformation matrix into a 3×3 homogeneous matrix, and calculate the inverse of the homogeneous matrix.

[0042] As one possible implementation, preprocessing includes grayscale conversion, image filtering, and histogram equalization.

[0043] As one possible implementation, median filtering is used for image filtering; and / or, contrast-limited adaptive histogram equalization is used for histogram equalization.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] 1. The elevator drum brake monitoring method proposed in this invention integrates machine vision and infrared thermal imaging. It preprocesses the acquired raw image, including grayscale conversion, image filtering, and histogram equalization. Grayscale conversion converts the color image to grayscale, accelerating the processing speed; filtering removes noise from the image, improving image quality; and histogram equalization enhances image contrast, making the key features of the brake more prominent. This ensures the accuracy and reliability of subsequent analysis.

[0046] 2. The elevator drum brake monitoring method proposed in this invention integrates machine vision and infrared thermal imaging. It takes the curvature maxima on the edge contour as corner points. The curvature maxima can accurately capture sharp changes in the contour, improve the geometric accuracy of corner detection, reduce noise interference through mathematical curvature calculation, is more robust than the traditional gradient method, directly associates geometric features, and reduces the computational complexity of feature matching or recognition.

[0047] 3. The elevator drum brake monitoring method proposed in this invention integrates machine vision and infrared thermal imaging. It performs image registration on the pre-processed visible light image and the pre-processed infrared thermal imaging image respectively, so that the images are aligned in space. This can remove the differences between the two images in terms of resolution, viewing angle and image size, so that the two images achieve the best geometric matching.

[0048] 4. This invention proposes a monitoring method for elevator drum brakes that integrates machine vision and infrared thermal imaging. By fusing the registered visible light image and the registered infrared thermal image, a fused image with richer information and higher quality can be generated, ensuring the accuracy and reliability of subsequent monitoring.

[0049] 5. This invention proposes a monitoring method for elevator drum brakes that integrates machine vision and infrared thermal imaging. The fused image is used for brake status and temperature monitoring. By analyzing the brake structure information in the fused image, it is possible to determine whether the brake has wear, deformation, or other abnormalities. Simultaneously, by combining the temperature information from the infrared image, we can monitor the brake's temperature distribution and trends in real time, promptly detect temperature anomalies, and prevent brake overheating-related malfunctions. Attached Figure Description

[0050] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0051] Figure 1 A flowchart of an elevator drum brake monitoring method integrating machine vision and infrared thermal imaging provided in an embodiment of the present invention;

[0052] Figure 2 These are the feature point maps of the preprocessed visible light image and the preprocessed infrared thermal imaging image in the embodiments of the present invention;

[0053] Figure 3 This is a diagram showing the matching feature points in an embodiment of the present invention.

[0054] Figure 4 This is a diagram showing the matching feature points after affine transformation in an embodiment of the present invention.

[0055] Figure 5 This is an image of the result obtained by using bilinear interpolation based on affine transformation in an embodiment of the present invention for image fusion operation;

[0056] Figure 6 This is an image obtained by performing image fusion using principal component analysis in an embodiment of the present invention.

[0057] Figure 7 This is a diagram showing the binarized image in an embodiment of the present invention.

[0058] Figure 8 This is an edge image obtained by performing edge detection on a binarized image using the Canny edge detection algorithm in an embodiment of the present invention;

[0059] Figure 9This is a screenshot of the program interface for detecting straight lines in an image using Hough transform in an embodiment of the present invention.

[0060] Figure 10 This is a screenshot of the brake status monitoring interface for the fused image in an embodiment of the present invention. Detailed Implementation

[0061] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0062] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0063] In this invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, "at least one of a, b, or c" can represent: a, b, c, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0064] This invention aims to provide a monitoring method for elevator drum brakes that integrates machine vision and infrared thermal imaging. This method enables rapid monitoring of the brake's operating status, ensuring the timeliness and accuracy of the monitoring data. Simultaneously, it can accurately acquire parameters such as the brake's rotation angle, speed, and acceleration, guaranteeing the reliability of the monitoring results. Furthermore, it can operate stably under different environmental conditions, maintaining the stability and consistency of the monitoring data.

[0065] See Figure 1A monitoring method for elevator drum brakes that integrates machine vision and infrared thermal imaging includes:

[0066] S1. Acquire the raw visible light image and raw infrared thermal image of the drum brake in operation;

[0067] As an example, an industrial camera is used to acquire raw visible light images, and an infrared camera is used to acquire raw infrared thermal images.

[0068] S2. Preprocess the original visible light image and the original infrared thermal imaging image to obtain the preprocessed visible light image and the preprocessed infrared thermal imaging image.

[0069] As one possible implementation, preprocessing includes grayscale conversion, image filtering, and histogram equalization.

[0070] As an example, a weighted average method is used for grayscale processing, converting the original color image into an 8-bit grayscale image for subsequent processing.

[0071] As one possible approach, median filtering is used for image filtering.

[0072] The purpose of image filtering is to control and eliminate noise by adjusting the frequency characteristics of the image, removing high-frequency components or enhancing low-frequency components. There are various filtering methods, including mean filtering, Gaussian filtering, and median filtering. This embodiment uses median filtering for image filtering. The gray values ​​of all pixels in the image are sorted to obtain a dataset. The value at the middle position is taken from the sorted dataset as the filtering output. This method can effectively handle sharp noise while preserving the image's detailed features and edge information.

[0073] As one possible implementation, an adaptive histogram equalization method with limited contrast is used for histogram equalization.

[0074] In the actual working environment of elevator brakes, due to the illumination of light sources and the imaging characteristics of industrial cameras, the acquired images may be affected by optical properties, exhibiting reflection and diffuse reflection phenomena. The image acquisition environment and imaging mode of industrial cameras may lead to some defects in the images, such as short exposure time, slow camera sensor response, and unsatisfactory ambient light, resulting in insufficient image brightness, dark grayscale values, and unclear contrast, affecting the accuracy and analysis effect of image recognition methods. Therefore, histogram equalization processing is needed to enhance image contrast. Histogram equalization processing includes: global histogram equalization, local adaptive histogram equalization, and contrast-limited adaptive histogram equalization. This embodiment uses the contrast-limited adaptive histogram equalization method for histogram equalization processing, using the slope of the transformation function to amplify the contrast near the pixel value. This slope is proportional to the slope of the neighborhood cumulative distribution function, and therefore also proportional to the pixel histogram of that region. Contrast-limited adaptive histogram equalization clips the histogram to a predetermined value before calculating the cumulative distribution function to limit amplification, thereby limiting the slope of the cumulative distribution function and thus the slope of the transformation function. When contrast amplification is limited, the appearance of noise is also limited.

[0075] This embodiment performs preprocessing on the acquired original image, including grayscale conversion, image filtering, and histogram equalization. Grayscale conversion converts the color image to a grayscale image, accelerating the processing speed; filtering removes noise from the image, improving image quality; and histogram equalization enhances the image contrast, making the key features of the actuator more prominent. This ensures the accuracy and reliability of subsequent analysis.

[0076] S3. Perform image registration on the preprocessed visible light image and the preprocessed infrared thermal imaging image to obtain the registered visible light image and the registered infrared thermal imaging image.

[0077] Because visible light images and infrared thermal images differ in resolution, viewing angle, and image size, image registration is necessary between the preprocessed visible light images and the preprocessed infrared thermal images. This involves identifying similar features between the two images, such as corners and edges, to establish their geometric transformation relationship. This transformation relationship can include various types such as translation, rotation, scaling, and affine transformations, to achieve the best geometric match between the two images.

[0078] As one possible implementation, S3 includes:

[0079] S30. Detect the edges of the preprocessed visible light image and the preprocessed infrared thermal imaging image respectively, extract the curvature maxima on the edge contour as corner points, and obtain multiple corner points;

[0080] As an example, the Canny edge detection algorithm is used to detect image edges. The Canny algorithm uses a Gaussian filter to smooth both the preprocessed visible light image and the preprocessed infrared thermal image to reduce the impact of noise on edge detection. The Gaussian filter preserves more image detail after smoothing the image. The two-dimensional Gaussian function is as follows:

[0081]

[0082] Where (x, y) are coordinates, and σ is the standard deviation of the Gaussian function, which determines the width of the Gaussian filter. When σ is large, the Gaussian filter performs stronger smoothing on the image, thus better suppressing noise, but may also lose some image details. Conversely, when σ is small, the filter has a weaker smoothing effect on the image, better preserving image details, but may not be as effective at suppressing noise. Next, the Sobel operator is used to calculate the gradient of each pixel in the image to accurately capture the intensity and direction of edges. The gradient formula is as follows:

[0083]

[0084]

[0085] The above formula gives the angle between the edge direction and the horizontal axis. By knowing the gradient direction and magnitude of each pixel, edge information in the image can be located and analyzed more accurately.

[0086] Then, non-maximum suppression is employed, using this gradient information to remove non-edge pixels, ensuring that only true edges are retained. This is achieved by comparing the gradient magnitude of each pixel with its neighboring pixels along its gradient direction. If a pixel's gradient magnitude is locally maximum along its gradient direction, then this pixel is considered part of an edge and is retained. Conversely, if a pixel's gradient magnitude is not locally maximum, then this pixel is suppressed or removed, resulting in a more accurate and refined edge image.

[0087] As an example, a scale-space angle detector is used to extract the curvature maxima on the edge contour as corner points. Curvature maxima can accurately capture sharp local changes in the contour, improve the geometric accuracy of corner detection, reduce noise interference through mathematical curvature calculation, are more robust than traditional gradient methods, directly associate geometric features, and reduce the computational complexity of feature matching or recognition.

[0088] S31. Obtain the coordinates of the contour points within the minimum point interval on both sides of each corner point, and perform a Gaussian weighted average of the coordinates of each corner point and the corresponding contour points to obtain multiple feature points;

[0089] See Figure 2 After processing by S31, the influence of contour burrs can be removed, the edges can be smoothed, and the contour lines of the same feature point in the preprocessed visible light image and the preprocessed infrared thermal image can be more similar.

[0090] S32. Extract feature point descriptors from the preprocessed visible light image and the preprocessed infrared thermal imaging image, respectively;

[0091] As an example, the Scale Invariant Feature Transform (SIFT) algorithm is used to detect keypoints in an image. SIFT detects these keypoints by constructing a Difference-of-Gaussian (DOG) scale-space. The DOG scale-space is obtained by subtracting Gaussian-filtered images at different scales, effectively highlighting local extrema in the image. Once a keypoint is detected, SIFT selects a fixed-size region around each keypoint and calculates a histogram of gradient directions for the pixels within that region. This histogram contains shape and texture information about the region surrounding the keypoint, forming a feature descriptor. SIFT descriptors are rotation-invariant and scale-invariant, meaning that even if the image is rotated or scaled, the corresponding keypoints can be found by matching these descriptors.

[0092] S33. Match the feature point descriptors of the preprocessed visible light image and the feature point descriptors of the preprocessed infrared thermal imaging image to obtain multiple matching points;

[0093] As an example, for a feature point A in the preprocessed visible light image, we find a feature point B in the preprocessed infrared thermal image that is most similar to its descriptor; for feature point B in the preprocessed infrared thermal image, we also find a feature point C in the preprocessed visible light image that is most similar to its descriptor. If A and C are the same feature point, then A and B are considered a correct matching pair.

[0094] S34. Scale and rotate multiple matching points to align the preprocessed visible light image and the preprocessed infrared thermal image in space, thereby obtaining the registered visible light image and the registered infrared thermal image.

[0095] See Figures 3 to 4 As an example, the matching feature points are scaled and rotated, a set of matching feature points are selected and their scale factor and relative orientation difference in the two images are calculated, the images are scaled according to the scale factor, and affine transformation is used according to the relative orientation difference to match the angle of one image with that of the other, so as to adapt to the possible scale and orientation differences between different images and align the images in space.

[0096] S4. Perform a fusion operation on the registered visible light image and the registered infrared thermal image to obtain the fused image;

[0097] Image fusion is used to combine information from multiple image sources from different sensors, different time points, or different viewpoints to generate a more informative and higher-quality image.

[0098] As one possible implementation, bilinear interpolation based on affine transformation is used for image fusion, and the resulting image is shown below. Figure 5 As shown, the method includes:

[0099] S401. Configure the fusion function, obtain the affine transformation matrix of the registered image based on the fusion function, and calculate the inverse matrix of the affine transformation matrix;

[0100] As one possible implementation, S401 includes:

[0101] S4010. Extract multiple matching point pairs from the registered image and remove abnormal matching point pairs based on the random sample consensus algorithm;

[0102] S4011. Call the cv2.estimateAffine2D function to calculate the optimal affine transformation matrix, which is a 2×3 affine transformation matrix;

[0103] S4012. Complete the 2×3 affine transformation matrix into a 3×3 homogeneous matrix, and calculate the inverse matrix of the homogeneous matrix based on the np.linalg.inv function.

[0104] S402. Based on the inverse matrix of the affine transformation matrix, the pixel coordinates in the registered infrared thermal imaging image are mapped to the new image, and the corresponding new image coordinate points are obtained;

[0105] S403. Perform bilinear interpolation at the new image coordinates to obtain the interpolated pixel values;

[0106] S404. Place the registered visible light image onto the new image to obtain the fused image.

[0107] As an example, the pixel coordinates (u, v) in the infrared image can be mapped onto the new image using the inverse of the affine transformation matrix, yielding the corresponding coordinates (ut, vt). Bilinear interpolation is then performed at the mapped coordinates (ut, vt) in the new image to obtain the interpolated pixel values. The fused image can then be obtained by directly placing the visible light image onto the new image. The script for this method is as follows:

[0108]

[0109]

[0110] As one possible implementation, principal component analysis image fusion is used for image fusion. The resulting image is shown below. Figure 6 As shown, the method includes:

[0111] S411. Extract the first feature vector of the registered visible light image and the second feature vector of the registered infrared thermal image, and combine the first feature vector and the second feature vector into a feature matrix;

[0112] As an example, MATLAB extracts the first feature vector from the registered visible light image and the second feature vector from the registered infrared thermal image, and combines the first and second feature vectors into a large feature matrix. Each row of this matrix represents a pixel, and each column represents a feature.

[0113] S411. Perform principal component analysis on the feature matrix to obtain the principal components;

[0114] S412. Configure the number of principal components to be retained, extract features from the original visible light image and the original infrared thermal image, and project the extracted features onto the retained principal components.

[0115] As an example, after constructing the feature matrix, PCA analysis is performed on the feature matrix to find the principal components. The number of principal components to be retained is then selected, and the original features are projected onto the principal components. The script for this fusion method is as follows:

[0116] %Assuming that kejianGray and hongwaiGray are both image matrices that have been read and converted to grayscale.

[0117] % Flatten all pixel values ​​of kejianGray into a one-dimensional array

[0118] zkejianVector = Flatten (visibleImageGray)

[0119] % Flattens all pixel values ​​of hongwaiGray into a one-dimensional array

[0120] zhongwaiVector=flatten(infraredImageGray)

[0121] % Creates an empty feature matrix that will contain two column vectors: one from the visible light image and one from the infrared image.

[0122] matrix = initialize an empty matrix

[0123] % Merges the pixel vectors of the visible light image and the infrared image into a matrix.

[0124] matrix = merge([zkejian,zhongwai])

[0125] %matrix is ​​now a two-dimensional matrix, where each row represents a pixel, and each column represents the pixel values ​​of the visible light image and the infrared image, respectively.

[0126] % Calculate the covariance matrix

[0127] covmatrix = Calculate the covariance (matrix)

[0128] % Calculate the eigenvectors and eigenvalues ​​of the covariance matrix

[0129] [vector, ~] = Calculates eigenvalues ​​and eigenvectors (covmatrix)

[0130] % Sort the feature vectors according to the size of their corresponding feature values

[0131] First, obtain the diagonal elements (i.e., eigenvalues) of the covariance matrix.

[0132] values ​​= Get the diagonal elements (covmatrix)

[0133] % Sort the feature values ​​in descending order and obtain the sort index.

[0134] [~,idx] = Sort the array(values, 'descending')

[0135] % Rearrange the feature vectors using the sort index

[0136] vector = rearrange(vector, idx)

[0137] % Select the number of principal components to retain (e.g., select the top N).

[0138] N=2% is just an example; you need to choose the value of N according to your specific situation.

[0139] % Extract the top N feature vectors as principal components

[0140] pc = extract columns(vector, 1 to N)

[0141] % Project the original features onto the principal components

[0142] pf = matrix multiplication(featureMatrix, pc).

[0143] S5. Perform brake status monitoring and temperature monitoring on the fused image to obtain monitoring results.

[0144] Before the drum brake is put into operation, a straight line mark is made on the brake drum. During the operation of the brake, the straight line mark rotates together with the brake drum. The rotation state of the brake can be calculated by the rotation of the straight line mark on the surface of the brake drum.

[0145] As one possible implementation, brake state monitoring is performed on the fused image, including:

[0146] S501. Binarize the fused image to obtain a binarized image;

[0147] Binarization sets the grayscale value of each pixel in an image to 0 or 255, effectively converting the image into a black and white format. It separates the foreground from the background; in this monitoring process, the straight line markings are the foreground, and the rest is the background. Binarization makes the straight line markings clearer, facilitating line detection. The binarized image looks like... Figure 7 As shown.

[0148] As an example, the Ostu method is used for binarization. The gray-level histogram of each frame of the fused image is calculated, the number of pixels at each gray level is counted, and all possible gray levels are iterated through. For each gray level, the intra-class variance between the foreground and background is calculated. Finally, the optimal threshold is selected based on the value of the intra-class variance to minimize it. This threshold is then used to perform binarization segmentation of the image, classifying pixels with gray values ​​greater than the threshold as foreground and pixels with gray values ​​less than or equal to the threshold as background.

[0149] S502. Perform edge detection on the binarized image to obtain the edge image;

[0150] See Figure 8 As a possible implementation, the Canny edge detection algorithm is still used to perform edge detection on the binarized image to obtain the edge image.

[0151] S503. Detect straight lines in edge images based on Hough transform, and calculate the rotational speed and acceleration of the brake based on the change in the angle between the straight lines in the current frame and the previous frame in the fused image.

[0152] As an example, the interface of a line detection program is as follows: Figure 9 As shown, after obtaining the slope of the line, the arctangent function can be used to convert the slope into an angle. In each frame, the angle and time of the previous frame are compared with the angle and time of the current frame to calculate the rate of change of rotational speed and acceleration.

[0153] S504. Configure the rotational speed threshold and acceleration threshold of the brake. When the calculated rotational speed exceeds the rotational speed threshold or the calculated acceleration exceeds the acceleration threshold, the brake status is determined to be abnormal.

[0154] As an example, a threshold range is determined before monitoring. When the brake's rotational speed or acceleration exceeds this threshold range, it is considered an abnormal state, and the abnormal state is recorded. The operating parameters are displayed in a line graph, showing the stages of the abnormal state as follows: Figure 10 As shown by the solid black line in the middle.

[0155] As one possible implementation, temperature monitoring of the fused image includes:

[0156] S511. Map the grayscale values ​​of the preprocessed infrared thermal imaging image to temperature values;

[0157] As an example, a calibration function is used to map grayscale values ​​to temperature values. This maps the grayscale levels in an image to the actual temperature range. Typical calibration functions, provided by camera manufacturers, associate pixel values ​​with their corresponding temperature values ​​to ensure an accurate mapping between the grayscale levels in the image and the actual temperature.

[0158] S512. Configure a temperature threshold. When the temperature value exceeds the temperature threshold, the brake temperature is determined to be abnormal.

[0159] As an example, after mapping grayscale values ​​to temperature values, the temperature change of the brake is determined based on the changes in grayscale values. A specified temperature threshold is set; when the temperature exceeds the set threshold, it is defined as a temperature anomaly. The abnormal temperature will be recorded and a temperature anomaly alarm will be triggered so that appropriate measures can be taken promptly.

[0160] This embodiment performs brake status monitoring and temperature monitoring on the fused image. By analyzing the brake structure information in the fused image, it is possible to determine whether the brake has wear, deformation, or other abnormalities. Simultaneously, by combining the temperature information from the infrared image, we can monitor the brake's temperature distribution and trends in real time, promptly detect temperature anomalies, and prevent brake overheating-related malfunctions.

[0161] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the description of the drawings, in carrying out the claimed invention. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several of the functions listed in the specification. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0162] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A method for monitoring elevator drum brakes that integrates machine vision and infrared thermal imaging, characterized in that, include: S1. Acquire raw visible light images and raw infrared thermal images of the drum brake in operation; S2. Preprocess the original visible light image and the original infrared thermal imaging image to obtain a preprocessed visible light image and a preprocessed infrared thermal imaging image; S3. Perform image registration on the preprocessed visible light image and the preprocessed infrared thermal imaging image to obtain the registered visible light image and the registered infrared thermal imaging image; S3 includes: S30. Detect the edges of the preprocessed visible light image and the preprocessed infrared thermal imaging image respectively, extract the curvature maxima on the edge contour as corner points, and obtain multiple corner points; S31. Obtain the coordinates of the contour points within the minimum point interval on both sides of each corner point, and perform a Gaussian weighted average of the coordinates of each corner point and the corresponding contour points to obtain multiple feature points; S32. Extract feature point descriptors from the preprocessed visible light image and the preprocessed infrared thermal image, respectively; S33. Match the feature point descriptors of the preprocessed visible light image and the feature point descriptors of the preprocessed infrared thermal image to obtain multiple matching point pairs; S34. Scale and rotate the plurality of matching point pairs to align the preprocessed visible light image and the preprocessed infrared thermal imaging image in space to obtain the registered visible light image and the registered infrared thermal imaging image. S4. Perform a fusion operation on the registered visible light image and the registered infrared thermal image to obtain a fused image; the image fusion operation is performed using a bilinear interpolation method based on affine transformation, including: S401. Configuring a fusion function, obtaining the affine transformation matrix of the registered image based on the fusion function, and calculating the inverse matrix of the affine transformation matrix; S401 includes: S4010. Extracting multiple matching point pairs in the registered image, and removing abnormal matching point pairs based on a random sample consensus algorithm; S4011. Calculating the optimal affine transformation matrix, which is a 2×3 affine transformation matrix; S4012. Completing the 2×3 affine transformation matrix into a 3×3 homogeneous matrix, and calculating the inverse matrix of the homogeneous matrix; S402. Based on the inverse matrix of the affine transformation matrix, the pixel coordinates in the registered infrared thermal imaging image are mapped to the new image, and the corresponding new image coordinate points are obtained; S403. Perform bilinear interpolation at the new image coordinate point to obtain the interpolated pixel value; S404. Place the registered visible light image onto the new image to obtain the fused image; S5. Perform brake status monitoring and temperature monitoring on the fused image to obtain monitoring results; brake status monitoring on the fused image includes: S501. Binarize the fused image to obtain a binarized image; S502. Perform edge detection on the binarized image to obtain an edge image; S503. Detect straight lines in the edge image based on Hough transform, and calculate the rotational speed and acceleration of the brake based on the change in the angle between the straight lines in the current frame and the previous frame in the fused image; S504. Configure the rotational speed threshold and acceleration threshold of the brake. When the calculated rotational speed exceeds the rotational speed threshold or the calculated acceleration exceeds the acceleration threshold, the brake status is determined to be abnormal.

2. The elevator drum brake monitoring method integrating machine vision and infrared thermal imaging according to claim 1, characterized in that, Image fusion is performed using principal component analysis, including: S411. Extract the first feature vector of the registered visible light image and the second feature vector of the registered infrared thermal image, and combine the first feature vector and the second feature vector into a feature matrix; S411. Perform principal component analysis on the feature matrix to obtain the principal components; S412. Configure the number of principal components to be retained, extract features from the original visible light image and the original infrared thermal image, and project the extracted features onto the retained principal components.

3. The elevator drum brake monitoring method integrating machine vision and infrared thermal imaging according to claim 1, characterized in that, Temperature monitoring of the fused image includes: S511. Map the grayscale values ​​of the preprocessed infrared thermal imaging image to temperature values; S512. Configure a temperature threshold. When the temperature value exceeds the temperature threshold, determine that the brake temperature is abnormal.

4. The elevator drum brake monitoring method integrating machine vision and infrared thermal imaging according to claim 1, characterized in that, An edge detection method based on gradient calculation is used to perform edge detection on the binarized image to obtain an edge image.

5. The elevator drum brake monitoring method integrating machine vision and infrared thermal imaging according to claim 1, characterized in that, The preprocessing includes grayscale conversion, image filtering, and histogram equalization.

6. The elevator drum brake monitoring method integrating machine vision and infrared thermal imaging according to claim 5, characterized in that, Image filtering is performed using median filtering; and / or histogram equalization is performed using an adaptive histogram equalization method with limited contrast.

Images