A method and device for monitoring elevator displacement based on machine vision

Abstract

The present invention discloses a method and device for monitoring hoist displacement based on machine vision. By monitoring the wire rope section near the head sheave, machine vision is used to determine the wire rope displacement, thereby indirectly measuring the cage displacement. Due to its non-contact nature and the presence of a support platform at the head sheave, the device is easy to install and implement, and can be readily implemented in existing industrial sites. The characteristic kernel K proposed in the detection method differs from the commonly used Gaussian kernel, resulting in better pinpoint accuracy. The detection method can achieve a computational speed exceeding 100 fps on Matlab for a properly captured 1920*1080 video, meeting the requirements of high-speed tracking.

Description

Technical Field

[0001] The present invention belongs to the field of displacement detection of industrial on-site hoisting equipment, and in particular relates to a method and device for monitoring hoist displacement based on machine vision. Background Art

[0002] Underground resource mining now mostly uses mine hoisting equipment. With the advancement of science and technology and the continuous improvement of the requirements for modernization of mine production, more and more attention is paid to improving the operating stability and safety of equipment. Improving the informatization and intelligent transformation of equipment are the top priorities of mine hoisting equipment upgrades.

[0003] The existing displacement detection methods for lifting containers mostly use a rotary encoder installed on the drum, and use the rotational angular displacement and diameter of the drum to calculate the displacement of the wire rope at the rope outlet end of the drum to estimate the displacement of the lifting container. However, the disadvantages of this method are: (1) the displacement calculation is not accurate enough due to the change in the winding diameter when the drum is wound with multiple layers of rope, and (2) the suspension rope section (the wire rope between the drum and the sheave) has obvious horizontal and vertical displacement fluctuations during the lifting operation, resulting in a low accuracy of the final estimated displacement of the lifting container. In order to overcome these problems, some experimental units will use high-precision sensors such as laser sensors or rope sensors to directly detect the position of the lifting container, but these sensors are greatly affected by the on-site environment. For example, laser sensors need to ensure that there is no obstruction between the laser emitter and the reflector, and the reflector surface is not covered with oil and dust. These methods are suitable for experimental and calibration use, but are not suitable for long-term use in industrial sites. The rapid development of machine vision has led to an increasing number of machine vision applications in industrial sites. Using machine vision methods to improve the convenience and effectiveness of lifting container displacement detection will be a good breakthrough point. However, existing methods often ignore the surface texture of the wire rope to detect its lateral displacement. Due to the significant self-similarity of the wire rope surface, detection along the wire rope's direction is easily interfered with by similar surfaces. Consequently, there is currently no robust and fast online method for detecting the longitudinal displacement of a wire rope. Given the difficulty of applying existing technologies in industrial settings and the resulting errors, there is a need for a fast, non-contact, online monitoring device and method for detecting the longitudinal displacement of a wire rope, which can be easily installed and implemented, and thereby indirectly detect the displacement of lifting equipment. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and device for monitoring the displacement of a hoist based on machine vision, which shoots a video of the vertical rope section (the wire rope section from the sheave to the hoisting container) at a fixed point, and obtains the wire rope displacement at the fixed point through image processing, thereby estimating the displacement of the container lifted by the hoist.

[0005] In order to achieve the above technical purpose, the technical solution of the present invention is:

[0006] A method for monitoring the displacement of an elevator based on machine vision comprises the following steps:

[0007] Step 1: Use a fixed camera to shoot a continuous image of a moving wire rope and a stationary background plate, and identify the wire rope area in the image. The background plate is a monochrome plate with a grayscale color difference from the wire rope.

[0008] Step 2: In the starting frame of the continuous image, the parameters of the feature kernel are selected based on the resolution and size of the wire rope area, and the feature kernel is calculated.

[0009] Step 3: In each frame of the continuous picture, the feature value is extracted based on the center of the wire rope area, and the position of the best feature is calculated as the template frame. Then, the position of the template frame in the next frame is predicted based on the motion trajectory before this frame. Then, the ROI area is selected in the next frame with the predicted position as the center. The template frame and ROI area are respectively extracted using the feature kernel to obtain the template frame feature and ROI area feature. Finally, the correlation response of the template frame feature on the ROI area feature is calculated. The distance of the position with the largest correlation response relative to the center position of the ROI area feature is the movement distance of the wire rope from this frame to the next frame.

[0010] In step 4, based on the image feature KT′ corresponding to the location with the maximum correlation response, calculate the correlation response KT′T of KT′ on the template frame feature KT to verify the results of step 3. If the maximum correlation response is found in the middle of KT′T, the verification is successful. Otherwise, adjust the parameters of the feature kernel K in step 2, and then return to step 2 and repeat until the verification is successful.

[0011] Step 5: Accumulate the calculated moving distance of the wire rope up to the current frame to obtain the total displacement of the wire rope, that is, the displacement of the lifting container.

[0012] In the method, in step 1, the wire rope area in the image is identified by first converting the captured continuous images into grayscale images, and then identifying the wire rope area in the image based on the difference between the background plate and the wire rope image.

[0013] The method, in step 3, the parameter of the feature kernel K is r size , r d and r c , where r size is the parameter that determines the first and second dimension size of the feature kernel K, r c To determine the parameter of the third dimension of the feature kernel K, r d is the radius of the center ring.

[0014] The size of the calculated feature kernel K is (2*r size +1,2*rsize +1,2*r c +1).

[0015]

[0016] in:

[0017]

[0018] Where x, y, and c are the coordinates of the 1st, 2nd, and 3rd dimensions of the feature kernel K, respectively; a is the radius of the central ring section of K; f(d) represents the function of d, and d is an intermediate parameter.

[0019] In the method, in step 2, the correlation response is obtained based on a fast correlation response calculation method, wherein the fast correlation response calculation method includes the following steps:

[0020] The input quantities for fast correlation response calculation are X and H0, that is, the correlation response of H0 on X is calculated, where X and H0 are both in matrix form:

[0021] H0 is rotated 180° to compute the convolution.

[0022] Construct a matrix mH of the same size as H0 and initialize it to all 1s.

[0023] Record the total number of H0 parameters len.

[0024] Flip H0 180° to obtain H, and then calculate the sum of all parameters in H and H.*H, namely eH and eHH, respectively.

[0025] The convolutions eX, eXX, and eXH of X and mH, X.*X and mH, and X and H are calculated respectively.

[0026] Calculate the variance of all sub-matrices of size H0 in X and arrange them in their original positions to obtain the variance matrix stdX.

[0027] Calculate the variance stdH of H.

[0028] To calculate the final correlation response of H0 on X, first multiply eX by eH and divide the result by len to get the quotient. Then subtract the quotient from eXH, divide the difference by stdX and stdH, and then divide by len. That is, (eXH-eX*eH / len). / stdX / stdH / len.

[0029] Among them, .* is dot product, which means element-by-element multiplication operation, and . / is dot division, which means element-by-element division operation.

[0030] The method, in step 3, uses feature kernels to extract features from the template frame and ROI region respectively to obtain template frame features and ROI region features, and finally calculates the correlation response of the template frame features on the ROI region features. The ROI region S is selected in the next frame with the predicted position as the center, and features K0 and K are used to extract features from T and S respectively to obtain KT and KS, and calculate the correlation response KST of KT on KS. Where K0 is a layer of the middle channel of K, that is, K0=K(:,:,r c +1).

[0031] In the method, in step 3, if the current frame is the first frame of a continuous picture, the position of the predicted template frame in the next frame is directly obtained by taking the position of the current template frame.

[0032] In the method, in step 4, the parameter of the feature kernel K in step 3 is adjusted by r d Gradually decrease from the initial value to the final value, where the initial value is r size 2 / 3 of the final value is r size 1 / 3 of.

[0033] In the method, in step 4, if the review cannot be successful, the one with the largest maximum response value in the iterative process is selected as the best result output.

[0034] An electronic device, comprising:

[0035] One or more processors.

[0036] a storage device for storing one or more programs,

[0037] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0038] A computer readable medium stores a computer program, which implements the aforementioned method when executed by a processor.

[0039] The technical effect of the present invention is that the suspension rope section near the sheave is used as the monitoring object, and the displacement result of the wire rope is obtained by machine vision method to realize indirect measurement of the cage displacement. Due to the non-contact characteristics and the fact that there is generally a support platform at the sheave, the method is easy to install and realize, and is easy to implement in existing industrial sites; the characteristic kernel K proposed by the monitoring method of the present invention is different from the commonly used Gaussian kernel, and the fixed-point effect is better; the correlation response verification method proposed by the monitoring method of the present invention can effectively avoid the interference of similar targets, especially for self-similar targets such as wire ropes, and can improve the accuracy of detection; for the monitoring method of the present invention, the calculation speed of the normally shot 1920*1080 video on Matlab can exceed 100fps, which meets the needs of high-speed tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 1 is a three-dimensional schematic diagram of a mine hoisting equipment displacement monitoring device based on video tracking provided by an embodiment of the present invention;

[0041] Figure 2 This is the actual steel wire rope and background plate area map obtained in step 2 provided in the embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below with reference to the embodiments.

[0043] like Figure 1 As shown, the video tracking-based displacement monitoring device for mine hoisting equipment provided in this embodiment includes an industrial camera 1, a suspension rope segment wire rope 2, and a background plate 3. The industrial camera and background plate are located on either side of the suspension rope segment wire rope, with the center of the industrial camera facing the center of the suspension rope segment wire rope. In this embodiment, the background plate is a light-colored, diffusely reflective, monochrome plate, which facilitates wire rope identification and extraction.

[0044] On the other hand, the present invention also provides a monitoring method using the above device, comprising the following steps:

[0045] Step 1: If Figure 1 An industrial camera is installed on the vertical rope segment, facing the rope. A background board is placed behind the rope to capture the rope in motion. In this example, the camera resolution used is 1920 x 1080, with a frame rate of 60 fps, and the captured images are in RGB format.

[0046] Step 2: After reading the image captured in step 1, convert it into a grayscale image and identify the wire rope area in the image based on the difference between the background plate and the wire rope image, such as Figure 2Since most steel wire ropes are silvery white and the grease on them turns grayish brown after a period of time, the image proportion of each color domain has little effect on the results, so grayscale images are uniformly used for calculation in this embodiment.

[0047] Step 3: Select the parameters of the feature kernel K (r size , r d , r c ), calculate the feature kernel K, the size is (2*r size +1,2*r size +1,2*r c +1).

[0048]

[0049] Where x, y, and c are the coordinates of the 1st, 2nd, and 3rd dimensions of the feature kernel K, respectively; a is the radius of the central ring section of K; f(d) represents the function of d, and d is an intermediate parameter.

[0050] Step 4: In each frame, extract feature values ​​near the center of the wire rope area, calculate the position of the best feature as the template frame T, predict the position of the template frame in the next frame based on the motion trajectory before this frame, select the ROI area S in the next frame with the predicted position as the center, use the feature kernels K0 and K for T and S respectively, so as to extract features to obtain KT and KS, and then calculate the correlation response KST of KT on KS (using the fast correlation response calculation method). The distance of the position with the largest correlation response relative to the center position is the movement distance v of the wire rope from this frame to the next frame j (Assume that the current frame is the jth frame). K0 is a layer of the middle channel of K, that is, K0=K(:,:,r c +1).

[0051] Currently, there are many methods for predicting motion trajectories, and they are relatively mature. For example, in this embodiment, the prediction of motion trajectories is based on the assumption that the pixel position of the object in the i-th frame is Relative to the pixel position of the object in the i-1th frame The change of (v xi ,v yi ), assuming v x0 =0,v y0 =0, the position change in the first frame is v x1 =0,v y1 = 0. Then the position of the template frame in the next frame is calculated using the motion trajectory prediction: In specific implementations, other motion trajectory prediction methods may also be used according to specific needs, such as Kalman filtering, differential autoregressive moving average model, Gaussian mixture model, neural network, etc.

[0052] The fast correlation response calculation method used in this embodiment is:

[0053] Input the matrix for which the correlation response needs to be calculated: X, H0; in this embodiment, the correlation response KST of KT on KS is calculated, so X is KS, H0 is KT, and the specific calculation process is:

[0054] H=rot90(H0,2);

[0055] mH=ones(size(H));

[0056] len=numel(H);

[0057] eH=sum(H,'all');

[0058] eHH=sum(H.*H,'all');

[0059] eX=convn(X,mH,"valid");

[0060] eXX=convn(X.*X,mH,"valid");

[0061] eXH=convn(X,H,"valid");

[0062] stdX=sqrt((eXX-eX.*eX / len) / len);

[0063] stdH=sqrt((eHH-eH*eH / len) / len);

[0064] score=(eXH-eX*eH / len). / stdX / stdH / len.

[0065] That is: H0 is rotated 180° to calculate the convolution;

[0066] Construct a matrix mH of the same size as H0 and initialize it to all 1s;

[0067] Record the total parameter value len of H0;

[0068] Flip H0 180° to obtain H, and then calculate the sum of all parameters in H and H.*H, namely eH and eHH respectively;

[0069] The convolutions eX, eXX, and eXH of X and mH, X.*X and mH, and X and H are calculated respectively;

[0070] Calculate the variance of all sub-matrices of size H0 in X and arrange them in their original positions to obtain the variance matrix stdX;

[0071] Calculate the variance stdH of H;

[0072] Calculate the final correlation response of H0 on X: First, multiply the result of eX by eH and divide it by len to get the quotient. Then subtract the quotient from eXH, divide the difference by stdX and stdH respectively, and then divide by len; that is, (eXH-eX*eH / len). / stdX / stdH / len;

[0073] Among them, .* is dot product, which means element-by-element multiplication operation, and . / is dot division, which means element-by-element division operation.

[0074] Step 5: Calculate the correlation response KT′T between the image feature KT′ corresponding to the position with the maximum correlation response, and review the calculation results of step 4. If the maximum value of the correlation response obtained in this step appears in the middle of KT′T, the review is successful. If the review fails, adjust the parameter r of the feature kernel K in step 3. d , repeat steps 3 and 4 until the verification is successful. Here the parameter r d The adjustment process is to adjust r d Gradually decrease from the initial value to the final value, where the initial value is r size 2 / 3 of the final value is r size 1 / 3 of r d If the verification fails when the value is reduced to the final value, the maximum response value in the iteration process is selected as the best result output.

[0075] Step 6: This is the predicted current wire rope displacement ps.

[0076] According to an embodiment of the present invention, the present invention further provides an electronic device and a computer-readable medium.

[0077] Electronic equipment includes:

[0078] one or more processors;

[0079] a storage device for storing one or more programs,

[0080] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0081] In practice, users can use electronic devices acting as terminal devices to interact with servers, also serving as electronic devices, over a network to receive or send messages. Terminal devices are generally various electronic devices equipped with a display and operated through a human-computer interface, including but not limited to smartphones, tablets, laptops, and desktop computers. Various specific application software can be installed on terminal devices as needed, including but not limited to web browsers, instant messaging software, social networking platforms, and shopping apps.

[0082] The server is a network service end that provides various services, such as providing computing services for image data transmitted from terminal devices. This allows for the detection and calculation of lifting displacement based on received wire rope images and other data, and returns the final calculation results to the terminal device.

[0083] The displacement monitoring method provided in this embodiment is generally executed by a server. In actual application, if necessary conditions are met, the terminal device can also directly execute the displacement monitoring.

[0084] Similarly, the computer-readable medium of the present invention stores a computer program thereon, and when the program is executed by a processor, the displacement monitoring method of the embodiment of the present invention is implemented.

Claims

1. A method for monitoring elevator displacement based on machine vision, characterized in that: The steps include: Step 1: Use a fixed camera to shoot a continuous image including the moving wire rope and the static background plate, and identify the wire rope area in the image; The background plate is a monochrome plate with a grayscale color difference from the wire rope; Step 2, in the starting frame of the continuous image, the parameters of the feature kernel are selected based on the resolution and size of the wire rope area, and the feature kernel is calculated; Step 3: In each frame of the continuous picture, the feature value is extracted based on the center of the wire rope area, and the position of the best feature is calculated as the template frame. Then, the position of the template frame in the next frame is predicted based on the motion trajectory before this frame. Then, the ROI area is selected in the next frame with the predicted position as the center. The template frame and ROI area are respectively extracted using the feature kernel to obtain the template frame feature and ROI area feature. Finally, the correlation response of the template frame feature on the ROI area feature is calculated. The distance of the position with the maximum correlation response relative to the center position of the ROI area feature is the movement distance of the wire rope from this frame to the next frame. Step 4: Based on the image feature KTμ corresponding to the position with the maximum correlation response, calculate the correlation response KT′T of KT′ on the template frame feature KT to verify the calculation result of step 3. If the maximum value of the obtained correlation response appears in the middle of KT′T, the verification is successful; otherwise, adjust the parameters of the feature kernel K in step 2, and then return to step 2 and repeat until the verification is successful. Step 5: Accumulate the calculated moving distance of the wire rope up to the current frame to obtain the total displacement of the wire rope, that is, the displacement of the lifting container.

2. The method according to claim 1, characterized in that In the step 1, the wire rope area in the image is identified by first converting the captured continuous images into grayscale images, and then identifying the wire rope area in the image based on the difference between the background plate and the wire rope image.

3. The method according to claim 1, characterized in that In step 3, the parameter of the feature kernel K is r size , r d and r c , where r size is the parameter that determines the first and second dimension size of the feature kernel K, r c To determine the parameter of the third dimension of the feature kernel K, r d is the radius of the center ring; The size of the calculated feature kernel K is (2*r size +1,2*r size +1,2*r c +1); in: Where x, y, and c are the coordinates of the 1st, 2nd, and 3rd dimensions of the feature kernel K, respectively; a is the radius of the central ring section of K; f(d) represents the function of d, and d is an intermediate parameter.

4. The method according to claim 1, wherein In step 2, the correlation response is obtained based on a fast correlation response calculation method, wherein the fast correlation response calculation method includes the following steps: The input quantities for fast correlation response calculation are X and H0, that is, the correlation response of H0 on X is calculated, where X and H0 are both in matrix form: H0 is rotated 180° to calculate the convolution; Construct a matrix mH of the same size as H0 and initialize it to all 1s; Record the total parameter value len of H0; Flip H0 180° to obtain H, and then calculate the sum of all parameters in H and H.*H, namely eH and eHH respectively; The convolutions eX, eXX, and eXH of X and mH, X.*X and mH, and X and H are calculated respectively; Calculate the variance of all sub-matrices of size H0 in X and arrange them in their original positions to obtain the variance matrix stdX; Calculate the variance stdH of H; Calculate the final correlation response of H0 on X: First, multiply the result of eX by eH and divide it by len to get the quotient. Then subtract the quotient from eXH, divide the difference by stdX and stdH respectively, and then divide by len; that is, (eXH-eX*eH / len). / stdX / stdH / len; Among them, .* is dot product, which means element-by-element multiplication operation, and . / is dot division, which means element-by-element division operation.

5. The method according to claim 4, characterized in that In the step 3, the template frame and ROI region are respectively extracted using feature kernels to obtain template frame features and ROI region features, and finally the correlation response of the template frame features on the ROI region features is calculated, and the ROI region S is selected in the next frame with the predicted position as the center, and the feature kernels K0 and K are respectively used to extract features from T and S to obtain KT and KS, and the correlation response KST of KT on KS is calculated; wherein K0 is a layer of the middle channel of K, that is, K0=K(:,:,r c +1).

6. The method according to claim 1, characterized in that In the step 3, if the current frame is the first frame of the continuous picture, the position of the predicted template frame in the next frame is directly obtained by taking the position of the current template frame.

7. The method according to claim 2, characterized in that In step 4, the parameters of the feature kernel K in step 3 are adjusted by changing r d Gradually decrease from the initial value to the final value, where the initial value is r size 2 / 3 of the final value is r size 1 / 3 of.

8. The method according to claim 5, characterized in that In step 4, if the review is unsuccessful, the result with the largest maximum response value during the iteration process is selected as the best result output.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

10. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images