Elevator door lock tongue engagement depth detection method and system based on computer vision

Abstract

The present invention provides a method and system for detecting the engagement depth of elevator floor door lock tongue based on computer vision, the method comprising: using a camera to obtain a video of the whole process of normal engagement of an elevator floor door lock; obtaining several pictures from the video, and using a classification algorithm to classify them into two categories according to whether the electrical contacts are in contact; selecting a picture of the electrical contact contact moment and clear picture from the picture set of the electrical contact contact, and using an edge detection algorithm to process the picture to obtain an edge image; using a key point acquisition algorithm to acquire key points of the edge image; taking the steel plate engaged by the lock hook as a reference, and calculating the engagement depth of the lock tongue according to the pixel coordinates of the key points. The method and system have the advantages of high measurement accuracy, simple operation, and no need to install additional detection equipment, etc., and can effectively solve the problem that the electrical contact contact moment is difficult to capture and the detection accuracy is low, providing a strong guarantee for the safe operation of the elevator.

Description

Technical Field

[0001] The present invention relates to the technical field of elevator detection, and in particular to a computer vision-based elevator floor door lock tongue engagement depth detection method and system. Background Art

[0002] During the routine maintenance and inspection of elevators, the engagement depth of the locking element of the elevator floor door lock is a critical safety parameter. According to relevant regulations, the engagement depth of the locking element of the elevator floor door lock when the electrical contact is closed is required to be no less than 7mm to ensure the normal closure of the elevator floor door and the safe operation of the elevator.

[0003] At present, common detection methods mainly include visual inspection, steel ruler or vernier caliper measurement, and installation of sensors and detectors on door locks. Traditional measurement methods such as visual inspection and steel ruler measurement cannot accurately determine the moment when the electrical contacts are on and off, resulting in strong subjective judgment and low measurement accuracy. The internal space of elevator door locks is narrow, and maintenance is usually carried out under power. Sensors, detectors and other devices are difficult to install in a small space. At the same time, there are safety hazards during the live installation process, which poses a threat to the life safety of maintenance personnel. Summary of the invention

[0004] In view of the above, the present invention provides a method and system for detecting the engagement depth of elevator floor door lock tongue based on computer vision, which has the advantages of high measurement accuracy, simple operation, and no need to install additional detection equipment. It can effectively solve the problem that it is difficult to capture the contact moment of electrical contacts and the detection accuracy is low, and provide strong guarantee for the safe operation of elevators.

[0005] The technical solution of the present invention:

[0006] In a first aspect, the present invention provides a method for detecting the engagement depth of an elevator door lock tongue based on computer vision, comprising the following steps:

[0007] Step S1: Use a camera to obtain a video of the entire process of the elevator door lock being normally engaged;

[0008] Step S2: obtaining several pictures from the video and using a classification algorithm to classify them into two categories according to whether the electrical contacts are in contact;

[0009] Step S3: Select a clear picture of the electrical contact contact moment from the electrical contact contact picture set, and process the picture using an edge detection algorithm to obtain an edge image;

[0010] Step S4, using a key point acquisition algorithm to acquire key points from the edge image;

[0011] Step S5, taking the steel plate engaged by the lock hook as a reference, the depth of engagement of the lock tongue is calculated according to the pixel coordinates of the key points.

[0012] Furthermore, step S2 specifically includes:

[0013] Step S21: using the output video as input, and acquiring several pictures from the video at a sampling frequency of 10 frames per second;

[0014] In step S22, the obtained images are taken as input, and the VGG16 classification algorithm is used to classify the images into two categories based on whether the electrical contacts are in contact or not, including images with the electrical contacts in contact and images without the electrical contacts in contact.

[0015] Furthermore, step S3 specifically includes:

[0016] Step S31, selecting a clear picture of the electrical contact contact moment from the electrical contact contact picture set;

[0017] In step S32, the image selected in the above step is used as input, and an edge detection algorithm based on a diffusion probability model is adopted. The output of the detection algorithm is an edge image of the image. The edge detection algorithm based on a diffusion probability model is a neural network algorithm implemented based on a pytorch framework.

[0018] Furthermore, step S4 specifically includes:

[0019] Step S41, receiving an edge image as input, wherein the edge image is an image of the elevator floor door lock tongue engagement process obtained after being processed by an edge detection algorithm;

[0020] Step S42, using a semi-automatic Shi-Tomasi corner point detection algorithm to obtain key points from the edge image, the key points required are mainly the top point of the lock tongue hook tip and two points that can significantly represent the vertical distance of the steel plate engaged by the lock tongue, wherein the distance between the key point at the upper position on the steel plate and the top point of the lock tongue hook tip should be able to represent the relative depth of the lock tongue engagement, and the line connecting the two key points on the steel plate should be in the vertical direction;

[0021] Step S43, after inputting the edge map, use the mouse to select a rectangular area at the edge image where the key point needs to be obtained. When the mouse is released, the Shi-Tomasi corner point detection algorithm is automatically used to obtain the most significant key point in the rectangular area, and the key point is numbered and the pixel coordinates of the key point are stored. The pixel coordinates of the key point are the two-dimensional coordinates represented by one pixel as a unit length, with the upper left corner of the image as the coordinate origin, the right as the positive x direction, and the downward as the positive y axis direction;

[0022] Step S44, each key point and its corresponding pixel coordinates are output, providing important basis and data for calculating the meshing depth.

[0023] Furthermore, step S5 specifically includes:

[0024] Step S51, calculating the absolute value of the difference between the vertical coordinates of the pixel coordinates of the two key points of the steel plate engaged with the lock tongue as a, and the absolute value of the difference between the vertical coordinates of the pixel coordinates of the top point at the lock tongue hook tip and the key point above the steel plate as b;

[0025] Step S52: Calculate the actual depth h of the engagement of the lock tongue based on the thickness c of the steel plate, and the calculation formula is: h=c*b / a.

[0026] In a second aspect, the present invention further provides an elevator floor door lock tongue engagement depth detection system based on computer vision, comprising:

[0027] A video acquisition module is used to use a camera to acquire a video of the entire process of the elevator floor door lock being normally engaged;

[0028] An image classification module, for obtaining a plurality of images from the video, and using a classification algorithm to classify the images into two categories according to whether the electrical contacts are in contact;

[0029] The edge detection module is used to select a clear picture of the electrical contact contact moment from the electrical contact contact picture set, and process the picture using an edge detection algorithm to obtain an edge image;

[0030] A key point acquisition module, used for acquiring key points of the edge image using a key point acquisition algorithm;

[0031] The depth calculation module is used to calculate the depth of engagement of the lock tongue according to the pixel coordinates of the key points, taking the steel plate engaged by the lock hook as a reference.

[0032] Furthermore, the image classification module specifically includes:

[0033] An image sampling unit, used to take the output video as input and acquire several images from the video at a sampling frequency of 10 frames per second, so as to ensure that the key moment of the engagement process of the lock tongue can be captured;

[0034] The classification processing unit is used to take the obtained several pictures as input, adopt the pre-trained VGG16 classification algorithm, and classify the pictures into two categories: pictures with electrical contacts in contact and pictures without electrical contacts, based on whether the electrical contacts are in contact.

[0035] Furthermore, the edge detection module specifically includes:

[0036] A picture screening unit, used for automatically selecting a clear picture of the moment when the electrical contact touches from a collection of pictures of the electrical contact touches;

[0037] The edge detection processing unit is used to take the above-selected picture as input, and process the picture using an edge detection algorithm based on a diffusion probability model to obtain an edge image of the picture as output, wherein the edge detection algorithm based on the diffusion probability model is a neural network algorithm implemented based on the pytorch framework.

[0038] Furthermore, the key point acquisition module specifically includes:

[0039] An input receiving unit, used to receive an edge image as input, wherein the edge image is an image of the elevator floor door lock tongue engagement process obtained after being processed by an edge detection algorithm, and the edge image clearly shows the engagement edge of the lock tongue and the steel plate;

[0040] An automatic detection unit is used to automatically detect key points of the edge image using a semi-automatic Shi-Tomasi corner point detection algorithm, where the required key points mainly include the top point of the lock tongue hook tip and two points that can significantly represent the vertical distance of the steel plate engaged by the lock tongue, wherein the distance between the key point at the upper position on the steel plate and the top point of the lock tongue hook tip is used to represent the relative depth of the lock tongue engagement, and the line connecting the two key points on the steel plate should be substantially in the vertical direction;

[0041] The auxiliary annotation unit is used to allow the user to select a rectangular area on the edge image where the key point needs to be obtained by the mouse after receiving the edge image. When the mouse is released, the unit automatically uses the Shi-Tomasi corner point detection algorithm to obtain the most significant key point in the selected rectangular area, and numbers the key point and stores its pixel coordinates. The pixel coordinates are two-dimensional coordinates represented by one pixel as a unit length, with the upper left corner of the image as the coordinate origin, the right as the positive direction of the x-axis, and the downward as the positive direction of the y-axis;

[0042] The output unit is used to output each key point and its corresponding pixel coordinates, providing important basis and data support for subsequent meshing depth calculation.

[0043] Furthermore, the depth calculation module specifically includes:

[0044] A pixel coordinate difference calculation unit, used to calculate the absolute value of the difference between the vertical coordinates of the pixel coordinates of two key points of the steel plate engaged by the lock tongue as a, and the absolute value of the difference between the vertical coordinates of the pixel coordinates of the top point at the lock tongue hook tip and the key point above the steel plate as b, wherein the key points include the top point at the lock tongue hook tip and two points that can significantly represent the vertical distance of the steel plate engaged by the lock tongue;

[0045] The depth calculation unit is used to calculate the actual depth h of the locking tongue engagement by using a proportional conversion method based on the known steel plate thickness c. The calculation formula is: h=c*b / a.

[0046] The computer vision-based elevator door lock tongue engagement depth detection method and system of the present invention have the following beneficial effects:

[0047] 1. Eliminate subjective judgment errors: By using the VGG16 classification algorithm to classify images of whether the electrical contacts are in contact, and selecting pictures of the moment when the electrical contacts are in contact for processing, the present invention can accurately judge the moment when the electrical contacts are in contact, thereby eliminating errors caused by manual subjective judgment and improving the accuracy of measurement.

[0048] 2. Improve measurement accuracy: The present invention uses an edge detection algorithm based on a diffusion probability model and a semi-automatic Shi-Tomasi corner point detection algorithm to obtain key points of the edge image when the lock tongue is engaged, and measure it at the pixel level. Compared with traditional measurement methods, such as visual inspection and steel ruler measurement, the measurement accuracy of the present invention has been significantly improved.

[0049] 3. Improve the safety of maintenance personnel: Traditional detection methods usually require maintenance personnel to contact and measure under live devices, which poses a great safety hazard. However, the present invention uses a non-contact computer vision method for detection, which does not require maintenance personnel to directly contact the elevator door lock, thereby greatly improving the safety of maintenance personnel.

[0050] 4. Reduced operating difficulty and detection cost: The present invention does not require the installation of additional instruments in the working environment of the lock tongue, which reduces the operating difficulty. At the same time, due to the use of computer vision technology, it reduces the dependence on traditional detection tools and also reduces the detection cost. This is undoubtedly a more economical and efficient detection solution for elevator maintenance units.

[0051] In summary, through this technical solution, the present invention eliminates the subjective judgment error caused by the difficulty in judging the contact moment of the electrical contact; this solution adopts computer vision method to measure at pixel-level accuracy, which improves the measurement accuracy compared with the traditional measurement method; this solution does not require maintenance personnel to contact and measure under the energized device, which improves the safety of maintenance personnel; this solution does not require the installation of instruments in the working environment of the lock tongue, which reduces the difficulty of operation and the cost of detection. The elevator floor door lock tongue engagement depth detection method based on computer vision provided by the present invention shows significant beneficial effects in eliminating subjective judgment errors, improving measurement accuracy, improving the safety of maintenance personnel, and reducing the difficulty of operation and detection costs, and has broad application prospects and promotion value.

[0052] The preferred embodiments of the present invention and their beneficial effects will be further described in detail in conjunction with specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but should not be construed as limiting the present invention. In the accompanying drawings:

[0054] Figure 1 It is a flow chart of a method for detecting engagement depth of elevator door lock tongue based on computer vision of the present invention;

[0055] Figure 2 It is a structural diagram of the elevator floor door lock tongue engagement depth detection system based on computer vision of the present invention. DETAILED DESCRIPTION

[0056] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0057] See also Figure 1 The present invention provides a method for detecting the engagement depth of an elevator door lock tongue based on computer vision, comprising the following steps:

[0058] Step S1: Use a camera to obtain a video of the entire process of the elevator door lock being normally engaged;

[0059] Step S2: obtain several pictures from the video and use the VGG16 classification algorithm to classify them into two categories according to whether the electrical contacts are in contact;

[0060] Step S3, selecting a clear picture of the electrical contact contact moment from the electrical contact contact picture set, and processing the picture using an edge detection algorithm based on a diffusion probability model to obtain an edge image;

[0061] Step S4: using the semi-automatic Shi-Tomasi corner detection algorithm to obtain key points of the edge image;

[0062] Step S5, taking the steel plate engaged by the lock hook as a reference, the depth of engagement of the lock tongue is calculated according to the pixel coordinates of the key points.

[0063] It can be understood that in step S2, the VGG16 classification algorithm can also use other classification algorithms to classify images.

[0064] In step S3, the edge detection algorithm based on the diffusion probability model may also use other edge detection algorithms to perform edge detection on the image.

[0065] In step S4, the semi-automatic Shi-Tomasi corner detection algorithm may also use other key point acquisition algorithms to capture the key points of the edge image.

[0066] In the above technical solution, steps S3 and S4 are cancelled, and the engagement depth of the elevator door lock tongue is detected by image segmentation.

[0067] The specific implementation details of each step are described in detail below.

[0068] Step S1: Use a camera to obtain a video of the entire process of the elevator door lock being normally engaged;

[0069] The S1 step specifically includes:

[0070] Step S11, obtaining a video of the whole process of normal engagement of the elevator floor door lock, wherein the camera is kept horizontally parallel to the steel plate engaged with the lock tongue during shooting, so as to ensure that the captured video can accurately reflect the process of the lock tongue engagement;

[0071] Step S12: provide sufficient lighting environment with appropriate angles to ensure that the lock tongue and the steel plate engaged by the lock tongue can be correctly and clearly identified, and at the same time ensure that the shadow will not have an excessive impact on the subsequent edge detection;

[0072] Step S13: verify whether the video captured fully displays the entire process of the bolt engagement, so as to ensure the accuracy and reliability of subsequent analysis;

[0073] The method of using a camera to shoot video to measure the engagement depth of the lock tongue solves the problem that it is difficult to use measuring tools or install measuring devices in the small space where the floor door lock is located. At the same time, this non-contact measurement solution is more conducive to the personal safety of the inspection personnel.

[0074] Step S2: obtain several pictures from the video and use the VGG16 classification algorithm to classify them into two categories according to whether the electrical contacts are in contact;

[0075] The S2 step specifically includes:

[0076] Step S21: using the output video as input, and acquiring several pictures from the video at a sampling frequency of 10 frames per second;

[0077] In step S22, the obtained images are used as input, and the VGG16 classification algorithm is used to classify the images into two categories based on whether the electrical contacts are in contact or not, including images of electrical contacts in contact and images of electrical contacts not in contact. Image classification plays an important role in determining whether the electrical contacts are in contact, and can solve the difficulty of manually determining whether the electrical contacts are in contact or not.

[0078] The beneficial effects of step S2 in the present invention are mainly reflected in the following aspects:

[0079] 1. Efficient image sampling: By setting the sampling frequency of 10 frames per second, several images can be quickly obtained from the video. This step effectively improves the efficiency of image processing and provides sufficient and high-quality image data for subsequent steps such as image classification and edge detection.

[0080] 2. Accurate image classification: Using the VGG16 classification algorithm, the images are accurately divided into images with and without electrical contacts based on whether the electrical contacts are in contact. This classification method not only solves the difficulty of manually judging whether there is contact, but also greatly improves the accuracy and reliability of classification, laying a solid foundation for subsequent edge detection and depth calculation steps.

[0081] 3. Optimize the detection process: The implementation of step S2 makes the entire detection process more optimized and efficient. Through automated image sampling and classification, manual intervention is reduced, detection speed and accuracy are improved, and detection costs are reduced. This is of great significance for real-time monitoring and troubleshooting of the engagement depth of elevator door lock tongues, and helps to improve the safety performance and operating efficiency of elevators.

[0082] In summary, the implementation of step S2 in the present invention provides strong support for the accurate detection of the engagement depth of the elevator door lock tongue through the beneficial effects of efficient image sampling, accurate image classification and optimized detection process, and has important practical application value.

[0083] Step S3, selecting a clear picture of the electrical contact contact moment from the electrical contact contact picture set, and processing the picture using an edge detection algorithm based on a diffusion probability model to obtain an edge image;

[0084] The S3 step specifically includes:

[0085] Step S31, selecting a clear picture of the electrical contact contact moment from the electrical contact contact picture set, can solve the problem that it is difficult for humans to judge whether the electrical contact is at the contact moment.

[0086] In step S32, the image selected in the above step is used as input, and an edge detection algorithm based on a diffusion probability model is used, and the output of the detection algorithm is an edge image of the image. The edge detection algorithm based on a diffusion probability model is a neural network algorithm implemented based on the pytorch framework, which can obtain a more accurate and finer edge image, making the meshing depth detection more accurate.

[0087] The implementation of step S3 in the present invention brings the following significant beneficial effects:

[0088] 1. Accurately capture the moment of contact: Step S31 uses automated means to accurately select clear pictures of the electrical contact moment from the pictures of electrical contact. This step effectively solves the problem that it is difficult for humans to accurately judge whether the electrical contact is at the moment of contact, improves the accuracy and efficiency of picture selection, and provides high-quality picture data for subsequent edge detection.

[0089] 2. Improve edge detection accuracy: Step S32 uses an edge detection algorithm based on a diffusion probability model to perform edge detection on the selected image to obtain a more accurate and detailed edge image. This algorithm is implemented based on the pytorch framework and fully utilizes the advantages of neural networks. It can more finely identify edge information in the image, thereby improving the accuracy of meshing depth detection. This is of great significance for the accurate evaluation and maintenance of the meshing state of the elevator door lock tongue.

[0090] In summary, the implementation of step S3 in the present invention provides strong support for the accurate detection of the engagement depth of the elevator door lock tongue by accurately capturing the contact moment and improving the edge detection accuracy, which helps to improve the safety performance and operation efficiency of the elevator.

[0091] Step S4: using the semi-automatic Shi-Tomasi corner detection algorithm to obtain key points of the edge image;

[0092] The S4 step specifically includes:

[0093] Step S41, receiving an edge image as input, wherein the edge image is an image of the elevator floor door lock tongue engagement process obtained after being processed by an edge detection algorithm;

[0094] Step S42, using the semi-automatic Shi-Tomasi corner point detection algorithm to obtain key points of the edge image. The key points required are mainly the top point of the lock tongue hook tip and two points that can significantly represent the vertical distance of the steel plate engaged by the lock tongue, among which the distance between the key point at the upper position on the steel plate and the top point of the lock tongue hook tip should be able to represent the relative depth of the lock tongue engagement, and the line connecting the two key points on the steel plate should be in the vertical direction.

[0095] Step S43, after inputting the edge map, use the mouse to select a rectangular area on the edge image where the key point needs to be obtained. When the mouse is released, the Shi-Tomasi corner point detection algorithm is automatically used to obtain the most significant key point in the rectangular area, and the key point is numbered and the pixel coordinates of the key point are stored. The method of obtaining key points by manually annotating the area is conducive to improving the accuracy of key point acquisition and has better universal adaptability to edge maps of poor quality. The pixel coordinates of the key point are the two-dimensional coordinates represented by one pixel as a unit length, with the upper left corner of the image as the coordinate origin, the right as the positive x direction, and the downward as the positive y axis. The pixel coordinates can be used to calculate the meshing depth detection at pixel-level accuracy.

[0096] Step S44, each key point and its corresponding pixel coordinates are output, providing important basis and data for calculating the meshing depth.

[0097] The implementation of step S4 in the present invention brings the following significant beneficial effects:

[0098] 1. Accurate key point acquisition: Through the semi-automatic Shi-Tomasi corner point detection algorithm in step S4, the required key points can be accurately obtained from the edge image, including the top point of the lock tongue hook tip and two key points that represent the vertical distance of the steel plate engaged by the lock tongue. This accurate key point acquisition method provides important data support for the subsequent calculation of the lock tongue engagement depth, improving the accuracy and reliability of the detection.

[0099] 2. Improve detection accuracy and efficiency: In step S43, the method of manually marking the area to obtain key points not only improves the accuracy of key point acquisition, but also makes the method more universal for edge images with poor quality. At the same time, the operation of selecting a rectangular area with the mouse and automatically detecting key points simplifies the detection process and improves detection efficiency.

[0100] 3. Pixel-level precision calculation: By using pixel coordinates to represent the position of key points, the meshing depth detection can be calculated at pixel-level precision. This high-precision calculation method makes the detection results more accurate and reliable, providing strong data support for the meshing status evaluation and maintenance of elevator door locks.

[0101] 4. Providing important basis for subsequent steps: The key points and their corresponding pixel coordinates output in step S44 provide important basis and data for the subsequent calculation of meshing depth.

[0102] In summary, the implementation of step S4 in the present invention provides strong support for the accurate detection of the engagement depth of the elevator door lock tongue through the beneficial effects of accurate key point acquisition, improved detection accuracy and efficiency, pixel-level precision calculation, and providing important basis for subsequent steps, which helps to improve the safety performance and operation efficiency of the elevator.

[0103] Step S5, taking the steel plate engaged by the lock hook as a reference, the depth of engagement of the lock tongue is calculated according to the pixel coordinates of the key points.

[0104] Step S5 specifically includes:

[0105] Step S51, calculate the absolute value of the difference between the vertical coordinates of the two key point pixel coordinates of the steel plate where the lock tongue is engaged, which is a, and the absolute value of the difference between the vertical coordinates of the top point at the lock tongue hook tip and the key point pixel coordinates above the steel plate, which is b.

[0106] Step S52, based on the thickness c of the steel plate (i.e., the actual distance between two key points of the steel plate) as a reference, calculate the actual depth h of the locking tongue engagement, and the calculation formula is: h=c*b / a.

[0107] The implementation of step S5 in the present invention provides an efficient and accurate method for accurately calculating the engagement depth of the elevator door lock tongue, and the specific beneficial effects are as follows:

[0108] 1. Improve calculation accuracy: By calculating the absolute value a of the difference between the vertical coordinates of the two key points of the steel plate where the lock tongue is engaged in step S51, and the absolute value b of the difference between the vertical coordinates of the pixel coordinates of the top point at the lock tongue hook tip and the key point above the steel plate, and then taking the thickness c of the steel plate (that is, the actual distance between the two key points of the steel plate) as a reference, the actual depth h of the lock tongue engagement is calculated using the formula h=c*b / a. This process fully considers the correspondence between the pixel information in the image and the actual physical size, thereby realizing the accurate calculation of the lock tongue engagement depth.

[0109] 2. Enhanced detection reliability: The method of the present invention is based on computer vision technology. By automatically processing and analyzing image data, it avoids the subjectivity and errors of manual detection and improves the reliability and accuracy of detection. At the same time, due to the use of advanced technologies such as semi-automatic Shi-Tomasi corner detection algorithm to obtain key points, the robustness and adaptability of detection are further enhanced.

[0110] 3. Improve elevator safety performance: The meshing state of the elevator door lock tongue is directly related to the safe operation of the elevator. By accurately calculating the meshing depth of the lock tongue, the present invention can timely discover potential poor meshing or fault conditions, providing strong support for the safety maintenance of the elevator. This not only helps to improve the safety performance of the elevator, but also prolongs the service life of the elevator and reduces maintenance costs.

[0111] Fourth, easy to implement and promote: The method of the present invention is based on existing image processing technology and algorithms, does not require complex hardware equipment and professional operating skills, and is easy to implement and promote in the field of elevator inspection and maintenance. At the same time, due to the advantages of high efficiency, accuracy and reliability, this method is expected to become an important technical means in the field of elevator safety inspection.

[0112] In summary, the implementation of step S5 in the present invention provides a strong guarantee for the safe operation of the elevator by accurately calculating the engagement depth of the lock tongue, enhancing detection reliability, improving the safety performance of the elevator, and facilitating implementation and promotion.

[0113] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0114] Based on the same inventive concept, an embodiment of the present application also provides an elevator floor door lock tongue engagement depth detection system based on computer vision, and the modules included are used to execute the steps in the corresponding embodiments of the elevator floor door lock tongue engagement depth detection method based on computer vision.

[0115] See also Figure 2 , the elevator floor door lock tongue engagement depth detection system based on computer vision includes:

[0116] A video acquisition module is used to use a camera to acquire a video of the entire process of the elevator floor door lock being normally engaged;

[0117] An image classification module, used to obtain a number of images from the video, and classify them into two categories according to whether the electrical contacts are in contact using a VGG16 classification algorithm;

[0118] An edge detection module is used to select a clear picture of the electrical contact contact moment from the electrical contact contact picture set, and process the picture using an edge detection algorithm based on a diffusion probability model to obtain an edge image;

[0119] A key point acquisition module, used for acquiring key points of the edge image using a semi-automatic Shi-Tomasi corner point detection algorithm;

[0120] The depth calculation module is used to calculate the depth of engagement of the lock tongue according to the pixel coordinates of the key points, taking the steel plate engaged by the lock hook as a reference.

[0121] It can be understood that in the image classification module, the VGG16 classification algorithm can also use other classification algorithms to classify images.

[0122] In the edge detection module, the edge detection algorithm based on the diffusion probability model can also use other edge detection algorithms to perform edge detection on the image.

[0123] In the key point acquisition module, the semi-automatic Shi-Tomasi corner point detection algorithm can also use other key point acquisition algorithms to capture the key points of the edge image.

[0124] In the above technical solution, the edge detection module and the key point acquisition module are cancelled, and the engagement depth of the elevator door lock tongue is detected by means of an image segmentation module.

[0125] The specific implementation details of each module of the system are described in detail below.

[0126] A video acquisition module is used to use a camera to acquire a video of the entire process of the elevator floor door lock being normally engaged;

[0127] The video acquisition module specifically includes:

[0128] A camera assembly is used to obtain a video of the whole process of the normal engagement of the elevator floor door lock. When shooting, the camera is kept parallel to the steel plate engaged with the lock tongue to ensure that the captured video can accurately reflect the process of the lock tongue engagement;

[0129] An illumination control unit is used to provide sufficient illumination at an appropriate angle to ensure that the lock tongue and the steel plate engaged by the lock tongue can be correctly and clearly identified, while ensuring that shadows do not have an excessive impact on subsequent edge detection;

[0130] The video integrity verification unit is used to verify whether the video captured fully displays the entire process of the bolt engagement, so as to ensure the accuracy and reliability of subsequent analysis;

[0131] The method of using a camera to shoot video to measure the engagement depth of the lock tongue solves the problem that it is difficult to use measuring tools or install measuring devices in the small space where the floor door lock is located. At the same time, this non-contact measurement solution is more conducive to the personal safety of the inspection personnel.

[0132] An image classification module, used to obtain a number of images from the video, and classify them into two categories according to whether the electrical contacts are in contact using a VGG16 classification algorithm;

[0133] The image classification module specifically includes:

[0134] An image sampling unit, used to take the output video as input and acquire several images from the video at a sampling frequency of 10 frames per second, so as to ensure that the key moment of the engagement process of the lock tongue can be captured;

[0135] The classification processing unit is used to take the obtained several pictures as input, adopt the pre-trained VGG16 classification algorithm, and classify the pictures into two categories: pictures with electrical contact and pictures without electrical contact, based on whether the electrical contact is in contact. The classification processing unit automatically analyzes the electrical contact status in the picture, thereby solving the difficulty of manually judging whether it is in contact, and improving the accuracy and efficiency of classification. The VGG16 classification algorithm has been specifically trained for the engagement process of the elevator floor door lock tongue to improve the recognition accuracy of the electrical contact contact status.

[0136] An edge detection module is used to select a clear picture of the electrical contact contact moment from the electrical contact contact picture set, and process the picture using an edge detection algorithm based on a diffusion probability model to obtain an edge image;

[0137] The edge detection module specifically includes:

[0138] The picture screening unit is used to automatically select pictures of the electrical contact contact moment and with clear images from the electrical contact contact picture set. This step solves the problem that it is difficult for humans to accurately judge whether the electrical contact is at the contact moment, and improves the accuracy and efficiency of picture selection;

[0139] The edge detection processing unit is used to take the above-selected picture as input, and process the picture using an edge detection algorithm based on a diffusion probability model to obtain an edge image of the picture as output. The edge detection algorithm based on the diffusion probability model is a neural network algorithm implemented based on the pytorch framework, which can more accurately identify and extract edge information in the image, making the obtained edge image more refined, thereby improving the accuracy of subsequent meshing depth detection.

[0140] A key point acquisition module, used for acquiring key points of the edge image using a semi-automatic Shi-Tomasi corner point detection algorithm;

[0141] The key point acquisition module specifically includes:

[0142] An input receiving unit, used to receive an edge image as input, wherein the edge image is an image of the elevator floor door lock tongue engagement process obtained after being processed by an edge detection algorithm, and the edge image clearly shows the engagement edge of the lock tongue and the steel plate;

[0143] An automatic detection unit, using a semi-automatic Shi-Tomasi corner point detection algorithm to automatically detect key points of the edge image, wherein the required key points mainly include the top point of the lock tongue hook tip and two points that can significantly characterize the vertical distance of the steel plate engaged by the lock tongue, wherein the distance between the key point at the upper position on the steel plate and the top point of the lock tongue hook tip is used to characterize the relative depth of engagement of the lock tongue, and the line connecting the two key points on the steel plate should be substantially in the vertical direction to ensure the accuracy of subsequent engagement depth calculation;

[0144] The auxiliary annotation unit, after receiving the edge image, allows the user to select a rectangular area on the edge image where the key point needs to be obtained by using the mouse. When the mouse is released, the unit automatically uses the Shi-Tomasi corner point detection algorithm to obtain the most significant key point in the selected rectangular area, and numbers the key point and stores its pixel coordinates. The pixel coordinates take the upper left corner of the image as the coordinate origin, the right as the positive direction of the x-axis, and the downward as the positive direction of the y-axis, with one pixel as a unit length. The key points obtained in this way have high accuracy and better universality for edge images with poor quality;

[0145] The output unit is used to output each key point and its corresponding pixel coordinates, providing important basis and data support for subsequent meshing depth calculation.

[0146] The depth calculation module is used to calculate the depth of engagement of the lock tongue according to the pixel coordinates of the key points, taking the steel plate engaged by the lock hook as a reference.

[0147] The depth calculation module specifically includes:

[0148] A pixel coordinate difference calculation unit, used to calculate the absolute value of the difference between the vertical coordinates of the pixel coordinates of two key points of the steel plate engaged by the lock tongue as a, and the absolute value of the difference between the vertical coordinates of the pixel coordinates of the top point at the lock tongue hook tip and the key point above the steel plate as b, wherein the key points include the top point at the lock tongue hook tip and two points that can significantly represent the vertical distance of the steel plate engaged by the lock tongue;

[0149] The depth calculation unit is used to calculate the actual depth h of the lock tongue engagement based on the known steel plate thickness c (i.e., the actual distance between two key points of the steel plate) by a proportional conversion method. The calculation formula is: h=c*b / a. The output of the depth calculation unit is the actual depth h of the lock tongue engagement, which provides important data support for the subsequent evaluation and maintenance of the elevator floor door lock tongue engagement status.

[0150] The computer vision-based elevator door lock tongue engagement depth detection method and system of the present invention have the following beneficial effects:

[0151] 1. Eliminate subjective judgment errors: By using the VGG16 classification algorithm to classify images of whether the electrical contacts are in contact, and selecting pictures of the moment when the electrical contacts are in contact for processing, the present invention can accurately judge the moment when the electrical contacts are in contact, thereby eliminating errors caused by manual subjective judgment and improving the accuracy of measurement.

[0152] 2. Improve measurement accuracy: The present invention uses an edge detection algorithm based on a diffusion probability model and a semi-automatic Shi-Tomasi corner point detection algorithm to obtain key points of the edge image when the lock tongue is engaged, and measure it at the pixel level. Compared with traditional measurement methods, such as visual inspection and steel ruler measurement, the measurement accuracy of the present invention has been significantly improved.

[0153] 3. Improve the safety of maintenance personnel: Traditional detection methods usually require maintenance personnel to contact and measure under live devices, which poses a great safety hazard. However, the present invention uses a non-contact computer vision method for detection, which does not require maintenance personnel to directly contact the elevator door lock, thereby greatly improving the safety of maintenance personnel.

[0154] 4. Reduced operating difficulty and detection cost: The present invention does not require the installation of additional instruments in the working environment of the lock tongue, which reduces the operating difficulty. At the same time, due to the use of computer vision technology, it reduces the dependence on traditional detection tools and also reduces the detection cost. This is undoubtedly a more economical and efficient detection solution for elevator maintenance units.

[0155] In summary, through this technical solution, the present invention eliminates the subjective judgment error caused by the difficulty in judging the contact moment of the electrical contact; this solution adopts computer vision method to measure at pixel-level accuracy, which improves the measurement accuracy compared with the traditional measurement method; this solution does not require maintenance personnel to contact and measure under the energized device, which improves the safety of maintenance personnel; this solution does not require the installation of instruments in the working environment of the lock tongue, which reduces the difficulty of operation and the cost of detection. The elevator floor door lock tongue engagement depth detection method based on computer vision provided by the present invention shows significant beneficial effects in eliminating subjective judgment errors, improving measurement accuracy, improving the safety of maintenance personnel, and reducing the difficulty of operation and detection costs, and has broad application prospects and promotion value.

[0156] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0157] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A computer vision-based elevator door lock tongue engagement depth detection method, characterized in that: The steps include: Step S1: Use a camera to obtain a video of the entire process of the elevator door lock being normally engaged; Step S2: obtaining several pictures from the video and using a classification algorithm to classify them into two categories according to whether the electrical contacts are in contact; Step S3: Select a clear picture of the electrical contact contact moment from the electrical contact contact picture set, and process the picture using an edge detection algorithm to obtain an edge image; Step S4, using a key point acquisition algorithm to acquire key points from the edge image; Step S5, taking the steel plate engaged by the lock hook as a reference, the depth of engagement of the lock tongue is calculated according to the pixel coordinates of the key points.

2. The computer vision-based elevator door lock tongue engagement depth detection method according to claim 1, characterized in that: The S2 step specifically includes: Step S21: using the output video as input, and acquiring several pictures from the video at a sampling frequency of 10 frames per second; In step S22, the obtained images are taken as input, and the VGG16 classification algorithm is used to classify the images into two categories based on whether the electrical contacts are in contact or not, including images with the electrical contacts in contact and images without the electrical contacts in contact.

3. The computer vision-based elevator door lock tongue engagement depth detection method according to claim 1, characterized in that: The S3 step specifically includes: Step S31, selecting a clear picture of the electrical contact contact moment from the electrical contact contact picture set; In step S32, the image selected in the above step is used as input, and an edge detection algorithm based on a diffusion probability model is adopted. The output of the detection algorithm is an edge image of the image. The edge detection algorithm based on a diffusion probability model is a neural network algorithm implemented based on a pytorch framework.

4. The computer vision-based elevator door lock tongue engagement depth detection method according to claim 1, characterized in that: The S4 step specifically includes: Step S41, receiving an edge image as input, wherein the edge image is an image of the elevator floor door lock tongue engagement process obtained after being processed by an edge detection algorithm; Step S42, using a semi-automatic Shi-Tomasi corner point detection algorithm to obtain key points from the edge image, the key points required are mainly the top point of the lock tongue hook tip and two points that can significantly represent the vertical distance of the steel plate engaged by the lock tongue, wherein the distance between the key point at the upper position on the steel plate and the top point of the lock tongue hook tip should be able to represent the relative depth of the lock tongue engagement, and the line connecting the two key points on the steel plate should be in the vertical direction; Step S43, after inputting the edge map, use the mouse to select a rectangular area at the edge image where the key point needs to be obtained. When the mouse is released, the Shi-Tomasi corner point detection algorithm is automatically used to obtain the most significant key point in the rectangular area, and the key point is numbered and the pixel coordinates of the key point are stored. The pixel coordinates of the key point are the two-dimensional coordinates represented by one pixel as a unit length, with the upper left corner of the image as the coordinate origin, the right as the positive x direction, and the downward as the positive y axis direction; Step S44, each key point and its corresponding pixel coordinates are output, providing important basis and data for calculating the meshing depth.

5. The computer vision-based elevator door lock tongue engagement depth detection method according to claim 4, characterized in that: Step S5 specifically includes: Step S51, calculating the absolute value of the difference between the vertical coordinates of the pixel coordinates of the two key points of the steel plate engaged with the lock tongue as a, and the absolute value of the difference between the vertical coordinates of the pixel coordinates of the top point at the lock tongue hook tip and the key point above the steel plate as b; Step S52: Calculate the actual depth h of the engagement of the lock tongue based on the thickness c of the steel plate, and the calculation formula is: h=c*b / a.

6. A computer vision-based elevator door lock tongue engagement depth detection system, characterized in that: include: A video acquisition module is used to use a camera to acquire a video of the entire process of the elevator floor door lock being normally engaged; An image classification module, for obtaining a plurality of images from the video, and using a classification algorithm to classify the images into two categories according to whether the electrical contacts are in contact; The edge detection module is used to select a clear picture of the electrical contact contact moment from the electrical contact contact picture set, and process the picture using an edge detection algorithm to obtain an edge image; A key point acquisition module, used for acquiring key points of the edge image using a key point acquisition algorithm; The depth calculation module is used to calculate the depth of engagement of the lock tongue according to the pixel coordinates of the key points, taking the steel plate engaged by the lock hook as a reference.

7. The computer vision-based elevator door lock tongue engagement depth detection system according to claim 6, characterized in that: The image classification module specifically includes: An image sampling unit, used to take the output video as input and acquire several images from the video at a sampling frequency of 10 frames per second, so as to ensure that the key moment of the engagement process of the lock tongue can be captured; The classification processing unit is used to take the obtained several pictures as input, adopt the pre-trained VGG16 classification algorithm, and classify the pictures into two categories: pictures with electrical contacts in contact and pictures without electrical contacts, based on whether the electrical contacts are in contact.

8. The computer vision-based elevator door lock tongue engagement depth detection system according to claim 6, characterized in that: The edge detection module specifically includes: A picture screening unit, used for automatically selecting a clear picture of the moment when the electrical contact touches from a collection of pictures of the electrical contact touches; The edge detection processing unit is used to take the above-selected picture as input, and process the picture using an edge detection algorithm based on a diffusion probability model to obtain an edge image of the picture as output, wherein the edge detection algorithm based on the diffusion probability model is a neural network algorithm implemented based on the pytorch framework.

9. The computer vision-based elevator door lock tongue engagement depth detection system according to claim 6, characterized in that: The key point acquisition module specifically includes: An input receiving unit, used to receive an edge image as input, wherein the edge image is an image of the elevator floor door lock tongue engagement process obtained after being processed by an edge detection algorithm, and the edge image clearly shows the engagement edge of the lock tongue and the steel plate; An automatic detection unit is used to automatically detect key points of the edge image using a semi-automatic Shi-Tomasi corner point detection algorithm, where the required key points mainly include the top point of the lock tongue hook tip and two points that can significantly represent the vertical distance of the steel plate engaged by the lock tongue, wherein the distance between the key point at the upper position on the steel plate and the top point of the lock tongue hook tip is used to represent the relative depth of the lock tongue engagement, and the line connecting the two key points on the steel plate should be substantially in the vertical direction; The auxiliary annotation unit is used to allow the user to select a rectangular area on the edge image where the key point needs to be obtained by the mouse after receiving the edge image. When the mouse is released, the unit automatically uses the Shi-Tomasi corner point detection algorithm to obtain the most significant key point in the selected rectangular area, and numbers the key point and stores its pixel coordinates. The pixel coordinates are two-dimensional coordinates represented by one pixel as a unit length, with the upper left corner of the image as the coordinate origin, the right as the positive direction of the x-axis, and the downward as the positive direction of the y-axis; The output unit is used to output each key point and its corresponding pixel coordinates, providing important basis and data support for subsequent meshing depth calculation.

10. The computer vision-based elevator door lock tongue engagement depth detection system according to claim 9, characterized in that: The depth calculation module specifically includes: A pixel coordinate difference calculation unit, used to calculate the absolute value of the difference between the vertical coordinates of the pixel coordinates of two key points of the steel plate engaged by the lock tongue as a, and the absolute value of the difference between the vertical coordinates of the pixel coordinates of the top point at the lock tongue hook tip and the key point above the steel plate as b, wherein the key points include the top point at the lock tongue hook tip and two points that can significantly represent the vertical distance of the steel plate engaged by the lock tongue; The depth calculation unit is used to calculate the actual depth h of the locking tongue engagement by using a proportional conversion method based on the known steel plate thickness c. The calculation formula is: h=c*b / a.

Images