Elevator full load detection method based on image recognition

Abstract

This invention discloses an image recognition-based elevator full-load detection method, comprising: Step 1: setting a reference scale on the elevator car wall, setting a camera inside the elevator car, and determining the actual coordinate data of the reference scale and the elevator car wall in the camera coordinate system; Step 2: acquiring images inside the elevator car in real time, and processing the images to obtain processed images; Step 3: obtaining the contours of the reference scale, the elevator car wall, and the corresponding areas of each person or object inside the elevator car, and determining the pixel coordinates and actual coordinates of each pixel on the contours; Step 4: determining whether the elevator is full. This invention uses image recognition to calculate whether there are still people standing inside the elevator car to determine whether it is full, solving the problem that existing weight detection methods cannot identify many types of actual full loads, and avoiding situations where the elevator is actually full but stops at multiple floors; in addition, this invention can be directly implemented based on existing monitoring cameras inside elevator cars, resulting in low cost.

Description

Technical Field

[0001] This invention relates to the field of elevator technology, and in particular to an elevator full-load detection method based on image recognition. Background Technology

[0002] During peak commuting hours, close to clocking in, the elevators are full but still stop on every floor – a bad experience that every office worker has undoubtedly had.

[0003] In the past, elevator load limits were determined by weight detection. Under this standard, if the elevator was full of children or loaded with a large amount of goods, it could not accommodate more users even if the weight limit had not been met. In other words, the existing elevator weight detection system could not identify many actual load conditions. Summary of the Invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide an elevator full-load detection method based on image recognition, so as to detect whether the elevator is actually full.

[0005] To address the aforementioned technical problems, this invention proposes an elevator full-load detection method based on image recognition, comprising:

[0006] Step 1: Set a reference scale on the elevator car wall, set a camera inside the elevator car, calibrate the camera inside the elevator car, and obtain the camera parameters, including intrinsic and extrinsic parameter matrices, distortion parameter matrices, and camera focal length; determine the actual coordinate data of the reference scale and the elevator car wall in the camera coordinate system;

[0007] Step 2: Real-time acquisition of images inside the elevator car, preprocessing the images to remove distortion, and then performing inverse perspective transformation to obtain the processed image;

[0008] Step 3: Use a preset edge detection algorithm to segment and extract the reference scale, elevator car wall, and people or objects inside the elevator car from the processed image. Determine the pixel coordinate data of the corresponding pixels on the edges of the reference scale, elevator car wall, and people or objects inside the elevator car, and perform linear fitting to obtain the contour of the corresponding region of the reference scale, elevator car wall, and people or objects inside the elevator car. Determine the pixel coordinates of each pixel on the contour of each region, and then calculate the actual coordinates in the camera coordinate system corresponding to the pixel coordinates.

[0009] Step 4: Calculate the actual distance between the nearest pixel on the outline of each person or object area in the elevator car and the outline of the nearest pixel on the outline of an adjacent person or object area, and calculate the actual distance between the elevator car wall and the nearest pixel on the outline of an adjacent person or object area, to obtain a set C of actual distances. If any value in set C is greater than the first preset value, then it is not fully loaded; otherwise, calculate the sum of the differences between each value in set C and the second preset value. If the sum is greater than the first preset value, then it is not fully loaded; otherwise, it is fully loaded.

[0010] Furthermore, in step 3, the actual coordinates corresponding to the target pixel are calculated according to the following formula:

[0011] ;

[0012] ;

[0013] Where X and Y are the actual coordinates of the target pixel in the camera coordinate system, f is the camera focal length, x and y are the pixel coordinates of the target pixel in the image, and Z is the distance from the actual position of the target pixel to the camera optical center, which is calculated using the following formula:

[0014] ;

[0015] Where W is the actual width of the reference ruler, and w is the pixel width of the reference ruler.

[0016] Furthermore, in step 4, the actual spacing between pixels is calculated according to the following formula:

[0017] ;

[0018] Where (X1,Y1,Z1) and (X2,Y2,Z2) are the actual coordinates of the two target pixels in the camera coordinate system, and d is the actual distance between them.

[0019] Furthermore, the camera is located at the top center of the elevator car.

[0020] Furthermore, the preset edge detection algorithm is the Sobel operator, the Canny operator, or the Laplacian operator.

[0021] Furthermore, in step 3, a linear fit is performed using polynomial fitting, Bézier curve fitting, or B-spline curve fitting to obtain the contour.

[0022] Furthermore, the first preset value ranges from 35 cm to 42 cm, and the second preset value ranges from 2 cm to 10 cm.

[0023] The beneficial effects of this invention are as follows: This invention uses image recognition to calculate whether there are still people standing in the elevator car to determine whether it is fully loaded, which solves the problem that existing weight detection cannot identify many types of actual full load, and avoids the situation where the elevator stops at each floor when it is actually fully loaded; in addition, this invention can be directly implemented based on existing monitoring cameras in the elevator car, which is low cost; this invention can eliminate the need for additional installation in existing elevators, and full load determination can be achieved through image recognition. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the elevator full-load detection method based on image recognition, according to an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the target area inside the elevator car according to an embodiment of the present invention. Detailed Implementation

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] In this embodiment of the invention, directional indicators (such as up, down, left, right, front, back, etc.) are only used to explain the relative positional relationship and movement of each component in a specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.

[0028] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features.

[0029] Please refer to Figure 1 The elevator full-load detection method based on image recognition in this embodiment of the invention includes steps 1 to 4.

[0030] Step 1: Set a reference ruler on the elevator car wall (the actual size of the reference ruler and its distance from the camera are known; it can be attached inside the elevator car). Set up the camera inside the elevator car. An existing surveillance camera in the elevator car can be used. This invention can be directly applied to existing elevator systems without additional hardware, saving costs. Preferably, the camera's wide-angle range is 75°–140°.

[0031] The camera inside the elevator car is calibrated (to eliminate camera distortion and improve image precision and accuracy). Camera parameters are obtained, including intrinsic and extrinsic parameter matrices, distortion parameter matrices, and camera focal length. The actual coordinates of the reference scale and the elevator car wall in the camera coordinate system are determined. The Zhang Zhengyou calibration method can be used for camera calibration. Preferably, the camera is positioned at the top center of the elevator car, which results in smaller calculation errors and more accurate full-load judgment.

[0032] Step 2: Real-time acquisition of images inside the elevator car, preprocessing the images to remove distortion, and then performing inverse perspective transformation to obtain the processed image. The processing in Step 2 can employ existing common perspective distortion elimination methods. This involves using a known point on the pixel plane and a perspective transformation matrix to obtain the transformed point. Existing perspective distortion elimination methods generally consist of three main steps: First, establishing a standard perspective blank coordinate system and drawing the standard pixel coordinates of the preset button corner points on this system. Second, the back projection algorithm. After obtaining the detection results of the corner point pixel coordinates, the inverse matrix of the camera's inherent parameters is used to transform the pixel coordinates of the button corner points into spatial coordinates on the normalized image plane at that time. Third, estimating camera motion from the distorted image to the standard perspective blank coordinate system. After obtaining the spatial coordinates, the Rodrigues equation is used to sample the rotation degree at intervals of 0.5 per axis, rotating the spatial coordinates of the distortion angle to obtain new spatial coordinates, forming a new spatial quadrilateral. When the lines of the new spatial quadrilateral are parallel to the standard spatial quadrilateral obtained from the standard perspective corner points, we can obtain the optimal pose for the camera movement. The fourth step is to form a new image without distortion. After obtaining the optimal pose, the same operation is performed on the corner point pixels to transform each pixel in the distorted image into new pixel coordinates. Then, by applying the inverse transformation, perspective distortion can finally be eliminated.

[0033] Step 3: Using a preset edge detection algorithm, segment and extract the reference scale, the four elevator car walls, and the target (i.e., person or object) inside the elevator car from the processed image. Determine the pixel coordinate data of the corresponding pixels on the edges of the reference scale, elevator car walls, and targets inside the elevator car, and perform linear fitting to obtain the contours of the corresponding regions of the reference scale, elevator car walls, and targets inside the elevator car. Determine the pixel coordinates of each pixel on the contour of each region (the set of pixels and their pixel coordinates on the contours of the regions of the reference scale, the four elevator car walls, and the people or objects inside the elevator car can be denoted as P). 参 P 壁1 P 壁2 P 壁3 P 壁4 P 人1 P人2 ..., P 物1 P 物2 ...), and then calculate the actual coordinates in the camera coordinate system corresponding to the pixel coordinates (the set of actual coordinates corresponding to each pixel on the outline of the scale, the four elevator car walls, and the area of ​​people or objects inside the elevator car can be denoted as R respectively). 参 R 壁1 R 壁2 R 壁3 R 壁4 R 人1 R 人2 , ..., R 物1 R 物2 ...).

[0034] In step 3, linear fitting is performed to obtain the region contour, which can eliminate some interference errors (such as hair at the edge of the corresponding area for a person), making the full load judgment more accurate and allowing the elevator to carry more people or objects within a reasonable range. Preferably, polynomial fitting, Bézier curve fitting, or B-spline curve fitting is used to obtain the contour through linear fitting. These existing fitting methods can be used to find a mathematical model that conforms to the actual contour, thereby better describing the shape of the object and improving the accuracy of the full load judgment in step 4.

[0035] As one implementation method, the preset edge detection algorithm is the Sobel operator, the Canny operator, or the Laplacian operator. This embodiment of the invention determines the object boundary based on changes in image pixel values; in specific implementations, OpenCV4 image segmentation can be used.

[0036] Step 4: Calculate the actual distance between the nearest pixels on the outline of each target region within the elevator car and the outline of adjacent target regions (this part is for calculating the nearest distance between adjacent targets (people and people or people and objects), please refer to...). Figure 2 For example, if person 3, person 2, object 1, and person 1 are all adjacent to each other, then from P... 人1 and P 人3 China, P 人2 and P 人3 China, P 人1 and P 物1 China, P 人2 and P 物1 P 物2 and P 物1 P 人2 and P 物2 P 人4 and P 物2 P 人2 and P 人4Find the two nearest pixels and calculate their actual distance. Also calculate the actual distance between the nearest pixels on the contours of the four elevator car walls and the adjacent target areas (this part is for calculating the nearest actual distance between the outermost target and the corresponding elevator car wall; please refer to...). Figure 2 The actual distances are calculated based on the following two parts: wall 1 with people 1, 2, and 3; wall 2 with people 1, object 1, and object 2; wall 3 with object 1, object 2, and person 4; and wall 4 with people 3 and 4. The actual distances are then calculated to obtain the set C of actual distances. If any value in set C is greater than the first preset value, then the elevator is not fully loaded. Otherwise, the sum of the differences between each value in set C and the second preset value is calculated. If the sum is greater than the first preset value, then the elevator is not fully loaded; otherwise, it is fully loaded. If a fully loaded situation is detected, it is fed back to the elevator control system so that the elevator does not need to stop at each floor.

[0037] As one implementation method, the first preset value ranges from 35 cm to 42 cm, and the second preset value ranges from 2 cm to 10 cm. The first preset value range corresponds to the shoulder width of an adult, and 40 cm can be used in specific implementations. If any value in set C is greater than the first preset value, it means that there is more space in the elevator car than an adult, and the elevator is not fully loaded. The range of the second preset value corresponds to the closest distance between people that people can accept. If it is less than this distance, it means that it is very crowded. Generally, 4 cm is selected. This distance will not make people feel very uncomfortable, and it also maximizes the elevator's loading capacity. When the sum of the differences between each value in set C and the second preset value is greater than the first preset value, it means that 1-2 more people can be loaded through displacement adjustment. If it is less, it means that the elevator cannot continue to be crowded. While ensuring the elevator's transportation efficiency during peak hours, it can also detect full load in a timely manner, and then feed back to the elevator control system so that the elevator does not need to stop at every floor.

[0038] As one implementation method, in step 3, the actual coordinates corresponding to the target pixel (i.e., the pixel to be calculated) are calculated according to the following formula:

[0039] ;

[0040] ;

[0041] Where X and Y are the actual coordinates of the target pixel in the camera coordinate system, f is the camera focal length, x and y are the pixel coordinates of the target pixel in the image, and Z is the distance from the actual position of the target pixel to the camera optical center, which is calculated using the following formula:

[0042] ;

[0043] Where W is the actual width of the reference ruler, and w is the pixel width of the reference ruler in the image.

[0044] In one implementation, step 4 calculates the actual spacing between pixels according to the following formula:

[0045] ;

[0046] Where (X1,Y1,Z1) and (X2,Y2,Z2) are the actual coordinates of the two target pixels in the camera coordinate system, and d is the actual distance between them.

[0047] Furthermore, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0048] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for detecting full load on an elevator based on image recognition, characterized in that, include: Step 1: Set a reference scale on the elevator car wall, set a camera inside the elevator car, calibrate the camera inside the elevator car, and obtain the camera parameters, including intrinsic and extrinsic parameter matrices, distortion parameter matrices, and camera focal length; determine the actual coordinate data of the reference scale and the elevator car wall in the camera coordinate system; Step 2: Real-time acquisition of images inside the elevator car, preprocessing the images to remove distortion, and then performing inverse perspective transformation to obtain the processed image; Step 3: Use a preset edge detection algorithm to segment and extract the reference scale, elevator car wall, and people or objects inside the elevator car from the processed image. Determine the pixel coordinate data of the corresponding pixels on the edges of the reference scale, elevator car wall, and people or objects inside the elevator car, and perform linear fitting to obtain the contour of the corresponding region of the reference scale, elevator car wall, and people or objects inside the elevator car. Determine the pixel coordinates of each pixel on the contour of each region, and then calculate the actual coordinates in the camera coordinate system corresponding to the pixel coordinates. Step 4: Calculate the actual distance between the nearest pixel on the outline of each person or object area in the elevator car and the outline of the nearest pixel on the outline of an adjacent person or object area, and calculate the actual distance between the elevator car wall and the nearest pixel on the outline of an adjacent person or object area, to obtain a set C of actual distances. If any value in set C is greater than the first preset value, then it is not fully loaded; otherwise, calculate the sum of the differences between each value in set C and the second preset value. If the sum is greater than the first preset value, then it is not fully loaded; otherwise, it is fully loaded.

2. The elevator full-load detection method based on image recognition as described in claim 1, characterized in that, In step 3, the actual coordinates of the target pixel are calculated using the following formula: ； ； Where X and Y are the actual coordinates of the target pixel in the camera coordinate system, f is the camera focal length, x and y are the pixel coordinates of the target pixel in the image, and Z is the distance from the actual position of the target pixel to the camera optical center, which is calculated using the following formula: ； Where W is the actual width of the reference ruler, and w is the pixel width of the reference ruler.

3. The elevator full-load detection method based on image recognition as described in claim 1, characterized in that, In step 4, the actual spacing between pixels is calculated using the following formula: ； Where (X1,Y1,Z1) and (X2,Y2,Z2) are the actual coordinates of the two target pixels in the camera coordinate system, and d is the actual distance between them.

4. The elevator full-load detection method based on image recognition as described in claim 1, characterized in that, The camera is located at the top center of the elevator car.

5. The elevator full-load detection method based on image recognition as described in claim 1, characterized in that, The preset edge detection algorithms are the Sobel operator, the Canny operator, or the Laplacian operator.

6. The elevator full-load detection method based on image recognition as described in claim 1, characterized in that, In step 3, a linear fit is performed using polynomial fitting, Bézier curve fitting, or B-spline curve fitting to obtain the contour.

7. The elevator full-load detection method based on image recognition as described in claim 1, characterized in that, The first preset value ranges from 35 cm to 42 cm, and the second preset value ranges from 2 cm to 10 cm.

Images