A depth image segmentation method, device, electronic equipment and storage medium

Abstract

The application discloses a depth image segmentation method and device, electronic equipment and storage medium. The depth image segmentation method comprises the following steps: converting a current frame depth image captured by a camera into a point cloud, determining a door detection area point cloud from the point cloud, traversing the door detection area point cloud according to the opening and closing direction of an elevator door, determining a point with a difference value of depth values greater than a preset depth threshold value from adjacent points in the door detection area point cloud as an edge point of the elevator door, determining a pixel point corresponding to the edge point in the current frame depth image to generate a door edge, and segmenting the current frame depth image by the door edge to obtain a depth image area not blocked by the elevator door as hall-out depth data of the elevator. The depth image is segmented by processing the current frame depth image to generate the door edge in the current frame depth image, the door edge can be identified in the depth image without an additional sensor to detect the position of the door, and an effective area of the depth data not blocked by the door can be segmented.

Description

[0001] This application claims priority to Chinese Patent Application No. 2022108131852, filed on July 11, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present invention relates to the field of image processing technology, and in particular to a depth image segmentation method, apparatus, electronic device and storage medium. Background Technology

[0003] To improve elevator safety and passenger experience, sensors for collecting depth data need to be installed above the elevator car doors and to monitor targets outside the hall in real time. Based on the monitored data, if there is movement of related objects, it can be assumed that there is a passenger movement trend, and door opening control should be implemented according to the elevator control strategy.

[0004] Throughout the entire cycle when the elevator door is not fully closed, the sensor continuously collects depth data. However, as the elevator door closes, it remains within the sensor's field of vision. The elevator door obstructs and interferes with targets outside the hall, which can easily cause the system to misidentify the elevator door as the target to be monitored.

[0005] Existing technology requires the use of additional sensors to detect elevator doors, thereby enabling other sensors monitoring the external environment to identify valid areas in depth data that are not obstructed by elevator doors. Summary of the Invention

[0006] This invention provides a depth image segmentation method, apparatus, electronic device, and storage medium to solve the problem in the prior art that requires additional sensors to detect elevator doors.

[0007] In a first aspect, embodiments of the present invention provide a depth image segmentation method for segmenting depth images captured by a camera installed inside an elevator car, comprising:

[0008] The current frame depth image captured by the camera is converted into a point cloud, and the exit detection area point cloud is determined from the point cloud;

[0009] Traverse the point cloud of the door detection area according to the opening and closing direction of the elevator door, and determine the points in the point cloud of the door detection area whose depth value difference with the adjacent points is greater than a preset depth threshold, so as to be the edge points of the elevator door.

[0010] In the current frame depth image, determine the pixel points corresponding to the edge points to generate the door edges;

[0011] The current frame depth image is segmented by the edge of the door to obtain a depth image region that is not blocked by the elevator door, which is used as the elevator's hall depth data.

[0012] Secondly, embodiments of the present invention also provide a depth image segmentation apparatus for segmenting depth images captured by a camera installed inside an elevator car, comprising:

[0013] The point cloud conversion module is used to convert the current frame depth image captured by the camera into a point cloud, and to determine the exit detection area point cloud from the point cloud;

[0014] The point cloud traversal module is used to traverse the point cloud of the door detection area according to the opening and closing direction of the elevator door, and determine the points in the point cloud of the door detection area whose depth difference with the adjacent points is greater than a preset depth threshold, so as to be the edge points of the elevator door.

[0015] A door edge generation module is used to determine the pixel points corresponding to the edge points in the current frame depth image in order to generate door edges;

[0016] The segmentation module is used to segment the current frame depth image by the edge of the door to obtain a depth image region that is not blocked by the elevator door, which serves as the elevator's hallway depth data.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0018] At least one processor; and

[0019] A memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the depth image segmentation method described in the first aspect.

[0021] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions that are used to cause a processor to execute the depth image segmentation method described in the first aspect.

[0022] This invention converts the current frame depth image captured by a camera into a point cloud, and determines the exit detection area point cloud from the point cloud. It then traverses the door detection area point cloud according to the opening and closing direction of the elevator door, and identifies points in the door detection area point cloud whose depth value difference with adjacent points is greater than a preset depth threshold as edge points of the elevator door. Furthermore, it determines the corresponding pixels of the edge points in the current frame depth image to generate the door edge. Finally, it segments the current frame depth image using the door edge to obtain the depth image region not blocked by the elevator door as the elevator hall depth data. By processing the current frame depth image to generate the door edge in the current frame depth image, the depth image is segmented. This method can identify the exit edge in the depth image without the need for additional sensors to detect the door position, thereby segmenting the effective area of ​​depth data not blocked by the door. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1A An illustration taken by a camera;

[0025] Figure 1B A diagram illustrating the depth image captured by a camera;

[0026] Figure 1C This is a schematic diagram of depth image segmentation;

[0027] Figure 1D A schematic diagram illustrating the principle of door edge detection;

[0028] Figure 2 This is a flowchart of the steps of the depth image segmentation method according to Embodiment 1 of the present invention;

[0029] Figure 3 This is a flowchart of the steps of the depth image segmentation method according to Embodiment 2 of the present invention;

[0030] Figure 4 This is a structural block diagram of the depth image segmentation device according to Embodiment 3 of the present invention;

[0031] Figure 5 This is a structural block diagram of the electronic device according to Embodiment 4 of the present invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] In this embodiment, the door edge of the elevator door refers to the door edge in the vertical direction.

[0034] like Figure 1A As shown, when the elevator car 1 is level with the floor, in order to monitor the environment of the waiting hall 2, a camera 3 is installed inside the elevator car 1. The camera 3 can monitor the environment of the waiting hall 2 to form a depth image, and make decisions and controls on the elevator door through depth image analysis.

[0035] like Figure 1B The diagram shows a depth image captured by camera 3. Camera 3 is mounted on the door beam of car 1, and its viewpoint covers part of the interior of car 1 and the waiting hall 2. Depth image analysis mainly focuses on analyzing the depth image of a portion of the waiting hall 2. However, during the closing process of the elevator door 12, the visible area of ​​the waiting hall 2 changes dynamically for camera 3 due to the obstruction of the elevator door 12. Therefore, it is necessary to segment the image region corresponding to the visible area of ​​the waiting hall 2 in each frame of the depth image to improve the efficiency of analyzing the environment of the waiting hall 2. In the prior art, additional sensors are needed to detect the edge of the elevator door 12. After identifying the edge position of the elevator door 12 in the depth image, the area not obstructed by the elevator door 12 is segmented from the depth image as the visible area of ​​the waiting hall 2. To solve the problem of needing additional sensors to detect the elevator door 12, this invention provides a depth image segmentation method according to the following embodiments.

[0036] Example 1

[0037] Figure 2 This is a flowchart of a depth image segmentation method provided in Embodiment 1 of the present invention. This embodiment is applicable to the segmentation of depth images to obtain depth images of the area outside the hall that is not blocked by elevator doors. This method can be executed by a depth image segmentation device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 2 As shown, this depth image segmentation method includes:

[0038] S201. Convert the current frame depth image captured by the camera into a point cloud, and determine the exit detection area point cloud from the point cloud.

[0039] In this embodiment, the camera can be one or more of a monocular camera, a binocular camera, and a multi-camera system. In addition, the camera can also be at least one of a color camera, a monochrome camera, and an infrared camera.

[0040] like Figure 1A and Figure 1B As shown, camera 3 can be installed inside elevator car 1 so that camera 3 can capture part of the area where waiting hall 2 is located. In one example, camera 3 can be installed in the middle of the top beam of the elevator car door of elevator car 1. Of course, camera 3 can also be installed in other positions, as long as it can capture at least part of the area of ​​waiting hall 2. This embodiment does not limit the installation position of camera 3.

[0041] Camera 3 can capture images at a preset frame rate during elevator operation and generate a depth image based on the images. Each pixel in the depth image is associated with the distance from the physical location of that pixel to the camera, and this distance is the depth value.

[0042] In an alternative embodiment, the current frame depth image can be converted into a raw point cloud in the camera's coordinate system. The raw point cloud is then converted into a point cloud in the world coordinate system based on the camera's installation location, intrinsic parameters, and extrinsic parameters. For details, refer to existing methods for converting images into point clouds, which will not be described in detail here.

[0043] This embodiment converts a depth image into a point cloud. Compared to a depth image, a point cloud has higher precision. Compared to directly using a depth image to identify door edges, using a point cloud can improve the precision and accuracy of door edge recognition.

[0044] In another alternative embodiment, the elevator door frame can be determined from the point cloud based on the camera's installation location and the elevator car's geometry. Points within the area enclosed by the elevator door frame in the point cloud are then defined as the door detection area point cloud. Figure 1B As shown, since the camera's installation position is fixed and the geometric dimensions of the elevator car 1 are fixed (e.g., the elevator door frame 11 is fixed), the position of the elevator door frame 11 relative to the camera is also relatively fixed. Therefore, the position of the elevator door frame 11 can be determined from the point cloud. The points within the area enclosed by the elevator door frame 11 are used as the door detection area point cloud. Since the elevator door 12 must move within the area enclosed by the elevator door frame 11, by determining the door detection area point cloud, the door edge can be identified by processing the door detection area point cloud. This can reduce the amount of data that needs to be processed, improve the efficiency of identifying door edges, and improve the efficiency of depth image segmentation.

[0045] S202. Traverse the point cloud of the door detection area according to the opening and closing direction of the elevator door, and determine the points in the point cloud of the door detection area whose depth value difference with the adjacent points is greater than the preset depth threshold, so as to be the edge points of the elevator door.

[0046] Optionally, after obtaining the point cloud of the door detection area, the point cloud of the door detection area can be projected onto the plane where the elevator door is located, and on this plane, according to the opening and closing direction of the elevator door, N neighboring points are taken at a set interval to calculate the average depth value of the N neighboring points. If the absolute value of the difference between the average depth values ​​calculated by two adjacent intervals is greater than the preset depth threshold, it indicates that the point cloud of the door edge of the elevator door has been traversed.

[0047] like Figure 1D As shown, for point A on the edge of elevator door 12 and point B on the floor of waiting hall 2, the depth from point A to camera 3 is much smaller than the depth from point B to camera 3. That is, the point cloud near the edge of the elevator door, including points on the edge of the door and points on the floor of waiting hall 2, will have a sudden change in depth value. Based on this characteristic, if the absolute value of the difference between the mean values ​​of depths calculated by two adjacent intervals is greater than the preset depth threshold, it indicates that the point cloud of the elevator door edge has been traversed.

[0048] S203. Determine the pixel points corresponding to the edge points in the current frame depth image to generate the gate edge.

[0049] In one example, the point cloud can be projected onto the current frame depth image. Each point in the point cloud has a corresponding pixel in the current frame depth image. After traversing the point cloud of the door detection region to obtain the door edge points of the elevator door, the pixel corresponding to the edge point can be determined in the current frame depth image to generate the door edge.

[0050] In an optional embodiment, all points in the point cloud of the door detection region can be traversed to obtain multiple edge points of the elevator door edge. After determining the pixel points corresponding to the multiple door edge points in the current frame depth image, the door edge can be obtained by connecting or fitting the multiple pixel points.

[0051] In another alternative embodiment, points at a specified height in the point cloud of the door detection area can be traversed to determine the points on the door edge at the specified height. Here, height can refer to the vertical distance to the camera; for example, the specified height can be the vertical distance from the top of the elevator door to the camera. This allows traversing the edge points at the top of the elevator door, such as... Figure 1CAs shown, since the camera 3 captures images from top to bottom to obtain a depth image, the door edge 121 of the elevator door is not vertical in the depth image, but has a certain angle with the vertical direction. This angle is related to the position of the door edge 121 and the installation position of the camera 3. After traversing to obtain the top point of the door edge of the elevator door, the pixel point of that point in the depth image can be determined, such as pixel point P1. Then, after determining the position of pixel point P1, the angle C is determined according to the installation position of the camera 3. The line that passes through pixel point P1 and makes an angle C with the vertical direction in the depth image is the door edge.

[0052] In another alternative embodiment, the top and bottom points of the elevator door edge can be obtained simultaneously. After determining the pixel points of the top and bottom points of the door edge, such as pixel points P1 and P2, connecting pixel points P1 and P2 will give the door edge.

[0053] In one example, when the elevator door is a double door, the camera is installed in the middle of the crossbeam of the elevator car door. After obtaining one door edge, the other door edge can be obtained by mirroring the door edge with the camera as the center.

[0054] S204. The current frame depth image is segmented by the edge of the door to obtain a depth image region that is not blocked by the elevator door, which is used as the elevator's hall depth data.

[0055] like Figure 1C In the depth image shown, the door edges include the first door edge P1-P2 and the second door edge P3-P4. The depth image can be segmented by the first door edge P1-P2 and the second door edge P3-P4 to obtain the depth image of the shaded area, which can be used as the depth data of the waiting hall that is not blocked by the elevator door, i.e., the area visible from the camera's perspective.

[0056] This invention converts the current frame depth image captured by a camera into a point cloud, and determines the exit detection area point cloud from the point cloud. It then traverses the door detection area point cloud according to the opening and closing direction of the elevator door, and identifies points in the door detection area point cloud whose depth value difference with adjacent points is greater than a preset depth threshold as edge points of the elevator door. Furthermore, it determines the corresponding pixels of the edge points in the current frame depth image to generate the door edge. Finally, it segments the current frame depth image using the door edge to obtain the depth image region not blocked by the elevator door as the elevator hall depth data. By processing the current frame depth image to generate the door edge in the current frame depth image, the depth image is segmented. This method can identify the exit edge in the depth image without the need for additional sensors to detect the door position, thereby segmenting the effective area of ​​depth data not blocked by the door.

[0057] Example 2

[0058] Figure 3 This is a flowchart of a depth image segmentation method provided in Embodiment 2 of the present invention. This embodiment optimizes Embodiment 1 as described above. Figure 3 As shown, this depth image segmentation method includes:

[0059] S301. Convert the current frame depth image captured by the camera into a point cloud, and determine the exit detection area point cloud from the point cloud.

[0060] S302. Obtain a preset list of index values. Each index value in the list represents the index position of the elevator door edge relative to the camera in the opening and closing direction of the elevator door.

[0061] The index value list can be a lookup table of index values ​​and index positions of the elevator door edge. For example, each index value represents the index position of the elevator door edge relative to the camera in the opening and closing direction of the elevator door.

[0062] like Figure 1C As shown, the elevator door is a double door. Camera 3 is located in the middle of the upper crossbeam of the elevator car. Starting from the center line where camera 3 is located, the index values ​​to the left of camera 3 are 1 to 30, and the index values ​​to the right of camera 3 are -1 to -30. Each index value corresponds to an index position. For example, the index position corresponding to index value 1 is 1 cm to the left of camera 3, and the index value 30 represents 30 cm to the left of camera 3. Of course, the above is just an example. The index position corresponding to index value 1 can also be 2 cm to the left of camera 3, etc. This embodiment does not limit this.

[0063] Those skilled in the art can pre-set an index value list based on the geometric dimensions of the elevator car and the installation location of the camera.

[0064] S303. In the opening and closing direction of the elevator door, traverse the points in the door detection area point cloud at the specified height contained in the index position corresponding to each current index value, and calculate the average depth value of the points contained in the index position corresponding to each index value as the average depth value corresponding to the current index value.

[0065] Specifically, points in the door detection area point cloud can be projected onto the plane where the elevator door is located, and the index position corresponding to each index value contains multiple points in the door detection area point cloud.

[0066] like Figure 1CAs shown, the index position corresponding to index value 1 can be the interval between index value 1 and index value 0, the index position corresponding to index value 30 can be the interval between index value 30 and index value 29, and so on. The specified height H1 can be determined based on the installation position of camera 3 and the top of the elevator door, so as to traverse the point cloud of the top of the elevator door in the point cloud of the door detection area during traversal.

[0067] During traversal, the points in the gate detection region point cloud at the specified height contained at the index position corresponding to each current index value can be traversed from left to right. That is, when traversing to the current index value, the position between the current index value and the next index value is determined as the index position corresponding to the current index value, and the depth value of the point cloud within that index position is obtained to calculate the average depth value, which is used as the average depth value of the current index value.

[0068] like Figure 1C As shown, the point cloud is traversed according to the opening and closing direction F of the elevator door. When the index value 30 is reached, the interval between the index value 30 and the index value 29 is determined as the index position corresponding to the index value 30. This index position includes the projection points of multiple points in the point cloud of the door detection area. The average depth value of the multiple points included can be calculated as the average depth value of the index value 30.

[0069] S304. Calculate the difference between the current index value and the average depth of the previous index value.

[0070] For example, such as Figure 1C As shown, when traversing to index 29, the difference between the depth average of index 29 and index 30 can be calculated.

[0071] S305. When the absolute value of the difference is greater than the preset depth threshold, the point corresponding to the current index value is determined as the edge point of the elevator door.

[0072] like Figure 1C As shown, if when traversing to index 29, the absolute value of the difference between the average depth of index 29 and index 30 is greater than the preset depth threshold, the point corresponding to index 29 can be determined as the edge point of the elevator door.

[0073] When the height specified in S303 is determined based on the installation position of camera 3 and the top of the elevator door, the edge point is the edge point of the top of the elevator door, such as... Figure 1B Point E1, when the specified height is determined based on the installation position of camera 3 and the bottom of the elevator door, is the bottom edge point of the elevator door. Figure 1B Point E2 in the image; of course, the specified height can also be the vertical distance from any point on the edge of the elevator door to camera 3, in order to detect any edge point on the edge of the door.

[0074] S306. Determine the first pixel point corresponding to the first edge point in the current frame depth image.

[0075] In one example, the edge points include a first edge point, which is the edge point at the top of the elevator door, such as... Figure 1B Point E1, as shown, has an index value corresponding to the first edge point. This index value can be mapped to the corresponding pixel in the current frame's depth image, which is the first pixel corresponding to the first edge point. Figure 1B Mid-edge point E1 at Figure 1C The corresponding pixel in the image is pixel P1.

[0076] S307. In the preset index value list - angle comparison table, find the target angle that matches the current index value.

[0077] The index value list - included angle can be a lookup table built based on the camera installation location and the geometry of the elevator door. This lookup table is used to find the angle between the edge of the elevator door and the vertical direction in the depth image when the elevator door is located at the index position corresponding to each index value.

[0078] After determining the first edge point, you can find the angle that matches the index value of the first edge point in the preset index value list - angle comparison table as the target angle C.

[0079] S308. Generate the first gate edge that passes through the first pixel in the current frame depth image, and the angle between the first gate edge and the vertical direction is the target angle.

[0080] like Figure 1C In the depth image shown, the first gate edge P1-P2 is generated. The first gate edge P1-P2 passes through the first pixel point P1 and makes an angle C with the vertical direction.

[0081] In one optional embodiment, when the elevator door is a double door, the door edge includes a first door edge and a second door edge. After generating the first door edge, the second door edge is obtained by mirroring the first door edge with the camera as the center in the current frame depth image. Specifically, as shown... Figure 1C As shown, the edges of the first door, P1-P2, are mirrored with respect to the center line of camera 3 to obtain the edges of the second door, P3-P4. Of course, the same method used to generate the edges of the first door, P1-P2, can also be used to generate the edges of the second door, P3-P4.

[0082] In another embodiment, the point clouds at the top and bottom of the elevator door can also be traversed to obtain the edge points at the top and bottom, such as... Figure 1BThe edge points E1 and E2 in the image are identified, and the corresponding pixels, such as pixels P1 and P2, are determined in the depth image. Connecting pixels P1 and P2 yields the first gate edge.

[0083] In another alternative embodiment, all points in the point cloud of the gate detection region can be traversed to obtain multiple edge points, and the gate edge can be fitted using these multiple edge points.

[0084] S309. Segment the current frame depth image by the door edge to obtain the depth image region not blocked by the elevator door, which is used as the elevator hall depth data.

[0085] like Figure 1C In the depth image shown, the door edges include the first door edge P1-P2 and the second door edge P3-P4. The depth image can be segmented by the first door edge P1-P2 and the second door edge P3-P4 to obtain the depth image of the shaded area, which can be used as the depth data of the waiting hall that is not blocked by the elevator door, i.e., the area visible from the camera's perspective.

[0086] In an optional embodiment, before segmenting the current frame depth image by the door edge, the position of the door edge in the previous frame depth image can be obtained, the difference between the position of the door edge in the current frame depth image and the position of the door edge in the previous frame depth image can be calculated to obtain the moving distance, and the moving speed of the door edge can be calculated based on the moving distance. It is further determined whether the moving speed is greater than a preset speed threshold. If so, the current frame depth image is segmented by the door edge in the previous frame depth image to obtain the depth image area not blocked by the elevator door as the elevator hall outside depth data. If not, S309 is executed.

[0087] Specifically, after generating the door edge in each frame of depth image, the door edge can be tracked to obtain its movement speed. For example, if the camera acquires depth images at a preset frame rate, the interval between acquiring two frames of depth images can be obtained. The movement distance is calculated by the difference between the positions of the door edge in the two frames of depth images. The difference between the movement distance and the interval is the movement speed of the door edge, i.e., the movement speed of the elevator door. If the movement speed is greater than a preset speed threshold, it indicates that interference may be causing the door edge detection error. For example, when rapidly drifting dust passes by the camera, the dust is identified as the door edge. In this case, the door edge of the previous frame of depth image can be used to segment the current frame of depth image, avoiding the problem of door edge detection errors caused by interference, which leads to inaccurate depth image segmentation. This improves the anti-interference performance of door edge detection and the accuracy of depth image segmentation.

[0088] In an optional embodiment, after determining the first pixel corresponding to the first edge point in S306, the target pixel where the door edge is located can be determined from each row of pixels based on the camera installation position and the geometric dimensions of the elevator car, and the depth data not blocked by the elevator door can be read from the depth image based on the target pixel.

[0089] Specifically, such as Figure 1C As shown, after determining the first pixel P1 and mirroring it along the center line of the camera to obtain the second pixel P3, since the distance (height) from the physical position of each row of pixels to the camera is known in the depth image based on the camera's installation position and the elevator car's dimensions in the vertical direction, and the physical distance between the first pixel P1 and the second pixel P2 can be calculated based on their index positions, the pixels at the door edge in each row of pixels can be calculated using the following geometric relationship:

[0090]

[0091] In the above formula, H1 is the height from the camera to the physical position of the row pixel where the first pixel P1 and the second pixel P2 are located, and L... 13 H1 is the distance between the physical positions of the first pixel P1 and the second pixel P2, which can be calculated from the index positions of the first pixel P1 and the second pixel P2. H2 is the height from the physical position of the Nth row pixel to the camera. L 56 L is the physical distance between two points on the first and second gate edges in the Nth row of pixels. The above formula can be used to calculate L. 56 , will L 56 After dividing the image along the center line of camera 3, two index positions can be obtained. Using the index positions and height H2, the pixel P5 of the first door edge in the Nth row and the pixel P6 of the second door edge in the Nth row can be located in the depth image. The data of the pixels between pixel P5 and pixel P6 in the Nth row are read as the area not blocked by the elevator doors, that is, the depth data of the waiting hall visible from the camera's perspective.

[0092] In this embodiment, after converting the current frame depth image into a point cloud and determining the exit detection region point cloud from the point cloud, an index value list is obtained. In the opening and closing direction of the elevator door, the points in the door detection region point cloud at a specified height, contained at the index position corresponding to each current index value, are traversed to calculate the average depth value of the points contained at each index position. This average depth value is used as the average depth value corresponding to the current index value. The difference between the average depth value of the current index value and the previous index value is calculated. When the absolute value of the difference is greater than a preset depth threshold, the point corresponding to the current index value is determined as the edge point of the elevator door, thus identifying the first edge in the current frame depth image. The first pixel corresponding to the edge point is used, and the target angle matching the current index value is found in the preset index value list - angle lookup table. The first door edge passing through the first pixel point is generated in the current frame depth image. The angle between the first door edge and the vertical direction is the target angle. The current frame depth image is segmented by the door edge to obtain the depth image area not blocked by the elevator door, which is used as the elevator hall depth data. On the one hand, the door edge can be identified in the depth image without the need for additional sensors to detect the position of the door, so as to segment the effective area of ​​depth data not blocked by the door. On the other hand, the pixel corresponding to the edge point is determined by traversing the index value, which has low computation and high efficiency.

[0093] Furthermore, by calculating the average depth value of the points contained in the index position corresponding to each index value, the depth value can be filtered by mean. This avoids the situation where gaps in the elevator door structure cause abrupt changes in depth value, which may lead to misjudgment as door edges, thus improving the accuracy of door edge detection.

[0094] Furthermore, the door edge is tracked, and when the moving speed of the door edge exceeds a preset speed threshold, the current frame depth image is segmented from the door edge in the previous frame depth image. This avoids the problem of door edge detection errors caused by high-speed moving objects such as dust, which leads to inaccurate depth image segmentation. This improves the anti-interference performance of door edge detection and the accuracy of depth image segmentation.

[0095] Example 3

[0096] Figure 4 This is a schematic diagram of a depth image segmentation device provided in Embodiment 3 of the present invention. Figure 4 As shown, the depth image segmentation device is used to segment depth images captured by cameras installed inside an elevator car, including:

[0097] The point cloud conversion module 401 is used to convert the current frame depth image captured by the camera into a point cloud, and to determine the exit detection area point cloud from the point cloud;

[0098] The point cloud traversal module 402 is used to traverse the point cloud of the door detection area according to the opening and closing direction of the elevator door, and determine the points in the point cloud of the door detection area whose depth value difference with the adjacent points is greater than a preset depth threshold, so as to be the edge points of the elevator door.

[0099] The door edge generation module 403 is used to determine the pixel points corresponding to the edge points in the current frame depth image in order to generate door edges;

[0100] The segmentation module 404 is used to segment the current frame depth image with the door edge to obtain a depth image region that is not blocked by the elevator door, as the elevator's hall outside depth data.

[0101] Optionally, the point cloud conversion module 401 includes:

[0102] The first point cloud generation unit is used to convert the current frame depth image into the original point cloud in the coordinate system of the camera;

[0103] The second point cloud generation unit is used to convert the original point cloud into a point cloud in the world coordinate system based on the installation position of the camera, the intrinsic parameters and extrinsic parameters of the camera.

[0104] Optionally, the point cloud conversion module 401 includes:

[0105] An elevator door frame determination unit is used to determine the elevator door frame from the point cloud based on the installation position of the camera and the geometric dimensions of the elevator car.

[0106] The door detection area point cloud determination unit is used to determine the points in the area enclosed by the elevator door frame as the door detection area point cloud in the point cloud.

[0107] Optionally, the point cloud traversal module 402 includes:

[0108] An index value list acquisition unit is used to acquire a preset index value list, wherein each index value in the index value list represents the index position of the edge of the elevator door relative to the camera in the opening and closing direction of the elevator door;

[0109] The point cloud traversal unit is used to traverse the points in the point cloud of the door detection area at a specified height contained at the position corresponding to each current index value in the opening and closing direction of the elevator door, so as to calculate the average depth value of the points contained at the index position corresponding to each index value as the average depth value corresponding to the current index value.

[0110] The difference calculation unit is used to calculate the difference between the current index value and the depth average of the previous index value.

[0111] An edge point determination unit is used to determine the point corresponding to the current index value as the edge point of the elevator door when the absolute value of the difference is greater than a preset depth threshold.

[0112] Optionally, the edge point includes a first edge point at the top of the elevator door within the field of view of the camera, and the door edge generation module 403 includes:

[0113] The first pixel point determination unit is used to determine the first pixel point corresponding to the first edge point in the current frame depth image;

[0114] The target angle determination unit is used to find the target angle that matches the current index value in a preset index value list - included angle lookup table;

[0115] The first gate edge generation unit is used to generate a first gate edge passing through the first pixel in the current frame depth image, wherein the angle between the first gate edge and the vertical direction is the target angle.

[0116] Optionally, the edge points include a first edge point at the top and a second edge point at the bottom of the elevator door within the field of view of the camera, and the door edge generation module 403 includes:

[0117] A pixel point determination unit is used to determine, in the current frame depth image, a first pixel point corresponding to the first edge point and a second pixel point corresponding to the second edge point;

[0118] The first gate edge generation unit is used to generate line segments with the first pixel and the second pixel as endpoints in the current frame depth image to obtain the first gate edge.

[0119] Optionally, the elevator door is a double door, and the door edge includes a first door edge and a second door edge. The door edge generation module 403 further includes:

[0120] The second gate edge generation unit is used to mirror the first gate edge in the current frame depth image with the camera as the center to obtain the second gate edge.

[0121] Optionally, it also includes:

[0122] The previous door edge position acquisition module is used to obtain the position of the door edge in the previous frame depth image;

[0123] The movement speed calculation module is used to calculate the difference between the position of the door edge in the current frame depth image and the position of the door edge in the previous frame depth image to obtain the movement distance, and to calculate the movement speed of the door edge based on the movement distance;

[0124] The movement speed determination module is used to determine whether the movement speed is greater than a preset speed threshold;

[0125] The segmentation module 402 includes:

[0126] The first segmentation unit is used to segment the current frame depth image from the door edge in the previous frame depth image to obtain a depth image region that is not blocked by the elevator door, which serves as the elevator's hallway depth data.

[0127] The depth image segmentation apparatus provided in this embodiment of the invention can execute the depth image segmentation method provided in Embodiment 1 and Embodiment 2 of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0128] Example 4

[0129] Figure 5 A schematic diagram of an electronic device 50 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0130] like Figure 5 As shown, the electronic device 50 includes at least one processor 51 and a memory, such as a read-only memory (ROM) 52 and a random access memory (RAM) 53, communicatively connected to the at least one processor 51. The memory stores computer programs executable by the at least one processor. The processor 51 can perform various appropriate actions and processes based on the computer program stored in the ROM 52 or loaded into the RAM 53 from storage unit 58. The RAM 53 can also store various programs and data required for the operation of the electronic device 50. The processor 51, ROM 52, and RAM 53 are interconnected via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0131] Multiple components in electronic device 50 are connected to I / O interface 55, including: input unit 56, such as keyboard, mouse, etc.; output unit 57, such as various types of monitors, speakers, etc.; storage unit 58, such as disk, optical disk, etc.; and communication unit 59, such as network card, modem, wireless transceiver, etc. Communication unit 59 allows electronic device 50 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0132] Processor 51 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 51 performs the various methods and processes described above, such as depth image segmentation methods.

[0133] In some embodiments, the depth image segmentation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 50 via ROM 52 and / or communication unit 59. When the computer program is loaded into RAM 53 and executed by processor 51, one or more steps of the depth image segmentation method described above may be performed. Alternatively, in other embodiments, processor 51 may be configured to perform the depth image segmentation method by any other suitable means (e.g., by means of firmware).

[0134] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0138] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0139] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0140] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0141] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0142] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A depth image segmentation method, characterized in that, The depth images captured by cameras installed inside the elevator car are used for segmentation, including: The current frame depth image captured by the camera is converted into a point cloud, and the exit detection area point cloud is determined from the point cloud; Traverse the point cloud of the door detection area according to the opening and closing direction of the elevator door, and determine the points in the point cloud of the door detection area whose depth value difference with the adjacent points is greater than a preset depth threshold, so as to be the edge points of the elevator door. In the current frame depth image, determine the pixel points corresponding to the edge points to generate the door edges; The current frame depth image is segmented by the edge of the door to obtain a depth image region that is not blocked by the elevator door, which is used as the elevator's hall depth data. The step of traversing the point cloud of the door detection area according to the opening and closing direction of the elevator door, and determining points in the point cloud whose depth difference with adjacent points is greater than a preset depth threshold, as edge points of the elevator door, includes: Obtain a preset list of index values, where each index value in the list represents the index position of the elevator door edge relative to the camera in the opening and closing direction of the elevator door; In the opening and closing direction of the elevator door, the points in the door detection area point cloud at a specified height included at the index position corresponding to each current index value are traversed to calculate the average depth value of the points included at the index position corresponding to each index value, which is used as the average depth value corresponding to the current index value. The specified height is the vertical distance from the top of the elevator door to the camera. During traversal, the point cloud at the top of the elevator door in the door detection area point cloud is traversed. The points included at the index position are the points in the multiple door detection area point clouds included at the index position after projecting the points in the door detection area point cloud onto the plane where the elevator door is located. Calculate the difference between the current index value and the average depth of the previous index value; When the absolute value of the difference is greater than a preset depth threshold, the point corresponding to the current index value is determined as the edge point of the elevator door. The edge point includes the first edge point of the top of the elevator door within the field of view of the camera.

2. The method as described in claim 1, characterized in that, The step of converting the current frame depth image captured by the camera into a point cloud includes: Convert the current frame depth image into the original point cloud in the coordinate system of the camera; The original point cloud is converted into a point cloud in the world coordinate system based on the installation location of the camera, the camera's intrinsic parameters, and extrinsic parameters.

3. The method as described in claim 1, characterized in that, Determining the exit detection region point cloud from the point cloud includes: The elevator door frame is determined from the point cloud based on the installation location of the camera and the geometric dimensions of the elevator car. In the point cloud, the points within the area enclosed by the elevator door frame are defined as the door detection area point cloud.

4. The method as described in claim 1, characterized in that, The step of determining the pixel corresponding to the edge point in the current frame depth image to generate the door edge includes: In the current frame depth image, determine the first pixel point corresponding to the first edge point; In the preset index value list - angle lookup table, find the target angle that matches the current index value; A first gate edge passing through the first pixel is generated in the current frame depth image, and the angle between the first gate edge and the vertical direction is the target angle.

5. The method as described in claim 1, characterized in that, The edge point also includes a second edge point at the bottom of the elevator door within the field of view of the camera. The step of determining the pixel corresponding to the edge point in the current frame depth image to generate the door edge includes: In the current frame depth image, determine the first pixel corresponding to the first edge point and the second pixel corresponding to the second edge point; In the current frame depth image, a line segment with the first pixel and the second pixel as endpoints is generated to obtain the first gate edge.

6. The method as described in claim 4 or 5, characterized in that, The elevator door is a double door, and the door edge includes a first door edge and a second door edge. The step of determining the pixel points corresponding to the edge points in the current frame depth image to generate the door edge further includes: In the current frame depth image, the edge of the first door is mirrored with the camera as the center to obtain the edge of the second door.

7. The method according to any one of claims 1-5, characterized in that, Before segmenting the current frame depth image by the door edge to obtain the depth image region not blocked by the elevator door, and using it as the elevator's hallway depth data, the process further includes: Get the location of the door edge in the previous frame depth image; The difference between the position of the door edge in the current frame depth image and the position of the door edge in the previous frame depth image is calculated to obtain the moving distance, and the moving speed of the door edge is calculated based on the moving distance; Determine whether the moving speed is greater than a preset speed threshold; If so, the step of segmenting the current frame depth image by the door edge to obtain a depth image region not blocked by the elevator door, as the elevator's hallway depth data, includes: The current frame depth image is segmented from the door edge in the previous frame depth image to obtain the depth image region not blocked by the elevator door, which is used as the elevator hall outside depth data; If not, perform the step of segmenting the current frame depth image by the door edge to obtain a depth image region not blocked by the elevator door, which is used as the elevator's hall outside depth data.

8. A depth image segmentation apparatus, characterized in that, The depth images captured by cameras installed inside the elevator car are used for segmentation, including: The point cloud conversion module is used to convert the current frame depth image captured by the camera into a point cloud, and to determine the exit detection area point cloud from the point cloud; The point cloud traversal module is used to traverse the point cloud of the door detection area according to the opening and closing direction of the elevator door, and determine the points in the point cloud of the door detection area whose depth difference with the adjacent points is greater than a preset depth threshold, so as to be the edge points of the elevator door. A door edge generation module is used to determine the pixel points corresponding to the edge points in the current frame depth image in order to generate door edges; The segmentation module is used to segment the current frame depth image by the edge of the door to obtain a depth image region that is not blocked by the elevator door, which serves as the elevator's hall depth data. The point cloud traversal module includes: An index value list acquisition unit is used to acquire a preset index value list, wherein each index value in the index value list represents the index position of the edge of the elevator door relative to the camera in the opening and closing direction of the elevator door; The point cloud traversal unit is used to traverse the points in the door detection area point cloud at a specified height corresponding to the position of each current index value in the opening and closing direction of the elevator door, so as to calculate the average depth value of the points contained in the index position corresponding to each index value as the average depth value corresponding to the current index value. The specified height is the vertical distance from the top of the elevator door to the camera. During traversal, the point cloud at the top of the elevator door in the door detection area point cloud is traversed. The points contained in the index position are the points in the multiple door detection area point clouds contained in the index position after projecting the points in the door detection area point cloud onto the plane where the elevator door is located. The difference calculation unit is used to calculate the difference between the current index value and the depth average of the previous index value. An edge point determination unit is used to determine the point corresponding to the current index value as the edge point of the elevator door when the absolute value of the difference is greater than a preset depth threshold. The edge point includes the first edge point of the top of the elevator door within the field of view of the camera.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the depth image segmentation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the depth image segmentation method according to any one of claims 1-7.

Images