A high-speed people counting method based on depth camera

Abstract

The application discloses a high-speed personnel access counting method based on a depth camera and belongs to the technical field of machine vision algorithms. The regular depth information obtained by the depth camera is calibrated without calibration; iteration adaptive segmentation is performed on each frame, and people are cut out one by one, the introduction of three-dimensional point cloud coordinate values avoids the front and rear people shielding problem caused by the inclined installation of the camera, can avoid the loss of people during tracking, is suitable for complex scenes such as large height difference of pedestrians, mutual leaning and front and rear shielding, has simple process, is stable and reliable, the whole process is to process two-dimensional regular depth data, and there is no algorithm with high time complexity, so the running speed is fast, the real-time performance is strong, meanwhile, the whole algorithm process does not use a large third-party library, required memory resources are small, can be better transplanted to an embedded system such as a single-chip microcomputer, and has higher practicability.

Description

Technical Field

[0001] This invention relates to the field of machine vision algorithm technology, specifically to a high-speed method for counting people entering and exiting based on a depth camera. Background Technology

[0002] Multi-target recognition and detection is the core of machine vision and has always been a widely discussed issue. With the intelligent development of society and the emergence of concepts such as smart cities, smart transportation, and smart furniture, multi-target recognition and detection has also developed rapidly.

[0003] For intelligent public transportation systems, such as buses with online monitoring, understanding the number of people entering and exiting the vehicle can be used for dynamic planning to avoid congestion. Public transportation departments can use basic information about passenger congestion to build optimal bus scheduling models. By improving transport scheduling based on passenger flow at stations during different operating hours, it will bring significant economic benefits.

[0004] Whether in shopping malls or high-rise residential buildings, elevators are an essential part. However, if elevators are always operating at full power, they consume a lot of energy and result in serious waste of resources. Adjusting the operating power of elevators according to the number of people in the elevator can save a lot of energy.

[0005] Traditional optical gratings and weight sensors suffer from drawbacks such as low measurement efficiency and poor accuracy, leading to significant errors during use. Furthermore, their low efficiency can cause congestion at building and elevator entrances, and even create safety hazards.

[0006] The use of RGB cameras for person recognition and detection is limited by lighting conditions, making them unusable in low-light environments. Adding supplemental lighting can also negatively impact elevator comfort. Existing methods using deep learning and neural networks for feature recognition are also limited by complex environments such as dense crowds, varying heights, and people leaning on each other. These methods involve processing large amounts of information and require significant computation, hindering real-time judgment. Therefore, this paper proposes a high-speed person entry and exit counting method based on a depth camera. Summary of the Invention

[0007] The technical problem to be solved by the present invention is: how to overcome the shortcomings of the existing technology and provide a high-speed personnel entry and exit counting method based on a depth camera.

[0008] The present invention solves the above-mentioned technical problems through the following technical solution, and the present invention includes the following steps:

[0009] Includes the following steps:

[0010] S1: The depth camera is fixedly installed and takes pictures of the scene vertically or at an angle to the ground. The depth camera is used to obtain a regular depth map for each frame. For each frame of the depth map, the frame is scanned according to the set scan step size Nstep1 to obtain the set of pixels Sp.

[0011] S2: Compare the depth values ​​of the pixels in the set Sp with the lowest ground threshold Hgth of the row or column. Pixels with depth values ​​less than the lowest ground threshold Hgth are stored in the potential head point set Shdp, and pixels with depth values ​​greater than the lowest ground threshold Hgth are stored in the impossible potential head point set Swdp according to the row or column. Sort the potential head point set Shdp in ascending order.

[0012] S3: Traverse the set of impossible potential head points Sgdp, take the mean value Hgaver of each row or column of pixels, update the current value Hg of the ground value set of the row or column of pixels in the next frame with the mean value Hgaver of each row or column of pixels, and subtract a constant Cmh from Hg to obtain the lowest ground threshold Hgth of each row or column in the next frame.

[0013] S4: Select points successively from the potential head point set Shdp, select the potential head center point P0(x, y), and perform iterative processing in the vicinity of this point until the Euclidean distance between two iterations is less than the set threshold. The iteration is completed, and the head center point and the minimum bounding rectangle Th of the head are determined based on the result of the last iteration.

[0014] S5: Starting from the minimum and maximum values ​​xmax, xmin, ymax, and ymin in the x-axis and y-axis directions of the minimum bounding rectangle Th of the human head, push lines along the positive and negative directions of the y-axis and x-axis respectively until the depth value of a point in the pixel set Sp is less than the set threshold in the corresponding direction, and obtain the minimum bounding rectangle Tp of the human body.

[0015] S6: Determine the positional overlap between the minimum bounding rectangle Tp of the human body obtained in step S5 and the already obtained set of potential human body regions Saph. If they do not overlap, add the element to the set Saph; otherwise, discard it.

[0016] S7: Expand the minimum bounding rectangle Tp obtained in step S6, delete the points in the set Shdp located within the expanded rectangle, and form a new set Shdp.

[0017] S8: Repeat steps S4 to S7 until the set Shdp is empty, complete the adaptive segmentation of the current frame, and denote the set Saph as the set of potential human body regions Scph of the current frame.

[0018] S9: Match each element in the current frame's potential human body region set Scph with the minimum bounding rectangle of each element in the historical frame's potential human region set Shph, and establish the intersection-union ratio (GIOU) matrix Mg and the intersection-union ratio distance penalty matrix DGIOU matrix Mdg.

[0019] S10: Match the elements of the intersection-union distance penalty matrix DGIOU matrix Mdg in descending order to complete the matching of set Scph and set Shph; if an element in set Shph is matched for multiple consecutive frames, the element is promoted to a human region; if an element in set Shph is not matched for multiple consecutive frames, the element is deleted; all human regions constitute the human region set Spp.

[0020] S11: Establish the motion trajectory of each element based on the center position of each element in set Spp;

[0021] S12: Determine the direction of entry and exit and count the number of elements in the set Spp by their movement trajectories;

[0022] S13: If an element in set Spp has already been counted, then delete that element.

[0023] Furthermore, in step S4, the iterative process is as follows:

[0024] S41: Take points successively from the potential head point set Shdp, select the center point P0(x,y) of the potential head, subtract a constant Chmax from the depth value Hh corresponding to the selected potential head center point P to obtain the highest head height threshold hmaxth, and add a constant Chmin to obtain the lowest head height threshold hminth.

[0025] S42: Starting from the center point P0(x, y), push lines along both the positive and negative directions of the y-axis and x-axis. For each pixel reached by the push line, store the number of pixels with a depth value less than the maximum head height threshold hmaxth in Pmaxth, and the number of pixels with a depth value greater than the minimum head height threshold hminth in Pminth. Select the threshold Nmax for the number of pixels exceeding the maximum head height and the threshold Nmin for the number of pixels below the minimum head height. If Pmaxth is greater than Nmax or Pminth is less than Nmin during the push line process in any direction, end the push line process and obtain the coordinate value N corresponding to the end boundary of the push line in that direction. i , where i = -x, x, -y, y, x represents the coordinate value corresponding to the end boundary of the line push along the positive x-axis, -x represents the coordinate value corresponding to the end boundary of the line push along the negative x-axis, y represents the coordinate value corresponding to the end boundary of the line push along the positive y-axis, and -y represents the coordinate value corresponding to the end boundary of the line push along the negative y-axis.

[0026] S43: The coordinate value N corresponding to the obtained end boundary of the push line. i Where i = -x, x, -y, y, update the center point P(x, y) of the potential head to P1(x1, y1), then the center point P1 of the new potential head is:

[0027] x1=(N x +N -x ) / 2

[0028] y1=(N y +N -y ) / 2

[0029] Select an iteration termination threshold Niter. If the absolute values ​​of both (x1-x) and (y1-y) are less than Niter, the iteration is complete. Using point P1 as the midpoint of the head, select the minimum bounding rectangle Th of the head, where the coordinates of the top-left corner of Th are (N... -x N -y ), with length and width of N respectively. x -N -x N y -N -y Otherwise, continue iterating and repeat the above process until the absolute values ​​of (x1-x) and (y1-y) are both less than Niter.

[0030] Furthermore, in step S5, the specific process is as follows:

[0031] S51: From the coordinates of the top-left corner of the smallest bounding rectangle Th of the human head, and its length and width, we can obtain the xmin, xmax, ymin, and ymax of Th, where:

[0032] xmin = N -x

[0033] xmax = N -x +N x -N -x

[0034] ymin = N -y

[0035] ymax = N -y +N y -N -y

[0036] Starting from (xmax, (ymin+ymax) / 2) and (xmin, (ymin+ymax) / 2) respectively, push the points along the positive and negative directions of the x-axis; starting from ((xmin+xmax) / 2, ymax) and ((xmin+xmax) / 2, ymin) respectively, push the points along the positive and negative directions of the y-axis.

[0037] S52: After pushing the points along the y-axis and x-axis, we obtain the smallest bounding rectangle Th of the human body, as well as the coordinates of its upper left corner, length, and width, which are Pleft(xp, yp), Nwidth, and Nheight, respectively.

[0038] Furthermore, in step S51, the specific process of pushing the point along the positive and negative directions of the x-axis is as follows:

[0039] S5101: Set the minimum height threshold hpminth, select an equal division ratio Nx, divide Th into Nx equal parts along the y-axis, obtaining Nx-1 dividing lines. Using xmax and xmin as the starting x coordinates, push points in both positive and negative directions along the y-direction values ​​of the Nx-1 dividing lines. During the pushing process, calculate the 3D point cloud coordinates of the corresponding pixel in Sp and the 3D point cloud coordinates of the corresponding pixel in Sp after adding a step size stepx along the pushing line. Calculate the x-axis distance dx between these two points. The formula for calculating dx is as follows:

[0040] dx=distdxmin*abs(depstepx-dep)

[0041] distdxmin=100*abs(x-cx) / ff

[0042] Where abs represents the absolute value of the value within the parentheses, depstepx represents the depth value of the pixel in Sp corresponding to the point after adding the step size stepx, dep represents the depth value of the pixel in Sp corresponding to the point, x represents the x value of the point, and ff and cx are the intrinsic parameters of the depth camera.

[0043] S5102: Set the threshold for the furthest 3D point in the x-direction distance to dxth. If the threshold for the number of points in the furthest 3D point in the x-direction distance is not met, set the threshold to Ndx. When the depth value of the pushed pixel within Sp is greater than the height threshold hpminth of the shortest person's body or the number of points in the furthest 3D point in the x-direction distance is greater than Ndx, end the pushing along that line and store the ending coordinates in the set Sendp. i , where i = -x, x;

[0044] S5103: After the positive and negative direction push lines are completed, the two sets of final boundary coordinate values ​​obtained will be sentp. x and Sendp -x Sort the coordinates by smallest value and then select the px-th coordinate value from each of the two sets of coordinate values ​​to obtain the coordinate value N corresponding to the end boundary. i Where i = -x, x;

[0045] The x-coordinate of the top-left corner and the width of the smallest bounding rectangle of the potential human body are:

[0046] xp=N -x

[0047] Nwidth = N x -N -x .

[0048] Furthermore, in step S51, the specific process of pushing the point along the positive and negative directions of the y-axis is as follows:

[0049] S5111: Obtained from xp and Nwidth values ​​along the x-axis:

[0050] xpmin = N -x

[0051] xpmax = N -x +Nwidth

[0052] The initial values ​​for ypmin and ypmax along the y-axis are as follows:

[0053] ypmin = ymin

[0054] ypmax = ymax

[0055] Take a step size stepy, and starting from points (xpmin, ypmax+stepy) and (xpmin, ypmin-stepy), push the points along the direction from xpmin to xpmax. During the pushing process, calculate the 3D point cloud coordinates of the corresponding pixel in Sp, and the 3D point cloud coordinates of the corresponding pixel in Sp after adding or subtracting stepy in the y-axis direction. Calculate the y-axis distance dy between these two points. The formula for calculating dy is as follows:

[0056] dy=distdymin*abs(depstepy-dep)

[0057] distdymin=100*abs(y-cy) / ff

[0058] Where abs represents the absolute value of the value within the parentheses, depstepy represents the depth value of the pixel in Sp corresponding to the point after adding the step size stepy, dep represents the depth value of the pixel in Sp corresponding to the point, y represents the y value of the point, and ff and cy are the intrinsic parameters of the depth camera.

[0059] S5112: Set the threshold for the furthest 3D point's y-direction distance to dyth, the threshold for the shortest person's height to hpminth, the threshold for the number of points that do not meet the minimum height requirement to Npminth, and the threshold for the number of points that do not meet the furthest 3D point's y-direction distance requirement to Ndy. During the point pushing process from xpmin to xpmax, if the number of points that do not meet the minimum height requirement is greater than Npminth or the number of points that do not meet the furthest 3D point's y-direction distance requirement is greater than Ndy, the point pushing process ends; otherwise, update ypmin and ypmin.

[0060] ypmin = ypmin - stepy

[0061] ypmax = ypmax + stepy

[0062] Add the step size stepy and continue the push point process as described above for the next time;

[0063] S5113: After the push point is completed, the y-axis coordinate and length of the top-left corner of the smallest bounding rectangle of the potential human body are:

[0064] yp = ypmin

[0065] Nheight = ypmax - ypmin.

[0066] Furthermore, in step S6, the already obtained potential human body regions are denoted as set Saph, and are set to an empty set before each frame is processed; the minimum bounding rectangle of the newly obtained potential human body is Tp.

[0067] The newly obtained minimum bounding rectangle of the potential human body is matched one by one with the minimum bounding rectangles of all elements in the set Saph. The intersection area of ​​the two rectangles is calculated as Si. The threshold of the maximum intersection area of ​​the two rectangles is taken as Nith. If Si is greater than Nith, it means that Th has been detected and is directly discarded; otherwise, it is considered that a new element has been detected, Th is stored in the set Saph, and the depth value of the pixel in Sp corresponding to the minimum bounding rectangle Tp of the human body is set to a flag value Ndf that is different from the original data.

[0068] Furthermore, in step S7, each element of set Saph is expanded according to a set ratio p2, and points in set Shdp located within the expanded rectangle are deleted to form a new set Shdp.

[0069] Furthermore, step S9 specifically includes the following process:

[0070] S91: Denote the potential human regions of the historical frames as a set Shph, and set it to an empty set during initialization; denote the potential human body regions of the current frame as a set Scph, which is the set of all potential human body regions in the current frame in step S8.

[0071] S92: For each element in the set Shph, based on its historical position rectangle information and the continuity of head movement, establish a rectangle intersection-union ratio matrix Mg. The value in the r-th row and c-th column is the ratio of the intersection area to the union area of ​​the minimum bounding rectangle of the r-th element of the set Shph and the minimum bounding rectangle of the c-th element of the set Scph.

[0072] S93: Based on the intersection-union ratio matrix Mg, subtract a distance penalty value Ndp to obtain the intersection-union ratio distance penalty matrix DGIOU matrix Mdg. The value in the r-th row and c-th column is the ratio of the intersection area to the union area of ​​the minimum bounding rectangle of the r-th element of set Shph and the minimum bounding rectangle of the c-th element of set Scph, minus a distance penalty value Ndp.

[0073] The intersection-union ratio matrix Mg is as follows:

[0074]

[0075] The intersection-union ratio (IUU) distance penalty matrix DGIOU matrix Mdg is as follows:

[0076]

[0077] Furthermore, step S10 specifically includes the following process:

[0078] S101: Denote the number of potential human regions in the historical frame as Nh; denote the number of potential human body regions in the current frame as Nc; take the minimum matching success threshold Ngth during the matching process; denote the row Rm where the maximum element of the intersection-union distance penalty matrix Mdg is located, and the column as Cn, indicating that the Cn-th element of the set Scph matches the Rm-th element of the set Shph.

[0079] S102: If the value of the intersection-union ratio matrix Mg at the corresponding position is greater than Ngth, the matching is considered successful; otherwise, the matching is considered unsuccessful. Then, the elements of the Rm-th row and Cn-th column of the intersection-union ratio distance penalty matrix Mdg are all set to a constant Nmin, where Nmin = -40.

[0080] S103: Repeat steps S101 and S102 until all elements in matrix Md are Nmin, indicating that the matching is complete. Let the number of matching pairs be Np.

[0081] Furthermore, if Np equals Nh, it means that all elements of set Shph match elements of set Scph, and set Scph generates (Nc-Nh) new elements; if Np equals Nc, it means that all elements of set Scph match elements of set Shph, and set Shph has (Nh-Nc) unmatched elements; if an element of set Shph does not match for a set of frames for a set of frames consecutively, it means that the person region has left the field of view, and the element is deleted from set Shph; all elements of set Scph and all remaining elements in set Shph constitute the potential person region set Shph for the next frame of the historical frame; if an element of set Shph matches for a set of frames consecutively, the potential person region is upgraded to a person region.

[0082] Compared with existing technologies, this invention has the following advantages: This method for counting people entering and exiting based on a depth camera obtains regular depth information from the depth camera, eliminating the need for calibration; for each frame, it scans the frame at a certain step size to obtain a set of pixels, eliminating the need to scan the entire frame and significantly reducing algorithm time complexity; when the camera is tilted or encounters complex scenes such as stairs, the ground will tilt. The current value of the ground value set for the next frame is obtained by subtracting a constant Cmh from the current frame's row or column, and the highest ground threshold for that row or column is updated in real time, resulting in high accuracy and applicability to many complex scenes; iterative adaptive segmentation is performed for each frame, extracting each person individually. The introduction of 3D point cloud coordinates avoids the problem of occlusion caused by tilted camera installation, thus avoiding issues during tracking. The algorithm is suitable for complex scenarios such as pedestrians with large height differences, leaning on each other, and occlusion. It has a simple process and is stable and reliable. It matches the potential person region set of the historical frame with the potential person body region set of the current frame using the intersection-union ratio distance penalty value to form the potential person region set of the historical frame for the next frame. For potential person regions that are matched multiple times consecutively, they are upgraded to person regions. The person region set is updated in real time, which has very good accuracy, stability and real-time performance. The entire process processes two-dimensional rule depth data and does not have algorithms with high time complexity, so it runs fast and has strong real-time performance. At the same time, the entire algorithm does not use large third-party libraries, requires less memory resources, and can be better ported to embedded systems such as microcontrollers, making it more practical. Attached Figure Description

[0083] Figure 1 This is a flowchart illustrating the personnel entry and exit counting method based on a depth camera in Embodiment 1 of the present invention.

[0084] Figure 2 This is a schematic diagram of the implementation process in Embodiment 1 of the present invention;

[0085] Figure 3This is a schematic diagram of the pedestrian flow situation taken vertically in Embodiment 2 of the present invention;

[0086] Figure 4 This is a schematic diagram of a single iteration of the human head in Embodiment 2 of the present invention;

[0087] Figure 5 This is a schematic diagram of iterative head segmentation in Embodiment 2 of the present invention;

[0088] Figure 6 This is a schematic diagram of the human body segmentation in Embodiment 2 of the present invention;

[0089] Figure 7 This is a schematic diagram of the oblique shooting of the crowd flow in Embodiment 3 of the present invention;

[0090] Figure 8 This is a schematic diagram of iterative head segmentation in Embodiment 3 of the present invention;

[0091] Figure 9 This is a schematic diagram of the human body segmentation in Embodiment 3 of the present invention;

[0092] Figure 10 This is a schematic diagram of the matching of the current frame and historical frames in Embodiment 4 of the present invention;

[0093] Figure 11 This is a schematic diagram of the entry / exit judgment in Embodiment 5 of the present invention;

[0094] Figure 12 This is a schematic diagram of entry / exit determination in Embodiment Six of the present invention. Detailed Implementation

[0095] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0096] Example 1

[0097] like Figure 1 As shown, this embodiment provides a technical solution: a high-speed personnel entry and exit counting method based on a depth camera, including... Figure 1 The steps in the process.

[0098] It should be noted that this invention is based on Visual Studio 2019 software and uses C++ language code to implement the entire implementation case system. It runs on a Windows 10 system with an AMD-R7-5800H CPU. It processes regular depth images with a resolution of 320*240 obtained by a single-frame depth camera. The processing time is within 0.1ms, and the counting accuracy is above 99%.

[0099] like Figure 2As shown, the depth camera is first fixedly installed, capturing the scene vertically or at an angle to the ground. The tilt angle α = 0–45°; here, α = 0° for vertical shooting and α = 30° for tilt shooting is used to obtain a regular depth image with a resolution of 320*240. The camera then counts people entering and exiting within the field of view in real time. Based on the elevator specifications, the elevator door height Hg is between 2000 and 3000 mm. Here, Hg is the current value of the ground value set in the first frame; we take Hg = 2400. A maximum ground threshold constant Cmh is chosen, ranging from 400 to 800; here, we take Cmh = 600. The minimum ground threshold value for the first frame is manually specified as Hgth = Hg - Cmh = 1800. All distance units mentioned above are in mm. In practical implementation, taking the camera tilted forward or backward as an example, the ground values ​​should be taken column-wise, but this is not a limitation.

[0100] Example 2

[0101] This embodiment is a detailed explanation of the single-frame adaptive segmentation step in the method of Embodiment 1, specifically the first implementation case of vertical shooting single-frame adaptive segmentation. The specific process is as follows:

[0102] (1) As Figure 3 As shown, a person enters the field of view, and the camera is in a vertical shooting position, at which point α = 0°.

[0103] For this frame, scan the frame at a certain scan step size Nstep = 10 to 80, here we take Nstep = 30, to obtain the pixel set Sp.

[0104] Compare the depth values ​​of pixels within Sp with Hgth. Pixels with depth values ​​less than Hgth are stored in the potential head point set Shdp, while pixels with depth values ​​greater than Hgth are stored in the impossible potential head point set Sndp in columns.

[0105] Traverse the impossible potential head point set Shdp, take the mean value Hgaver of each column of pixels, update the current value Hg of the ground value set of that column in the next frame with the mean value Hgaver of each column of pixels, subtract a constant Cmh from Hg to obtain the lowest ground threshold Hgth of each column in the next frame, and sort the potential head point set Shdp in ascending order.

[0106] It should be noted that the above process provides a method for regionalizing ground values, but it is not limited to this. Generally, the direction of the long side of the camera is taken as front and back, and the direction of the short side of the camera is taken as left and right. When the camera is tilted front and back, the ground value set should be selected by row, and when the camera is tilted left and right, the ground value set should be selected by column. In this embodiment, taking the left and right tilt of the camera as an example, the ground value should be selected by column.

[0107] (2) Figure 4As shown, points are successively selected from the potential head point set Shdp, and the center point P0(x, y) of the potential head is selected. The depth value Hh corresponding to the selected potential head center point is subtracted by a constant Chmax = 100 - 200, here Chmax = 150, to obtain the maximum head height threshold hmaxth. Then, a constant Chmin = 100 - 200 is added, here Chmin = 200, to obtain the minimum head height threshold hminth.

[0108] Starting from the center point P0(x, y), lines are pushed along both the positive and negative y-axis. The number of pixels within Sp whose depth value is less than the maximum head height threshold hmaxth is stored in Pmaxth, and the number of pixels whose depth value is greater than the minimum head height threshold hminth is stored in Pminth. The number of pixels exceeding the maximum head height threshold Nmax and below the minimum head height threshold Nmin is selected. If, during the pushing process in any direction, Pmaxth is greater than Nmax or Pminth is less than Nmin, the pushing process ends, and the coordinate value N corresponding to the end boundary of the pushing process in that direction is obtained. i , where i=-x,x,-y,y.

[0109] The coordinate value N corresponding to the end boundary of the obtained push line. i Let i = -x, x, -y, y, to update the center point P1(x1, y1) of the potential head. Then the new center point P1 of the potential head is:

[0110] x1=(N x +N -x ) / 2

[0111] y1=(N y +N -y ) / 2

[0112] Select the iteration termination threshold Niter = 1 to 10, here Niter = 5. Since the absolute values ​​of (x1-x) and (y1-y) are both greater than Niter, update the depth value Hh corresponding to the potential head center point to the depth value of the pixel point within Sp corresponding to the coordinates of point P1, and continue the iteration.

[0113] (3) Figure 5 As shown, after two iterations, the difference between the x-axis coordinates and the y-axis coordinates of points P1 and P2 are both less than Niter. The iteration is complete, and the coordinates of the top left corner Pleft (xleft, yleft), the width Nwidth, and the length Nheight of the minimum bounding rectangle Th of the human head are obtained.

[0114] (4) Figure 6As shown, the coordinates of the top-left corner and the length and width of the smallest bounding rectangle Th of the human head can be used to obtain the xmin, xmax, ymin, and ymax of Th, where:

[0115] xmin = N -x

[0116] xmax = N -x +N x -N -x

[0117] ymin = N -y

[0118] ymax = N -y +N y -N -y

[0119] Starting from (xmax, (ymin+ymax) / 2) and (xmin, (ymin+ymax) / 2) respectively, push the points along the positive and negative directions of the x-axis. Starting from ((xmin+xmax) / 2,ymax) and ((xmin+xmax) / 2,ymin) respectively, push the points along the positive and negative directions of the y-axis.

[0120] Along the positive and negative directions of the x-axis:

[0121] Set the minimum height threshold hpminth, and select an equal division ratio Nx = 4 to 8. Here, Nx = 6 can be chosen. Divide Th into Nx = 6 equal parts along the y-axis, obtaining Nx-1 = 5 dividing lines. Using xmax and xmin, push points in both positive and negative directions along the y-direction values ​​of the Nx-1 dividing lines. During the pushing process, calculate the 3D point cloud coordinates of the corresponding pixel in Sp and the 3D point cloud coordinates of the corresponding pixel in Sp after adding a step size stepx along the pushing line. Calculate the x-axis distance dx between these two points. The calculation of dx can be, but is not limited to:

[0122] dx=distdxmin*abs(depstepx-dep)

[0123] distdxmin=100*abs(x-cx) / ff

[0124] In the above formula, abs represents the absolute value of the value inside the parentheses, depstepx represents the depth value of the pixel in Sp corresponding to the point after adding the step size stepx, dep represents the depth value of the pixel in Sp corresponding to the point, x represents the x value of the point, and ff and cx are the intrinsic parameters of the depth camera.

[0125] The x-direction distance threshold for the (farthest 3D point) 3D point cloud coordinates is set to dxth = 15~30. Here, dxth = 20 is chosen, which does not meet the threshold of Ndx for the number of points in the x-direction distance of the (farthest 3D point) 3D point cloud coordinates. When the depth value of the pushed pixel within Sp is greater than the height threshold hpminth of the shortest human body or the number of points in the x-direction distance of the farthest 3D point is greater than Ndx, the pushing along that line ends, and the ending coordinate value is stored in the set Sendp. i , where i = -x, x.

[0126] After the positive and negative lines are pushed out, the two sets of end boundary coordinate values ​​obtained will be sentp. x and Sendp -x Sort the coordinates by their smallest value, and select the coordinates from px = Nx / 2-2 to Nx / 2+2 from each of the two sets of coordinates. Here, we take the coordinate value of px = 3 to obtain the coordinate value N corresponding to the end boundary. i , where i = -x, x.

[0127] Then, the x-coordinate of the top-left corner and the width of the smallest bounding rectangle of the potential human body are:

[0128] xp=N -x

[0129] Nwidth = N x -N -x

[0130] Along the positive and negative directions of the y-axis:

[0131] The xp and Nwidth values ​​obtained along the x-axis can be used to obtain:

[0132] xpmin = N -x

[0133] xpmax = N -x +Nwidth

[0134] Initially select ypmin = ymin in the y-axis direction

[0135] ypmax = ymax

[0136] Take a step size `stepy`, and starting from points (xpmin, ypmax+stepy) and (xpmin, ypmin-stepy), push the point along the direction from xpmin to xpmax. During the pushing process, calculate the 3D point cloud coordinates of the corresponding pixel within `Sp` and the 3D point cloud coordinates of the corresponding pixel within `Sp` after adding or subtracting the step size `stepy` in the y-axis direction. Calculate the y-axis distance `dy` between these two points. The calculation of `dy` can be, but is not limited to:

[0137] dy=distdymin*abs(depstepy-dep)

[0138] distdymin=100*abs(y-cy) / ff

[0139] In the above formula, abs represents the absolute value of the value inside the parentheses, depstepy represents the depth value of the pixel in Sp corresponding to the point after adding the step size stepy, dep represents the depth value of the pixel in Sp corresponding to the point, y represents the y value of the point, and ff and cy are the intrinsic parameters of the depth camera.

[0140] The threshold for the farthest 3D point in the y-direction distance is set to dyth = 15~30, and here we take dyth = 25. The threshold for the shortest person's body height is hpminth. The threshold for the number of points that do not meet the shortest person's body height requirement is Npminth. The threshold for the number of points that do not meet the threshold for the farthest 3D point in the y-direction distance is Ndy.

[0141] During the initial point push from xpmin to xpmax, the following conditions were not met: the number of points with the shortest body height is less than Npminth, and the number of points with the y-direction distance from the farthest 3D point is less than Ndy. Update.

[0142] ypmin = ypmin - stepy

[0143] ypmax = ypmax + stepy

[0144] Add the step size stepy and continue to push the point a second time as described above.

[0145] During the second step of pushing points from xpmin to xpmax, the following conditions are not met: the number of points with the shortest body height is less than Npminth, and the number of points with the y-direction distance from the farthest 3D point is less than Ndy. Update.

[0146] ypmin = ypmin - stepy

[0147] ypmax = ypmax + stepy

[0148] During the third step of pushing points from xpmin to xpmax, the number of points with the shortest body height does not exceed Npminth, so an update is needed.

[0149] ypmin = ypmin - stepy

[0150] ypmax = ypmax + stepy

[0151] The conclusion is reached. The y-coordinate and length of the top-left corner of the smallest bounding rectangle of the potential human body are:

[0152] yp = ypmin

[0153] Nheight = ypmax - ypmin

[0154] After pushing the points along the y-axis and x-axis, we obtain the smallest bounding rectangle Th of the human body, with the coordinates of its top left corner and its length and width as Pleft(xp, yp), Nwidth, and Nheight.

[0155] (5) The potential human body regions that have been obtained are denoted as set Saph. If the set Saph is empty, the rectangle Th is pushed into the potential human body regions and denoted as set Saph.

[0156] (6) Each element of set Saph is expanded by a ratio p2 = 1.1 to 1.3, here p2 = 1.15. Points in set Shdp located within the expanded rectangle are deleted to form a new set Shdp.

[0157] (7) At this point, the set Shdp is empty, and the adaptive segmentation of the frame is complete.

[0158] Example 3

[0159] This embodiment is a detailed explanation of the single-frame adaptive segmentation step in the method of Embodiment 1, specifically the first implementation case of single-frame adaptive segmentation for tilted shooting. The specific process is as follows:

[0160] (1) As Figure 7 As shown, the two people come into view, and the camera is tilted at α = 30° to capture the scene.

[0161] For this frame, scan the frame at a certain scan step size Nstep = 10 to 80, here we take Nstep = 30, to obtain the pixel set Sp.

[0162] The depth values ​​of pixels within Sp are compared with Hgth. Pixels with depth values ​​less than Hgth are stored in the potential head point set Shdp, while pixels with depth values ​​greater than Hgth are stored in the impossible potential head point set Swdp in columns.

[0163] Traverse the set of impossible potential head points Sgdp, take the mean value Hgaver of each column of pixels, update the current value Hg of the ground value set of that column in the next frame with the mean value Hgaver of each column of pixels, and subtract a constant Cmh from Hg to obtain the lowest ground threshold Hgth of each column in the next frame.

[0164] (2) Figure 4 As shown, the single iteration process is consistent with the process (3) in the implementation case 1 of the vertical shooting single frame adaptive segmentation.

[0165] (3) Figure 8As shown, after two iterations, the difference between the x-axis coordinates and the y-axis coordinates of points Pleft1 and Pleft2 are both less than Niter. The iteration is complete, and the coordinates of the upper left corner Pll(xl, yl), the width Nlwidth, and the length Nlheight of the minimum bounding rectangle Th1 of the human head are obtained.

[0166] (4) Figure 9 As shown, the coordinates of the top-left corner and the length and width of the smallest bounding rectangle Th1 of the human head can be used to obtain the xmin, xmax, ymin, and ymax of Th1, where:

[0167] xmin = xl

[0168] xmax = xl + Nlwidth

[0169] ymin = yl

[0170] ymax = yl + Nlheight

[0171] Use xmax and xmin to push the points along the positive and negative directions of the x-axis, and use ymax and ymin to push the points along the positive and negative directions of the y-axis, respectively.

[0172] Along the positive and negative directions of the x-axis:

[0173] Set the minimum height threshold hpminth, and select an equal division ratio Nx = 4 to 8, here Nx = 5. Divide Th into Nx = 5 parts along the y-axis, obtaining Nx-1 = 4 dividing lines. Using xmax and xmin, push points in both positive and negative directions along the y-direction values ​​of the Nx-1 dividing lines. During the pushing process, calculate the 3D point cloud coordinates of the corresponding pixel in Sp and the 3D point cloud coordinates of the corresponding pixel in Sp after adding a step size stepx along the pushing line. Calculate the x-axis distance dx between these two points, where dx can be calculated in various ways, but is not limited to:

[0174] dx=distdxmin*abs(depstepx-dep)

[0175] distdxmin=100*abs(x-cx) / ff

[0176] In the above formula, abs represents the absolute value of the value inside the parentheses, depstepx represents the depth value of the pixel in Sp corresponding to the point after adding the step size stepx, dep represents the depth value of the pixel in Sp corresponding to the point, x represents the x value of the point, and ff and cx are the intrinsic parameters of the depth camera.

[0177] The threshold for the furthest 3D point's x-direction distance is set to dxth = 15-30. Here, dxth = 20 is chosen, which does not meet the threshold of the number of points the furthest 3D point can reach in the x-direction. Since the depth value of the pixels within Sp reached in the first, second, third, and fourth pushes in the negative direction is greater than the threshold of the shortest person's height hpminth, the push is terminated, and the final coordinates are stored in the set Sendp. i Where i = -x, x. When the depth value of the first, second, third, and fourth pushes in the positive direction to the pixel within p is greater than the minimum height threshold hpminth, the pushing ends, and the final coordinate values ​​are stored in the set Sendp. i , where i = -x, x.

[0178] After the positive and negative lines are pushed out, the two sets of end boundary coordinate values ​​obtained will be sentp. x and Sendp -x Sort the coordinates by their smallest value, and select the coordinates from px = Nx / 2-2 to Nx / 2+2 from each of the two sets of coordinates. Here, we take the coordinate value of px = 3 to obtain the coordinate value N corresponding to the end boundary. i , where i = -x, x.

[0179] Then, the x-coordinate of the top-left corner and the width of the smallest bounding rectangle of the potential human body are:

[0180] xp=N -x

[0181] Nwidth = N x -N -x

[0182] Along the positive and negative directions of the y-axis:

[0183] The xp and Nwidth values ​​obtained along the x-axis can be used to obtain:

[0184] xpmin = N -x

[0185] xpmax = N -x +Nwidth

[0186] Initially select ypmin = ymin in the y-axis direction

[0187] ypmax = ymax

[0188] Take a step size `stepy`, and starting from points (xpmin, ypmax+stepy) and (xpmin, ypmin-stepy), push the point along the direction from xpmin to xpmax. During the pushing process, calculate the 3D point cloud coordinates of the corresponding pixel within `Sp` and the 3D point cloud coordinates of the corresponding pixel within `Sp` after adding or subtracting the step size `stepy` in the y-axis direction. Calculate the y-axis distance `dy` between these two points. The calculation of `dy` can be, but is not limited to:

[0189] dy=distdymin*abs(depstepy-dep)

[0190] distdymin=100*abs(y-cy) / ff

[0191] In the above formula, abs represents the absolute value of the value inside the parentheses, depstepy represents the depth value of the pixel in Sp corresponding to the point after adding the step size stepy, dep represents the depth value of the pixel in Sp corresponding to the point, y represents the y value of the point, and ff and cy are the intrinsic parameters of the depth camera.

[0192] The threshold for the furthest 3D point in the y-direction distance is set to dyth = 15~30, here we take dyth = 25. The threshold for the shortest person's body height is hpminth. The threshold for the number of points that do not meet the shortest person's body height requirement is Npminth = 5~30, here we take Npminth = 20. The threshold for the number of points that do not meet the furthest 3D point in the y-direction distance requirement is Ndy = 5-30, here we take Ndy = 20.

[0193] During the initial stepping from xpmin to xpmax, the number of points at the shortest person's height did not exceed Npminth. Update:

[0194] ypmin = ypmin - stepy

[0195] ypmax = ypmax + stepy

[0196] The final calculation point is as follows: the y-coordinate of the top-left corner and the height of the smallest bounding rectangle of the potential human body are:

[0197] yp = ypmin

[0198] Nheight = ypmax - ypmin

[0199] After pushing the points along the y-axis and x-axis, the smallest bounding rectangle of the human body, Tp1, is obtained. The coordinates of its upper left corner and its length and width are Ppll(xp, yp), width Nplwidth, and length Nplheight.

[0200] (5) The potential human body region that has been obtained is denoted as set Saph (empty). The rectangle Tp1 is pushed into the potential human body region and denoted as set Saph. The depth value of the pixel point set Sp corresponding to the boundary line of the smallest bounding rectangle Tp of the human body is set to a flag value Ndf that is different from the original value. Here, Ndf = 11 is taken.

[0201] (6) The element Tp1 in the set Saph is expanded by a ratio p2 = 1 to 1.1, here p2 = 1.05. The points in the set Shdp located within the expanded rectangle are deleted to form a new set Shdp.

[0202] (7) At this point, the set Shdp is not empty. Continue with the adaptive segmentation of this frame.

[0203] (8) Use the already updated collection Shdp, such as Figure 4 As shown, the single iteration process is consistent with the process (3) in the first case of vertical shooting single-frame adaptive segmentation implementation.

[0204] (9) such as Figure 8 As shown, after two iterations, the difference between the x-axis coordinates and the y-axis coordinates of points Pright1 and Pright2 are both less than Niter, the iteration is complete, and the coordinates of the upper left corner Prr(xr, yr), width Nwidth, and length Nheight of the minimum bounding rectangle Th1 of the human head are obtained.

[0205] (10) such as Figure 9 As shown, the coordinates of the top-left corner and the length and width of the smallest bounding rectangle Th2 of the human head can be used to obtain the xmin, xmax, ymin, and ymax of Th2, where:

[0206] xmin = xr

[0207] xmax = xr + Nrwidth

[0208] ymin=yr

[0209] ymax = yr + Nrheight

[0210] Use xmax and xmin to push the points along the positive and negative directions of the x-axis, and use ymax and ymin to push the points along the positive and negative directions of the y-axis, respectively.

[0211] Along the positive and negative directions of the x-axis:

[0212] Set the minimum height threshold hpminth, and select an equal division ratio Nx = 4 to 8, here Nx = 5. Divide Th into Nx = 5 parts along the y-axis, obtaining Nx-1 = 4 dividing lines. Using xmax and xmin, push points in both positive and negative directions along the y-direction values ​​of the Nx-1 dividing lines. During the pushing process, calculate the 3D point cloud coordinates of the corresponding pixel in Sp and the 3D point cloud coordinates of the corresponding pixel in Sp after adding a step size stepx along the pushing line. Calculate the x-axis distance dx between these two points, where dx can be calculated in various ways, but is not limited to:

[0213] dx=distdxmin*abs(depstepx-dep)

[0214] distdxmin=100*abs(x-cx) / ff

[0215] In the above formula, abs represents the absolute value of the value inside the parentheses, depstepx represents the depth value of the pixel in Sp corresponding to the point after adding the step size stepx, dep represents the depth value of the pixel in Sp corresponding to the point, x represents the x value of the point, and ff and cx are the intrinsic parameters of the depth camera.

[0216] The threshold for the furthest 3D point's x-direction distance is set to dxth = 15-30. Here, dxth = 20 is chosen, which does not meet the threshold of Ndx for the number of points in the furthest 3D point's x-direction distance. The depth values ​​of the pixels within Sp pushed in the first, second, and third negative directions have been set to the flag value Ndf = 11. The depth value of the pixel within Sp pushed in the fourth direction is greater than the threshold hpminth for the shortest person's body height, so the pushing ends, and the ending coordinates are stored in the set Sendp. i Where i = -x, x. When the depth value of the first, second, third, and fourth pushes in the positive direction to the pixel within p is greater than the minimum height threshold hpminth, the pushing ends, and the final coordinate values ​​are stored in the set Sendp. i , where i = -x, x.

[0217] After the positive and negative lines are pushed out, the two sets of end boundary coordinate values ​​obtained will be sentp. x and Sendp -x Sort the coordinates by their smallest value and select the coordinates from the two sets of coordinates from px = Nx / 2-2 to Nx.

[0218] / 2+2 positions, here we take the coordinate value of px=3, to obtain the coordinate value N corresponding to the end boundary. i , where i = -x, x.

[0219] Then, the x-coordinate of the top-left corner and the width of the smallest bounding rectangle of the potential human body are:

[0220] xp=N -x

[0221] Nwidth = N x -N -x

[0222] Along the positive and negative directions of the y-axis:

[0223] The xp and Nwidth values ​​obtained along the x-axis can be used to obtain:

[0224] xpmin = N -x

[0225] xpmax = N -x +Nwidth

[0226] Initially select ypmin = ymin in the y-axis direction

[0227] ypmax = ymax

[0228] Take a step size `stepy`, and starting from points (xpmin, ypmax+stepy) and (xpmin, ypmin-stepy), push the point along the direction from xpmin to xpmax. During the pushing process, calculate the 3D point cloud coordinates of the corresponding pixel within `Sp` and the 3D point cloud coordinates of the corresponding pixel within `Sp` after adding or subtracting the step size `stepy` in the y-axis direction. Calculate the y-axis distance `dy` between these two points. The calculation of `dy` can be, but is not limited to:

[0229] dy=distdymin*abs(depstepy-dep)

[0230] distdymin=100*abs(y-cy) / ff

[0231] In the above formula, abs represents the absolute value of the value inside the parentheses, depstepy represents the depth value of the pixel in Sp corresponding to the point after adding the step size stepy, dep represents the depth value of the pixel in Sp corresponding to the point, y represents the y value of the point, and ff and cy are the intrinsic parameters of the depth camera.

[0232] The threshold for the furthest 3D point in the y-direction distance is set to dyth = 15~30, here we take dyth = 25. The threshold for the shortest person's body height is hpminth. The threshold for the number of points that do not meet the minimum body height requirement is Npminth = 5~30, here we take Npminth = 20. The threshold for the number of points that do not meet the minimum 3D point in the y-direction distance requirement is Ndy = 5~30, here we take Ndy = 20.

[0233] During the initial point push from xpmin to xpmax, the number of points with the shortest body height does not exceed Npminth, so an update is needed.

[0234] ypmin = ypmin - stepy

[0235] ypmax = ypmax + stepy

[0236] The final calculation point is as follows: the y-coordinate of the top-left corner and the height of the smallest bounding rectangle of the potential human body are:

[0237] yp = ypmin

[0238] Nheight = ypmax - ypmin

[0239] After pushing the points along the y-axis and x-axis, we obtain the smallest bounding rectangle Tp2 of the human body, with the coordinates of its top left corner and its length and width as Prleft(xp, yp), Nrwidth, and Nrheight.

[0240] (11) The potential human body regions that have been obtained are denoted as set Saph. There is an element Tp1. The rectangle Tp2 is overlapped with Tp1. The intersection area of ​​the two rectangles is calculated as Si. The threshold of the maximum intersection area of ​​the two rectangles is Nith = Tp1 * Ci, where Ci = 0.5 ~ 0.7. Here, Ci = 0.6 is taken. Since Si is less than Nith, Tp2 is a new element and is stored in the potential human body regions denoted as set Saph.

[0241] (12) The Tp2 in the set Saph is expanded by a ratio p2 = 1 to 1.1, here p2 = 1.05. The points in the set Shdp located within the expanded rectangle are deleted to form a new set Shdp.

[0242] (13) At this point, the set Shdp is empty. End the adaptive segmentation of this frame.

[0243] Example 4

[0244] This embodiment is a detailed explanation of the current frame and historical frame matching step in the method of Embodiment 1, that is, the first implementation case of current frame and historical frame matching. The specific process is as follows:

[0245] (1) As Figure 10 As shown, the potential human regions of historical frames are denoted as set Shph. Suppose that there are two elements T1 and T2 in set Shph at this time. T1 has matched for 5 consecutive frames, forming human region Spp. T2 has matched for 4 consecutive frames. The potential human regions of the current frame are denoted as set Scph. Set Scph has two elements, T3 and T4.

[0246] For each element Ti in set Shph, match it with each element Tj in set Shph, and calculate the intersection-union ratio (GIOU) matrix and the intersection-union ratio distance penalty matrix DGIOU of the minimum bounding rectangles of the two elements:

[0247] The elements of the GIOU matrix are:

[0248] GIOU = Si / Su

[0249] Where Si is the area of ​​the intersection of the minimum bounding rectangles of Ti and Tj, and Su is the area of ​​the union of the minimum bounding rectangles of Ti and Tj;

[0250] The elements of the DGIOU matrix are:

[0251] DGIOU=Si / Su-Ndp

[0252] Where Si is the area of ​​the intersection of the minimum bounding rectangles of Ti and Tj, Su is the area of ​​the union of the minimum bounding rectangles of Ti and Tj, and Ndp is the penalty value for the distance between the two elements, which can be taken as:

[0253] Ndp=k1*abs(ctopr-ptopr) / Npheight-k2*abs(ctopc-ptopc) / Npwidth

[0254] Where k1 = 2.0~5.0, here we take k1 = 2.0, k2 = 2.0~5.0, here we take k2 = 4.0, abs means taking the absolute value of the value in parentheses, ctopr means the y value of the center point of the minimum bounding rectangle of the potential human body in the current frame, ptopr means the y value of the center point of the minimum bounding rectangle of the human body in the historical frame, Npheight means the height of the minimum bounding rectangle of the human body in the historical frame; ctopc means the x value of the center point of the minimum bounding rectangle of the potential human body in the current frame, ptopc means the x value of the center point of the minimum bounding rectangle of the human body in the historical frame, and Npwidth means the width of the minimum bounding rectangle of the human body in the historical frame.

[0255] Construct an intersection-union ratio (GIOU) matrix Mg, where the value in the r-th row and c-th column is the intersection-union ratio of the r-th element of set Shph to the c-th element of set Scph.

[0256]

[0257] Construct an intersection-union ratio (IUU) distance penalty matrix Mdg, where the value in the r-th row and c-th column is the IUU of the r-th element of set Shph and the c-th element of set Scph minus the distance penalty value.

[0258]

[0259] (2) The number of potential human regions in the historical frame is Nh = 2; the number of potential human body regions in the current frame is Nc = 2.

[0260] The minimum successful intersection-union ratio (IU) threshold Ngth is set to 0.05-0.2 during the matching process. Here, Ngth is set to 0.15. The row containing the largest element of the IU distance penalty matrix Mdg is R2, and the column is C2. The GIOU value at the corresponding position in the IU matrix Mg is 0.56, which is greater than Ngth. This indicates that the C2th element of set Scph matches the R2th element of set Shph.

[0261] The elements of the R2 row and C2 column of the intersection-union distance penalty matrix Mdg can be set to a sufficiently small constant Nmin, here Nmin = -40.

[0262]

[0263] The row containing the largest element of the intersection-union ratio distance penalty matrix Mdg is R1, the column is C1, and the GIOU value of the corresponding position in the intersection-union ratio matrix Mg is 0.42, which is greater than Ngth. This indicates that the C1th element of set Scph matches the R1th element of set Shph.

[0264] The elements of the R2 row and C2 column of the intersection-union distance penalty matrix Mdg can be set to a sufficiently small constant Nmin, here Nmin = -40.

[0265]

[0266] If all elements in matrix Md are Nmin, it means the matching is complete, and the number of matching pairs is Np = 2.

[0267] Np = Nh = Nc, the two elements T1 and T2 in set Shph and the two elements T3 and T4 in set Scph are completely matched, with no additions or losses of elements.

[0268] All elements of set Scph are added to the historical frames of the matching elements in set Shph, forming the potential human region set Shph for the next historical frame.

[0269] If elements T1 and T2 of the Shph set match for three consecutive frames, these two elements are upgraded to human regions.

[0270] All human regions constitute the human region set Spp, and at this time, the set Spp contains two elements, historical frame data T1 and T2.

[0271] (3) Figure 10As shown, based on the data stored in Spp, the motion trajectories tra1 and tra2 of the human region set Spp are established.

[0272] Example 5

[0273] This embodiment is a detailed explanation of the entry / exit judgment step in the method of Embodiment 1, namely, the first implementation case of entry / exit judgment. The specific process is as follows:

[0274] like Figure 11 As shown, two entry and exit standard lines can be established. We can define the left standard line (L1) as the entry standard line and the right standard line (L2) as the exit standard line in the camera's field of view. We can define a person as exiting if they appear on the left side of L1 and move to the right side of L2, and as entering if they appear on the right side of L2 and move to the left side of L1. tra1 appears on the left side of L1 and moves to the right side of L2, indicating a person has exited. tra2 appears on the right side of L2 and moves to the left side of L1, indicating a person has entered. tra3 appears at L2 and stops between L1 and L2, not reaching the left side of L1, therefore it is not considered a person entering. When a person's entry or exit is detected, the data corresponding to that person in the potential person region set Shph from the person's historical frames is deleted.

[0275] Example 6

[0276] This embodiment is a detailed explanation of the entry / exit judgment step in the method of Embodiment 1, which is the second implementation case of entry / exit judgment. The specific process is as follows:

[0277] like Figure 12 As shown, two entry and exit standard lines can be established. We can define the left standard line (L1) as the entry standard line and the right standard line (L2) as the exit standard line in the camera's field of view. We can define a person as exiting if they appear on the left side of L1 and move to the right side of L2, and as entering if they appear on the right side of L2 and move to the left side of L1. If tra1 appears on the left side of L1 but does not reach the right side of L2, it is not considered an exit. If tra2 enters from the right side of L2 but does not reach the left side of L1, it is not considered an entry. If tra3 appears on the right side of L2, reaches the left side of L1, and continues from the left side of L1 to the right side of L2, it is considered an entry and an exit. When a person's entry or exit is determined, the data relative to that person in the potential person region set Shph of the person's historical frames is deleted.

[0278] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A high speed people ingress egress counting method based on depth camera, characterized in that, The method comprises the following steps: S1: a depth camera is fixedly installed, and a scene is shot vertically or obliquely to the ground, regular depth maps of each frame are obtained by using the depth camera, for each frame of the depth map, a set of pixel points Sp is obtained by scanning the frame at a set scanning step Nstep1; S2: the depth value of each pixel point in the set Sp is compared with the lowest ground threshold Hgth of the row or column, the pixel points less than the lowest ground threshold Hgth are stored in a potential head point set Shdp, the pixel points greater than the lowest ground threshold Hgth are stored in an impossible potential head point set Sgdp, and the potential head point set Shdp is sorted from small to large; S3: the impossible potential head point set Sgdp is traversed, the average value Hgaver of each row or column of pixel points is taken, the average value Hgaver of each row or column of pixel points is used to update the current value Hg of the ground value set of each row or column of pixel points of the next frame, and the lowest ground threshold Hgth of each row or column of the next frame is obtained by subtracting a constant Cmh from Hg; S4: a potential head center point P0(x, y) is taken from the potential head point set Shdp, iterative processing is performed around the point until the Euclidean distance of the results of two iterations is less than a set threshold, the iteration is completed, and the result of the last iteration is used to determine the head center point and a minimum circumscribed rectangle Th of the head; S5: the minimum maximum values xmax, xmin, ymax and ymin of the x-axis and y-axis directions of the minimum circumscribed rectangle Th of the head are taken as starting points, the pixel points are pushed in the positive and negative directions of the y-axis and x-axis respectively until the depth value of a point in the pixel point set Sp is less than a set threshold in the corresponding direction, and a minimum circumscribed rectangle Tp of the body of the person is obtained; S6: the minimum circumscribed rectangle Tp of the body of the person obtained in step S5 is subjected to position overlap judgment with the already obtained potential body region set Saph, if the minimum circumscribed rectangle Tp of the body of the person is not overlapped with the potential body region set Saph, the minimum circumscribed rectangle Tp of the body of the person is added to the set Saph, otherwise, the minimum circumscribed rectangle Tp of the body of the person is discarded; S7: the retained minimum circumscribed rectangle Tp of the body of the person in step S6 is expanded, the points in the set Shdp located in the expanded rectangle are deleted, and a new set Shdp is formed; S8: steps S4 to S7 are repeatedly performed until the set Shdp is empty, adaptive segmentation of the current frame is completed, and the set Saph is recorded as a potential body region set Scph of the current frame; S9: each element in the potential body region set Scph of the current frame is matched with the minimum circumscribed rectangle of each element in the potential body region set Shph of the historical frame, an intersection over union GIOU matrix Mg and an intersection over union distance penalty matrix DGIOU matrix Mdg are established; S10: elements of the intersection over union distance penalty matrix DGIOU matrix Mdg are matched in turn from large to small, and the matching of the set Scph and the set Shph is completed; if any element in the set Shph is matched for a plurality of consecutive frames, the element is upgraded to a person region; if any element in the set Shph is not matched for a plurality of consecutive frames, the element is deleted; and all the person regions constitute a person region set Spp. S11: Establishing the motion trajectory of each element in the set Spp according to the center position of the element; S12: Judging the in-out direction and counting by the motion trajectory of the element in the set Spp; S13: If any element in the set Spp has been counted, deleting the element.

2. A high speed people counting method based on depth camera according to claim 1, characterized in that: In the step S4, the iteration process is specifically as follows: S41: Taking the points from the set Shdp one by one, selecting the center point P0(x, y) of the potential head, subtracting a constant Chmax from the depth value Hh corresponding to the selected center point P of the potential head to obtain the highest head height threshold hmaxth, and adding a constant Chmin to obtain the shortest head height threshold hminth; S42: Taking the center point P0(x, y) as the starting point, respectively pushing points along the positive and negative directions of the y-axis and the x-axis, storing the number of points with a depth value less than the highest head height threshold hmaxth in Pmaxth, and the number of points with a depth value greater than the lowest head height threshold hminth in Pminth, selecting a threshold Nmax exceeding the number of points of the highest head height, and a threshold Nmin below the number of points of the lowest head height; if Pmaxth is greater than Nmax or Pminth is less than Nmin in the process of pushing points in any direction, ending the pushing process, and obtaining the coordinate value N corresponding to the pushing end boundary in the direction i wherein i=-x, x, -y, y, x represents the coordinate value corresponding to the pushing end boundary along the positive direction of the x-axis, -x represents the coordinate value corresponding to the pushing end boundary along the negative direction of the x-axis, y represents the coordinate value corresponding to the pushing end boundary along the positive direction of the y-axis, and -y represents the coordinate value corresponding to the pushing end boundary along the negative direction of the y-axis; S43: end the coordinate value N corresponding to the obtained push point end boundary i Wherein, i=-x, x, -y, y, update the center point P0(x, y) of the potential human head to P1(x1, y1), then the center point P1 of the new potential human head is: x1 = (N x + N -x ) / 2 y1 = (N y + N -y ) / 2 Select the iteration end threshold Niter, if the absolute value of (x1-x) and the absolute value of (y1-y) are less than Niter, the iteration is completed, take P1 point as the midpoint of the head, select the minimum circumscribed rectangle Th of the head, wherein the left upper corner coordinate point of Th is (N -x , N -y ), the length and width are N x -N -x , N y -N -y ; otherwise continue iteration, repeat the above process until the absolute value of (x1-x) and the absolute value of (y1-y) are less than Niter.

3. A high speed people counting method based on depth camera according to claim 2, characterized in that: In the step S5, the specific process is as follows: S51: The xmin, xmax, ymin and ymax of Th can be obtained from the coordinates of the upper left corner of the minimum circumscribed rectangle Th of the head and the length and width, wherein: xmin = N -x xmax = N -x + N x - N -x ymin = N -y ymax = N -y + N y - N -y Taking (xmax, (ymin+ymax) / 2) and (xmin, (ymin+ymax) / 2) as the starting points respectively, pushing the points along the positive and negative directions of the x-axis, and taking ((xmin+xmax) / 2, ymax) and ((xmin+xmax) / 2, ymin) as the starting points respectively, pushing the points along the positive and negative directions of the y-axis; S52: After the points are pushed along the x-axis and y-axis, the minimum circumscribed rectangle Th of the body is obtained, and the coordinates of the upper left corner, the length and the width of the rectangle are Pleft(xp, yp), Nwidth and Nheight respectively.

4. A high speed people counting method based on depth camera according to claim 3, characterized in that: In the step S51, the specific process of pushing the points along the positive and negative directions of the x-axis is as follows: S5101: Set the shortest body height threshold hpminth, select the equal proportion N F , divide Th into N F parts along the y-axis direction, obtain N F -1 boundary lines, take xmax and xmin as the starting x coordinates, and push the points in the positive and negative directions along the y direction values of the N F -1 boundary lines, respectively. In the process of pushing the points, calculate the three-dimensional point cloud coordinate values of the pixel points in Sp corresponding to the point and the three-dimensional point cloud coordinate values of the pixel points in Sp corresponding to the point after adding the stepx to the point along the pushing direction. Calculate the distance value dx in the x-axis direction of the two points, and the calculation formula of dx is as follows: dx=distdxmin*abs(depstepx-dep) distdxmin=100*abs(x-cx) / ff Wherein, abs represents the absolute value of the value in the parentheses, depstepx represents the depth value of the pixel point in Sp corresponding to the point after adding the step length stepx, dep represents the depth value of the pixel point in Sp corresponding to the point, x represents the x value of the point, ff and cx are the internal parameters of the depth camera; S5102: set the farthest three-dimensional point x-direction distance threshold value as dxth, and the farthest three-dimensional point x-direction distance point number threshold value as Ndx. When the depth value of the pixel point pushed to Sp is greater than the minimum height threshold value hpminth of the person or the number of points in the farthest three-dimensional point x-direction distance is greater than Ndx, end the pushing of the point in the current pushing direction, and store the coordinate value of the end into the set Sendp i , wherein i = -x, x; S5103: After the positive and negative direction push point ends, the obtained two end boundary coordinate value sets Sendp x and Sendp -x are sorted from small to large respectively, and the coordinate values of the first px positions are selected from the two coordinate value sets respectively to obtain the coordinate values N corresponding to the end boundary i , where i=-x, x; Then the x-axis coordinate of the upper left corner of the minimum circumscribed rectangle of the potential body and the width are: xp = N -x Nwidth = N x - N -x .

5. A high speed people counting method based on depth camera as claimed in claim 3, wherein: In the step S51, the specific process of pushing the points along the positive and negative directions of the y-axis is as follows: S5111: Obtaining the following values by using xp and Nwidth obtained along the x-axis direction: xpmin = N -x xpmax = N -x + Nwidth The initial ypmin and ypmax in the y-axis direction are as follows: ypmin=ymin ypmax=ymax Taking a step length stepy, taking points (xpmin, ypmax+stepy) and (xpmin, ypmin-stepy) as the starting points respectively, pushing the points along the direction from xpmin to xpmax, calculating the three-dimensional point cloud coordinate values of the pixel points in Sp corresponding to the points and the three-dimensional point cloud coordinate values of the pixel points in Sp corresponding to the points after adding and subtracting the step length stepy in the y-axis direction in the process of pushing the points, and calculating the y-axis distance dy of the two points, the calculation formula of dy is as follows: dy=distdymin*abs(depstepy-dep) distdymin = 100 * abs (y - cy) / ff wherein, abs represents the absolute value of the value in the parentheses, depstepy represents the depth value of the pixel point corresponding to Sp after the point plus the step length stepy, dep represents the depth value of the pixel point corresponding to Sp, y represents the y value of the point, ff and cy are the intrinsic parameters of the depth camera; S5112: set the farthest three-dimensional point y direction distance threshold dyth, the shortest human body height threshold hpminth, the number of points not meeting the shortest human body height threshold Npminth, and the number of points not meeting the farthest three-dimensional point y direction distance threshold Ndy. In the process of pushing the point in the direction from xpmin to xpmax, if the number of points not meeting the shortest human body height threshold is greater than Npminth or the number of points not meeting the farthest three-dimensional point y direction distance threshold is greater than Ndy, the pushing of the point is ended, otherwise, ypmin and ypmin are updated: ypmin = ypmin - stepy ypmax = ypmax + stepy plus stepy, continue the next pushing process according to the above process; S5113: after the pushing of the point is ended, the y axis coordinate of the upper left corner of the potential human body minimum bounding rectangle and the length are: yp = ypmin Nheight = ypmax - ypmin.

6. A high speed people counting method based on depth camera according to claim 3, characterized in that: In the step S6, the potential human body region obtained is recorded as a set Saph, which is set as an empty set before each frame is processed; and the newly obtained potential human body minimum bounding rectangle is Tp; The newly obtained potential human body minimum bounding rectangle is matched with all the elements in the set Saph one by one, the intersection area of the two rectangular frames is calculated as Si, and the maximum intersection area threshold of the two rectangular frames is taken as Nith. If Si is greater than Nith, it indicates that Th has been detected, and is directly discarded; otherwise, it is considered that a new element is detected, and Th is stored in the set Saph. The depth value of the pixel point in Sp corresponding to the human body minimum bounding rectangle frame Tp is set as a flag value Ndf different from the original data.

7. A high speed people ingress-egress counting method based on depth camera as claimed in claim 6, wherein: In the step S7, each element of the set Saph is expanded by a set proportion value p2, the points in the set Shdp located in the expanded rectangle are deleted, and a new set Shdp is formed.

8. A high speed people counting method based on depth camera according to claim 7, characterized in that: In the step S9, the following process is specifically included: S91: the potential human region of the historical frame is recorded as a set Shph, which is set as an empty set at the initialization; and the potential human body region of the current frame is recorded as a set Scph, which is the set of all potential human body regions of the current frame in the step S8; S92: for each element in the set Shph, a rectangular intersection ratio matrix Mg is established according to the historical position rectangular frame information and the continuity of the human head movement. The value in the rth row and the cth column is the ratio of the intersection area to the union area of the minimum bounding rectangle of the rth element of the set Shph and the minimum bounding rectangle of the cth element of the set Scph. S93: According to the intersection-over-union matrix Mg, a distance penalty value Ndp is subtracted to obtain an intersection-over-union distance penalty matrix DGIOU matrix Mdg, the value of the rth row and the cth column of which is the ratio of the intersection area of the minimum bounding rectangle of the rth element of the set Shph and the minimum bounding rectangle of the cth element of the set Scph to the union area minus a distance penalty value Ndp; Wherein, the intersection-over-union matrix Mg is as follows: The intersection-over-union distance penalty matrix DGIOU matrix Mdg is as follows:

9. A high speed people ingress-egress counting method based on depth camera as claimed in claim 8, wherein: In the step S10, the following processes are specifically included: S101: The number of potential human body regions of the historical frame is recorded as Nh, the number of potential human body regions of the current frame is recorded as Nc, the minimum matching success intersection-over-union threshold value Ngth in the matching process is taken, and the row Rm and the column Cn of the maximum element of the intersection-over-union distance penalty matrix Mdg are recorded, indicating that the Cnth element of the set Scph matches the Rmth element of the set Shph; S102: If the value of the intersection-over-union matrix Mg at the corresponding position is greater than Ngth, it is considered that the matching is successful, otherwise it is considered that the matching fails; then the elements of the Rmth row and the Cnth column of the intersection-over-union distance penalty matrix Mdg are both set to a constant Nmin, Nmin = -40; S103: Repeat the above step S101 and step S102 operations until all elements in the matrix Md are Nmin, which indicates that the matching is completed, and the number of matching pairs is recorded as Np.

10. The method of claim 9, wherein: If Np is equal to Nh, it indicates that all elements of the set Shph match the elements of the set Scph, and the set Scph generates (Nc-Nh) new elements; if Np is equal to Nc, it indicates that all elements of the set Scph match the elements of the set Shph, and the set Shph has (Nh-Nc) elements that are not matched; if any element of the set Shph is not matched for a continuous number of frames, it is considered that the human region exits the field of view, and the element is deleted from the set Shph; all elements of the set Scph and all remaining elements in the set Shph constitute the potential human region set Shph of the next frame of the historical frame; if any element of the set Shph is matched for a continuous number of frames, the potential human region is upgraded to a human region.

Images