Crowd counting method, device and system
The depth and amplitude images are obtained through image sensors, and the extreme value points and target classification are detected using sliding windows, which solves the problem of low accuracy and efficiency of traffic statistics in the prior art, and achieves more efficient calculations and more accurate traffic statistics.
Patent Information
- Application Number
- CN202011062769.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-30
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-09-30
AI Technical Summary
The existing flow statistics technology has low accuracy and efficiency in ambient light changes and crowded people, and 3D point cloud data is easily affected by the body of the characters and leads to false detection.
The depth image and amplitude image are acquired based on the image sensor, and the candidate head area is determined through the sliding window to detect the extreme points in the depth image, and the amplitude image is used to classify the target to realize the traffic statistics.
The calculation accuracy and efficiency of traffic statistics are improved, the influence of overexposure and overdarkness of light is avoided, and the complexity and false detection rate of target segmentation are reduced.
Smart Images

Figure CN112257520B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device and system for counting human traffic. Background Art
[0002] With the rapid development of the new retail industry, the statistics of passenger flow data are changing from manual observation to intelligent sensor statistics. Statistical methods based on computer vision technology can scientifically and effectively analyze passenger flow in time and space, and make business decisions quickly and timely. Statistics of passenger flow in different time periods enable managers to accurately plan future activities, determine time, manpower, etc., thereby improving service quality.
[0003] At present, 2D vision algorithms or 3D point cloud data are mainly used for crowd counting. However, when using 2D target recognition algorithms to identify people in RGB images, it is easily affected by changes in ambient light. When the number of people is too dense, even if the model is complex enough, target segmentation is difficult and the amount of calculation is large, which reduces the efficiency and accuracy of crowd counting. Crowd counting using 3D point cloud data will be affected by the body parts of people and is prone to false detection. Therefore, the existing crowd counting technology still has the problem of low accuracy and calculation efficiency. Summary of the invention
[0004] In view of this, an object of the present invention is to provide a method, device and system for counting human flow, which can improve the calculation accuracy and efficiency of human flow counting.
[0005] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:
[0006] In a first aspect, an embodiment of the present invention provides a method for counting human flow, comprising: acquiring a depth image and an amplitude image of a target area based on an image sensor; wherein the image sensor is arranged directly above the target area; performing extreme point detection on the depth distance in the depth image based on a sliding window, and determining each candidate head area in the depth image; acquiring each image to be detected in the amplitude image based on each candidate head area, performing target classification on the images to be detected, and obtaining a result of counting human flow in the target area; wherein the image to be detected is the corresponding candidate head area in the amplitude image.
[0007] Further, an embodiment of the present invention provides a first possible implementation manner of the first aspect, wherein the step of performing extreme point detection on the depth distance in the depth image based on a sliding window to determine each candidate head region in the depth image includes: obtaining the depth distance corresponding to each pixel point in the depth image; determining the window size of the sliding window based on the intrinsic parameters of the image sensor; locating each extreme value region in the depth image according to the window size of the sliding window and the depth distance corresponding to each pixel point; wherein the extreme value region is the minimum value region of the depth distance corresponding to each pixel point in the depth image; calculating the area intersection-union ratio between each of the extreme value regions, and determining each candidate head region according to the area intersection-union ratio.
[0008] Further, an embodiment of the present invention provides a second possible implementation of the first aspect, wherein the step of locating each extreme value area in the depth image according to the window size of the sliding window and the depth distance corresponding to each pixel point includes: determining the average depth distance in each sliding window based on the depth distance corresponding to each pixel point, the window size of the sliding window and a preset sliding step, and obtaining a sliding window mean matrix; determining the center point coordinates of each sliding window in the depth image based on the window size of the sliding window and the preset sliding step, and obtaining a coordinate index matrix corresponding to the sliding window mean matrix; determining the coordinate size of each extreme value area in the depth image based on the sliding window mean matrix and the coordinate index matrix; wherein the coordinate size includes the center point coordinates and pixel size of the extreme value area.
[0009] Further, an embodiment of the present invention provides a third possible implementation of the first aspect, wherein the step of determining the coordinate size of each extreme value area in the depth image based on the sliding window mean matrix and the coordinate index matrix includes: determining each minimum depth distance in the sliding window mean matrix; wherein the minimum depth distance is less than each adjacent average depth distance value; obtaining the center point coordinates corresponding to the sliding window where each minimum depth distance is located from the coordinate index matrix, and obtaining the center point coordinates of each extreme value area; based on each minimum depth distance, the internal parameters of the image sensor and the first calculation formula, obtaining the pixel length of each extreme value area; wherein the pixel length is the number of pixel columns of the extreme value area, and the first calculation formula is:
[0010]
[0011] Based on each of the minimum depth distances, the internal parameters of the image sensor and the second calculation formula, the pixel width of each of the extreme value regions is obtained; wherein the pixel width is the number of pixel rows in the extreme value region, and the second calculation formula is:
[0012]
[0013] Among them, W i is the pixel length of the i-th extreme value area, L i is the pixel width of the ith extreme value region, Head w Head is the preset physical length of the head. l is the preset head physical width, I cols is the number of pixel columns of the depth image, I rows is the number of pixel rows of the depth image, α is the horizontal field angle of the image sensor, β is the vertical field angle of the image sensor, H i is the i-th minimum depth distance, and round is the rounding operation.
[0014] Further, an embodiment of the present invention provides a fourth possible implementation of the first aspect, wherein the step of determining the window size of the sliding window based on the intrinsic parameters of the image sensor includes: acquiring a preset physical size of the head; wherein the physical size of the head includes a physical length of the head and a physical width of the head; and determining the number of pixels corresponding to the physical length of the head based on the intrinsic parameters of the image sensor and a third calculation formula; wherein the third calculation formula is:
[0015]
[0016] Based on the internal parameters of the image sensor and a fourth calculation formula, the number of pixels corresponding to the physical width of the head is determined; wherein the fourth calculation formula is:
[0017]
[0018] Among them, W pixel is the number of pixels corresponding to the physical length of the head, L pixel is the number of pixels corresponding to the physical width of the head, Head w is the physical length of the head, Head l is the physical width of the head, I cols is the number of pixel columns of the depth image, I rows is the number of pixel rows of the depth image, α is the horizontal field angle of the image sensor, β is the vertical field angle of the image sensor, H install is the vertical distance between the image sensor and the plane where the person is located, H avg is the average height of pedestrians, and round is a rounding operation; based on the number of pixels corresponding to the physical length of the head and the number of pixels corresponding to the physical width of the head, the window size of the sliding window is determined.
[0019] Further, an embodiment of the present invention provides a fifth possible implementation of the first aspect, wherein the step of obtaining each image to be detected in the amplitude image based on each candidate head area, performing target classification on the image to be detected, and obtaining a pedestrian flow statistical result in the target area includes: based on the coordinate size of each candidate head area, obtaining an image within each coordinate size area from the amplitude image to obtain each image to be detected; inputting each image to be detected into a pre-trained neural network model to obtain a classification result of each target to be detected; wherein the neural network model is trained based on pre-labeled human amplitude image samples; and determining the pedestrian flow statistical result in the target area based on the classification result of each target to be detected.
[0020] Further, an embodiment of the present invention provides a sixth possible implementation of the first aspect, wherein the step of determining the pedestrian flow statistics result in the target area based on the classification results of each of the targets to be detected includes: determining the number of people in the target area based on the classification results of each of the targets to be detected; and using the number of people as the pedestrian flow statistics result in the target area.
[0021] Further, an embodiment of the present invention provides a seventh possible implementation of the first aspect, wherein the crowd flow statistics result includes the number of people entering and leaving; a warning line is set in the depth image; the step of determining the crowd flow statistics result in the target area based on the classification result of each target to be detected includes: determining each person head area in the depth image based on the classification result of each target to be detected; determining the moving trajectory of each person head area in the target area based on each person head area in multiple consecutive frames of depth images; determining the number of people entering and leaving the target area based on the moving trajectory of each person head area in the target area and the position of the warning line.
[0022] Further, an embodiment of the present invention provides an eighth possible implementation of the first aspect, wherein a first warning line, a second warning line and a third warning line are set in the depth image; the step of determining the amount of people entering and leaving the target area based on the moving trajectory of each of the person's head areas in the target area and the position of each of the warning lines includes: when the moving trajectory of the person's head area intersects with the first warning line, the second warning line and the third warning line in sequence, adding one to the count value of the amount of people entering; when the moving trajectory of the person's head area intersects with the third warning line, the second warning line and the first warning line in sequence, adding one to the count value of the amount of people leaving; based on the count value of the amount of people entering and the count value of the amount of people leaving within a preset time, the amount of people entering and leaving the target area is obtained.
[0023] Furthermore, an embodiment of the present invention provides a ninth possible implementation of the first aspect, wherein the method further includes: when the moving trajectory of the head area of the person intersects with the warning line, triggering a preset alarm device to sound an alarm.
[0024] In a second aspect, an embodiment of the present invention further provides a pedestrian flow counting device, comprising: an image acquisition module, used to acquire a depth image and an amplitude image of a target area based on an image sensor; wherein the image sensor is arranged directly above the target area; an area determination module, used to perform extreme point detection on the depth image based on a sliding window, and determine each candidate head area in the depth image; a flow counting module, used to acquire each image to be detected in the amplitude image based on each candidate head area, perform target classification on the image to be detected, and obtain a pedestrian flow counting result of the target area; wherein the image to be detected includes the candidate head area in the amplitude image.
[0025] In a third aspect, an embodiment of the present invention provides a people flow counting system, comprising: an image sensor and a controller, the controller comprising a processor and a storage device; a computer program is stored on the storage device, and the computer program, when executed by the processor, executes the method described in any one of the first aspects.
[0026] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the steps of the method described in any one of the above-mentioned first aspects.
[0027] The embodiment of the present invention provides a method, device and system for counting human flow, the method comprising: obtaining a depth image and an amplitude image of a target area based on an image sensor; wherein the image sensor is arranged directly above the target area; performing extreme point detection on the depth distance in the depth image based on a sliding window to determine each candidate head area in the depth image; obtaining each image to be detected in the amplitude image based on each candidate head area, performing target classification on the image to be detected, and obtaining a human flow counting result of the target area; wherein the image to be detected is the corresponding candidate head area in the amplitude image. In the method, by obtaining the depth image and the amplitude image of the target area, the influence of overexposure and overdarkness on imaging is avoided. Since the depth image carries the depth distance information corresponding to each pixel point, by detecting the extreme point of the depth distance in the depth image based on the sliding window, each candidate head area in the depth image can be obtained without target segmentation. By further classifying the corresponding candidate head area in the amplitude image, the number of head areas in the target area can be determined, thereby quickly and accurately obtaining the human flow counting result, and improving the calculation accuracy and efficiency of human flow counting.
[0028] Other features and advantages of the embodiments of the present invention will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned techniques of the embodiments of the present invention.
[0029] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0031] Figure 1 A flow chart of a method for counting human traffic provided by an embodiment of the present invention is shown;
[0032] Figure 2 A depth image of a target area provided by an embodiment of the present invention is shown;
[0033] Figure 3 An amplitude image of a target area provided by an embodiment of the present invention is shown;
[0034] Figure 4 A schematic diagram of a warning line provided by an embodiment of the present invention is shown;
[0035] Figure 5 A schematic diagram of the structure of a human flow counting device provided by an embodiment of the present invention is shown;
[0036] Figure 6 A schematic diagram of the structure of a controller provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be described below in conjunction with the drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.
[0038] At present, crowd counting based on 2D visual algorithms usually uses RGB images as processing data. RGB images are easily affected by changes in ambient light, which affects the accuracy of crowd counting. In addition, the 2D target recognition algorithm uses a deep learning model, which has high training and hardware costs. When the number of people in the image is too dense, even if the model is complex enough, target segmentation is difficult and the amount of calculation is large. Crowd counting based on 3D point clouds searches for extreme points and fits circles or ellipses to determine the head area. This head area determination method is easily affected by limbs and relies on point cloud size features to easily cause false detection. Therefore, considering that the existing crowd counting technology still has the problem of low accuracy and computational efficiency, in order to improve this problem, the embodiments of the present invention provide a crowd counting method, device and system, which can be used to improve the computational accuracy and efficiency of crowd counting. The following is a detailed introduction to the embodiments of the present invention.
[0039] This embodiment provides a method for counting human traffic. Figure 1 The flow chart of the method for counting people flow shown in the figure can be applied to a controller (such as a computer or a mobile terminal, etc.) that is connected to the image sensor for communication. The method mainly includes the following steps S102 to S106:
[0040] Step S102: acquiring a depth image and an amplitude image of the target area based on an image sensor.
[0041] The above image sensor can simultaneously capture the depth image and amplitude image of the target area, and the resolution and acquisition range of the depth image and the amplitude image are the same (that is, the pixel coordinates of each target object in the depth image are equal to the corresponding pixel coordinates of each target object in the amplitude image). The above amplitude image is an infrared amplitude map obtained by acquiring the infrared light amplitude of each person in the target area. The above image sensor can be a solid-state laser radar, which can capture the depth image and amplitude image of the target area at the same time.
[0042] The target area can be any area where people flow statistics are required, see Figure 2 The depth image of the target area shown, Figure 2 The depth image of the target area is shown in FIG. 1 , and the grayscale value of the pixel in the depth image can reflect the depth distance corresponding to the pixel. Figure 3 Amplitude image of the target area shown, Figure 3 The amplitude image of the target area is shown in FIG. Figure 2 The content is the same as Figure 2 and Figure 3 The pixel dimensions are also the same.
[0043] In order to facilitate crowd counting and prevent people in the image from blocking each other, the above-mentioned image sensor can be set directly above the target area. The image sensor directly above the target area can capture the heads of each person in the target area, avoiding the situation where people are blocked and it is difficult to count the flow of people, thereby improving the accuracy of crowd counting.
[0044] In order to improve the computational efficiency of crowd counting, the depth image and amplitude image can be filled with holes and denoised by median filtering to reduce the interference of noise points. Considering that the target of detection and tracking in the crowd counting task is pedestrians, the ground and low objects will increase the amount of calculation and recognition interference. Therefore, the depth threshold can be set according to the height range information of children and adults (for example, the depth threshold can be 1m~2m), and the depth image and amplitude image are directly filtered based on the depth threshold. Pixels whose depth distance is not outside the depth threshold are filtered out to eliminate the interference of the ground and low objects on the crowd counting calculation and improve the computational efficiency.
[0045] Step S104: performing extreme point detection on the depth distance in the depth image based on the sliding window to determine each candidate head region in the depth image.
[0046] Since the position of the image sensor is fixed, in the captured image of the person, the distance between the person's head area and the image sensor is the smallest. The depth image carries the depth distance information of each pixel point. By detecting the minimum value of the depth distance corresponding to each pixel point in the depth image, the candidate head area in the depth image that may be the head area can be determined.
[0047] The above sliding window can traverse each pixel in the depth image. By using the sliding window to detect the minimum value of the depth distance corresponding to each pixel in the depth image, it is possible to prevent the minimum value point of the depth distance from being missed, avoid the influence of noise (directly searching for extreme points in the depth image will be affected by noise), and thus avoid missing the head area. Compared with using the target recognition algorithm to identify the head area, using the sliding window to detect the candidate head area does not require target segmentation of the depth image. When the people in the depth image are relatively dense, each candidate head area can also be accurately detected, avoiding the problem of reducing the accuracy of target segmentation due to dense people and thus reducing the accuracy of human flow statistics.
[0048] Step S106 , obtaining each to-be-detected image in the amplitude image based on each candidate head region, performing target classification on the to-be-detected image, and obtaining a human flow statistics result of the target region.
[0049] Based on the position and size of each candidate head region in the depth image, images of the same size are obtained from the same position in the amplitude image to obtain each image to be detected. By performing target classification on each image to be detected in the amplitude image, it is possible to identify whether each image to be detected is a head region. According to the target classification results of each image to be detected in the amplitude image, the number of head regions in the amplitude image can be obtained, and then the number of head regions in the target region can be obtained, thereby obtaining the human flow statistics result (the human flow statistics result includes the number of people in the target region).
[0050] For example, the above-mentioned depth image includes a first candidate head area, a second candidate head area and a third candidate head area. According to the center point coordinates and pixel sizes (length and width, in pixels) of the first candidate head area, the second candidate head area and the third candidate head area, the first image to be detected, the second image to be detected and the third image to be detected corresponding to each candidate head area are obtained from the amplitude image, wherein the center point coordinates and pixel size of the first candidate head area are the same as those of the first image to be detected, and the target object in the first candidate head area is the same target object as the target object in the first image to be detected, and the position and size of the target object in the depth image and the amplitude image are the same, and so on, the second candidate head area has the same center point coordinates and pixel size as the second image to be detected, and the third candidate head area has the same center point coordinates and pixel size as the third image to be detected.
[0051] The above-mentioned crowd counting method provided in this embodiment avoids the influence of overexposure and darkness on imaging by acquiring the depth image and amplitude image of the target area. In addition, compared with the color RGB image, only human features can be detected in the depth image and amplitude image, which can protect the privacy of the user. Therefore, the above-mentioned crowd counting method can also be applied to areas with high user privacy requirements where the acquisition of RGB images is prohibited. Since the depth image carries the depth distance information corresponding to each pixel point, by detecting the extreme points of the depth distance in the depth image based on the sliding window, each candidate head area in the depth image can be obtained without target segmentation. By further classifying the corresponding candidate head areas in the amplitude image, the number of head areas in the target area can be determined, thereby quickly and accurately obtaining the crowd counting results, thereby improving the calculation accuracy and efficiency of the crowd counting.
[0052] In order to accurately determine each candidate head region in the depth image, this embodiment provides an implementation method for detecting extreme points of depth distance in the depth image based on a sliding window to determine each candidate head region in the depth image. Specifically, the following steps (1) to (4) may be referred to for execution:
[0053] Step (1): Obtain the depth distance corresponding to each pixel in the depth image.
[0054] The above-mentioned depth image carries the depth distance corresponding to each pixel point, and the depth distance is the vertical distance from the image sensor to each pixel point on the surface of the photographed person.
[0055] Step (2): Determine the window size of the sliding window based on the intrinsic parameters of the image sensor.
[0056] When designing the window size of the sliding window, it is necessary to consider parameters such as the installation height, field of view, resolution, and average height of the person of the image sensor. In order to determine the appropriate window size, first obtain the preset head physical size, which includes the preset head physical length and the preset head physical width. The above-mentioned preset head physical size can be the average head physical size obtained from multiple measurement experiments, such as the preset head physical size can be 15cm*15cm. The unit of the above-mentioned head physical size is meter or centimeter. For the convenience of calculation, the preset head physical size can be converted into pixel size according to the internal parameters of the image sensor, that is, the unit of the head size is converted from meter or centimeter to pixel.
[0057] Based on the internal parameters of the image sensor and the third calculation formula, the number of pixels corresponding to the physical length of the head is determined. The third calculation formula is:
[0058]
[0059] Based on the internal parameters of the image sensor and the fourth calculation formula, the number of pixels corresponding to the physical width of the head is determined. The fourth calculation formula is:
[0060]
[0061] Among them, W pixel is the number of pixels corresponding to the physical length of the head, L pixel is the number of pixels corresponding to the physical width of the head, Head w is the physical length of the head, Head l is the physical width of the head, I cols is the number of pixel columns of the depth image, I rows is the number of pixel rows of the depth image, α is the horizontal field of view of the image sensor, β is the vertical field of view of the image sensor, H install is the vertical distance between the image sensor and the plane where the person is located (that is, the vertical distance between the image sensor and the ground), round is a rounding operation. Since the number of pixels is usually an integer, round is used to round the number of pixels to an integer. H avg is the average height of pedestrians, which can be obtained based on the average height of multiple pedestrians in the target area. According to the triangle theorem, the preset head pixel size corresponding to the preset head physical size can be calculated. The preset head pixel size includes the number of pixels corresponding to the head physical length and the number of pixels corresponding to the head physical width, that is, the pixel length of the preset head pixel size is W. pixel , pixel width is L pixel .
[0062] The window size of the sliding window is determined based on the number of pixels corresponding to the physical length of the head and the number of pixels corresponding to the physical width of the head. The above sliding window is a rectangular frame with a certain size. In order to obtain the extreme value area in the depth image based on the sliding window, the window size of the sliding window can be determined based on the above preset head pixel size. For example, the window size of the sliding window can be 1 / 2 to 1 / 3 of the above preset head pixel size, and the preset sliding step of the sliding window can be 1 / 2 of the above window size. The upper left corner of the sliding window rectangular frame is overlapped with the upper left corner of the depth image to obtain the first sliding window, and the sliding window is moved to the right by the preset sliding steps in sequence. When the sliding window moves to the rightmost end of the depth image, the sliding window is moved downward by the preset sliding step, and so on, until each pixel point in the depth image is traversed using the sliding window.
[0063] Step (3): Locate each extreme value region in the depth image according to the window size of the sliding window and the depth distance corresponding to each pixel point.
[0064] Considering that pedestrians walking in the target area will cause body contact, the pedestrian segmentation task is more difficult, and the head regions are independent of each other. Therefore, head region detection becomes the basis of the pedestrian flow statistics task. Since the head region is an extreme value in the imaging area, based on this assumption, searching for the extreme point position can achieve head positioning.
[0065] The above extreme value region is the minimum value region of the depth distance corresponding to each pixel in the depth image. The minimum value region in the depth image is determined according to the depth distance of the pixel points in each sliding window, which can be specifically performed by referring to the following steps 1) to 3):
[0066] Step 1): Based on the depth distance corresponding to each pixel point, the window size of the sliding window and the preset sliding step, the average depth distance in each sliding window is determined to obtain a sliding window mean matrix.
[0067] Based on the window size of the sliding window and the preset sliding step, the position of each sliding window in the depth image can be used to calculate the average depth distance corresponding to each pixel point in the window when the sliding window is at each position, and the average depth distance in each sliding window is obtained. According to the position of each sliding window and the corresponding average depth distance, the sliding window mean matrix corresponding to the depth image is obtained, that is, each element in the sliding window mean matrix is the average depth distance in each sliding window.
[0068] Step 2): Based on the window size of the sliding window and the preset sliding step, the coordinates of the center points of each sliding window in the depth image are determined to obtain a coordinate index matrix corresponding to the sliding window mean matrix.
[0069] According to the window size of the sliding window and the preset sliding step, the center point coordinates of each sliding window in the depth image can be obtained. For example, when the number of pixels in each row of the sliding window is w, the number of pixels in each column is l, and the preset sliding step is x, the center point coordinates of the first sliding window in the first row can be obtained as (w / 2, l / 2), the center point coordinates of the second sliding window in the first row are ((w / 2)+x, l / 2), the center point coordinates of the nth sliding window in the first row are ((w / 2)+x*n, l / 2), and the center point coordinates of the nth sliding window in the mth row are ((w / 2)+x*n, (l / 2)+x*n).
[0070] According to the coordinates of the center points of each sliding window in the depth image, the coordinate index matrix corresponding to the sliding window mean matrix is obtained. Each element in the sliding window mean matrix corresponds to each element in the coordinate index matrix one by one. For example, the coordinates of the center point of the sliding window where the first average depth distance in the sliding window mean matrix is located are the coordinate values of the first element in the coordinate index matrix. Similarly, the coordinates of the center point of the sliding window where the average depth matrix in the i-th row and j-th column of the sliding window mean matrix is located are the coordinates of the i-th row and j-th column element in the coordinate index matrix.
[0071] Since directly searching for extreme points of depth distance in the depth image is easily affected by noise, a sliding window is used to obtain the local area mean instead of a single extreme point, and there is overlap between the sliding windows, so as to obtain the sliding window mean matrix and the coordinate index matrix. According to the sliding window mean matrix and the coordinate index matrix, each pixel point in the depth image can be traversed, thereby improving the accuracy of extreme area positioning.
[0072] Step 3): Determine the coordinate size of each extreme value region in the depth image based on the sliding window mean matrix and the coordinate index matrix.
[0073] The above coordinate size includes the center point coordinates and pixel size of the extreme value area (the pixel size is the number of pixel columns and pixel rows of the extreme value area). Since the image sensor is set directly above the target area, the head of the person is closest to the image sensor, and the depth distance corresponding to the pixel of the head image in the depth image is the smallest. Therefore, the minimum depth distances in the sliding window mean matrix are first determined, and the minimum depth distances are less than the adjacent average depth distance values. Each average depth distance in the sliding window mean matrix is compared with its 8 adjacent elements (the edge element is compared with its 3 adjacent elements). When the average depth distance of a certain position is less than or equal to the adjacent 8 elements, the average depth distance of the position is determined to be the minimum depth distance. Considering the influence of the preset sliding step of the sliding window, the extreme value area may not be accurately located. According to the coordinate index and the sliding window, the neighborhood of each average depth distance in the sliding window mean matrix is searched locally, which further improves the accuracy of the extreme value area position determination.
[0074] The coordinates of the center point corresponding to the sliding window where each minimum depth distance is located are obtained from the coordinate index matrix to obtain the coordinates of the center point of each extreme value region. For example, when the first minimum depth distance is the i-th row and j-th column element in the sliding window mean matrix, the i-th row and j-th column element of the coordinate index matrix is the coordinates of the center point of the sliding window where the first minimum depth distance is located, and is also the coordinates of the center point of the first extreme value region.
[0075] Taking into account the different sizes of head areas when imaging people of different heights, the pixel size (including pixel length and pixel width) of each extreme value area is calculated according to the preset head physical size, image sensor installation height, image sensor field of view and each minimum depth distance. Since each extreme value area may be a head area, the pixel size corresponding to each extreme value area when it is a head image can be accurately calculated according to the average depth distance of each extreme value area.
[0076] Based on each minimum depth distance, the internal parameters of the image sensor and the first calculation formula, the pixel length of each extreme value area is obtained, wherein the pixel length is the number of pixel columns in the extreme value area (that is, the number of pixels in each row of the extreme value area). The above first calculation formula is:
[0077]
[0078] Based on each minimum depth distance, the intrinsic parameter of the image sensor and the second calculation formula, the pixel width of each extreme value area is obtained, wherein the pixel width is the number of pixel rows in the extreme value area (that is, the number of pixels in each column of the extreme value area). The above second calculation formula is:
[0079]
[0080] Among them, W i is the pixel length of the i-th extreme value area, L i is the pixel width of the ith extreme value region, Head w Head is the preset physical length of the head. l is the preset head physical width, I cols is the number of pixel columns of the depth image, I rows is the number of pixel rows of the depth image, α is the horizontal field of view of the image sensor, β is the vertical field of view of the image sensor, H i is the i-th minimum depth distance, and round is the rounding operation.
[0081] Step (4): Calculate the area intersection-to-union ratio between each extreme value region, and determine the candidate head region based on the area intersection-to-union ratio.
[0082] In order to avoid searching for duplicate areas in extreme value areas, the area intersection and union ratio of each extreme value area is calculated according to the pixel size of each extreme value area. The calculation formula of the area intersection and union ratio of each extreme value area is:
[0083]
[0084] Among them, IOU is the intersection-over-union ratio, I s is the area intersection of two extreme value regions, U s is the area union of the two extreme value regions.
[0085] When the IoU ratio of two extreme value regions is greater than the preset threshold, the two extreme value regions are considered to be repeatedly selected, and the region with the smaller average depth distance between the two extreme value regions is taken as the candidate head region, and the extreme value region with the larger average depth distance is deleted; when the IoU ratio of two extreme value regions is less than the preset threshold, the two extreme value regions are respectively taken as candidate head regions.
[0086] Since the above-mentioned candidate head regions are suspected head regions, in order to further accurately determine the head region in the depth image and then count the flow of people, this embodiment provides an implementation method of obtaining each image to be detected in the amplitude image based on each candidate head region, performing target classification on the image to be detected, and obtaining the statistical result of the flow of people in the target area. Specifically, refer to the following steps 1 to 3:
[0087] Step 1: Based on the coordinate size of each candidate head region, the image within each coordinate size region is obtained from the amplitude image to obtain each image to be detected.
[0088] The coordinate size of each candidate head region in the depth image obtained above is obtained, and the corresponding rectangular image and image are intercepted from the amplitude image according to the size of each candidate head region as the image to be detected. For example, when the center point coordinates of the first candidate head region are (a, b), the pixel length is w1, and the pixel width is l1, a rectangular frame with the center point coordinates of (a, b), the pixel length is w1, and the pixel width is l1 is drawn from the amplitude image, and the image within the rectangular frame is intercepted to obtain the first image to be detected. Correspondingly, the image to be detected corresponding to each candidate head region is intercepted from the amplitude image.
[0089] Step 2: Input each image to be detected into the pre-trained neural network model to obtain the classification results of each target to be detected.
[0090] The neural network model is trained based on pre-labeled human amplitude image samples. The amplitude image is a grayscale image collected by the image sensor under laser illumination. It is not affected by changes in ambient light and can obtain valid data in darkness and strong light. By replacing the RGB image with the amplitude image as the training and detection data of the neural network model, the head area in the image to be detected can be accurately identified, thereby improving the accuracy of target classification.
[0091] In a specific embodiment, the above-mentioned neural network model can be a convolutional neural network. Since the above-mentioned target classification only needs to perform a binary classification operation on the image to be detected (detecting whether the image to be detected is a head area or a non-head area), no deep model is required. However, the shallow model is a weak classifier and is prone to false detection. In order to ensure the recognition accuracy and real-time operation of the classifier, the above-mentioned neural network model can adopt a shallow convolutional neural network based on the Boosting idea. The Boosting idea is equivalent to connecting each weak classifier in parallel, and the input image runs independently in each weak classifier to obtain a prediction result, and the final result is obtained by weighted addition according to the error rate of each weak classifier. The above-mentioned neural network model can use three shallow convolutional neural networks with the same structure for cascading. For example, when the pixel size of the input image is 48×48×1, the operating parameters of each layer of the convolutional neural network are shown in Table 1 below:
[0092] Table 1 Convolutional neural network operating parameters of each network layer
[0093] Network layer type Network layer structure Sliding step length Activation Function Output size Convolutional Layer 3×3×8 1 Relu6 46×46×8 Pooling Layer 2×2 2 Relu6 23×23×8 Convolutional Layer 3×3×16 2 Relu6 11×11×16 Pooling Layer 2×2 2 Relu6 5×5×16 Fully connected layer 400×128 -- Tanh 1×128 Fully connected layer 128×2 -- Tanh 1×2
[0094] Taking into account that the extreme points of the head are affected by the body posture and are prone to missed detection, it is known through observation of the data that when the head is missed, an extreme point area appears near its neighborhood. Therefore, when the recognition result of the image to be detected corresponding to the above-mentioned candidate head area is a non-head area, the extreme point area in the neighborhood of the candidate head area can be further identified to determine whether it is a head area.
[0095] Step 3: Determine the human flow statistics results in the target area based on the classification results of each target to be detected.
[0096] In one embodiment, the above-mentioned human flow statistics result includes the current number of people in the target area. The above-mentioned image sensor can collect a panoramic image of the target area, determine the number of people in the target area based on the classification results of each target to be detected, and use the number of people as the human flow statistics result in the target area. The number of head areas in the target area identified is determined according to the classification results of each target to be detected, and the number of head areas is the number of people in the target area, thereby obtaining the human flow statistics result in the target area.
[0097] In another embodiment, the above-mentioned human flow statistics result includes the number of people entering and leaving the target area, and a warning line is set in each depth image collected by the above-mentioned image sensor; when determining the human flow statistics result in the target area based on the classification result of each target to be detected, the following steps a to c can be specifically referred to:
[0098] Step a: Determine the head area of each person in the depth image based on the classification results of each target to be detected.
[0099] When the target classification result of the target to be detected is the head area, the center point coordinates and pixel size of the target to be detected in the amplitude image are obtained, and the head area of each person is framed in the depth image according to the center point coordinates and pixel size of the target to be detected in the amplitude image.
[0100] Step b: Based on the head regions of the characters in the continuous multi-frame depth images, determine the movement trajectory of the head regions of the characters in the target area.
[0101] When counting the number of people entering and leaving the target area, it is necessary to set up image sensors at each entrance and exit of the target area, use the image sensors to collect depth images and amplitude images in real time, determine the head area of each person in each depth image, and calculate the movement trajectory of each head area based on the coordinates of the center point of each person's head area in each of two consecutive depth images. When determining the movement trajectory of the head area of a person, it is necessary to match whether the head area of the person between frames belongs to the same person. It is assumed that the person moves at an approximately uniform speed during walking, that is, the person whose head area center point coordinates are closest between two frames belongs to the same person. The distance calculation formula is as follows:
[0102]
[0103] Where dis is the distance between the center points of the head area of the character, x 1,i is the row coordinate of the center point of the head area of the i-th person in the previous depth image, y 1,i is the center point column coordinate of the head area of the i-th person in the previous frame depth image, x 2,i is the row coordinate of the center point of the head area of the i-th person in the depth image of the next frame, y 2,i The coordinates of the center point of the head area of the i-th person in the next frame of depth image are listed. According to the coordinates of the center point of the head area of the same person in multiple consecutive frames of depth images, the movement trajectory of the head area of the person can be obtained.
[0104] Step c: determining the amount of people entering and leaving the target area based on the movement trajectory of the head area of each person in the target area and the position of the warning line.
[0105] Head area collision detection is the basis for realizing two-way statistics of pedestrian flow. By setting multiple warning lines, when a person's head area passes through the warning lines, the direction of pedestrian movement can be determined as entering or leaving based on the order in which the person's head area passes through the warning lines.
[0106] In a specific implementation, the depth image may be provided with a first warning line, a second warning line, and a third warning line. When the moving trajectory of the head area of the person intersects with the first warning line, the second warning line, and the third warning line in sequence, the count value of the amount of person entering is increased by one. Among them, the first warning line is a warning line far away from the center of the target area, the third warning line is a warning line close to the center of the target area, the second warning line is located between the first warning line and the third warning line, and the first warning line, the second warning line, and the third warning line are parallel to each other. According to the moving trajectory of the head area of the person and the coordinates of each warning line, the order in which the head area of the person passes through each warning line can be obtained. Considering that the head area may be inaccurately positioned, setting only a single warning line may result in multiple triggering of the head, and three warning lines are set according to the pixel size of the head area, and the distance between the warning lines is greater than the pixel width of the head area to avoid miscounting. When the moving trajectory of the head area of the person intersects only with the first warning line, it indicates that the target area has not been entered.
[0107] When the moving trajectory of the character's head area intersects with the third warning line, the second warning line and the first warning line in sequence, the count value of the character's departure amount is increased by 1. Since the first warning line is a warning line far from the center of the target area, and the third warning line is a warning line close to the center of the target area, when the task head area first passes through the third warning line, and then passes through the second warning line and the first warning line, it indicates that the character is moving away from the target area, that is, the target character is about to leave the target area, and the count value of the character's departure amount is increased by 1. See FIG. Figure 4 The warning line diagram shown is Figure 4 is an image of one of the entrances and exits of the target area. Figure 4 The three warning lines in the picture are the third warning line, the second warning line and the first warning line from top to bottom. Figure 4 The moving trajectory of the head area of the person in the target area intersects with the third warning line and the second warning line in turn, and it is determined that the person is about to leave the target area, and the count value of the person's departure amount is increased by one.
[0108] Based on the count value of the number of people entering and the count value of the number of people leaving within the preset time, the number of people entering and the number of people leaving the target area are obtained. The preset time may be the opening time of the target area, and the human flow statistics result of the target area is obtained by the count value of the number of people entering and the count value of the number of people leaving the target area during the opening time.
[0109] In practical applications, image sensors can be set inside the target area and at the entrances and exits of the target area respectively. Based on the depth images and amplitude images collected by each image sensor, the number of people inside the target area, the number of people entering the target area, and the number of people leaving the target area can be comprehensively counted.
[0110] In a specific implementation, an alarm line is also provided in the above-mentioned depth image. This embodiment also provides an implementation for performing a safety alarm according to the moving trajectory of the above-mentioned head area of the person: when the moving trajectory of the head area of the person intersects with the alarm line, the preset alarm device is triggered to alarm. When a pedestrian passes the alarm line, the controller controls the preset alarm device to alarm to warn the pedestrian to leave or issue a safety warning. The above-mentioned alarm line can be an alarm line set according to the actual needs of the target area. The area in the target area where pedestrians are prohibited from entering is set as the alarm area (the alarm area can be of any shape), and the edge contour of the alarm area is provided with an alarm line. According to the alarm area, a functional relationship of the alarm line can be obtained. According to the moving trajectory of the head area of the pedestrian and the functional relationship of the alarm line, it can be determined whether the curve of the moving trajectory and the functional relationship has an intersection. When the head area of the pedestrian contacts the alarm line, that is, the moving trajectory and the alarm line intersect, it indicates that the pedestrian is about to enter the alarm area and trigger an alarm. The above-mentioned preset alarm device can alarm by sound, light or text reminder, such as by warning sound, broadcasting "do not enter" or flashing red light and other alarm methods to remind pedestrians.
[0111] The above-mentioned crowd counting method provided in this embodiment avoids the influence of ambient light on imaging by collecting depth images and amplitude images, realizes rapid detection and identification of the head area, thereby realizing real-time statistics of the crowd flow in the target area, and can also accurately count the number of people entering and leaving the target area according to the movement trajectory of the head area, thereby improving the accuracy of crowd counting.
[0112] Corresponding to the human flow counting method provided in the above embodiment, the embodiment of the present invention provides a human flow counting device, see Figure 5 The structure diagram of a human flow counting device shown in FIG. 1 includes the following modules:
[0113] The image acquisition module 51 is used to acquire a depth image and an amplitude image of the target area based on an image sensor; wherein the image sensor is arranged directly above the target area.
[0114] The region determination module 52 is used to perform extreme point detection on the depth image based on a sliding window to determine each candidate head region in the depth image.
[0115] The traffic statistics module 53 is used to obtain each image to be detected in the amplitude image based on each candidate head area, perform target classification on the image to be detected, and obtain the human traffic statistics result of the target area; wherein the image to be detected includes the candidate head area in the amplitude image.
[0116] The above-mentioned people flow counting device provided by the present embodiment avoids the influence of overexposure and too dark light on imaging by acquiring the depth image and amplitude image of the target area. Since the depth image carries the depth distance information corresponding to each pixel point, each candidate head area in the depth image can be obtained by detecting the extreme points of the depth distance in the depth image based on the sliding window, without the need for target segmentation. By further classifying the corresponding candidate head areas in the amplitude image, the number of head areas in the target area can be determined, thereby quickly and accurately obtaining the people flow counting result, thereby improving the calculation accuracy and efficiency of the people flow counting.
[0117] In one embodiment, the above-mentioned area determination module 52 is further used to obtain the depth distance corresponding to each pixel point in the depth image; determine the window size of the sliding window based on the internal parameters of the image sensor; locate each extreme value area in the depth image according to the window size of the sliding window and the depth distance corresponding to each pixel point; wherein the extreme value area is the minimum value area of the depth distance corresponding to each pixel point in the depth image; calculate the area intersection and union ratio between each extreme value area, and determine the candidate head area according to the area intersection and union ratio.
[0118] In one embodiment, the above-mentioned area determination module 52 is further used to determine the average depth distance in each sliding window based on the depth distance corresponding to each pixel point, the window size of the sliding window and the preset sliding step, and obtain the sliding window mean matrix; based on the window size of the sliding window and the preset sliding step, determine the center point coordinates of each sliding window in the depth image, and obtain the coordinate index matrix corresponding to the sliding window mean matrix; based on the sliding window mean matrix and the coordinate index matrix, determine the coordinate size of each extreme value area in the depth image; wherein the coordinate size includes the center point coordinates and pixel size of the extreme value area.
[0119] In one embodiment, the region determination module 52 is further used to determine each minimum depth distance in the sliding window mean matrix; wherein the minimum depth distance is less than each adjacent average depth distance value; obtain the center point coordinates corresponding to the sliding window where each minimum depth distance is located from the coordinate index matrix to obtain the center point coordinates of each extreme value region; based on each minimum depth distance, the internal parameters of the image sensor and the first calculation formula, obtain the pixel length of each extreme value region; wherein the pixel length is the number of pixel columns of the extreme value region, and the first calculation formula is:
[0120]
[0121] Based on each minimum depth distance, the internal parameters of the image sensor and the second calculation formula, the pixel width of each extreme value area is obtained; wherein the pixel width is the number of pixel rows in the extreme value area, and the second calculation formula is:
[0122]
[0123] Among them, W i is the pixel length of the i-th extreme value area, L i is the pixel width of the ith extreme value region, Head w Head is the preset physical length of the head. l is the preset head physical width, I cols is the number of pixel columns of the depth image, I rows is the number of pixel rows of the depth image, α is the horizontal field of view of the image sensor, β is the vertical field of view of the image sensor, H i is the i-th minimum depth distance, and round is the rounding operation.
[0124] In one embodiment, the region determination module 52 is further used to obtain a preset physical size of the head; wherein the physical size of the head includes a physical length of the head and a physical width of the head; based on the intrinsic parameters of the image sensor and a third calculation formula, the number of pixels corresponding to the physical length of the head is determined; wherein the third calculation formula is:
[0125]
[0126] Based on the internal parameters of the image sensor and the fourth calculation formula, the number of pixels corresponding to the physical width of the head is determined; wherein the fourth calculation formula is:
[0127]
[0128] Among them, W pixel is the number of pixels corresponding to the physical length of the head, L pixel is the number of pixels corresponding to the physical width of the head, Head w is the physical length of the head, Head l is the physical width of the head, I cols is the number of pixel columns of the depth image, I rows is the number of pixel rows of the depth image, α is the horizontal field of view of the image sensor, β is the vertical field of view of the image sensor, H install H is the vertical distance between the image sensor and the plane where the person is located, avg is the average height of pedestrians, and round is the rounding operation; the window size of the sliding window is determined based on the number of pixels corresponding to the physical length of the head and the number of pixels corresponding to the physical width of the head.
[0129] In one embodiment, the above-mentioned traffic statistics module 53 is further used to obtain images within each coordinate size area from the amplitude image based on the coordinate size of each head candidate area to obtain each image to be detected; input each image to be detected into a pre-trained neural network model to obtain a classification result of each target to be detected; wherein the neural network model is trained based on pre-labeled human amplitude image samples; and determine the human flow statistics result in the target area based on the classification result of each target to be detected.
[0130] In one embodiment, the traffic statistics module 53 is further used to determine the number of people in the target area based on the classification results of each target to be detected; and use the number of people as the result of the human traffic statistics in the target area.
[0131] In one embodiment, the above-mentioned human flow statistics include the number of people entering and leaving; a warning line is set in the depth image; the above-mentioned traffic statistics module 53 is further used to determine the head area of each person in the depth image based on the classification result of each target to be detected; based on the head area of each person in the continuous multi-frame depth image, determine the moving trajectory of each head area of the person in the target area; based on the moving trajectory of each head area of the person in the target area and the position of the warning line, determine the number of people entering and leaving the target area.
[0132] In one embodiment, a first warning line, a second warning line and a third warning line are provided in the above-mentioned depth image; the above-mentioned traffic statistics module 53 is further used to increase the count value of the amount of people entering when the moving trajectory of the person's head area intersects with the first warning line, the second warning line and the third warning line in sequence; when the moving trajectory of the person's head area intersects with the third warning line, the second warning line and the first warning line in sequence, the count value of the amount of people leaving is increased by one; based on the count value of the amount of people entering and the count value of the amount of people leaving within a preset time, the amount of people entering and leaving the target area is obtained.
[0133] In one implementation, an alarm line is further provided in the depth image, and the device further comprises:
[0134] The alarm module is used to trigger a preset alarm device to sound an alarm when the moving trajectory of the person's head area intersects with the alarm line.
[0135] The above-mentioned crowd counting device provided in this embodiment avoids the influence of ambient light on imaging by collecting depth images and amplitude images, realizes rapid detection and identification of the head area, thereby realizing real-time statistics of the crowd flow in the target area, and can also accurately count the number of people entering and leaving the target area according to the movement trajectory of the head area, thereby improving the accuracy of crowd flow statistics.
[0136] The implementation principle and technical effects of the device provided in this embodiment are the same as those of the aforementioned embodiments. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.
[0137] Corresponding to the method and device provided in the above-mentioned embodiment, an embodiment of the present invention further provides a people flow counting system, which includes: an image sensor and a controller, the controller includes a processor and a storage device, and the above-mentioned image sensor is arranged directly above the target area.
[0138] The controller includes a processor and a storage device, such as Figure 6 As shown in the schematic diagram of the controller structure, the controller includes a processor 61 and a memory 62. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, the steps of the method provided in the above embodiment are implemented.
[0139] See also Figure 6 The controller further includes: a bus 64 and a communication interface 63, and the processor 61, the communication interface 63 and the memory 62 are connected via the bus 64. The processor 61 is used to execute executable modules stored in the memory 62, such as computer programs.
[0140] The memory 62 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.
[0141] The bus 64 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0142] Among them, the memory 62 is used to store programs, and the processor 61 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 61, or implemented by the processor 61.
[0143] The processor 61 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 61 or the instructions in the form of software. The above processor 61 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 62, and the processor 61 reads the information in the memory 62 and completes the steps of the above method in combination with its hardware.
[0144] An embodiment of the present invention provides a computer-readable medium, wherein the computer-readable medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method described in the above embodiment.
[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned embodiment, and will not be repeated here.
[0146] The computer program product of the pedestrian flow counting method, device and system provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.
[0147] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0148] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0149] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.
[0150] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A method for counting human traffic, characterized in that: include: S102, acquiring a depth image and an amplitude image of a target area based on an image sensor; wherein the image sensor is disposed directly above the target area; S104, performing extreme point detection on the depth distance in the depth image based on a sliding window to determine each candidate head region in the depth image; S106, acquiring each image to be detected in the amplitude image based on each candidate head region, performing target classification on the image to be detected, and obtaining a human flow statistics result of the target area; wherein the image to be detected is the corresponding candidate head region in the amplitude image; Step S104 includes: S1042, obtaining a depth distance corresponding to each pixel point in the depth image; S1044, determining a window size of the sliding window based on an internal parameter of the image sensor; S1046, locating each extreme value region in the depth image according to the window size of the sliding window and the depth distance corresponding to each pixel point; wherein the extreme value region is a minimum value region of the depth distance corresponding to each pixel point in the depth image; S1048, calculating the area intersection-and-union ratios between the extreme value regions, and determining each candidate head region according to the area intersection-and-union ratios; Step S1044 includes: Obtaining preset physical dimensions of the head; wherein the physical dimensions of the head include a physical length and a physical width of the head; Based on the internal parameters of the image sensor and the third calculation formula, the number of pixels corresponding to the physical length of the head is determined; wherein the third calculation formula is: Based on the internal parameters of the image sensor and a fourth calculation formula, the number of pixels corresponding to the physical width of the head is determined; wherein the fourth calculation formula is: Among them, W pixel is the number of pixels corresponding to the physical length of the head, L pixel is the number of pixels corresponding to the physical width of the head, Head w is the physical length of the head, Head l is the physical width of the head, I cols is the number of pixel columns of the depth image, I rows is the number of pixel rows of the depth image, α is the horizontal field angle of the image sensor, β is the vertical field angle of the image sensor, H install is the vertical distance between the image sensor and the plane where the person is located, H avg is the average height of pedestrians, and round is the integer operation; Determining a window size of the sliding window based on the number of pixels corresponding to the physical length of the head and the number of pixels corresponding to the physical width of the head; Step S1046 includes: S10462, based on the depth distance corresponding to each pixel point, the window size of the sliding window and the preset sliding step, determine the average depth distance in each sliding window to obtain a sliding window mean matrix; S10464, based on the window size of the sliding window and the preset sliding step, determine the coordinates of the center points of each of the sliding windows in the depth image, and obtain a coordinate index matrix corresponding to the sliding window mean matrix; S10466, determining the coordinate size of each extreme value region in the depth image based on the sliding window mean matrix and the coordinate index matrix; wherein the coordinate size includes the center point coordinates and pixel size of the extreme value region; Step S10466 includes: Determine each minimum depth distance in the sliding window mean matrix; wherein the minimum depth distance is smaller than each adjacent average depth distance value; Obtaining the center point coordinates corresponding to the sliding windows where each of the minimum depth distances is located from the coordinate index matrix, and obtaining the center point coordinates of each of the extreme value regions; Based on each of the minimum depth distances, the internal parameters of the image sensor and the first calculation formula, the pixel length of each of the extreme value areas is obtained; wherein the pixel length is the number of pixel columns of the extreme value area, and the first calculation formula is: Based on each of the minimum depth distances, the internal parameters of the image sensor and the second calculation formula, the pixel width of each of the extreme value regions is obtained; wherein the pixel width is the number of pixel rows in the extreme value region, and the second calculation formula is: Among them, W i is the pixel length of the i-th extreme value area, L i is the pixel width of the ith extreme value region, Head w Head is the preset physical length of the head. l is the preset head physical width, I cols is the number of pixel columns of the depth image, I rows is the number of pixel rows of the depth image, α is the horizontal field angle of the image sensor, β is the vertical field angle of the image sensor, H i is the i-th minimum depth distance, and round is the rounding operation; Step S106 includes: Based on the coordinate size of each candidate head region, images within each coordinate size region are acquired from the amplitude image to obtain each image to be detected; Input each to-be-detected image into a pre-trained neural network model to obtain a classification result of each to-be-detected target; wherein the neural network model is trained based on pre-labeled human amplitude image samples; The human flow statistics result in the target area is determined based on the classification result of each of the targets to be detected.
2. The method according to claim 1, characterized in that The step of determining the result of the human flow statistics in the target area based on the classification result of each of the targets to be detected includes: Determining the number of people in the target area based on the classification results of each of the targets to be detected; The number of people is used as the result of the human flow statistics in the target area.
3. The method according to claim 1, characterized in that: The human flow statistics result includes the number of people entering and leaving; a warning line is set in the depth image; The step of determining the result of the human flow statistics in the target area based on the classification result of each of the targets to be detected includes: Determine each person's head area in the depth image based on the classification results of each of the targets to be detected; Based on each of the head regions of the person in the continuous multiple-frame depth image, determining a moving trajectory of each of the head regions of the person in the target area; The amount of people entering and leaving the target area is determined based on the movement trajectory of each of the head areas of the people in the target area and the position of the warning line.
4. The method according to claim 3, characterized in that: The depth image is provided with a first warning line, a second warning line and a third warning line; The step of determining the amount of people entering and leaving the target area based on the movement trajectory of each of the head areas of the people in the target area and the position of the warning line includes: When the moving track of the head area of the person intersects with the first warning line, the second warning line and the third warning line in sequence, the count value of the person's entry amount is increased by one; When the moving track of the head area of the person intersects with the third warning line, the second warning line and the first warning line in sequence, the count value of the person's departure amount is increased by one; Based on the count value of the number of people entering and the count value of the number of people leaving within a preset time, the number of people entering and the number of people leaving the target area are obtained.
5. The method according to claim 3, characterized in that: An alarm line is also provided in the depth image, and the method further comprises: When the moving track of the head area of the person intersects with the alarm line, a preset alarm device is triggered to sound an alarm.
6. A people flow counting system, characterized in that: include: An image sensor and a controller, the controller including a processor and a storage device; The storage device stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 5 is executed.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are executed.
