Pedestrian counting method, device, equipment, and computer-readable storage medium

Abstract

The present invention belongs to the field of image processing technology, and particularly relates to a pedestrian counting method, device, equipment, and computer-readable storage medium, wherein the method is based on a depth image, and the method comprises the following steps: obtaining a current image, determining a connected domain of the current image; establishing an object list of the current image according to the connected domain; performing behavior discrimination and appearance discrimination on the objects in the object list respectively; when the behavior discrimination result satisfies a first preset value, and the appearance discrimination result satisfies a second preset value, the object is judged to be a person. The method can improve the accuracy of pedestrian detection by simultaneously performing behavior discrimination and appearance discrimination on the objects in the object list of the current image, and using the discrimination results of the two as the criteria for judgment.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a pedestrian counting method, device, equipment, and computer-readable storage medium. Background Art

[0002] At present, the passenger flow counting methods based on vision are generally divided into two categories, one is the passenger flow counting method based on RGB images, and the other is the passenger flow counting method based on depth images. The passenger flow counting method based on depth images usually performs convolution based on the depth image, calculates the HOG (Histogram of Oriented Gradient) value of the fixed ROI (Region Of Interest), and makes a right or wrong judgment through a machine learning model. However, there are several problems with this type of recognition method: 1. The depth image has less information and it is difficult to distinguish some human-like objects; 2. The convolution calculation consumes a lot of computing resources; 3. The HOG value of the fixed ROI has a low recognition rate for people at different distances.

[0003] Based on this, this invention is proposed. Summary of the invention

[0004] In view of the above technical problems, the present invention provides a pedestrian counting method, which performs behavior discrimination and appearance discrimination on the depth image respectively, and uses the discrimination results of the two as the judgment criteria at the same time, which can improve the accuracy of pedestrian detection. The specific scheme is as follows:

[0005] In a first aspect, a pedestrian counting method is provided, characterized in that the method is based on a depth image, and the method comprises the following steps:

[0006] Get the current image and determine the connected domain of the current image;

[0007] According to the connected domain, establish the object list of the current image;

[0008] Perform behavior and appearance discrimination on the objects in the object list;

[0009] When the behavior identification result satisfies the first preset value and the appearance identification result satisfies the second preset value, the object is determined to be a person.

[0010] Furthermore, the obtaining of the current image and determining the connected domain of the current image includes:

[0011] Read the depth image in the memory, determine the background image, obtain the foreground image of the current image based on the background image, and determine the connected domain based on the foreground image.

[0012] Further, determining the connected domain according to the foreground image includes:

[0013] Determine whether each pixel in the foreground image is a valid point in a preset order, and determine the connected domain according to the valid points, which specifically includes the following steps:

[0014] Determine the first effective point;

[0015] Traversing each pixel point on the edge of the first effective point, determining the pixel points that meet the first preset condition as the in-domain points of the first effective point, and marking the in-domain points of the first effective point, the first effective point and its in-domain points constitute a connected domain;

[0016] Traversing each pixel point on the edge of the intra-domain point of the first effective point, determining the pixel points that meet the first preset condition as the intra-domain points of the first effective point, and marking the intra-domain points of the first effective point, the first effective point and its intra-domain points constitute a connected domain;

[0017] Determine a new first valid point according to the preset order and repeat the above steps.

[0018] Furthermore, establishing an object list of the current image according to the connected domain includes:

[0019] Determine one by one whether the largest three-dimensional circumscribed cuboid of the connected domain meets the second preset condition, and if so, add it to the object list of the current image.

[0020] Furthermore, the objects in the current image object list are compared one by one with all the objects in the previous frame image object list. If the distance between the centroids of two objects is less than a third preset value, the two objects are marked as the same object.

[0021] Furthermore, the behavior identification includes:

[0022] The maximum displacement between the first frame image and the last frame image of the same object is calculated, and the behavior discrimination result is obtained based on the displacement.

[0023] Furthermore, the shape discrimination includes calculating the HOG values ​​of the objects in the object list of the current image one by one, and inputting them into a trained two-dimensional vector machine model, and the result output by the two-dimensional vector machine is the shape discrimination result.

[0024] In a second aspect, the present invention provides a device, characterized in that it includes a processor and a memory; wherein the processor implements the above-mentioned method when executing a computer program stored in the memory.

[0025] In a third aspect, the present invention provides a device, characterized in that it comprises the device described above

[0026] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the method described above

[0027] The present invention can improve the accuracy of pedestrian detection by simultaneously performing behavior discrimination and appearance discrimination on the objects in the current image object list and using the discrimination results of the two as the judgment standard. The method is not only aimed at the features of the current frame image, but also links the features of the previous and next frame images, and comprehensively utilizes the image features and the connections between them to improve the accuracy of pedestrian detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the method of the present invention;

[0029] Figure 2 is a background image in one embodiment;

[0030] Figure 3 is a current image in one embodiment;

[0031] Figure 4 is a foreground image in one embodiment;

[0032] Figure 5 Specific steps for implementing step S1 of the method of the present invention;

[0033] Figure 6 It is a schematic diagram of the module structure of the device of the present invention;

[0034] Figure 7 It is a schematic diagram of the module structure of the device of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0036] The passenger flow counting method based on depth images usually performs convolution on the depth images and calculates the HOG (Histogram of Oriented Gradient) value of a fixed ROI (Region Of Interest), and makes a right or wrong judgment through a machine learning model. However, there are several problems with this recognition method: 1. The depth image has less information and it is difficult to distinguish some human-like objects; 2. The convolution calculation consumes a lot of computing resources; 3. The HOG value of the fixed ROI has a low recognition rate for people at different distances.

[0037] In view of the above technical problems, the present invention provides a pedestrian counting method, which is based on a depth image and includes the following steps: Figure 1 As shown:

[0038] Step S1: Obtain the current image and determine the connected domain of the current image;

[0039] The above step S1 specifically includes the following steps: Figure 5 As shown:

[0040] Step S11: Read the depth image in the memory and determine the background image;

[0041] Step S12: obtaining a foreground image of the current image according to the background image;

[0042] Step S13: Determine the connected domain according to the foreground image.

[0043] Generally, after a depth camera captures an image, it will store it in its memory, so obtaining the current image is to read the image from the memory. Since the camera is fixed in one position when performing pedestrian counting, such as the entrance of some public places (such as shopping malls, amusement parks, etc.), the background part of the image captured by the camera is unchanged, and only the foreground image changes. Therefore, when processing the image, only the foreground image needs to be processed. The foreground image contains the area of ​​interest, and the foreground image data is smaller and cleaner, which is more conducive to data analysis and extraction.

[0044] To obtain the foreground image of the current image, you need to obtain the background image. You can obtain the foreground image by subtracting the current image from the background image. Generally, one of the first few frames when the computer is turned on is selected as the background image. When selecting, as long as the image is not all 0 or 1, it can be determined as the background image. After obtaining the current image from the memory, subtract the current image from the background image to obtain the foreground image of the current image. At this time, determine the connected domain based on the foreground image. Figure 2 As shown, it is the selected background image, and Figure 3 is the current image, Figure 4 Then Figure 3 foreground image.

[0045] Step S13 is specifically as follows: determining whether each pixel point in the foreground image is a valid point in a preset order, and determining a connected domain based on the valid points.

[0046] In this embodiment, the preset order may be from top to bottom column by column, or from left to right row by row.

[0047] When judging whether the pixel is a valid point according to the preset order, such as Figure 4As shown, most of the points in the processed foreground image have a distance value of 0. Therefore, when judging whether a pixel is a valid point, as long as the distance value of the pixel is not 0, the pixel is considered to be a valid point.

[0048] As an optional embodiment, judging whether each pixel point in the foreground image is a valid point in a preset order and determining a connected domain according to the valid points specifically includes the following steps:

[0049] Step S131: determining the first effective point;

[0050] The first valid point is the first pixel point whose distance is not 0 and is scanned in a preset order.

[0051] Step S132: traverse each pixel point on the edge of the first effective point, determine the pixel points that meet the first preset condition as the in-domain points of the first effective point, and mark the in-domain points of the first effective point. The first effective point and its in-domain points form a connected domain;

[0052] In this embodiment, the pixel points on the edge of the first effective point refer to the pixel points in a circle around the first effective point. If the first effective point is located at the edge of the image, the number of pixel points on the edge of the first effective point is 3. If the first effective point is located in the middle of the image, the number of points on the edge of the first effective point is 8. When the first effective point is located at other positions in the image, the number of points on the edge of the first effective point can also be other numbers.

[0053] Step S133: traverse each pixel point on the edge of the intra-domain point of the first effective point, determine the pixel points that meet the first preset condition as the intra-domain points of the first effective point, and mark the intra-domain points of the first effective point. The first effective point and its intra-domain points form a connected domain;

[0054] In this embodiment, some of the pixel points on the edge of the in-domain points of the first valid point have been marked as in-domain points. For the marked points, there is no need to repeatedly judge the mark, and only the unmarked pixel points need to be judged.

[0055] Step S134: determine a new first valid point according to a preset order, and repeat the above steps S132 and S133.

[0056] In this embodiment, S133 is a repetitive step. As long as a certain pixel point is determined to be a point within the domain of the first valid point, S133 needs to be executed until a new point within the domain of the first valid point cannot be determined. At this time, the connected domain of the first valid point is constructed and S134 is started.

[0057] When the first valid point is determined according to the preset order, the pixel points in the image are judged one by one according to the preset order, and the pixel points in the domain that have been determined as the first valid point do not need to be judged again.

[0058] In this embodiment, the first preset condition is that the distance difference between adjacent points is less than a first threshold value. The first threshold value can be specifically set according to different situations or requirements. In this embodiment, the first threshold value can be set to 10 cm.

[0059] In this embodiment, the distance of a pixel point refers to the distance between a point corresponding to the pixel point in a real physical space and a depth camera.

[0060] In this embodiment, the pixel point is not marked means that the pixel point is not a point within the domain of other first valid points.

[0061] Step S2: Establish an object list of the current image based on the connected domain;

[0062] Step S2 specifically includes the following steps:

[0063] After the connected domain of the foreground image is judged, several connected domains will be obtained. Some of these connected domains are depth images of people, and some are not. Therefore, in this step, the obtained connected domains are further screened. The specific screening steps are as follows:

[0064] Determine one by one whether the largest three-dimensional circumscribed cuboid of the connected domain meets the second preset condition, and if so, add it to the object list of the current image.

[0065] Among the several connected domains obtained, if it is a depth image obtained by photographing a person, then the length, width, and height of the person must be satisfied. According to this principle, the largest three-dimensional circumscribed cuboid of each connected domain can be determined first, and then it can be determined whether it satisfies the second preset condition. If so, it will be included in the object list of the current image.

[0066] In this embodiment, the second preset condition is to obtain a set of threshold ranges for the length, width and height of a human body based on statistical results, and compare the length, width and height of the largest three-dimensional circumscribed cuboid of the connected domain with the above threshold ranges. If the length, width and height of the largest three-dimensional circumscribed cuboid all fall within the threshold range, the connected domain is included in the current object list. If one of the length, width or height does not fall within the threshold range, it is removed. For example, when the height of the largest three-dimensional external cuboid corresponding to a connected domain exceeds 2.5m, the connected domain is removed.

[0067] Since the images taken by the depth camera are continuous between the previous and next frames, when a pedestrian passes through the shooting area, the pedestrian will be captured in several consecutive frames. Therefore, when counting pedestrians, determining whether the objects in the previous and next frames are the same object is also a key step.

[0068] In one embodiment, the following judgment is performed: objects in the current image object list are compared one by one with all objects in the previous frame image object list. If the distance between the centroids of two objects is less than a third preset value, the two objects are marked as the same object.

[0069] There are several objects in the current image object list and the previous frame image object list respectively, and the number may be the same or different. At this time, the objects in the current image list are compared with all the objects in the previous frame image object list one by one to determine whether it is one of the objects in the previous frame image object list. Since pedestrians move when passing through the shooting area, when determining whether an object is the same object as the object in the previous frame image, first calculate the distance between a certain object in the current image object list and the center of mass of all objects in the previous frame image object list to obtain the two objects with the smallest distance, and then determine whether the distance is less than the second threshold. If so, it is considered that the object moves with the same object in the previous frame image.

[0070] Since the previous frame image and the current image are both planar images, in order to calculate the distance between the centroids of objects in the previous and next two frames of images, they need to be converted to the same coordinates for calculation. In this embodiment, the camera world coordinates can be used, and after conversion, the calculation is performed according to the following formula:

[0071]

[0072] In the above formula, N is the distance between the center of mass of the objects in the object list in the previous and next frame images in the camera world coordinate system, (x, y, z) is the coordinate of the center of mass of an object in the current image in the camera world coordinate system, and (x1, y1, z1) is the coordinate of the center of mass of an object in the previous frame image in the camera world coordinate system.

[0073] Step S3: Performing behavior discrimination and appearance discrimination on the objects in the object list respectively;

[0074] The method in the above embodiment is used to realize dynamic tracking of objects and continuously maintain an object list. The objects in this list will be subsequently subjected to two types of discrimination: behavior discrimination and appearance discrimination.

[0075] In a specific embodiment, behavior identification specifically includes the following steps:

[0076] The maximum displacement between the first frame image and the last frame image of the same object is calculated, and the behavior discrimination result is obtained based on the displacement.

[0077] Because pedestrians will definitely move, objects that do not move will be eliminated. Behavior discrimination will judge the maximum displacement between the first frame and the last frame of the same object. Because the normal stride of a person is 0.37 times the height, the displacement of one step is 50cm. Therefore, as long as the maximum displacement is greater than 50cm, it can meet the minimum specification of behavior discrimination.

[0078] In a specific embodiment, the shape determination specifically includes the following steps:

[0079] The HOG values ​​of the objects in the object list of the current image are calculated one by one, and are input into the trained two-dimensional vector machine model. The output of the two-dimensional vector machine is the shape discrimination result.

[0080] The HOG value is the Histogram of Oriented Gradients, which is widely used in image processing and is a common method used by those skilled in the art. Therefore, the detailed calculation method of the HOG value will not be described here.

[0081] In the above embodiment, the trained two-dimensional vector machine model refers to a trained model machine. The training process of a general model machine is as follows:

[0082] Positive / negative sample collection → Separate foreground → Filter foreground image → Segment foreground image → Extract candidate object image → Calculate HOG feature value for object image → Label the data with positive and negative values ​​→ Use SVM model machine for machine learning → Output training model

[0083] By inputting the HOG value of the object in the object list of the current image into the trained two-dimensional vector machine model, the output result of the two-dimensional vector machine is the shape discrimination result.

[0084] Step S4: When the behavior identification result satisfies the first preset value and the appearance identification result satisfies the second preset value, the object is determined to be a person.

[0085] The first preset value and the second preset value can be set according to the accuracy required by the method and actual needs. The behavior discrimination result is A and the appearance discrimination result is B. In this embodiment, when A>50cm and B>70% are satisfied at the same time, the object is judged to be a person.

[0086] By simultaneously performing behavior and appearance discrimination on the objects in the current image object list and using the discrimination results of both as the judgment criteria, the accuracy of pedestrian detection can be improved. This method not only targets the features of the current frame image, but also links the features of the previous and next frame images, and comprehensively utilizes the image features and the connections between them to improve the accuracy of pedestrian detection.

[0087] Accordingly, the present application also discloses a device, such as Figure 6As shown, it includes a processor 21 and a memory 22; wherein, when the processor 21 executes the computer program stored in the memory 22, the method described in any one of the above embodiments is implemented.

[0088] Furthermore, the embodiment of the present application also discloses a device 20, such as Figure 7 As shown, it includes the device in the aforementioned embodiment.

[0089] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0090] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A pedestrian counting method, characterized in that: The method is based on a depth image and comprises the following steps: Get the current image and determine the connected domain of the current image; According to the connected domain, establish the object list of the current image; Perform behavior and appearance discrimination on the objects in the object list; When the behavior discrimination result satisfies the first preset value, and the appearance discrimination result satisfies the second preset value, the object is determined to be a person; The step of obtaining the current image and determining the connected domain of the current image includes: Read the depth image in the memory, determine the background image, obtain the foreground image of the current image based on the background image, and determine the connected domain based on the foreground image; Determining the connected domain according to the foreground image includes: Determine whether each pixel in the foreground image is a valid point in a preset order, and determine the connected domain based on the valid points, specifically including the following steps: Determine the first effective point; Traversing each pixel point on the edge of the first effective point, determining the pixel points that meet the first preset condition as the in-domain points of the first effective point, and marking the in-domain points of the first effective point, the first effective point and its in-domain points constitute a connected domain; Traversing each pixel point on the edge of the intra-domain point of the first effective point, determining the pixel points that meet the first preset condition as the intra-domain points of the first effective point, and marking the intra-domain points of the first effective point, the first effective point and its intra-domain points constitute a connected domain; Determine a new first valid point according to the preset order and repeat the above steps.

2. The pedestrian counting method according to claim 1, characterized in that: The process of establishing an object list of the current image according to the connected domain includes: Determine one by one whether the largest three-dimensional circumscribed cuboid of the connected domain meets the second preset condition, and if so, add it to the object list of the current image.

3. The pedestrian counting method according to claim 2, characterized in that: The objects in the current image object list are compared one by one with all the objects in the previous frame image object list. If the distance between the centroids of two objects is less than a third preset value, the two objects are marked as the same object.

4. The pedestrian counting method according to claim 3, characterized in that: The behavior identification includes: The maximum displacement between the first frame image and the last frame image of the same object is calculated, and the behavior discrimination result is obtained based on the displacement.

5. The pedestrian counting method according to claim 4, characterized in that: The shape discrimination includes calculating the HOG values ​​of the objects in the object list of the current image one by one, and inputting them into a trained two-dimensional vector machine model. The result output by the two-dimensional vector machine is the shape discrimination result.

6. A device, characterized in that: The method comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the method according to any one of claims 1 to 5 is implemented.

7. A device, characterized in that: Comprising the device as claimed in claim 6.

8. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images