Queuing statistics method and device based on multi-position detection, equipment and medium

Abstract

The application provides a queuing statistics method and device based on multi-part detection, equipment and medium, by acquiring picture data and preprocessing; using a detection model to detect the head, upper body, lower body, and full body of each person in the preprocessed picture data, and obtaining a set; recombining each set according to the intersection between each part to obtain a set corresponding to each person respectively; calculating the vertex, center point, bottom point, and inclination according to the coordinates of the set elements; arranging each set to obtain a list T, performing intersection judgment, screening, and sorting on each element in the list T, and performing approximate straight line judgment to determine the queuing list and the corresponding number of people in the queue. The application can realize the queuing counting function only with one 2D image, and for pedestrians in a dense scene, the head, upper body, lower body, full body, and other parts that can only be partially visible can be detected, so that the detection and accuracy under occlusion can be improved.

Description

Technical Field

[0001] This invention relates to the fields of image detection and image tracking technology. In particular, it relates to a queuing statistics method, apparatus, device, and medium based on multi-site detection. Background Technology

[0002] With the application of machine learning in computer vision, especially deep neural network algorithms based on CNN convolutional networks, pedestrian detection algorithms for general scenarios have become quite mature. Common algorithms such as the YOLO series, SSD series, and Faster R-CNN have all been successfully applied in pedestrian detection. Common deep learning-based crowd counting or crowd density estimation methods have also been greatly improved, with crowd counting methods used in shopping malls and tourist attractions saving manpower and reducing costs. However, the above-mentioned crowd counting methods are used to assess the number of all people present in a scene, and are difficult to apply to specialized application scenarios, such as filtering out the number of people queuing in an image. In addition, crowd estimation methods are based on the overall characteristics of people, which cannot accurately determine whether a person is in a queue.

[0003] The presence and number of people queuing within a given scenario is a crucial requirement. This is directly relevant for tasks such as queuing to buy goods in a shopping mall or counting queues of irrelevant individuals when entering or exiting various locations like tourist attractions. Therefore, this paper proposes a method for detecting and counting queues. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a queueing statistics method, apparatus, device and medium based on multi-site detection, to solve the problem of crowd counting in the prior art.

[0005] To achieve the above and other related objectives, the present invention provides a queuing statistics method based on multi-site detection, the method comprising: step S1: acquiring image data and performing preprocessing;

[0006] Step S2: Use a detection model to detect the head, upper body, lower body, and full body of each person in the preprocessed image data. Represent each part using a preset data structure containing coordinate information, and obtain sets A, B, C, and D corresponding to the head, upper body, lower body, and full body, respectively. Step S3: Based on the intersection of the parts, reorganize and merge sets A, B, C, and D to obtain sets Pi, Pk, Pm, and Pn corresponding to each individual, and place them all into the overall set P. Step S4: Calculate the vertex, center point, base point, and inclination of each set Pi based on the coordinates of the elements in each set Pi within the overall set P. Step S5: Sort the elements in each set Pi in ascending or descending order according to the horizontal axis size to obtain list T. Perform intersection judgment, filtering, and sorting on the elements in list T to obtain list R. Step S6: Perform approximate straight line judgment on each element in list R to determine the queuing list and the corresponding number of people in the queue.

[0007] In one embodiment of the present invention, the acquisition and preprocessing of image data includes: resetting the image size of the acquired image data to [3, M, N]; where 3 is the number of channels after resetting, M is the height after resetting, and N is the width after resetting; the image data consists of three M*N matrices, and the matrix values ​​are the pixel values ​​of the image data; the pixel values ​​are subtracted from the mean of each of the three matrices, and then divided by the variance of each of the three matrices to obtain the normalized image data.

[0008] In one embodiment of the present invention, the acquisition of the detection model includes: labeling the head, upper body, lower body, and full body of all persons in each preprocessed image data; dividing the labeled image data into a training set and a test set according to a certain ratio; selecting multiple target detection algorithms and iteratively training the image data and labeled target parts in the training set until the loss function of the training set no longer decreases and the average precision of the test set no longer increases, so as to obtain the training model corresponding to each of the target detection algorithms; testing each training model on the test set, and selecting the training model with the average precision as the detection model.

[0009] In one embodiment of the present invention, the preset data structure is: {class,[x1,y1,x2,y2]}; where class represents the target category, taking values ​​1-4, corresponding to head, upper body, lower body, and full body respectively; [x1,y1,x2,y2] represents the target coordinates, where x1 is the upper left horizontal coordinate, y1 is the upper left vertical coordinate, x2 is the lower right horizontal coordinate, and y2 is the lower right horizontal coordinate.

[0010] In one embodiment of the present invention, the method of step S3 includes: step S301: creating sets Pi, Pk, Pm, and Pn to represent individuals corresponding to each element in sets A, B, C, and D, and placing them into the total set P; step S302: sequentially searching for intersecting elements in all previous sets that satisfy a preset intersection area ratio with each corresponding individual in the current set, in the order from set D to set A; step S303: moving the intersecting elements in the previous sets to the set corresponding to the individual, and deleting the intersecting elements from the previous sets; step S304: repeating the above steps to obtain sets Pi, Pk, Pm, and Pn corresponding to individuals respectively.

[0011] In one embodiment of the present invention, the method for obtaining set Pi in step S3 includes: Step S311: Creating set Pi to represent the individuals corresponding to each element di in set D, adding each element di to set Pi, and placing set Pi into the total set P; Step S312: Calculating the area of ​​intersection between each element cj and di in set C, divided by the area of ​​cj itself, to obtain the area ratio sj; Step S313: If the area ratio sj is greater than a preset value Sc, it indicates that cj and di belong to the same person, adding cj to set Pi, and deleting cj from set C; Step S314: According to steps S312 and S313, calculating the intersection between each element bj and di in set B, adding bj that satisfies the preset intersection area ratio to set Pi, and deleting bj from set B; Step S315: According to steps S312 and S313, calculating the intersection between each element aj and di in set A, adding aj that satisfies the preset intersection area ratio to set Pi, and deleting aj from set A.

[0012] In one embodiment of the present invention, the method of step S313 further includes: if, after iterating through set C, set Pi contains multiple elements cj from set C, then select the cj with the largest area ratio intersecting with di, and delete the remaining cj from set Pi and add them back to set C; and / or, if, after iterating through set C, set Pi contains multiple elements cj with the same area ratio intersecting, then calculate the distance between the center point of cj and di, retain the cj corresponding to the minimum center point distance, and delete the remaining cj from set Pi and add them back to set C.

[0013] In one embodiment of the present invention, the formula for calculating the distance between the center points of cj and di is as follows: d center =sqrt{[(x1 cj +x2 cj ) / 2-(x1 di +x2 di ) / 2]*2+[(y1 cj +y2 cj ) / 2-(y1di +y2 di ) / 2]*2};where x1, x2, y1, y2 represent the coordinates of elements cj and di respectively; sqrt represents the square root calculation.

[0014] In one embodiment of the present invention, the step of performing intersection judgment, filtering, and sorting on each element in list T to obtain list R includes: creating a set Qi to represent the set of all elements that intersect with element ti in list T, and adding ti to set Qi; sequentially judging whether sets Q1 to Qi-1 already contain ti, and if a set Qk contains ti, deleting set Qi and designating Qk as the set to which ti belongs; sequentially calculating the intersection-union ratio of element tj with ti in list T, and if the intersection-union ratio is greater than a preset value S. iou If tj intersects with ti, then tj is added to set Qi or set Qk. Then, each element ti in list T is evaluated sequentially. If the number of elements in its corresponding set Qi is less than a preset threshold Sq, then ti is considered not to be in a queue, and ti and set Qi are removed from list T. For any element ti in list T, the elements qj in the set Qi that intersects with ti are sorted, and then arranged in ascending or descending order based on the distance between the center points of qj and ai, resulting in a sorted list R.

[0015] In one embodiment of the present invention, step S6 includes: sorting each element in list R by neighborhood and judging any one or more of the following: tilt, center point line, vertex line, and bottom point line; if there are a preset threshold Sn elements in list R that satisfy at least one of the following judgment conditions: tilt, center point line, vertex line, and bottom point line, then list R is determined to be a queue, and the number of elements in list R is the number of people in the queue.

[0016] To achieve the above and other related objectives, this invention provides a queue statistics device based on multi-site detection. The device includes: an acquisition module for acquiring image data and preprocessing it; a processing module for using a detection model to detect the head, upper body, lower body, and full body of each person in the preprocessed image data, representing each site using a preset data structure containing coordinate information, and acquiring sets A, B, C, and D corresponding to the head, upper body, lower body, and full body respectively; recombining and merging sets A, B, C, and D according to the intersection between each site to obtain sets Pi, Pk, Pm, and Pn corresponding to each person respectively, and placing them all into a total set P; calculating the vertex, center point, base point, and inclination of each set Pi according to the coordinates of the elements in each set Pi in the total set P; sorting the elements in each set Pi in the total set P in ascending or descending order according to the size of the horizontal axis of the center point of each set Pi to obtain a list T; performing intersection judgment, filtering, and sorting on the elements in list T to obtain a list R; and performing approximate straight line judgment on the elements in list R to determine the queue list and the corresponding number of people in the queue.

[0017] To achieve the above and other related objectives, the present invention provides a computer device, the device comprising: a memory and a processor; the memory storing a computer program, and the processor executing the computer program to implement the method described above.

[0018] To achieve the above and other related objectives, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0019] As described above, the present invention provides a queuing statistics method, apparatus, device, and medium based on multi-site detection. This involves acquiring image data and preprocessing it; using a detection model to detect the head, upper body, lower body, and full body of each person in the preprocessed image data; representing each site using a preset data structure containing coordinate information; and obtaining sets A, B, C, and D corresponding to the head, upper body, lower body, and full body, respectively; recombining and merging sets A, B, C, and D according to the intersection of the sites to obtain sets Pi, Pk, Pm, and Pn corresponding to each individual, and placing them all into a total set P; calculating the vertex, center point, base point, and inclination of each set Pi based on the coordinates of the elements in each set Pi within the total set P; arranging the elements in each set Pi in ascending or descending order according to the horizontal axis size to obtain a list T; performing intersection judgment, filtering, and sorting on the elements in list T to obtain a list R; and performing approximate straight line judgment on the elements in list R to determine the queuing list and the corresponding number of people in the queue.

[0020] It has the following beneficial effects:

[0021] This invention can achieve relatively accurate queue counting using only a single 2D image. In densely populated scenes where pedestrians are occluded with each other, it can detect multiple parts such as the head, upper body, lower body, and whole body that are only partially visible. This can improve the detection and accuracy under occlusion and further save manpower and costs for queue counting. Attached Figure Description

[0022] Figure 1 The diagram shown is a flowchart of a queuing statistics method based on multi-site detection in one embodiment of the present invention.

[0023] Figure 2 The diagram shown is a flowchart of step S3 in one embodiment of the present invention.

[0024] Figure 3 The diagram shows a flowchart of obtaining the set Pi in step S3 of one embodiment of the present invention.

[0025] Figure 4 The diagram shown is a schematic representation of a queueing statistics device based on multi-site detection in one embodiment of the present invention.

[0026] Figure 5 The diagram shown is a structural schematic of a computer device according to one embodiment of the present invention. Detailed Implementation

[0027] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0028] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0029] Existing crowd counting methods are mainly used to count the total number of people in a scene, but they are not applicable to specific scenarios. For example, when counting the number of people in a queue, it is necessary to exclude people who are not in the same queue, which is something that existing crowd counting methods cannot solve.

[0030] To address the aforementioned issues, this invention proposes a queue counting method based on multi-part detection. This invention can achieve relatively accurate queue counting using only a single 2D image. In dense scenes where pedestrians are occluded with each other, it can detect multiple parts such as the head, upper body, lower body, and whole body, which are only partially visible. This can improve the detection and accuracy under occlusion, thereby further saving manpower and costs in queue counting.

[0031] like Figure 1 The diagram illustrates a flowchart of a queuing statistics method based on multi-site detection according to an embodiment of the present invention. As shown, the method includes:

[0032] Step S1: Acquire image data and perform preprocessing.

[0033] In one embodiment of this application, the acquisition and preprocessing of image data includes:

[0034] Step S101: Reset the image size of the acquired image data to [3, M, N]; where 3 is the number of channels after reset, M is the height after reset, and N is the width after reset; the image data is 3 M*N matrices, and the matrix value is the pixel value of the image data;

[0035] Step S102: Subtract the mean of each of the three matrices from the pixel value, and then divide by the variance of each of the three matrices to obtain the normalized image data.

[0036] For example, for input image data, let the input shape be (C, H, W) (C represents the number of channels, H represents the height, and W represents the width). First, reset the size of the image data to (3, M, N), for example, to (3, 512, 512), where 3 represents the number of channels after reset, 512 represents the height after adjustment, and 512 represents the width after reset. At this time, the image data becomes three matrices of size 512*512, and the matrix values ​​are the pixel values ​​of the image. Subtract the mean of each of the three matrices (123.675, 116.28, 103.53) from the pixel values, and then divide by the variance of each of the three matrices (58.395, 57.12, 57.375) to obtain the normalized image data, which is still (3, 512, 512).

[0037] Step S2: Use the detection model to detect the head, upper body, lower body, and full body of each person in the preprocessed image data. Represent each part using a preset data structure containing coordinate information, and obtain sets A, B, C, and D corresponding to the head, upper body, lower body, and full body, respectively.

[0038] In one embodiment of this application, the method for acquiring and training the detection model includes:

[0039] Step S201: Label the head, upper body, lower body, and full body of all people in each preprocessed image data;

[0040] Step S202: Divide the labeled image data into a training set and a test set according to a certain ratio;

[0041] Step S203: Select multiple object detection algorithms and iteratively train the image data and labeled target parts in the training set until the loss function of the training set no longer decreases and the average accuracy of the test set no longer increases, so as to obtain the training model of each object detection algorithm.

[0042] Step S204: Test each trained model on the test set, and save the training model with the average precision as the detection model.

[0043] In one or more implementable embodiments, this invention proposes an algorithm for queuing judgment and queuing number counting in a crowd, based on a general object detection method. The acquisition and training methods of the detection model can be exemplified as follows:

[0044] Step 1: Image Annotation: For an image dataset, label the coordinates of the head, upper body, lower body, and full body of all people in the image. The coordinates are composed of the top left and bottom right corners of the target (x1, y1, x2, y2), where x1 is the x-coordinate of the top left corner, y1 is the y-coordinate of the top left corner, x2 is the x-coordinate of the bottom right corner, and y2 is the x-coordinate of the bottom right corner.

[0045] Step 2: Data partitioning: Let the total amount of data be L. The data can be partitioned into training set and test set according to a certain ratio, such as 8:2.

[0046] Step 3: Training: Select an object detection algorithm, including any one or more of YOLOv4, YOLOv5, YOLOv7, and Cascade R-CNN. For example, in some embodiments, this invention uses YOLOv4. Iterative training is performed on the image data and labeled targets in the training set until the loss function of the training set no longer decreases and the mean average precision (mAP) of the test set no longer improves, thus obtaining the training model corresponding to each of the object detection algorithms. Preferably, during training, the model weights are saved every 10 iterations.

[0047] Step 4: Select a model: Test the training model saved during the training process on the test set, and select the model with the highest mAP to save as the BEST model, which will be used as the detection model.

[0048] In one embodiment of this application, the preset data structure is as follows:

[0049] {class,[x1,y1,x2,y2]};

[0050] Wherein, class represents the target category, with values ​​from 1 to 4, corresponding to head, upper body, lower body, and full body respectively; [x1,y1,x2,y2] represents the target coordinates, where x1 is the x-coordinate of the upper left corner, y1 is the y-coordinate of the upper left corner, x2 is the x-coordinate of the lower right corner, and y2 is the x-coordinate of the lower right corner.

[0051] In simple terms, each part can be represented by the data structure above, which includes the category of the corresponding part, which can be used to divide different sets of parts; it also includes coordinate information, such as x1 being the x-coordinate of the upper left corner, y1 being the y-coordinate of the upper left corner, x2 being the x-coordinate of the lower right corner, y2 being the x-coordinate of the lower right corner, etc., which can be used for subsequent calculations of vertices, center points, base points, and inclination.

[0052] Furthermore, the present invention also classifies the detected different parts into sets of corresponding parts, such as all head targets as set {A}, all upper body targets as set {B}, all lower body targets as set {C}, and all full-body targets as set {D}.

[0053] Step S3: Based on the intersection between each part, reorganize and merge sets A, B, C, and D respectively to obtain sets Pi, Pk, Pm, and Pn corresponding to individuals, and put them all into the total set P.

[0054] In one embodiment of this application, the method of step S3 includes:

[0055] Step S301: Create sets Pi, Pk, Pm, and Pn to represent the individuals corresponding to each element in sets A, B, C, and D, and put them into the overall set P;

[0056] Step S302: Following the order from set D to set A, sequentially find the elements in all previous sets that satisfy the preset intersection area ratio with each corresponding individual in the current set;

[0057] Step S303: Move the intersecting elements in the previous set to the set corresponding to the individual, and delete the intersecting elements from the previous set;

[0058] Step S304: Repeat the above steps to obtain sets Pi, Pk, Pm, and Pn corresponding to each individual.

[0059] In simple terms, let set D sequentially determine the elements that intersect with the preceding sets C, B, and A, and move the intersecting elements to D. Then let set C sequentially determine the elements that intersect with the preceding sets B and A, and move the intersecting elements to C. Then let set B sequentially determine the elements that intersect with the preceding set A, and move the intersecting elements to B. Finally, new sets Pi, Pk, Pm, and Pn corresponding to individuals are formed.

[0060] Specifically, the method for obtaining the set Pi in step S3 includes:

[0061] Step S311: Create sets Pi to represent the individuals corresponding to each element di in set D, add each element di to set Pi, and put set Pi into the total set P;

[0062] Step S312: Calculate the area of ​​intersection between each element cj and di in set C, divided by the area of ​​cj itself, to obtain the area ratio sj;

[0063] Step S313: If the area ratio sj is greater than the preset value Sc, it means that cj and di belong to the same person. Add cj to set Pi and delete cj from set C.

[0064] In one embodiment of this application, the method of step S313 further includes:

[0065] A. If, after iterating through set C, set Pi contains multiple elements cj from set C, then select the cj with the largest area ratio that intersects with di, and remove the remaining cj from set Pi and add them back to set C.

[0066] And / or,

[0067] B. If, after iterating through set C, there are multiple elements cj with the same cross-area ratio in set Pi, then calculate the distance between cj and the center point of di, retain the cj with the minimum center point distance, and delete the remaining cj from set Pi and add them back to set C.

[0068] Step S314: Based on steps S312 and S313, calculate the intersection of each element bj with di in set B in turn, add bj that satisfies the preset intersection area ratio to set Pi, and delete bj from set B.

[0069] Step S315: Based on steps S312 and S313, calculate the intersection of each element aj with di in set A, add aj that satisfies the preset intersection area ratio to set Pi, and delete aj from set A.

[0070] In some embodiments, the operations performed sequentially on the elements di within the set {D} can be exemplified as follows:

[0071] First, create a set {Pi} to represent the person corresponding to di, add di to {Pi}, and add {Pi} to {P}.

[0072] Then, calculate the area of ​​intersection between element cj and di in set {C}, divided by the area of ​​cj itself, to obtain the area ratio sj:

[0073] sj=area(cj∩di) / area(cj);

[0074] Here, area refers to the calculated area. If sj is greater than the preset threshold Sc, then cj and di are considered to belong to the same person. cj is added to {Pi} and removed from the set {C}.

[0075] If, after iterating through set {C}, {Pi} contains multiple elements cj from {C}, then first select the cj corresponding to the maximum value of the area ratio pj of di, and remove the remaining cj from {Pi} and add them to {C}; and / or, if multiple identical pj values ​​exist, then calculate the distance d_center between cj and the center point of di, using the following formula:

[0076] d center =sqrt{[(x1 cj +x2 cj ) / 2-(x1 di +x2 di ) / 2]*2+[(y1 cj +y2 cj ) / 2-(y1 di +y2 di ) / 2]*2};

[0077] Where x1, x2, y1, and y2 represent the coordinates of elements cj and di, respectively; sqrt represents the square root calculation. Then, retain d. center The minimum value corresponds to cj, and the remaining cs are removed from {Pi} and added to {C}.

[0078] Next, using the same method as steps S312 and S313, determine the relationship between elements bj and di in set {B}, and add bj that meets the conditions to {Pi}.

[0079] Finally, using the same method as steps S312 and S313, determine the relationship between elements aj and di in set {A}, and add aj that meets the conditions to {Pi}.

[0080] In some embodiments, the operations performed sequentially on elements di within the set {C} can be exemplified as follows:

[0081] Create a set {Pk} to represent the person corresponding to ck, add ck to {Pk}, and add {Pk} to {P}.

[0082] Use the same method as steps S312 and S313 to determine the relationship between elements bj and ck in set {B}, and add bj that meets the conditions to {Pk}.

[0083] Use the same method as steps S312 and S313 to determine the relationship between elements aj and ck in set {A}, and add aj that meets the conditions to {Pk}.

[0084] In some embodiments, the operations performed sequentially on elements di within set {B} can be exemplified as follows:

[0085] Create a set {Pm} to represent the person corresponding to bm, add bm to {Pm}, and add {Pm} to {P}.

[0086] Use the same method as steps S312 and S313 to determine the relationship between elements aj and bm in set {A}, and add aj that meets the conditions to {Pm}.

[0087] In some embodiments, the operations performed sequentially on elements di within set {A} can be exemplified as follows:

[0088] Create a set {Pn} to represent the person corresponding to an, add an to {Pn}, and add {Pn} to {P}.

[0089] Step S4: Calculate the vertex, center point, base point, and slope of each set Pi based on the coordinates of the elements in each set Pi within the total set P.

[0090] It should be noted that the coordinates used to calculate the vertex, center point, bottom point, and inclination are coordinate data detected by the monitoring model in step S2 and represented by the preset data structure {class,[x1,y1,x2,y2]}.

[0091] In one embodiment of this application, the method for calculating vertices in step S4 includes:

[0092] If set Pi contains element ai from set A, and let the coordinates of ai be [x1, y1, x2, y2], then the formula for calculating the vertex vi of set Pi is: vi = [(x1 + x2) / 2, y1];

[0093] If set Pi does not contain an element ai from set A, then vertex vi of set Pi is empty.

[0094] In one embodiment of this application, the method for calculating the center point in step S4 includes:

[0095] If set Pi contains elements bi from set B, ci from set C, and di from set D, and let the coordinates of bi be [bx1, by1, bx2, by2], the coordinates of ci be [cx1, cy1, cx2, cy2], and the coordinates of di be [dx1, dy1, dx2, dy2], then the formula for calculating the center point is as follows:

[0096] center i x ={[(bx1+bx2) / 2+(cx1+cx2) / 2] / 2+(dx1+dx2) / 2} / 2;

[0097] center i y ={[(by1+by2) / 2+(cy1+cy2) / 2] / 2+(dy1+dy2) / 2} / 2;

[0098] Among them, center i x The x-coordinate of the center point is represented by center i. y Represents the ordinate of the center point;

[0099] If set Pi does not contain element bi from set B or element ci from set C, but set Pi contains element di from set D, then the formula for calculating the center point is as follows:

[0100] center i x = (dx1 + dx2) / 2;

[0101] center i y = (dy1+dy2) / 2;

[0102] If set Pi does not contain element di from set D, but contains element bi from set B and element ci from set C, then the formula for calculating the center point is as follows:

[0103] center i x =[(bx1+bx2) / 2+(cx1+cx2) / 2] / 2

[0104] center i y =[(by1+by2) / 2+(cy1+cy2) / 2] / 2

[0105] If set Pi does not fall under the above conditions, then the center point is empty.

[0106] In one embodiment of this application, the method for calculating the base point in step S4 includes:

[0107] If set Pi contains elements ci from set C and di from set D, and let the coordinates of ci be [cx1, cy1, cx2, cy2] and the coordinates of di be [dx1, dy1, dx2, dy2], then the base of set Pi is calculated using the following formula:

[0108] bottom i x =[(cx1+cx2) / 2+(dx1+dx2) / 2] / 2;

[0109] bottom i y = (cy2+dy2) / 2;

[0110] Among them, bottom i x Represents the x-coordinate of the bottom point, bottom i y Indicates the x-coordinate of the base point;

[0111] If set Pi contains only one of the elements ci from set C or di from set D, and its coordinates are [x1, y1, x2, y2], the base point of set Pi is calculated using the following formula:

[0112] bottom i x = (x1+x2) / 2;

[0113] bottom i y =y2;

[0114] If set Pi does not fall under the above conditions, then the bottom point is empty.

[0115] In one embodiment of this application, the method for calculating the tilt angle in step S4 includes:

[0116] Let the coordinates of the vertex be [vx,vy], the coordinates of the center point be [cx,cy], and the coordinates of the base point be [bx,by].

[0117] If set Pi has vertices and a center point, then the formula for calculating the slope in is:

[0118] in=arctan(|cy-vy| / |cx-vx|);

[0119] Where || represents the absolute value, and arctan represents the arctangent;

[0120] If set Pi has a base point and a center point, the formula for calculating the slope in is:

[0121] in=arctan(|cy-by| / |cx-bx|);

[0122] If set Pi has a vertex, a base, and a center, the formula for calculating the slope in is:

[0123] in=[arctan(|cy-vy| / |cx-vx|)+arctan(|cy-by| / |cx-bx|)] / 2;

[0124] If set Pi does not fall under the above conditions, then the slant in is empty.

[0125] Step S5: Sort the elements in the set P in ascending or descending order according to the size of the horizontal axis of the center point of each set Pi to obtain list T. Then, perform intersection judgment, filtering, and sorting on the elements in list T to obtain list R.

[0126] In simple terms, we first obtain list T by sorting. Specifically, we arrange the elements {Pi} in set {P} in ascending or descending order according to the size of the horizontal axis centered at pi, forming list T. Then, we obtain list R by performing intersection analysis, filtering, and sorting on the elements in list T.

[0127] In one embodiment of this application, the step of performing intersection judgment, filtering, and sorting on the elements in list T to obtain list R includes:

[0128] A. Create a set Qi to represent the set of all elements that intersect with elements ti in list T, and add ti to set Qi;

[0129] B. Determine in turn whether sets Q1 to Qi-1 contain ti. If a set Qk contains ti, delete set Qi and designate Qk as the set to which ti belongs.

[0130] C. Calculate the intersection-union ratio (IUR) of elements tj and ti in list T sequentially. If the IUR is greater than the preset value S... iou If tj intersects with ti, then tj is added to set Qi or set Qk;

[0131] D. Sequentially judge the element ti in list T. If the number of its corresponding set Qi is less than the preset threshold number Sq, then ti is considered not to be in a queue, and ti and set Qi are deleted from list T.

[0132] E. For any element ti in list T, sort the elements qj in the set Qi that intersects with ti, and arrange them in ascending or descending order based on the distance between the center points of qj and ai, to obtain the sorted list R.

[0133] Specifically, the following are examples of operations performed on the elements ti within the list {T} in sequence:

[0134] The steps for determining intersection include: First, create a set {Qi} to represent the set of all elements that intersect with ti, and add ai to {Qi}.

[0135] Next, check whether sets {Q1} to {Qi-1} already contain ti. If any set {Qk} contains ti, delete set {Qi} and designate {Qk} as the set to which ti belongs.

[0136] Then, calculate the intersection-union ratio (iOU) of elements tj and ti in list T, where j > i. The counting formula is as follows:

[0137] iou=A∩B / (A∪BA∩B);

[0138] If IoU is greater than the preset threshold S iou , such as S iou If the value is 0.1, then tj is considered to intersect with ti, and tj is added to {Qi} or {Qk}.

[0139] The filtering steps include: sequentially judging the element ti in the list {T}. If the number of its corresponding {Qi} elements is less than the preset threshold number Sq, such as Sq being 3, then ti is considered not to be in a queue, and ti and {Qi} are deleted from the list T.

[0140] The sorting steps include: for any element ti in the list {A}, sort the elements qj in the set {Qi} that intersects with ti, and arrange them in ascending or descending order based on the distance between the center points of qj and ti, to obtain the sorted list {Ri}.

[0141] Step S6: Perform an approximate straight line judgment on each element in list R to determine the queue list and the corresponding number of people in the queue.

[0142] In one embodiment of this application, step S6 includes:

[0143] A. Sort each element in list R by neighborhood and determine one or more of the following: slant, center point line, vertex line, and bottom point line.

[0144] In one or more embodiments, the determination of tilt in step S6 includes:

[0145] Loop through list R and check element rj. If the difference in slant between rj and set Pi is greater than a preset threshold So, such as So being 30, then determine that rj is not in the queue of set Pi and delete rj from list R.

[0146] In one or more embodiments, the determination of the center point line in step S6 includes:

[0147] Connect the center point of element rj in list R with the center point of rj+1 to form a straight line L. j Connect the center point of rj+1 with the center point of rj+2 to obtain the straight line L. j+1 Calculate L j With Lj+1 The included angle α is given by the formula:

[0148] a=arccos[(L j *L j+1 ) / (||L j ||*||L j+1 ||)];

[0149] Among them, L j *L j+1 Table dot product, ||L j ||and||L j+1 || Table-based modulo calculation;

[0150] If angle a is less than the preset angle threshold Sa, such as Sa being 30, then it is considered that rj, rj+1, and rj+2 can be connected by a straight line, and rj, rj+1, and rj+2 are determined to be in the same queue.

[0151] In one or more embodiments, the determination of the vertex line in step S6 includes:

[0152] Connect the center point of element rj in list R to the vertex rj+1 to form a straight line L. j Connecting the vertex of rj+1 to the vertex of rj+2 forms a straight line L. j+1 Calculate L j With L j+1 The included angle b is given by the formula:

[0153] b = arccos[(L j *L j+1 ) / (||L j ||*||L j+1 ||)];

[0154] Among them, L j *L j+1 Table dot product, ||L j ||and||L j+1 || Table-based modulo calculation;

[0155] If angle b is less than the preset angle threshold Sb, such as Sb being 30, then it is considered that rj, rj+1, and rj+2 can be connected into a straight line, and rj, rj+1, and rj+2 are determined to be in the same queue.

[0156] In one or more embodiments, the determination of the bottom point line in step S6 includes:

[0157] Connect the center point of element rj in list R with the base point of rj+1 to form a straight line L. j Connect the base points of rj+1 and rj+2 to obtain the straight line L. j+1 Calculate L jWith L j+1 The included angle c is given by the formula:

[0158] c = arccos[(L j *L j+1 ) / (||L j ||*||L j+1 ||)];

[0159] Among them, L j *L j+1 Table dot product, ||L j ||and||L j+1 || Table-based modulo calculation;

[0160] If angle c is less than the preset angle threshold Sc, such as Sc being 30, then it is considered that rj, rj+1, and rj+2 can be connected into a straight line, and rj, rj+1, and rj+2 are determined to be in a queue.

[0161] B. If list R contains a preset threshold Sn elements that satisfy at least one of the following judgment conditions: inclination, center point straight line, vertex straight line, and bottom point straight line, then list R is determined to be a queue, and the number of elements in list R is the number of people in the queue.

[0162] Specifically, if at least a preset threshold Sn of elements in the list {Ri} satisfy one of the following lines: the center point line, the vertex line, or the bottom point line, then {Ri} is considered a queue. The final result is a list {T} and a sublist {Ri}, where {Ri} may or may not exist. The final existing list {Ri} is the queue list, and the number of elements in list {Ri} represents the number of people in the queue.

[0163] In summary, this invention can achieve relatively accurate queue counting using only a single 2D image. In densely populated scenes where pedestrians are occluded with each other, it can detect multiple parts such as the head, upper body, lower body, and whole body that are only partially visible, thereby improving the detection and accuracy under occlusion and further saving manpower and costs for queue counting.

[0164] like Figure 4 The figure shows a schematic diagram of a queue counting device based on multi-location detection according to an embodiment of the present invention. As shown, the queue counting device 400 based on multi-location detection includes:

[0165] The acquisition module 401 is used to acquire image data and perform preprocessing.

[0166] The processing module 402 is used to detect the head, upper body, lower body, and full body of each person in the preprocessed image data using a detection model. Each part is represented using a preset data structure containing coordinate information, and sets A, B, C, and D corresponding to the head, upper body, lower body, and full body are obtained respectively. Based on the intersection between the parts, sets A, B, C, and D are recombined and merged to obtain sets Pi, Pk, Pm, and Pn corresponding to each individual, and all are placed into the overall set P. Based on the coordinates of each element in set Pi within the overall set P, the vertex, center point, base point, and inclination of each set Pi are calculated. The elements in each set Pi are sorted in ascending or descending order according to the horizontal axis size to obtain list T. The elements in list T are then subjected to intersection judgment, filtering, and sorting to obtain list R. Finally, the elements in list R are subjected to approximate straight-line judgment to determine the queuing list and the corresponding number of people in the queue.

[0167] It is understandable that the device 400, through the operation of its various modules, can achieve the following: Figure 1 The queueing statistics method based on multi-site detection is described above.

[0168] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, processing module 402 can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and its functions can be called and executed by a processing element of the above device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0169] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to form a system-on-a-chip (SOC).

[0170] like Figure 5 The figure shows a schematic diagram of a computer device according to an embodiment of the present invention. As shown, the computer device 500 includes a memory 501 and a processor 502. The memory 501 stores a computer program, and the processor 502 executes the computer program to implement the following... Figure 1 The queueing statistics method based on multi-site detection is described above.

[0171] The memory 501 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0172] The processor 502 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0173] In one embodiment of the present invention, a computer-readable storage medium stores a data transmission program thereon, which, when executed by a processor, implements as follows: Figure 1 The queueing statistics method based on multi-site detection is described above.

[0174] As will be understood by those skilled in the art, all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned image processing program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0175] These computer programs can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0176] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0177] In summary, the present invention provides a queuing statistics method, apparatus, device, and medium based on multi-site detection, which can effectively overcome the various shortcomings of the prior art and has high industrial application value.

[0178] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for queuing statistics based on multi-location detection, characterized by, The method comprises: Step S1: obtaining picture data and preprocessing; Step S2: using a detection model to detect the head, upper body, lower body, and full body of each person in the preprocessed picture data, using a preset data structure containing coordinate information to represent each part, and obtaining sets A, B, C, and D corresponding to the head, upper body, lower body, and full body respectively; Step S3: recombining sets A, B, C, and D according to the intersection between each part to obtain sets Pi, Pk, Pm, and Pn corresponding to the individual respectively, and putting them all into a total set P; The method of step S3 comprises: Step S301: creating sets Pi, Pk, Pm, and Pn to represent the individuals corresponding to the elements in sets A, B, C, and D respectively, and putting them into the total set P; Step S302: in the order of sets D to A, sequentially finding the intersection part elements in each previous set that satisfy the preset intersection area ratio with each corresponding individual in the current set; Step S303: moving the intersection part elements in the previous set to the set corresponding to the corresponding individual, and deleting the intersection part elements from the previous set; Step S304: repeating the above steps to obtain sets Pi, Pk, Pm, and Pn corresponding to the individual respectively; Step S4: calculating the vertex, center point, bottom point, and inclination of each set Pi according to the coordinates of the elements in each set Pi in the total set P; Step S5: arranging the center points of each set Pi in the total set P in ascending or descending order according to the size of the horizontal axis to obtain a list T, and performing intersection judgment, screening, and sorting on each element in the list T to obtain a list R; Step S6: performing approximate straight line judgment on each element in the list R to determine the queuing list and the corresponding number of people in line.

2. The multi-part detection based queuing statistics method of claim 1, wherein, The preprocessing of the obtained picture data comprises: resetting the picture size of the obtained picture data to [3, M, N]; wherein 3 is the number of channels after resetting, M is the height after resetting, and N is the width after resetting; the picture data is a 3M*N matrix, and the matrix value is the pixel value of the picture data; subtracting the mean value of each of the three matrices from the pixel value and dividing by the variance of each of the three matrices to obtain the normalized picture data.

3. The multi-part detection based queuing statistics method of claim 1, wherein, The obtaining of the detection model comprises: labeling the head, upper body, lower body, and full body of all people in each preprocessed picture data; dividing the labeled picture data into a training set and a test set according to a certain proportion; selecting a plurality of target detection algorithms, and iteratively training the picture data and the labeled target parts in the training set until the loss function of the training set no longer decreases and the average precision mean of the test set no longer improves, to obtain a training model corresponding to each target detection algorithm; testing each training model on the test set, and selecting the training with the highest average precision mean as the detection model.

4. The multi-part detection based queuing statistics method of claim 1, wherein, The preset data structure is: {class, [x1, y1, x2, y2]}; Wherein, class represents the target category, and takes values 1-4, corresponding to head, upper body, lower body, and full body respectively; [x1, y1, x2, y2] represents the target coordinates, x1 is the horizontal coordinate of the upper left corner, y1 is the vertical coordinate of the upper left corner, x2 is the horizontal coordinate of the lower right corner, and y2 is the vertical coordinate of the lower right corner.

5. The multi-part detection based queuing statistics method of claim 1, wherein, The method for obtaining the set Pi in the step S3 comprises: Step S311: creating a set Pi to represent individuals corresponding to each element di in the set D, adding each element di to the set Pi, and putting the set Pi into a total set P; Step S312: sequentially calculating the area ratio sj of the intersection of each element cj in the set C and di divided by the area of cj itself; Step S313: if the area ratio sj is greater than a preset value Sc, it indicates that cj and di belong to the same individual, cj is added to the set Pi, and cj is deleted from the set C; Step S314: according to steps S312 and S313, the intersection of each element bj in the set B and di is sequentially calculated, and bj satisfying the preset intersection area ratio is added to the set Pi, and bj is deleted from the set B; Step S315: according to steps S312 and S313, the intersection of each element aj in the set A and di is sequentially calculated, and aj satisfying the preset intersection area ratio is added to the set Pi, and aj is deleted from the set A.

6. The multi-part detection based queuing statistics method of claim 5, wherein, The method of step S313 further comprises: If after looping through the set C, the set Pi contains multiple elements cj in the set C, the element cj with the maximum intersection area ratio with di is selected, and the remaining cj is deleted from the set Pi and added back to the set C; And / or, If after looping through the set C, there are multiple elements cj in the set Pi with the same intersection area ratio, the center point distance of cj and di is calculated, and the cj corresponding to the minimum center point distance is retained, and the remaining cj is deleted from the set Pi and added back to the set C.

7. The multi-part detection based queuing statistics method of claim 6, wherein, The center point distance calculation formula of cj and di is as follows: d center =sqrt{[(x1 cj +x2 cj ) / 2-(x1 di +x2 di ) / 2]*2+[(y1 cj +y2 cj ) / 2-(y1 di +y2 di ) / 2]*2}; Wherein, x1, x2, y1, y2 represent the coordinates of elements cj and di respectively; sqrt represents square root calculation.

8. The multi-part detection based queuing statistics method of claim 1, wherein, The intersection judgment, screening, and sorting of each element in the list T to obtain the list R comprise: Creating a set Qi to represent the intersection of all elements of an element ti in the list T, and adding ti to the set Qi; Judging whether the set Q1 to the set Qi-1 contains ti, if a certain set Qk contains ti, deleting the set Qi, and specifying Qk as the set to which ti belongs; Sequentially calculating the intersection union ratio of elements tj and ti in the list T, if the intersection union ratio is greater than a preset value S iou, it is considered that tj and ti intersect, and tj is added to the set Qi or the set Qk; In sequence, if the number of the set Qi corresponding to the element ti in the list T is less than a preset threshold number Sq, it is considered that ti is not in a queue, and ti and the set Qi are deleted from the list T; For any element ti in the list T, the elements qj in the set Qi intersecting with ti are sorted in ascending or descending order according to the distance between the center points of qj and ai, to obtain a sorted list R.

9. The multi-part detection based queuing statistics method of claim 1, wherein, The step S6 comprises: The neighborhood sorting of each element in the list R is performed to determine any one or more of the inclination, the center point straight line, the vertex straight line, and the bottom point straight line; If a preset threshold number Sn of elements in the list R satisfy the determination condition of at least one of the inclination, the center point straight line, the vertex straight line, and the bottom point straight line, it is determined that the list R is a queuing queue, and the number of elements in the list R is the number of people in the queue.

10. A queuing statistics device based on multi-location detection, characterized by, The device comprises: An acquisition module configured to acquire picture data and perform preprocessing; A processing module configured to detect the head, upper body, lower body, and full body of each person in the preprocessed picture data using a detection model, represent each part using a preset data structure containing coordinate information, and acquire sets A, B, C, and D corresponding to the head, upper body, lower body, and full body respectively; recombine the sets A, B, C, and D according to the intersection between each part to obtain sets Pi, Pk, Pm, and Pn corresponding to each person respectively, and put them all into a total set P; The sets Pi, Pk, Pm, and Pn are created to represent the corresponding persons of the elements in the sets A, B, C, and D respectively, and are put into the total set P; in the order of the sets D to A, the intersection part elements of all previous sets and each corresponding person in the current set are found in sequence, the intersection part elements in the previous set are moved to the set corresponding to the corresponding person, and the intersection part elements are deleted from the previous set; the above steps are repeated to obtain the sets Pi, Pk, Pm, and Pn corresponding to each person respectively; The center point, bottom point, inclination, and vertex of each set Pi are calculated according to the coordinates of the elements in each set Pi in the total set P; the center points of the sets Pi in the total set P are arranged in ascending or descending order according to the size of the horizontal axis to obtain a list T, and the intersection, screening, and sorting of the elements in the list T are performed to obtain a list R; the approximate straight line of each element in the list R is determined to determine the queuing list and the corresponding number of people in the queue.

11. A computer device, characterized by The device comprises a memory and a processor; the memory stores a computer program, and the processor executes the computer program to implement the method of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is executed by a processor to implement the method of any one of claims 1-9.

Images