A queue counting method and system based on fusion of multiple information features

Abstract

The present invention discloses a queue counting method and system based on the fusion of multiple information features. The method comprises the following steps: performing monocular vision target detection and tracking; using the position information of each tracked head in each frame image as an extraction target, obtaining high-level features of the extraction target, wherein the high-level features include trajectory features, position features, time features, and speed features of the extraction target; performing calculation of the adjacency matrix E of the extraction target, including: calculating the trajectory similarity matrix E c , position similarity matrix E l , the temporal similarity matrix E t , and the speed similarity matrix E s ; Calculate the actual number of queue groups for N frames of imagery. This invention comprehensively addresses the existing technical issues of failing to cluster and differentiate between passenger flows and objects in the same queue group, and lacking the use of more complex and accurate computer vision algorithms and systems to help internet platforms improve operational efficiency and dynamically understand customer information.

Description

Technical Field

[0001] The present invention relates to a queue counting method and system based on the fusion of multiple information features, relates to traffic statistics, and belongs to the field of artificial intelligence technology. Background Art

[0002] With the rise and development of the service industry, a variety of consumer activities are emerging. During peak hours, many businesses such as hospitals, supermarkets, banks, and fast food restaurants face challenges in accurately understanding customer trends, such as long checkout lines, delayed window openings, and an inability to estimate the actual number of people in line. Current surveillance technologies have already played a significant role in assisting business operations. For example, head-counting cameras offer high accuracy, but installation requirements are complex and often limited to lobby entrances. AI-powered smart cameras that perform facial recognition and analysis pose the risk of infringing customer privacy. Furthermore, these AI recognition cameras can only analyze individual objects and cannot distinguish and analyze groups of identical objects. If one wants to predict the number of people remaining in a supermarket checkout line, the aforementioned methods will not be able to meet such complex counting requirements. This is because many people in the queue may be shopping in groups, such as couples, parents with children, or friends.

[0003] Therefore, using more complex and accurate computer vision algorithms and systems can help Internet platforms improve operational efficiency, dynamically understand customer information, and play a very important role in the development of the industry.

[0004] Patent 1 discloses a passenger flow counting device, passenger flow counting method, and storage medium. The patent publication number is CN109460811A. It specifically discloses that the number of passengers entering and leaving is counted by installing a distance measuring sensor and an infrared sensor. Due to the limitations of the sensor itself, it needs to be installed in a specific area. The installation conditions are harsh and it is not as versatile as ordinary cameras. Patent 2 discloses a passenger flow counting device based on binocular vision. The publication number is CN202058221U. It discloses that passenger flow is counted by binocular vision system and technology, mainly used for passenger flow monitoring in public transportation systems. The accuracy of the binocular matching algorithm will limit the accuracy of passenger flow counting, and the efficiency of operation and use is difficult to guarantee. Patent 3 discloses a crowd tracking and pedestrian flow counting method and device. The publication number is CN104751491A. It uses RGB-D cameras for pedestrian tracking and passenger flow counting. RGB-D can provide distance information of the target, but the detection accuracy of the depth camera will be greatly reduced for targets at a distance of more than 4m. Patent 4, with publication number CN103164858B, discloses a method for segmenting and tracking contiguous crowds based on superpixels and graph models. This method utilizes image segmentation methods, combined with prior knowledge of head shape, to segment the human body, converting the individual regions to be analyzed into an undirected connected graph. This method provides valuable insights for crowd clustering in this patent. However, super-resolution methods are not robust to scenes with varying lighting conditions and are prone to segmentation errors.

[0005] The existing technologies including the above patents do not cluster and differentiate the same group of targets in the passenger flow and queue. Summary of the Invention

[0006] The present invention aims to overcome the technical shortcomings of the prior art and solve the aforementioned technical problems by proposing a queue counting method and system based on the fusion of multiple information features. Using a simple monocular camera and computer vision technology, the present invention provides a set of passenger flow statistics and queue counting methods and systems, helping operators analyze and understand the number of queues and queuing conditions in real time. The system can also cluster queues belonging to the same group and output the actual number of queue groups. A queue belonging to the same group refers to a group of people queuing together.

[0007] The present invention specifically adopts the following technical solution: a queue counting method based on the fusion of multiple information features, comprising the following steps:

[0008] Step SS1: Performing monocular target detection and tracking, including: acquiring N consecutive frames of images, using a convolutional neural network to detect the position of the head and shoulders of people in each frame, tracking multiple targets using the DeepSort tracking algorithm, obtaining the position information of each head in each frame as the extracted target, and generating a target sequence T;

[0009] Step SS2: Performing behavioral analysis on each sequence in the target sequence T to obtain high-level features of the extracted target, wherein the high-level features include trajectory features, position features, time features, and speed features of the extracted target;

[0010] Step SS3: Execute the calculation of the adjacency matrix E of the extracted target, including: calculating the trajectory similarity matrix E c , position similarity matrix E l , the temporal similarity matrix E t , and the speed similarity matrix E s , E is calculated as follows,

[0011]

[0012] Where E={e ij} is a normalized n×n symmetric matrix that represents the correlation between each extracted target, that is, the probability of belonging to the same group of targets, e ij The larger the value, the higher the probability that target i and target j belong to the same group of targets. n represents the number of detected targets, and w c represents the weight of the trajectory similarity matrix; w l Represents the weight of the position similarity matrix; w t represents the weight of the temporal similarity matrix; w s represents the weight of the speed similarity matrix;

[0013] Step SS4: Execute the calculation of the actual queued group number set of N frames of images, including: calculating the Laplacian matrix L of the adjacency matrix E of the extracted target, using the graph cut algorithm to obtain the optimal cluster decomposition, and finally outputting the actual queued group number set.

[0014] As a preferred embodiment, step SS1 specifically includes:

[0015] The method based on convolutional neural network is used to detect the position of the head and shoulders of a person in the picture, which specifically includes: using a 19-layer convolutional neural network to perform target detection in each frame image, detecting the part above the shoulders to the top of the head, and outputting target frame information l(u p ,v p ,w p ,h p ), where (u p ,v p ) is the center point coordinate, w p and h p Represent the width and height of the target box respectively;

[0016] The method of tracking multiple detected targets using the DeepSort tracking algorithm specifically includes: finding the matching box of the next frame for the target detected in each frame image; if no matching box is found, the current frame is used as the starting point to be counted into the trajectory; and the target sequence detected in N frames of image is finally output as:

[0017] T={t1,t2,t i ,...,t n},

[0018] t i ={l s ,l s+1 ,...,l e},1≤s≤e≤N;

[0019] Where n represents the number of detected targets and N represents the number of frames.

[0020] As a preferred embodiment, the step SS2 specifically includes: detecting the i-th target t in the target sequence T detected in the N frames of image. i , calculate the following high-level features:

[0021] Trajectory feature A that characterizes the target's movement over time c , A c ={p i},

[0022] Position feature B that represents the target position c , B c Represents trajectory feature A c The first coordinate in

[0023] Time feature D c , D c It is a number between 1 and N, indicating the frame in which the target begins to appear on the screen;

[0024] Speed ​​characteristic S c , according to the trajectory feature A c The calculated number is shown in formula (2):

[0025]

[0026] Among them, Ω represents the Euclidean distance, are the coordinates of the i-th target in the target sequence T.

[0027] As a preferred embodiment, the step SS3 specifically includes: the adjacency matrix E is composed of the trajectory similarity matrix E c , position similarity matrix E l , time similarity matrix E tand speed similarity matrix E s Weighted, where:

[0028] Trajectory similarity matrix E c ={1-a ij}, a ij It is the sum of the squared differences between the trajectory characteristics of any target i and the trajectory characteristics of target j, normalized to 0 to 1. The smaller the trajectory difference between the two targets, the greater the probability that they belong to the same group;

[0029] Position similarity matrix E l ={1-l ij}, l ij The Euclidean distance between the positional feature values ​​of any target i and the positional feature values ​​of target j is normalized from 0 to 1. The closer the positions of two targets are, the greater the probability that they belong to the same group.

[0030] Temporal similarity matrix E t ={1-t ij}, t ij It is the L1 distance between the time characteristic value of any target i and the time characteristic value of target j. The closer the time of appearance of two targets is, the greater the probability that they belong to the same group.

[0031] Speed ​​similarity matrix E s ={1-s ij}, s ij It is the L1 distance between the speed of any target i and the speed of target j, normalized to 0 to 1. The closer the speeds of two targets are, the greater the probability that they belong to the same group.

[0032] As a preferred embodiment, step SS4 specifically includes:

[0033] Step SS41: Using the adjacency matrix E as the relationship matrix between the standard targets, the degree information matrix D is obtained, as shown in formula (3); the degree information matrix D is combined with the adjacency matrix E to obtain the Laplacian matrix L, as shown in formula (4). The Laplacian matrix L is a symmetric matrix.

[0034]

[0035] L=D -1 / 2 (DE)D -1 / 2 Formula (4)

[0036] Step SS42: Perform SVD decomposition on the Laplacian matrix L to obtain the eigenvalue λ and eigenvector v; calculate the eigenvector z = D 1 / 2v, classify the samples i corresponding to the vectors less than 0 in the transformed eigenvector z into one category, recalculate the energy matrix E' and Laplacian matrix L' and eigenvector z' between the samples i' corresponding to the vectors greater than 0, and classify the samples corresponding to the components less than 0 in the eigenvector z' into a new category, and thus recursively until there is no longer a component greater than 0 in the eigenvector z' or the dimension of the eigenvector z' is 1, the recursion ends, and the targets divided each time form a new target group C (c1, c2, ..., c k ), and finally the number of groups k in the entire queue is obtained.

[0037] The present invention also proposes a queue counting system based on the fusion of multiple information features, including:

[0038] The target detection and tracking module specifically performs: performing monocular vision target detection and tracking, including: acquiring N consecutive frames of images, using a convolutional neural network to detect the position of the head and shoulders of people in each frame, using the DeepSort tracking algorithm to track multiple targets, obtaining the position information of each head in each frame as the extracted target, and generating a target sequence T;

[0039] The target feature extraction module specifically performs the following steps: performing behavioral analysis on each sequence in the target sequence T to obtain high-level features of the extracted target, including trajectory features, position features, time features, and speed features of the extracted target;

[0040] The adjacency matrix calculation module specifically performs the following operations: Calculate the adjacency matrix E of the extracted target, including: Calculate the trajectory similarity matrix E c , position similarity matrix E l , the temporal similarity matrix E t , and the speed similarity matrix E s , E is calculated as follows,

[0041]

[0042] Where E={e ij} is a normalized n×n symmetric matrix that represents the correlation between each extracted target, that is, the probability of belonging to the same group of targets, e ij The larger the value, the higher the probability that target i and target j belong to the same group of targets. n represents the number of detected targets, and w c represents the weight of the trajectory similarity matrix; w l Represents the weight of the position similarity matrix; w t represents the weight of the temporal similarity matrix; w s represents the weight of the speed similarity matrix;

[0043] Calculate the actual number of groups module: Execute the calculation of the actual number of groups in the queue of N frames of images, including: calculating the Laplacian matrix L of the adjacency matrix E of the extracted target, using the graph cut algorithm to obtain the optimal cluster decomposition, and finally outputting the actual number of groups in the queue.

[0044] As a preferred embodiment, the target detection and tracking module specifically includes:

[0045] The method based on convolutional neural network is used to detect the position of the head and shoulders of a person in the picture, which specifically includes: using a 19-layer convolutional neural network to perform target detection in each frame image, detecting the part above the shoulders to the top of the head, and outputting target frame information l(u p ,v p ,w p ,h p ), where (u p ,v p ) is the center point coordinate, w p and h p Represent the width and height of the target box respectively;

[0046] The method of tracking multiple detected targets using the DeepSort tracking algorithm specifically includes: finding the matching box of the next frame for the target detected in each frame image; if no matching box is found, the current frame is used as the starting point to be counted into the trajectory; and the target sequence detected in N frames of image is finally output as:

[0047] T={t1,t2,t i ,...,t n},

[0048] t i ={l s ,l s+1 ,...,l e},1≤s≤e≤N;

[0049] Where n represents the number of detected targets and N represents the number of frames.

[0050] As a preferred embodiment, the target feature extraction module specifically includes: for the i-th target t in the target sequence T detected in the N frames of image, i , calculate the following high-level features:

[0051] Trajectory feature A that characterizes the target's movement over time c , A c ={p i},

[0052] Position feature B that represents the target position c , Bc Represents trajectory feature A c The first coordinate in

[0053] Time feature D c , D c It is a number between 1 and N, indicating the frame in which the target begins to appear on the screen;

[0054] Speed ​​characteristic S c , according to the trajectory feature A c The calculated number is shown in formula (2):

[0055]

[0056] Among them, Ω represents the Euclidean distance, are the coordinates of the i-th target in the target sequence T.

[0057] As a preferred embodiment, the adjacency matrix calculation module specifically includes: the adjacency matrix E is composed of the trajectory similarity matrix E c , position similarity matrix E l , time similarity matrix E t and speed similarity matrix E s Weighted, where:

[0058] Trajectory similarity matrix E c ={1-a ij}, a ij It is the sum of the squared differences between the trajectory characteristics of any target i and the trajectory characteristics of target j, normalized to 0 to 1, based on the fact that the smaller the trajectory difference between two targets, the greater the probability that they belong to the same group;

[0059] Position similarity matrix E l ={1-l ij}, l ij The Euclidean distance between the positional feature value of any target i and the positional feature value of target j is normalized from 0 to 1. The closer the two positions are, the greater the probability that they belong to the same group.

[0060] Temporal similarity matrix E t ={1-t ij}, t ij It is the L1 distance between the time feature value of any target i and the time feature value of target j. The closer the time of appearance of two targets is, the greater the probability that they belong to the same group.

[0061] Speed ​​similarity matrix E s ={1-s ij}, s ijis the L1 distance between the speed of any target i and the speed of target j, normalized to 0 to 1, based on the fact that the closer the speeds of two targets are, the greater the probability that they belong to the same group.

[0062] As a preferred embodiment, the module for calculating the actual number of groups specifically includes: taking the adjacency matrix E as the relationship matrix between the standard targets, and obtaining its degree information matrix D, as shown in formula (3); combining the degree information matrix D with the adjacency matrix E to obtain the Laplacian matrix L, as shown in formula (4), where the Laplacian matrix L is a symmetric matrix;

[0063]

[0064] L=D -1 / 2 (DE)D -1 / 2 Formula (4)

[0065] Perform SVD decomposition on the Laplacian matrix L to obtain the eigenvalue λ and eigenvector v; calculate the eigenvector z = D 1 / 2 v, classify the samples i corresponding to the vectors less than 0 in the transformed eigenvector z into one category, recalculate the energy matrix E' and Laplacian matrix L' and eigenvector z' between the samples i' corresponding to the vectors greater than 0, and classify the samples corresponding to the components less than 0 in the eigenvector z' into a new category, and thus recursively until there is no longer a component greater than 0 in the eigenvector z' or the dimension of the eigenvector z' is 1, the recursion ends, and the targets divided each time form a new target group C (c1, c2, ..., c k ), and finally the number of groups k in the entire queue is obtained.

[0066] The beneficial effects achieved by the present invention are as follows: the present invention aims to solve the technical demand for modeling the problem that the existing technology does not cluster and distinguish the same group of targets in the passenger flow and queues, and uses more complex and accurate computer vision algorithms and systems to help Internet platforms improve operational efficiency, dynamically understand customer information, and perform monocular vision target detection and tracking, including: obtaining N consecutive frames of images, using a convolutional neural network-based method to detect the position of the head and shoulders of people in each frame of image, using the DeepSort tracking algorithm to track the detected multiple targets to obtain a target sequence T; extracting high-level features of each target in T, the high-level features including the trajectory features, position features, time features, and speed features of the extracted targets; and performing calculations of the adjacency matrix E of the extracted targets, including: calculating the trajectory similarity matrix E c , position similarity matrix E l , the temporal similarity matrix E t , and the speed similarity matrix E s, execute the calculation of the actual queue group number set of N frames of images, including: calculating the Laplacian matrix L of the adjacency matrix E of the extracted target, using the graph cut algorithm to obtain the optimal clustering decomposition, and finally outputting the actual queue group number set, using computer vision technology to perform passenger flow statistics and counting, which has great benefits for the video surveillance industry; secondly, the present invention proposes for the first time to decompose and cluster the targets in the picture through feature extraction, target transformation and graph theory methods, and calculate the actual queue group number, which plays an important role in many actual scenarios, and solves the technical blind spot in the existing technology that there is no clustering and differentiation of passenger flow and targets in the same group in the queue. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a flow chart of a queue counting method based on the fusion of multiple information features of the present invention. DETAILED DESCRIPTION

[0068] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0069] Example 1: Figure 1 As shown, the present invention proposes a queue counting method based on the fusion of multiple information features, comprising the following steps:

[0070] Step SS1: Performing monocular target detection and tracking, including: designing a cache queue T to store N consecutive frames of images for subsequent detection, tracking, feature extraction, and adjacency matrix calculation; using a convolutional neural network-based method to detect the position of the head and shoulders of the person in each frame; using the DeepSort tracking algorithm to track multiple targets; obtaining the position information of each head in each frame as the target sequence T detected in the N frames; the size of N is set according to different scenarios and is generally set between 150 and 250;

[0071] Step SS2: extracting high-level features of the target using the position information of each tracked head in each frame image, wherein the high-level features include trajectory features, position features, time features, and speed features of the target;

[0072] Step SS3: Execute the calculation of the adjacency matrix E of the extracted target, including: calculating the trajectory similarity matrix E c , position similarity matrix E l , the temporal similarity matrix E t , and the speed similarity matrix E s , E is calculated as follows,

[0073]

[0074] Where E={e ij} is a normalized n×n symmetric matrix that represents the correlation between each extracted target, that is, the probability of belonging to the same group of targets, e ij The larger the value, the higher the probability that target i and target j belong to the same group of targets. n represents the number of detected targets, and w c represents the weight of the trajectory similarity matrix; w l Represents the weight of the position similarity matrix; w t represents the weight of the temporal similarity matrix; w s represents the weight of the speed similarity matrix;

[0075] Step SS4: Execute the calculation of the actual queued group number set of N frames of images, including: calculating the Laplacian matrix L of the adjacency matrix E of the extracted target, using the graph cut algorithm to obtain the optimal cluster decomposition, and finally outputting the actual queued group number set.

[0076] As a preferred embodiment, step SS1 specifically includes:

[0077] The method based on convolutional neural network is used to detect the position of the head and shoulders of a person in the picture. Specifically, a 19-layer convolutional neural network is used to detect the target in each frame image, and only the part above the shoulders to the top of the head is detected, and the target frame information l(u p ,v p ,w p ,h p ), where (u p ,v p ) is the center point coordinate, w p and h p Represent the width and height of the target box respectively;

[0078] The method of tracking multiple detected targets using the DeepSort tracking algorithm specifically includes: finding the matching box of the next frame for the target detected in each frame image; if no matching box is found, the current frame is used as the starting point to be counted into the trajectory; and the target sequence detected in N frames of image is finally output as:

[0079] T={t1,t2,t i ,...,t n},

[0080] t i ={l s ,l s+1 ,...,l e},1≤s≤e≤N;

[0081] Where n represents the number of detected targets and N represents the number of frames.

[0082] As a preferred embodiment, the step SS2 specifically includes: detecting the i-th target t in the target sequence T detected in the N frames of image. i , calculate the following high-level features:

[0083] Trajectory feature A that characterizes the target's movement over time c , A c ={p i}, Any detected target will inevitably leave a track in each frame of the image;

[0084] Position feature B that represents the target position c , B c Represents trajectory feature A c The first coordinate in

[0085] Time feature D c , D c It is a number between 1 and N, indicating the frame in which the target begins to appear on the screen;

[0086] Speed ​​characteristic S c , according to the trajectory feature A c The calculated number is shown in formula (2):

[0087]

[0088] Among them, Ω represents the Euclidean distance, are the coordinates of the i-th target in the target sequence T.

[0089] As a preferred embodiment, the step SS3 specifically includes: the adjacency matrix E is composed of the trajectory similarity matrix E c , position similarity matrix E l , time similarity matrix E t and speed similarity matrix E s Weighted, where:

[0090] Trajectory similarity matrix E c ={1-a ij}, a ij It is the sum of the squared differences between the trajectory features of any target i and the trajectory features of target j, normalized to 0 to 1. This is because the smaller the trajectory difference between two targets, the greater the probability that they belong to the same group;

[0091] Position similarity matrix E l ={1-l ij}, l ijThe Euclidean distance between the positional feature value of any target i and the positional feature value of target j is normalized from 0 to 1. This is because the closer the positions of two objects are, the greater the probability that they belong to the same group.

[0092] Temporal similarity matrix E t ={1-t ij}, t ij is the L1 distance between the time feature value of any target i and the time feature value of target j. This is because the closer the time of appearance of two targets is, the greater the probability that they belong to the same group.

[0093] Speed ​​similarity matrix E s ={1-s ij}, s ij is the L1 distance between the speed of any target i and the speed of target j, normalized to 0 to 1. This is because the closer the speeds of two targets are, the greater the probability that they belong to the same group.

[0094] As a preferred embodiment, step SS4 specifically includes:

[0095] Step SS41: Using the adjacency matrix E as the relationship matrix between the standard targets, the degree information matrix D is obtained, as shown in formula (3); the degree information matrix D is combined with the adjacency matrix E to obtain the Laplacian matrix L, as shown in formula (4). The Laplacian matrix L is a symmetric matrix.

[0096]

[0097] L=D -1 / 2 (DE)D -1 / 2 Formula (4)

[0098] Step SS42: Perform SVD decomposition on the Laplacian matrix L to obtain the eigenvalue λ and eigenvector v; calculate the eigenvector z = D 1 / 2 v, classify the samples i corresponding to the vectors less than 0 in the transformed eigenvector z into one category, recalculate the energy matrix E' and Laplacian matrix L' and eigenvector z' between the samples i' corresponding to the vectors greater than 0, and classify the samples corresponding to the components less than 0 in the eigenvector z' into a new category, and thus recursively until there is no longer a component greater than 0 in the eigenvector z' or the dimension of the eigenvector z' is 1, the recursion ends, and the targets divided each time form a new target group C (c1, c2, ..., c k ), and finally the number of groups k in the entire queue is obtained.

[0099] The present invention also proposes a queue counting system based on the fusion of multiple information features, including:

[0100] The target detection and tracking module specifically performs the following steps: performing monocular target detection and tracking, including acquiring N frames of images, detecting the position of the head and shoulders of people in each frame using a convolutional neural network-based method, tracking the detected multiple targets using the DeepSort tracking algorithm, and obtaining the position information of each head in each frame as a sequence T of targets detected in the N frames of images;

[0101] The target feature extraction module specifically performs the following operations: using the position information of each tracked head in each frame of the image as the extraction target, obtaining the high-level features of the extracted target, which include the trajectory features, position features, time features, and speed features of the extracted target;

[0102] The adjacency matrix calculation module specifically performs the following operations: Calculate the adjacency matrix E of the extracted target, including: Calculate the trajectory similarity matrix E c , position similarity matrix E l , the temporal similarity matrix E t , and the speed similarity matrix E s , E is calculated as follows,

[0103]

[0104] Where E={e ij} is a normalized n×n symmetric matrix that represents the correlation between each extracted target, that is, the probability of belonging to the same group of targets, e ij The larger the value, the higher the probability that target i and target j belong to the same group of targets. n represents the number of detected targets, and w c represents the weight of the trajectory similarity matrix; w l Represents the weight of the position similarity matrix; w t represents the weight of the temporal similarity matrix; w s represents the weight of the speed similarity matrix;

[0105] Calculate the actual number of groups module: Execute the calculation of the actual number of groups in the queue of N frames of images, including: calculating the Laplacian matrix L of the adjacency matrix E of the extracted target, using the graph cut algorithm to obtain the optimal cluster decomposition, and finally outputting the actual number of groups in the queue.

[0106] As a preferred embodiment, the target detection and tracking module specifically includes:

[0107] The method based on convolutional neural network is used to detect the position of the head and shoulders of a person in the picture, which specifically includes: using a 19-layer convolutional neural network to perform target detection in each frame image, detecting the part above the shoulders to the top of the head, and outputting target frame information l(u p ,vp ,w p ,h p ), where (u p ,v p ) is the center point coordinate, w p and h p Represent the width and height of the target box respectively;

[0108] The method of tracking multiple detected targets using the DeepSort tracking algorithm specifically includes: finding the matching box of the next frame for the target detected in each frame image; if no matching box is found, the current frame is used as the starting point to be counted into the trajectory; and the target sequence detected in N frames of image is finally output as:

[0109] T={t1,t2,t i ,...,t n},

[0110] t i ={l s ,l s+1 ,...,l e},1≤s≤e≤N;

[0111] Where n represents the number of detected targets and N represents the number of frames.

[0112] As a preferred embodiment, the target feature extraction module specifically includes: for the i-th target t in the target sequence T detected in the N frames of image, i , calculate the following high-level features:

[0113] Trajectory feature A that characterizes the target's movement over time c , A c ={p i},

[0114] Position feature B that represents the target position c , B c Represents trajectory feature A c The first coordinate in

[0115] Time feature D c , D c It is a number between 1 and N, indicating the frame in which the target begins to appear on the screen;

[0116] Speed ​​characteristic S c , according to the trajectory feature A c The calculated number is shown in formula (2):

[0117]

[0118] Among them, Ω represents the Euclidean distance, are the coordinates of the i-th target in the target sequence T.

[0119] As a preferred embodiment, the adjacency matrix calculation module specifically includes: the adjacency matrix E is composed of the trajectory similarity matrix E c , position similarity matrix E l , time similarity matrix E t and speed similarity matrix E s Weighted, where:

[0120] Trajectory similarity matrix E c ={1-a ij}, a ij It is the sum of the squared differences between the trajectory characteristics of any target i and the trajectory characteristics of target j, normalized to 0 to 1. The smaller the trajectory difference between the two targets, the greater the probability that they belong to the same group;

[0121] Position similarity matrix E l ={1-l ij}, l ij The Euclidean distance between the positional feature values ​​of any target i and the positional feature values ​​of target j is normalized from 0 to 1. The closer the positions of two targets are, the greater the probability that they belong to the same group.

[0122] Temporal similarity matrix E t ={1-t ij}, t ij It is the L1 distance between the time characteristic value of any target i and the time characteristic value of target j. The closer the time of appearance of two targets is, the greater the probability that they belong to the same group.

[0123] Speed ​​similarity matrix E s ={1-s ij}, s ij It is the L1 distance between the speed of any target i and the speed of target j, normalized to 0 to 1. The closer the speeds of two targets are, the greater the probability that they belong to the same group.

[0124] As a preferred embodiment, the module for calculating the actual number of groups specifically includes: taking the adjacency matrix E as the relationship matrix between the standard targets, and obtaining its degree information matrix D, as shown in formula (3); combining the degree information matrix D with the adjacency matrix E to obtain the Laplacian matrix L, as shown in formula (4), where the Laplacian matrix L is a symmetric matrix;

[0125]

[0126] L=D -1 / 2 (DE)D -1 / 2 Formula (4)

[0127] Perform SVD decomposition on the Laplacian matrix L to obtain the eigenvalue λ and eigenvector v; calculate the eigenvector z = D 1 / 2 v, classify the samples i corresponding to the vectors less than 0 in the transformed eigenvector z into one category, recalculate the energy matrix E' and Laplacian matrix L' and eigenvector z' between the samples i' corresponding to the vectors greater than 0, and classify the samples corresponding to the components less than 0 in the eigenvector z' into a new category, and thus recursively until there is no longer a component greater than 0 in the eigenvector z' or the dimension of the eigenvector z' is 1, the recursion ends, and the targets divided each time form a new target group C (c1, c2, ..., c k ), and finally the number of groups k in the entire queue is obtained.

[0128] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0129] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A queue counting method based on the fusion of multiple information features, characterized in that: The steps include: Step SS1: Monocular vision target detection and tracking, including: obtaining continuous N Frame image, use convolutional neural network to detect the position of the head and shoulders of the people in each frame, use DeepSort tracking algorithm to track multiple targets, obtain the position information of each head in each frame image as the extraction target, and generate target sequence T ; Step SS2: Target sequence T Performing behavioral analysis on each sequence in the process to obtain high-level features of the extracted target, wherein the high-level features include trajectory features, position features, time features, and speed features of the extracted target; Step SS3: Calculate the adjacency matrix of the extracted target E , including: calculating the trajectory similarity matrix , position similarity matrix , the temporal similarity matrix , and the speed similarity matrix , E The calculation method is as follows, ； in, It is a normalized Symmetric matrix, representing the correlation between each extracted target, that is, the probability of belonging to the same group of targets, The larger the target, the i and goals j The higher the probability of belonging to the same group of targets, n Indicates the number of detected targets. Represents the weight of the trajectory similarity matrix; Represents the weight of the position similarity matrix; represents the weight of the temporal similarity matrix; represents the weight of the speed similarity matrix; Step SS4: Calculation N The actual number of queued groups of frame images, including: the adjacency matrix of the extracted target E Calculate its Laplacian matrix L , using the graph cut algorithm to obtain the optimal cluster decomposition, and finally outputting the actual number of queued groups; the step SS4 specifically includes: Step SS41: Convert the adjacency matrix E As the relationship matrix between standard targets, the degree information matrix is ​​obtained D , see formula (3); degree information matrix D Joint adjacency matrix E Obtain the Laplacian matrix L , see formula (4), Laplacian matrix L is a symmetric matrix; ； Step SS42: Laplacian matrix L Perform SVD decomposition to obtain eigenvalues and eigenvectors v ; Calculate the eigenvector , the transformed feature vector z The samples corresponding to the vectors less than 0 in i Classify into one category, for samples corresponding to vectors greater than 0 Recalculate the sample The energy matrix between and the Laplacian matrix and the eigenvector , and the eigenvector The samples whose components are less than 0 are divided into a new category, and the eigenvectors are recursively divided into two categories. There are no longer any components or eigenvectors greater than 0 in When the dimension is 1, the recursion ends, and the targets divided each time form a new target group C ( c 1 , c 2 , ..., c k ), and finally get the number of groups in the entire queue k .

2. A queue counting method based on fusion of multiple information features according to claim 1, characterized in that: The step SS1 specifically includes: The method based on convolutional neural network is used to detect the position of the head and shoulders of a person in the picture. Specifically, a 19-layer convolutional neural network is used to detect the target in each frame image, detect the part above the shoulders to the top of the head, and output the target frame information. ,in are the center point coordinates, and Represent the width and height of the target box respectively; The method of tracking multiple detected targets using DeepSort tracking algorithm specifically includes: finding the matching box of the next frame for the target detected in each frame image, and if no matching box is found, the current frame is used as the starting point to be counted in the trajectory, and finally output N The target sequence detected in the frame image is: ， ； in, n Indicates the number of detected targets. N Indicates the frame number.

3. The queue counting method based on fusion of multiple information features according to claim 1 is characterized in that: The step SS2 specifically includes: N Target sequence detected in frame image T Middle goals , calculate the following high-level features: Characterize the trajectory characteristics of the target's movement over time , , ; Positional features that characterize the target location , Represents trajectory characteristics The first coordinate in Time characteristics , From 1 to N A number between , indicating the frame in which the target begins to appear in the picture; Speed ​​characteristics , according to the trajectory characteristics The calculated number is shown in formula (2): ； in, represents the Euclidean distance, is the target sequence T Middle The coordinates of the target.

4. The queue counting method based on fusion of multiple information features according to claim 1 is characterized in that: The step SS3 specifically includes: adjacency matrix E By the trajectory similarity matrix , position similarity matrix , temporal similarity matrix and speed similarity matrix Weighted, where: Trajectory similarity matrix ={1- }, Any target i Trajectory characteristics and goals j The sum of the squared differences of the trajectory features is normalized to 0 to 1. The smaller the trajectory difference between the two targets, the greater the probability that they belong to the same group; Position similarity matrix ={1- }, Any target i The position feature value and target j The Euclidean distance of the position feature values ​​is normalized from 0 to 1. The closer the positions of two targets are, the greater the probability that they belong to the same group. Temporal similarity matrix ={1- }, Any target i Time characteristic values ​​and targets j The time characteristic value of Distance: the closer the time when two targets appear, the greater the probability that they belong to the same group; Speed ​​similarity matrix ={1- }, Any target i Speed ​​and target j Speed Distance, normalized to 0 to 1, the closer the speeds of two targets are, the greater the probability that they belong to the same group.

5. A queue counting system based on the fusion of multiple information features, characterized in that: include: The target detection and tracking module specifically performs: monocular vision target detection and tracking, including: obtaining continuous N Frame image, use convolutional neural network to detect the position of the head and shoulders of the people in each frame, use DeepSort tracking algorithm to track multiple targets, obtain the position information of each head in each frame image as the extraction target, and generate target sequence T ; Extract target feature module, specifically: T Performing behavioral analysis on each sequence in the process to obtain high-level features of the extracted target, wherein the high-level features include trajectory features, position features, time features, and speed features of the extracted target; Calculate the adjacency matrix module, specifically perform: calculate the adjacency matrix of the extracted target E , including: calculating the trajectory similarity matrix , position similarity matrix , the temporal similarity matrix , and the speed similarity matrix , E The calculation method is as follows, ； in, It is a normalized Symmetric matrix, representing the correlation between each extracted target, that is, the probability of belonging to the same group of targets, The larger the target, the i and goals j The higher the probability of belonging to the same group of targets, n Indicates the number of detected targets. Represents the weight of the trajectory similarity matrix; Represents the weight of the position similarity matrix; represents the weight of the temporal similarity matrix; represents the weight of the speed similarity matrix; Calculate the actual number of groups module: Calculate N The actual number of queued groups of frame images, including: the adjacency matrix of the extracted target E Calculate its Laplacian matrix L , use the graph cut algorithm to obtain the optimal cluster decomposition, and finally output the actual number of queued groups; the module for calculating the actual number of groups specifically includes: converting the adjacency matrix E As the relationship matrix between standard targets, the degree information matrix is ​​obtained D , see formula (3); degree information matrix D Joint adjacency matrix E Obtain the Laplacian matrix L , see formula (4), Laplacian matrix L is a symmetric matrix; ； Laplacian matrix L Perform SVD decomposition to obtain eigenvalues and eigenvectors v ; Calculate the eigenvector , the transformed feature vector z The samples corresponding to the vectors less than 0 in i Classify into one category, for samples corresponding to vectors greater than 0 Recalculate the sample The energy matrix between and the Laplacian matrix and the eigenvector , and the eigenvector The samples whose components are less than 0 are divided into a new category, and the eigenvectors are recursively divided into two categories. There are no longer any components or eigenvectors greater than 0 in When the dimension is 1, the recursion ends, and the targets divided each time form a new target group C ( c 1 , c 2 , ..., c k ), and finally get the number of groups in the entire queue k .

6. A queue counting system based on fusion of multiple information features according to claim 5, characterized in that: The target detection and tracking module specifically includes: The method based on convolutional neural network is used to detect the position of the head and shoulders of a person in the picture. Specifically, a 19-layer convolutional neural network is used to detect the target in each frame image, detect the part above the shoulders to the top of the head, and output the target frame information. ,in are the center point coordinates, and Represent the width and height of the target box respectively; The DeepSort tracking algorithm is used to track the detected multiple targets, specifically including: finding the matching box of the next frame for the target detected in each frame image, if not found, then taking the current frame as the starting point and counting it into the trajectory, and finally outputting N The target sequence detected in the frame image is: ， ； in, n Indicates the number of detected targets. N Indicates the frame number.

7. The queue counting system based on fusion of multiple information features according to claim 5, characterized in that: The target feature extraction module specifically includes: N Target sequence detected in frame image T Middle goals , calculate the following high-level features: Characterize the trajectory characteristics of the target's movement over time , , ; Positional features that characterize the target location , Represents trajectory characteristics The first coordinate in Time characteristics , From 1 to N A number between , indicating the frame in which the target begins to appear in the picture; Speed ​​characteristics , according to the trajectory characteristics The calculated number is shown in formula (2): ； in, represents the Euclidean distance, p i is the first The coordinates of the target.

8. The queue counting system based on fusion of multiple information features according to claim 5, characterized in that: The adjacency matrix calculation module specifically includes: E By the trajectory similarity matrix , position similarity matrix , temporal similarity matrix and speed similarity matrix Weighted, where: Trajectory similarity matrix ={1- }, Any target i Trajectory characteristics and goals j The sum of the squared differences of the trajectory features is normalized to 0 to 1. The smaller the trajectory difference between the two targets, the greater the probability that they belong to the same group; Position similarity matrix ={1- }, Any target i The position feature value and target j The Euclidean distance of the position feature values ​​is normalized from 0 to 1. The closer the positions of two targets are, the greater the probability that they belong to the same group. Temporal similarity matrix ={1- }, Any target i Time characteristic values ​​and targets j The time characteristic value of Distance: the closer the time when two targets appear, the greater the probability that they belong to the same group; Speed ​​similarity matrix ={1- }, Any target i Speed ​​and target j Speed Distance, normalized to 0 to 1, the closer the speeds of two targets are, the greater the probability that they belong to the same group.

Images