A method, device and system for detecting queuing time

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a queue duration detection method, device and system. The method comprises: obtaining a set of video image frames detected at a starting point and its corresponding facial feature set and a set of video image frames detected at a target point to be detected and its corresponding facial feature set; judging whether the facial feature set corresponding to the set of video image frames detected at the target point to be detected matches the facial feature set corresponding to the set of video image frames detected at the starting point to be detected; if they match, obtaining the time difference between the target point to be detected and the starting point to be detected, and determining the current time required to queue. The technical solution of the present invention can not only count the passenger queue time in real time, but also match the facial features to make the detection result more accurate. The present invention only needs to be implemented by using two video acquisition devices, so that the use and maintenance cost of the system is not high.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a queue duration detection method, device and system. Background Art

[0002] In recent years, deep learning technologies represented by convolutional neural networks have been widely used in various artificial intelligence tasks, such as object classification, face recognition, and pedestrian identity re-identification. The breakthrough progress made by convolutional neural networks in these problems stems from the powerful expression ability brought by their hierarchical learning structure. In the application scenario of passenger flow statistics, passenger queue time is a very important parameter. Since the passenger flow speed is unknown, there is an error in estimating the queue time. The passenger flow speed is usually set based on field experience. If the passenger flow speed changes, it is often impossible to update it in time.

[0003] Since the waiting time of customers is a very important information, a plan for customer flow diversion can be made by estimating the waiting time of customers. If the customer flow is too crowded, it is easy for customers to give up queuing, resulting in a decrease in the transaction rate. In places such as subways and parks, it may even cause stampedes.

[0004] However, estimating the waiting time of customers is also a very difficult problem. First, it is difficult to collect information about customer flow. The traditional method is to place gates at the entrance, count the number of customers, and then estimate the waiting time. Secondly, customers may make various changes while queuing, such as cutting in line or giving up. In addition, when the flow of customers is very crowded, there is a problem of occlusion between customers, resulting in incomplete information.

[0005] In the process of implementing the prior art, the inventors found that the prior art has at least the following technical problems:

[0006] The existing technology does not accurately detect the passenger queue time, and cannot make real-time judgments based on actual conditions to give real-time queue time. It also requires the cooperation of multiple hardware devices, which makes the use and maintenance costs of the system high. Summary of the invention

[0007] The purpose of the present invention is to provide a queue time detection method, device and system to overcome the defects in the prior art that the passenger queue time detection is not accurate enough, and cannot be judged in real time according to the actual situation to give the real-time queue time, and requires the cooperation of multiple hardware devices to achieve it, which makes the use and maintenance cost of the system high.

[0008] In order to solve the above technical problems, an embodiment of the present invention provides a queuing duration detection method, comprising:

[0009] Obtain a set of video image frames to be detected at the starting point and its corresponding set of facial features and a set of video image frames to be detected at the target point to be detected and its corresponding set of facial features;

[0010] Determine whether the facial feature set corresponding to the set of video image frames detected at the target point to be detected matches the facial feature set corresponding to the set of video image frames detected at the starting point to be detected;

[0011] If they match, the time difference between the target point to be detected and the starting point to be detected is obtained to determine the current queuing time.

[0012] An embodiment of the present invention further provides a queuing duration detection device, including.

[0013] An information acquisition unit, used to acquire a set of video image frames to be detected at a starting point and a set of facial features corresponding thereto, and a set of video image frames to be detected at a target point to be detected and a set of facial features corresponding thereto;

[0014] A judging unit, used to judge whether the facial feature set corresponding to the set of video image frames detected at the target point to be detected matches the facial feature set corresponding to the set of video image frames detected at the starting point to be detected;

[0015] The queuing time determination unit is used to obtain the time difference between the target point to be detected and the starting point to be detected if there is a match, and determine the current queuing time.

[0016] An embodiment of the present invention further provides a queuing duration detection system, comprising: the queuing duration detection device as described above.

[0017] The present invention provides a method, device and system for detecting the length of a queue. The present invention obtains a set of video image frames detected at a starting point and its corresponding set of facial features and a set of video image frames detected at a target point to be detected and its corresponding set of facial features; determines whether the set of facial features corresponding to the set of video image frames detected at the target point to be detected matches the set of facial features corresponding to the set of video image frames detected at the starting point to be detected; if they match, obtains the time difference between the target point to be detected and the starting point to be detected, and determines the current time required to queue. The technical solution of the present invention can not only count the passenger queue time in real time, but also match the facial features to make the detection result more accurate. The present invention only needs to be implemented by using two video acquisition devices, so that the use and maintenance costs of the system are not high. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of a queuing duration detection method provided by an embodiment of the present invention;

[0019] Figure 2 It is a structural schematic diagram of a queuing duration detection device provided by an embodiment of the present invention;

[0020] Figure 3 It is a queue diagram of a queue duration detection method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the present invention clearer, the following will be described in detail with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that in the various embodiments of the present invention, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed for protection in the claims of the present application can be implemented.

[0022] The first embodiment of the present invention relates to a method for detecting the queuing time. The specific process is as follows: Figure 1 The method comprises:

[0023] 101: Obtain a set of video image frames to be detected at a starting point and a set of facial features corresponding thereto and a set of video image frames to be detected at a target point to be detected and a set of facial features corresponding thereto;

[0024] 102: Determine whether the facial feature set corresponding to the set of video image frames detected at the target point to be detected matches the facial feature set corresponding to the set of video image frames detected at the starting point to be detected;

[0025] 103: If they match, the time difference between the target point to be detected and the starting point to be detected is obtained to determine the current queuing time.

[0026] The step of obtaining a set of video image frames to be detected at the starting point and a set of facial features corresponding thereto and a set of video image frames to be detected at the target point to be detected and a set of facial features corresponding thereto comprises:

[0027] Set the discrete time point difference value;

[0028] Obtain a set of video image frames to be detected at the starting point and a set of corresponding facial features;

[0029] The face feature set corresponding to the set of video image frames detected at the target point to be detected is determined according to the discrete time point difference and the face feature set corresponding to the set of video image frames detected at the starting point to be detected.

[0030] The step of determining whether the facial feature set corresponding to the set of video image frames detected at the target point to be detected matches the facial feature set corresponding to the set of video image frames detected at the starting point to be detected comprises:

[0031] Set the similarity threshold of the facial feature set;

[0032] Obtain a set similarity distance between feature vectors of a face feature set corresponding to a set of video image frames detected at the target point to be detected and a face feature set corresponding to a set of video image frames detected at the starting point to be detected; the set similarity distance includes: improved Hausdorff distance, Jaccard distance or Occhi coefficient.

[0033] Determining whether the set similarity distance is less than the face feature set similarity threshold;

[0034] If the set similarity distance is less than the facial feature set similarity threshold, it is considered that they are different snapshots of the same group of people and clustering is performed.

[0035] Among them, the method includes:

[0036] Set correction intensity threshold;

[0037] According to the modified intensity threshold, the modified current queuing time is obtained.

[0038] For example, suppose the queue time is continuously changing. The queue time difference determined above is T0. The time difference between the two previous calculations is T -2 and T -1 The corrected queue time is

[0039] T=(1-α)T0+α(2×T -1 -T -2 )

[0040] Among them, α is the correction strength threshold. The larger α is, the more the corrected queue time is constrained by the previous state. Here, α can be set to 0.3. After adding the correction term, the estimated value of the queue time is more stable, eliminating the influence of a small number of outliers.

[0041] Based on the above Figure 1 In the specific embodiment shown, the preset image parameters are as follows: the preset similarity threshold of the face feature set is 0.2; using the deep learning algorithm, the face in the image is identified, that is, the face in the video image frame set to be detected at the starting point and the face in the video image frame set to be detected at the target point to be detected. The face feature extracted by the model is a 256-dimensional vector, denoted as f. For the queue area such as Figure 3 The set of all extracted features in the image at the i-th moment at the entrance shown (the set of video image frames to be detected at the starting point) is denoted as F i ={f i,k|k=1,2,3...}, k is the face number in the image. For all the extracted feature sets in the image at the jth moment at the exit (i.e., the image in the video image frame set at the target point to be detected), record it as F' j ={f j,k |k=1,2,3...}, then the specific implementation process of the queuing time detection method is as follows Figure 2 As shown, the specific process is as follows:

[0042] 1. Calculate the matching degree of face sets at time i and time j

[0043] For F i ={f i,k |k=1,2,3...} and F' j ={f j,k |k=1,2,3...}, calculate the distance H of the face set.

[0044] The method we are familiar with for calculating the matching degree of two sets is the Hausdorff distance. That is:

[0045] H(A,B)=max(h(A,B),h(B,A))

[0046] in:

[0047]

[0048]

[0049] ‖·‖ is the distance form between point set A and point set B (can be L1 or L2 distance), a, b are the feature vectors in sets A and B.

[0050] The Hausdorff distance can measure the distance between two point sets, but it is greatly affected by outliers. That is, even if most people in F and F' correspond, as long as there are outliers in both point sets, the Hausdorff distance will be a large value, which cannot express the fact that most points in F and F' are close. In practical applications, this situation is very likely to occur. That is, when we match faces captured by two cameras, there will be unmatched people almost all the time. Some people are in the camera at the entrance of the area but not captured by the exit camera, or vice versa. This will have a great impact on our calculations.

[0051] Therefore, an improved Hausdorff distance metric is proposed, which can accurately describe the similarity between two point sets.

[0052] H'(A,B)=max(h(A,B),h(B,A))

[0053] in

[0054]

[0055]

[0056] The sigmoid function is used to reduce the impact of outliers. Because we calculate the Euclidean distance between two vectors, ||ab||∈(0,+∞). Even if we average all distances, it will still cause adverse effects when the distance is large. Use sigmoid for normalization.

[0057] For some a∈A, let but

[0058]

[0059] Another benefit of using sigmoid is that the value range of H'(A,B) we calculated becomes (0,1), which makes it easier for us to design the threshold in the next step.

[0060] 2. Time point sampling strategy: if the starting point to be detected is i, then the time sampling method for the target point to be detected is j is as follows:

[0061] The two cameras capture data in real time, so there are two sets F and F' at each moment. Obviously, it is unrealistic to directly match the sets of the two cameras by brute force. We designed a time point sampling strategy to sample the moments i and j of the two cameras and compare the improved Hausdorff distance of the sets F and F' at moments i and j. We specify the threshold as 0.2. When H'(F,F') is less than 0.2, it is considered that moments i and j match, and ij is taken as the estimated queue time.

[0062] The sampling strategy at the following time points:

[0063] For example: take a fixed time point i every 10 minutes;

[0064] Let j be 5s, 10s, ..., 50s after time i;

[0065] Calculate H'(F,F')

[0066] If H'(F,F')<0.2, the corresponding time i and j are the required values.

[0067] Otherwise, j is taken as 5m, 10m, ..., 50m after time i;

[0068] Calculate H'(F,F')

[0069] If H'(F,F')<0.5, take j every 1m before and after the corresponding j, and take 10 new j in total;

[0070] Calculate H'(F,F')

[0071] When H'(F,F') takes the minimum value, the corresponding time i and j are the desired

[0072] When the queuing speed is slow, even if we use a slightly larger time interval, we can still find the same group of people as at time i. Therefore, we can use less computing resources and get reasonable matching results.

[0073] The second embodiment of the present invention relates to a queuing time detection device, such as Figure 2 As shown, the device comprises:

[0074] An information acquisition unit 201 is used to acquire a set of video image frames to be detected at a starting point and a set of facial features corresponding thereto and a set of video image frames to be detected at a target point to be detected and a set of facial features corresponding thereto;

[0075] A judging unit 202 is used to judge whether the facial feature set corresponding to the set of video image frames detected at the target point to be detected matches the facial feature set corresponding to the set of video image frames detected at the starting point to be detected;

[0076] The queuing time determining unit 203 is used to obtain the time difference between the target point to be detected and the starting point to be detected if there is a match, and determine the current queuing time.

[0077] Among them, the information acquisition unit is also used to set the discrete time point difference value; obtain the set of video image frames detected at the starting point and the set of facial features corresponding to them; determine the set of facial features corresponding to the set of video image frames detected at the target point to be detected based on the discrete time point difference value and the set of facial features corresponding to the set of video image frames detected at the starting point.

[0078] Among them, the judgment unit is also used to set a similarity threshold of a facial feature set; obtain the set similarity distance of the feature vectors of the facial feature set corresponding to the set of video image frames detected at the target point to be detected and the facial feature set corresponding to the set of video image frames detected at the starting point to be detected; judge whether the set similarity distance is less than the facial feature set similarity threshold; if the set similarity distance is less than the facial feature set similarity threshold, it is considered that they are different shots of the same group of people and clustering is performed.

[0079] The device further comprises:

[0080] A threshold setting unit, used to set a correction intensity threshold;

[0081] The correction unit is used to obtain the corrected current queuing time according to the correction strength threshold.

[0082] The third embodiment of the present invention relates to a queuing duration detection system, which includes: the queuing duration detection device as described above.

[0083] The present invention provides a method, device and system for detecting the length of a queue. The present invention obtains a set of video image frames detected at a starting point and its corresponding set of facial features and a set of video image frames detected at a target point to be detected and its corresponding set of facial features; determines whether the set of facial features corresponding to the set of video image frames detected at the target point to be detected matches the set of facial features corresponding to the set of video image frames detected at the starting point to be detected; if they match, obtains the time difference between the target point to be detected and the starting point to be detected, and determines the current time required to queue. The technical solution of the present invention can not only count the passenger queue time in real time, but also match the facial features to make the detection result more accurate. The present invention only needs to be implemented by using two video acquisition devices, so that the use and maintenance costs of the system are not high.

[0084] It is not difficult to find that the above first embodiment is a device embodiment corresponding to the second embodiment, and the second embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.

[0085] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other units in this embodiment.

[0086] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0087] For the convenience of description, the above device is described by dividing the functions into various units / modules. Of course, when implementing the present invention, the functions of each unit / module can be implemented in the same or multiple software and / or hardware.

[0088] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM). The above is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for detecting the length of queue, It is characterized in that include: Obtain a set of video image frames to be detected at the starting point and its corresponding set of facial features and a set of video image frames to be detected at the target point to be detected and its corresponding set of facial features; Determine whether the facial feature set corresponding to the set of video image frames detected at the target point to be detected matches the facial feature set corresponding to the set of video image frames detected at the starting point to be detected; If they match, the time difference between the target point to be detected and the starting point to be detected is obtained to determine the current queuing time; Set correction intensity threshold; The current queuing time is corrected according to the correction intensity threshold and the queuing time determined twice before, and the corrected current queuing time is obtained, which is as follows: T=(1-α)T 0 +α(2×T -1 -T -2 ) Among them, α is the modified intensity threshold, T -2 and T -1 is the queue time determined in the first two times, T 0 The current queue time; The step of determining whether the facial feature set corresponding to the set of video image frames detected at the target point to be detected matches the facial feature set corresponding to the set of video image frames detected at the starting point to be detected, comprises: Set the similarity threshold of facial feature set; Based on the improved Hausdorff distance, the set similarity distance of the feature vectors of the face feature set corresponding to the set of video image frames detected at the target point to be detected and the face feature set corresponding to the set of video image frames detected at the starting point to be detected is obtained, as follows: H'(A,B)=max(h(A,B),h(B,A)), where In the above formula, A and B are the face feature set corresponding to the set of video image frames detected at the starting point and the face feature set corresponding to the set of video image frames detected at the target point to be detected, respectively; a and b are the feature vectors in set A and set B, respectively; ‖·‖ is the distance norm; the sigmoid function is used for normalization processing; mean() represents the average function; max() represents the maximum value function; and min() represents the minimum value function; Determining whether the set similarity distance is less than the face feature set similarity threshold; If the set similarity distance of the two sets is less than the facial feature set similarity threshold, it is considered that the two facial feature sets are from the same group of people; The time for obtaining the set of detected video image frames at the starting point to be detected and its corresponding facial feature set and the set of detected video image frames at the target point to be detected and its corresponding facial feature set is determined by a time point sampling strategy, and the difference between the corresponding time of the target point to be detected and the starting point to be detected is used as the estimated queuing time; the time point sampling strategy includes: taking a fixed time point as the first time corresponding to the starting point to be detected at regular intervals, and taking a time that is an integer multiple of five seconds after the first time as the second time corresponding to the target point to be detected, and calculating the set similarity distance of the feature vectors of the facial feature set corresponding to the set of detected video image frames obtained at each second time and the first time respectively based on the improved Hausdorff distance, if the set similarity distance is less than the first facial feature set similarity threshold, then the corresponding second time and first time are the times corresponding to the target point to be detected and the starting point to be detected respectively; otherwise, the time five minutes after the first time is taken as the time corresponding to the target point to be detected and the starting point to be detected; The moments that are integer multiples of are taken as the updated second moments corresponding to the target point to be detected, and the set similarity distances of the feature vectors of the facial feature set corresponding to the detection video image frame set obtained at each updated second moment and the first moment are calculated based on the improved Hausdorff distance. If the set similarity distance is less than the second facial feature set similarity threshold, and the second facial feature set similarity threshold is greater than the first facial feature set similarity threshold, then the moments corresponding to every one minute before and after each updated second moment are taken as the re-updated second moments corresponding to the target point to be detected, and the set similarity distances of the feature vectors of the facial feature set corresponding to the detection video image frame set obtained at each re-updated second moment and the first moment are calculated based on the improved Hausdorff distance. The re-updated second moment corresponding to the minimum value in the set similarity distance and the corresponding first moment are respectively the moments corresponding to the target point to be detected and the starting point to be detected.

2. The queuing time detection method according to claim 1, It is characterized in that The step of obtaining a set of video image frames to be detected at a starting point and a set of facial features corresponding thereto and a set of video image frames to be detected at a target point to be detected and a set of facial features corresponding thereto comprises: Set the discrete time point difference value; Obtain a set of video image frames to be detected at the starting point and a set of corresponding facial features; The face feature set corresponding to the set of video image frames detected at the target point to be detected is determined according to the discrete time point difference and the face feature set corresponding to the set of video image frames detected at the starting point to be detected.

3. A queuing time detection device, It is characterized in that include: An information acquisition unit, used to acquire a set of video image frames to be detected at a starting point and a set of facial features corresponding thereto, and a set of video image frames to be detected at a target point to be detected and a set of facial features corresponding thereto; A judging unit, used to judge whether the facial feature set corresponding to the set of video image frames detected at the target point to be detected matches the facial feature set corresponding to the set of video image frames detected at the starting point to be detected; and also used to set a similarity threshold of the facial feature set; Based on the improved Hausdorff distance, the set similarity distance of the feature vectors of the face feature set corresponding to the set of video image frames detected at the target point to be detected and the face feature set corresponding to the set of video image frames detected at the starting point to be detected is obtained, as follows: H'(A,B)=max(h(A,B),h(B,A)), where In the above formula, A and B are respectively the facial feature set corresponding to the set of video image frames detected at the starting point to be detected and the facial feature set corresponding to the set of video image frames detected at the target point to be detected, a and b are respectively the feature vectors in set A and set B, ‖·‖ is the distance paradigm, the sigmoid function is used for normalization processing, mean() represents the average function, max() represents the maximum function, and min() represents the minimum function; it is determined whether the set similarity distance is less than the facial feature set similarity threshold; if the set similarity distance of the two sets is less than the facial feature set similarity threshold, it is considered that the two facial feature sets are the same group of people; A queuing time determination unit, configured to obtain the time difference between the target point to be detected and the starting point to be detected if there is a match, and determine the current queuing time; A threshold setting unit, used to set a correction intensity threshold; The correction unit is used to correct the current queuing time according to the correction strength threshold and the queuing time determined twice before, and obtain the corrected current queuing time, as shown in the following formula: T=(1-α)T 0 +α(2×T -1 -T -2 ) Among them, α is the modified intensity threshold, T -2 and T -1 is the queue time determined in the first two times, T 0 The current queue time; The queuing time determination unit is used to determine the time to obtain the detection video image frame set at the starting point to be detected and its corresponding face feature set and the detection video image frame set at the target point to be detected and its corresponding face feature set through a time point sampling strategy, and the difference between the corresponding time of the target point to be detected and the starting point to be detected is used as the estimated queuing time; the time point sampling strategy includes: taking a fixed time point as the first time corresponding to the starting point to be detected at regular intervals, and taking the time that is an integer multiple of five seconds after the first time as the second time corresponding to the target point to be detected, and calculating the set similarity distance of the feature vectors of the face feature set corresponding to the detection video image frame set obtained at each second time and the first time based on the improved Hausdorff distance, if the set similarity distance is less than the first face feature set similarity threshold, then the corresponding second time and first time are the times corresponding to the target point to be detected and the starting point to be detected respectively; otherwise, the first time is calculated. The moments that are integer multiples of five minutes thereafter are taken as the updated second moments corresponding to the target point to be detected. The set similarity distances of the feature vectors of the facial feature set corresponding to the detection video image frame set obtained at each updated second moment and the first moment are calculated based on the improved Hausdorff distance. If the set similarity distance is less than the second facial feature set similarity threshold, and the second facial feature set similarity threshold is greater than the first facial feature set similarity threshold, then the moments corresponding to every one minute before and after each updated second moment are taken as the re-updated second moments corresponding to the target point to be detected. The set similarity distances of the feature vectors of the facial feature set corresponding to the detection video image frame set obtained at each re-updated second moment and the first moment are calculated based on the improved Hausdorff distance. The re-updated second moment corresponding to the minimum value in the set similarity distance and the corresponding first moment are taken as the moments corresponding to the target point to be detected and the starting point to be detected, respectively.

4. The queuing time detection device according to claim 3, It is characterized in that The information acquisition unit is also used to set the discrete time point difference value; obtain the set of video image frames detected at the starting point to be detected and the set of facial features corresponding to them; and determine the set of facial features corresponding to the set of video image frames detected at the target point to be detected based on the discrete time point difference value and the set of facial features corresponding to the set of video image frames detected at the starting point to be detected.

5. A queuing time detection system, It is characterized in that A queuing duration detection device as claimed in claim 3 or 4.

Images