Stay duration detection method, device and server

Abstract

The present disclosure provides a stay duration detection method, device and server. The method comprises: sequentially obtaining each frame image of a video stream, detecting all persons in a first image by using a target detection algorithm, and generating a first detection box for the detected persons; processing the first image and the corresponding first detection box by using a tracking algorithm to obtain an added person set, a matched person set and an unmatched person set of the first image; if the added person set is not empty, or the matched person set is not empty and the tracking number reaches a threshold, detecting the first image by using a face detection algorithm, and generating a second detection box corresponding to a face; matching the first detection box and the second detection box of the first image; performing face recognition on the face in the second detection box to obtain person information of the face in the second detection box; and determining the stay duration of each person photographed by the video stream according to the person information detected from each frame image of the video stream.

Description

Technical Field

[0001] This disclosure relates to the field of security technology, and in particular to a method, device and server for detecting dwell time. Background Technology

[0002] With the rapid development of artificial intelligence technology, face detection and recognition, as a typical artificial intelligence technology, has been increasingly applied to the field of surveillance and security. Summary of the Invention

[0003] This disclosure provides a method, apparatus, and server for detecting dwell time. The technical solution is as follows:

[0004] This disclosure provides at least one embodiment of a method for detecting dwell time, the method comprising:

[0005] Each frame of the video stream is acquired sequentially, and all people in the first image are detected using an object detection algorithm. A first detection box is generated for the detected people. The first image is any frame of the video stream.

[0006] The first image and its corresponding first detection box are processed using a tracking algorithm to obtain a set of newly added people, a set of matched people, and a set of unmatched people in the first image.

[0007] If the newly added set of people is not empty, or the set of matched people is not empty and the number of tracking times reaches a threshold, then the face detection algorithm is used to detect the first image and generate a second detection box corresponding to the face.

[0008] Match the first detection box and the second detection box of the first image;

[0009] Perform face recognition on the face in the second detection frame to obtain the person information of the face in the second detection frame;

[0010] Based on the person information detected in each frame of the video stream, the dwell time of each person captured by the video stream is determined.

[0011] Optionally, matching the first detection box and the second detection box of the first image includes:

[0012] Obtain the vertex coordinates of the first detection box, the vertex coordinates of the second detection box, and the coordinates of the facial feature points in the second detection box in the first image;

[0013] Based on the vertex coordinates of the first detection box and the coordinates of the facial feature points in the second detection box, the inclusion relationship between each second detection box and each first detection box is determined. The inclusion relationship means that the facial feature points in the second detection box are all located within the first detection box.

[0014] If there is a one-to-one containment relationship between the first detection box and the second detection box, then it is determined that there is a one-to-one matching relationship between the first detection box and the second detection box.

[0015] If there is a one-to-many or many-to-one inclusion relationship between the first detection box and the second detection box, then the one-to-one matching relationship between the first detection box and the second detection box is determined based on the area of ​​the first detection box and the second detection box that have a one-to-many or many-to-one inclusion relationship.

[0016] Optionally, determining the one-to-one matching relationship between the first and second detection boxes based on the areas of the first and second detection boxes that have a one-to-many or many-to-one inclusion relationship includes:

[0017] The correlation coefficient between the first and second detection boxes that have a one-to-many or many-to-one inclusion relationship is calculated using the following formula:

[0018]

[0019] Among them, Face i Pedestrian represents the area of ​​the i-th second detection box. j Represents the area of ​​the j-th first detection box, Face i ∩Pedestrian j P represents the overlap area between the i-th second detection box and the j-th first detection box. ij This represents the correlation coefficient between the i-th second detection box and the j-th first detection box;

[0020] Based on the correlation coefficient between the first and second detection boxes, the one-to-one matching relationship between the first and second detection boxes is determined using the Hungarian algorithm or the KM algorithm.

[0021] Optionally, the step of performing face recognition on the face in the second detection frame to obtain the person information of the face in the second detection frame includes:

[0022] Extract the feature vector of the face in the second detection frame;

[0023] The extracted feature vectors are compared with the feature vectors of each category in the face feature database, and the Euclidean distance is calculated.

[0024] Based on the Euclidean distance between the extracted feature vector and the feature vectors of each category in the face feature database, the recognition probability of the face in the second detection frame and each category in the face feature database is determined.

[0025] Based on the recognition probabilities of the face in the second detection frame and various categories in the face feature database, the person information corresponding to the category associated with the face in the second detection frame is determined.

[0026] Optionally, determining the recognition probability of the face in the second detection frame relative to each category in the face feature database based on the Euclidean distance between the extracted feature vector and the feature vectors of each category in the face feature database includes:

[0027] The recognition probability of the face in the second detection frame and each category in the face feature database is calculated using the following formula:

[0028] P = α·P total +(1-α)·P best ;

[0029] Among them, P total P refers to the ratio of the number of feature vectors of a face in the second detection frame whose Euclidean distance to all feature vectors of the first category in the face feature database is less than the Euclidean distance threshold, to the total number of feature vectors in the first category, where the first category is any category in the face feature database; best The optimal feature probability of the face in the second detection frame; α is a coefficient;

[0030]

[0031] Where N is the total number of feature vectors of the first category, Num is the number of vectors less than the Euclidean distance threshold, d is the Euclidean distance, and threshold is the Euclidean distance threshold.

[0032]

[0033] Where, d min This refers to the minimum Euclidean distance between the feature vector of the face in the second detection frame and the feature vectors of all faces in the first category in the face feature database.

[0034] At least one embodiment of this disclosure provides a dwell time detection device, the device comprising:

[0035] The object detection algorithm module is used to sequentially acquire each frame of the video stream, detect all people in the first image using the object detection algorithm, and generate a first detection box for the detected people. The first image is any frame of the video stream.

[0036] The tracking algorithm module is used to process the first image and the corresponding first detection box using a tracking algorithm to obtain the set of newly added people, the set of matched people, and the set of unmatched people in the first image.

[0037] The face detection algorithm module is used to detect the first image using the face detection algorithm and generate a second detection box corresponding to the face if the newly added person set is not empty, or the already matched person set is not empty and the number of tracking times reaches a threshold.

[0038] A matching module is used to match the first detection box and the second detection box of the first image;

[0039] The face recognition module is used to perform face recognition on the face in the second detection frame and obtain the person information of the face in the second detection frame;

[0040] The determination module is used to determine the dwell time of each person captured by the video stream based on the person information detected in each frame of the video stream.

[0041] Optionally, the matching module is configured to obtain the vertex coordinates of the first detection box, the vertex coordinates of the second detection box, and the coordinates of the facial feature points in the second detection box in the first image; determine the inclusion relationship between each second detection box and each first detection box based on the vertex coordinates of the first detection box and the coordinates of the facial feature points in the second detection box, wherein the inclusion relationship means that all facial feature points in the second detection box are located within the first detection box; if there is a one-to-one inclusion relationship between the first detection box and the second detection box, then it is determined that there is a one-to-one matching relationship between the first detection box and the second detection box; if there is a one-to-many or many-to-one inclusion relationship between the first detection box and the second detection box, then a one-to-one matching relationship between the first detection box and the second detection box is determined based on the area of ​​the first detection box and the second detection box with a one-to-many or many-to-one inclusion relationship.

[0042] Optionally, the face recognition module is used to extract feature vectors of faces in the second detection frame; compare the extracted feature vectors with feature vectors of each category in the face feature database, and calculate the Euclidean distance; determine the recognition probability of the face in the second detection frame with each category in the face feature database based on the Euclidean distance between the extracted feature vectors and feature vectors of each category in the face feature database; and determine the person information corresponding to the category associated with the face in the second detection frame based on the recognition probability of the face in the second detection frame with each category in the face feature database.

[0043] At least one embodiment of this disclosure provides a server, the server including a processor and a memory, the memory storing at least one piece of program code, the program code being loaded and executed by the processor to implement the dwell time detection method as described above.

[0044] At least one embodiment of this disclosure provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement the dwell time detection method as described in any of the preceding claims.

[0045] The beneficial effects of the technical solutions provided in this disclosure are:

[0046] In this embodiment, a target detection algorithm detects all people in the first image and generates a first detection box for each detected person. A tracking algorithm is used to obtain a set of newly added people, a set of matched people, and a set of unmatched people in the first image. Face detection and subsequent face recognition are only performed when the set of newly added people is not empty, or when the set of matched people is not empty and the number of tracking attempts reaches a threshold. Then, the identified people and faces are matched to ensure accurate detection of the duration of each person's stay. This design allows for periodic face detection even when there are no newly added people but there are already matched people, avoiding the need for face detection and recognition for every image when there are no newly added people, thus saving processing resources. Furthermore, since this scheme does not require face detection and recognition for every frame, it avoids errors in duration statistics caused by the inability to continuously perform face detection and recognition when a frontal face image is unavailable. It also avoids the problem of wasting processing resources by continuously performing detection and recognition when a frontal face image is unavailable. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart of a dwell time detection method provided in an embodiment of this disclosure;

[0049] Figure 2 This is a flowchart of a dwell time detection method provided in an embodiment of this disclosure;

[0050] Figure 3 This is a schematic diagram of the detection frame provided in an embodiment of this disclosure;

[0051] Figure 4 This is a classification diagram provided in the embodiments of this disclosure;

[0052] Figure 5 This is a schematic diagram of the function curve provided in the embodiments of this disclosure;

[0053] Figure 6 This is a block diagram of a dwell time detection device provided in an embodiment of this disclosure;

[0054] Figure 7 This is a structural block diagram of a server provided in an embodiment of this disclosure. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0056] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” “third,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising” or “including” and similar terms mean that the elements or objects preceding “comprising” or “including” encompass the elements or objects listed following “comprising” or “including” and their equivalents, but do not exclude other elements or objects.

[0057] When capturing video images, related technologies often struggle to obtain frontal images of people, making recognition impossible. Furthermore, the acquisition process is susceptible to interference from lighting, occlusion, and external objects, leading to a decrease in recognition accuracy. Consequently, the way these technologies identify and detect faces in each frame results in a waste of processing resources.

[0058] Figure 1 This is a flowchart of a dwell time detection method provided in an embodiment of this disclosure. See also... Figure 1 The method includes:

[0059] 101: Sequentially acquire each frame of the video stream, use an object detection algorithm to detect all people in the first image, and generate a first detection box for the detected people. The first image is any frame of the video stream.

[0060] In this embodiment of the disclosure, the dwell time detection method can be executed by a server. The server obtains the video stream captured by the monitoring device (e.g., a camera), detects people in the area captured by the monitoring device based on the video stream, and determines the dwell time of the people in the area.

[0061] In this embodiment of the disclosure, the server processes each frame of the video stream using steps 101 to 105, thereby ultimately determining the duration of a person's stay based on the person's information in multiple frames.

[0062] 102: The first image and the corresponding first detection box are processed using a tracking algorithm to obtain the set of newly added people, the set of matched people, and the set of unmatched people in the first image.

[0063] The newly added people set refers to the set of people added to the first image compared to the previous frame. The matched people set refers to the set of people who appeared in the previous image and reappear in the first image. The unmatched people set refers to the set of people who appeared in the previous frame but did not reappear in the first image, indicating that the people may have left the field of view. In this case, the dwell time is calculated by subtracting the entry time from the departure time. Of course, to increase the reliability of the detection statistics, people entering the unmatched people set are not immediately considered to be in a departing state. Instead, people are considered to be in a departing state only if they remain in the unmatched people set for a certain period of time. People in the unmatched people set may also be detected and tracked again and return to the matched people set. The above three sets are obtained by processing the previous frame image, the first image, and their respective first detection boxes using a tracking algorithm.

[0064] In this embodiment of the disclosure, each person identified by the object detection algorithm can be assigned an identifier, which corresponds to the vertex coordinates of the first detection box corresponding to the person. The aforementioned set of newly added persons and set of matched persons can be sets of identifiers.

[0065] 103: If the newly added set of people is not empty, or the set of matched people is not empty and the number of tracking times reaches a threshold, then the face detection algorithm is used to detect the first image and generate a second detection box corresponding to the face.

[0066] Here, "reaching the threshold for the number of times the matched person set has been tracked" means that when the number of consecutively processed image frames reaches the threshold, for example, 60 consecutively processed frames, all faces in the first image are detected and matched. Once the threshold is reached, the tracking count is restarted. This design allows for periodic face detection algorithms even when no new people are added but matched people already exist, avoiding the need for face detection and recognition for every image when no new people are added, thus saving processing resources.

[0067] If the set of newly added characters is empty and the set of matched characters is empty, then the method process stops.

[0068] If the newly added set of people is empty and the already matched set of people is not empty but the number of tracking attempts has not reached the threshold, then the method process is stopped.

[0069] The individuals in the newly added and matched sets are all detected within the detection range. In addition to the newly added and matched sets, an unknown set of individuals, i.e., individuals outside the detection range, is also detected. For individuals in the unknown set (corresponding to a first detection box), if they are outside the detection range for more than a certain period of time, they are deleted from the unknown set.

[0070] It is worth noting that the object detection algorithm, tracking algorithm and face detection algorithm mentioned in this disclosure are all mature algorithms, and this disclosure does not limit them.

[0071] 104: Match the first detection box and the second detection box of the first image.

[0072] This means matching the outline of a person with their face, so that when performing facial recognition later, it can be determined which person is being identified.

[0073] 105: Perform face recognition on the face in the second detection frame to obtain the person information of the face in the second detection frame.

[0074] Here, the information about a person can include their name and other identity information, as well as the time information when the person was in that area.

[0075] The person information can be obtained from the database based on the face recognition results, while the time information of the person entering the area is obtained from the time of the first appearance of the first detection box corresponding to the second detection box in the video frame.

[0076] In this embodiment of the disclosure, the server can record the recognition and processing results of each frame of the image, including the newly added set of people and the set of matched people in each frame of the image. In this way, it is easy to determine the entry time based on the identifier of the person corresponding to the first detection box matched by the second detection box after face recognition.

[0077] 106: Based on the person information detected in each frame of the video stream, determine the dwell time of each person captured by the video stream.

[0078] The duration of a character's presence can be determined by the time of their first and last appearance in the image.

[0079] In this embodiment, a target detection algorithm detects all people in the first image and generates a first detection box for each detected person. A tracking algorithm is used to obtain a set of newly added people, a set of matched people, and a set of unmatched people in the first image. Face detection and subsequent face recognition are only performed when the set of newly added people is not empty, or when the set of matched people is not empty and the number of tracking attempts reaches a threshold. Then, the identified people and faces are matched to ensure accurate detection of the duration of each person's stay. This design allows for periodic face detection even when there are no newly added people but there are already matched people, avoiding the need for face detection and recognition for every image when there are no newly added people, thus saving processing resources. Furthermore, since this scheme does not require face detection and recognition for every frame, it avoids errors in duration statistics caused by the inability to continuously perform face detection and recognition when a frontal face image is unavailable. It also avoids the problem of wasting processing resources by continuously performing detection and recognition when a frontal face image is unavailable.

[0080] Figure 2 This is a flowchart of a dwell time detection method provided in an embodiment of this disclosure. See also... Figure 2 The method includes:

[0081] 201: Sequentially acquire each frame of the video stream, use an object detection algorithm to detect all people in the first image, and generate a first detection box for the detected people. The first image is any frame of the video stream.

[0082] In this embodiment of the disclosure, the dwell time detection method can be executed by a server. The server obtains the video stream captured by the monitoring device (e.g., a camera), detects people in the area captured by the monitoring device based on the video stream, and determines the dwell time of the people in the area.

[0083] In this embodiment of the disclosure, the server processes each frame of the video stream using steps 201 to 211, thereby ultimately determining the duration of a person's stay based on the person information in multiple frames.

[0084] 202: Use a tracking algorithm to process the first image and the corresponding first detection box to obtain the set of newly added people, the set of matched people, and the set of unmatched people in the first image.

[0085] The newly added people set refers to the set of people added to the first image compared to the previous frame. The matched people set refers to the set of people who appeared in the previous image and reappear in the first image. The unmatched people set refers to the set of people who appeared in the previous frame but did not reappear in the first image, indicating that the people may have left the field of view. In this case, the dwell time is calculated by subtracting the entry time from the departure time. Of course, to increase the reliability of the detection statistics, people entering the unmatched people set are not immediately considered to be in a departing state. Instead, people are considered to be in a departing state only if they remain in the unmatched people set for a certain period of time. People in the unmatched people set may also be detected and tracked again and return to the matched people set. The above three sets are obtained by processing the previous frame image, the first image, and their respective first detection boxes using a tracking algorithm.

[0086] In this embodiment of the disclosure, each person identified by the object detection algorithm can be assigned an identifier, which corresponds to the vertex coordinates of the first detection box corresponding to the person. The aforementioned set of newly added persons and set of matched persons can be sets of identifiers.

[0087] 203: If the newly added set of people is not empty, or the set of matched people is not empty and the number of tracking times reaches a threshold, then the face detection algorithm is used to detect the first image and generate a second detection box corresponding to the face.

[0088] Here, "reaching the threshold for the number of times the matched person set has been tracked" means that when the number of consecutively processed image frames reaches the threshold, for example, 60 consecutively processed frames, all faces in the first image are detected and matched. Once the threshold is reached, the tracking count is restarted. This design allows for periodic face detection algorithms even when no new people are added but matched people already exist, avoiding the need for face detection and recognition for every image when no new people are added, thus saving processing resources.

[0089] If the set of newly added characters is empty and the set of matched characters is empty, then the method process stops.

[0090] If the newly added set of people is empty and the already matched set of people is not empty but the number of tracking attempts has not reached the threshold, then the method process is stopped.

[0091] The individuals in the newly added and matched sets are all detected within the detection range. In addition to the newly added and matched sets, an unknown set of individuals, i.e., individuals outside the detection range, is also detected. For individuals in the unknown set (corresponding to a first detection box), if they are outside the detection range for more than a certain period of time, they are deleted from the unknown set.

[0092] 204: Obtain the vertex coordinates of the first detection box, the vertex coordinates of the second detection box, and the coordinates of the facial feature points in the second detection box in the first image.

[0093] In one possible implementation of this disclosure, both the first detection box and the second detection box are rectangles, and the size of the first detection box is larger than that of the second detection box.

[0094] Figure 3 This is a schematic diagram of a first detection frame and a second detection frame provided in an embodiment of this disclosure. See also... Figure 3 The first detection box includes A1, A2, A3, and A4, and the second detection box includes B1, B2, B3, and B4. Both the first and second detection boxes are rectangles, and each detection box includes four vertices. The server can obtain the coordinates of the four vertices of each detection box.

[0095] In another possible implementation of this disclosure, the first and second detection boxes may also be of other shapes, and there is no limitation thereto.

[0096] In addition, within each second detection frame, the server performs face detection, face alignment, and face feature extraction using a face detection algorithm, which identifies five key feature points of the face, such as the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth. The server then obtains the coordinates of these five feature points.

[0097] 205: Based on the vertex coordinates of the first detection box and the coordinates of the face feature points in the second detection box, determine the inclusion relationship between each second detection box and each first detection box. The inclusion relationship means that the face feature points in the second detection box are all located within the first detection box.

[0098] In one possible implementation of this disclosure, all facial feature points are located within their corresponding second detection boxes. Subsequent examples will be illustrated using the assumption that all facial feature points are located within their corresponding second detection boxes.

[0099] In another possible implementation of this disclosure, facial feature points may be partially located within the corresponding second detection box, without limitation.

[0100] 206: If there is a one-to-one containment relationship between the first detection box and the second detection box, then it is determined that there is a one-to-one matching relationship between the first detection box and the second detection box.

[0101] See Figure 3 If the first detection box A1 and the second detection box B1 have a one-to-one inclusion relationship, then it is determined that the first detection box A1 and the second detection box B1 have a one-to-one matching relationship.

[0102] 207: If there is a one-to-many or many-to-one inclusion relationship between the first detection box and the second detection box, then the one-to-one matching relationship between the first detection box and the second detection box is determined based on the area of ​​the first detection box and the second detection box that have a one-to-many or many-to-one inclusion relationship.

[0103] See Figure 3 The first detection box A2 has an inclusion relationship with the second detection boxes B2, B3, and B4 respectively, that is, there is a one-to-many inclusion relationship. The first detection boxes A2, A3, and A4 have an inclusion relationship with the second detection box B4 respectively, that is, there is a many-to-one inclusion relationship.

[0104] The object detection algorithm obtains a first set of detection boxes A{A1,A2,A3,A4}, and the face detection algorithm obtains a second set of detection boxes B{B1,B2,B3,B4}. Traversing sets A and B, if the coordinates of all five key feature points of a second detection box in set B are within a first detection box in set A, then they are considered a potential containment relationship, resulting in set C{A1B1,A2B2,A2B3,A2B4,A3B3,A3B4,A4B4}. Since A1B1 is a one-to-one match, we directly consider A1 and B1 to have a one-to-one matching relationship. The remaining elements in set C are all one-to-many or many-to-many matching relationships, which can be processed as follows:

[0105] The first step is to calculate the correlation coefficient between the first and second detection boxes that have a one-to-many or many-to-one inclusion relationship using the following formula:

[0106]

[0107] Among them, Face i Pedestrian represents the area of ​​the i-th second detection box. j Represents the area of ​​the j-th first detection box, Face i ∩Pedestrian j P represents the overlap area between the i-th second detection box and the j-th first detection box. ij This represents the correlation coefficient between the i-th second detection box and the j-th first detection box.

[0108] Here, i and j are both positive integers. For example, if the number of first detection boxes to be matched using step 207 is a and the number of second detection boxes is b, where a and b are both positive integers, then the value of i ranges from 0 to a and the value of j ranges from 0 to b.

[0109] Following the first step, we can obtain the correlation coefficient set C{PA2B2,PA2B3,PA2B4,PA3B3,PA3B4,PA4B4}, as shown in the table below:

[0110] B2 B3 B4 A2 PA2B2 PA2B3 PA2B4 A3 0 PA3B3 PA3B4 A4 0 0 PA4B4

[0111] The correlation coefficient matrix is ​​as follows:

[0112]

[0113] The second step is to determine the one-to-one matching relationship between the first detection box and the second detection box based on the correlation coefficient between the first detection box and the second detection box using the Hungarian algorithm or the KM (Kuhn-Munkres) algorithm.

[0114] Using the Hungarian algorithm or the KM algorithm to calculate the optimal matching relationship of the above correlation coefficient matrix, a reliable matching result is {A2B2, A3B3, A4B4}. Therefore, the final matching result is {A1B1, A2B2, A3B3, A4B4}.

[0115] 208: Extract the feature vector of the face in the second detection frame.

[0116] For example, the server can perform face detection, face alignment, and face feature extraction using a face detection algorithm to obtain a 512-dimensional face feature vector.

[0117] Of course, the 512 dimensions here are just an example; it could also be a feature vector with other dimensions.

[0118] 209: Compare the extracted feature vectors with the feature vectors of each category in the face feature database, and calculate the Euclidean distance.

[0119] That is, the Euclidean distance between the extracted feature vector and each feature vector in the face feature database is calculated.

[0120] Prior to this step, the method may also include: acquiring a facial feature database.

[0121] For example, a face feature database is trained using known sample data. Assume the sample data has M face categories, with N images per category (the number of images per category is not necessarily the same), resulting in a total of M·N images, where M and N are both positive integers. For each image in the dataset, a face detection algorithm is used to detect the face region and locate key points (five key points: left eye, right eye, nose, left corner of mouth, and right corner of mouth). Then, affine transformations are used for face alignment, and a face feature extraction algorithm is used to extract a 512-dimensional feature vector for each face. This yields M·N 512-dimensional feature vectors, and these feature vectors and their corresponding category labels constitute the face feature database for that data sample. Here, one category can correspond to one person.

[0122] 210: Determine the recognition probability of the face in the second detection frame and each category in the face feature database based on the Euclidean distance between the extracted feature vector and the feature vector of each category in the face feature database.

[0123] For example, step 210 may include:

[0124] The recognition probability of the face in the second detection frame and each category in the face feature database is calculated using the following formula:

[0125] P = α·P total +(1-α)·P best ;

[0126] Among them, P total This refers to the ratio of the number of feature vectors of the face in the second detection frame whose Euclidean distance to all feature vectors of the first category in the face feature database is less than the Euclidean distance threshold, to the total number of feature vectors of the first category. The purpose is to consider the overall recognition accuracy and avoid using only the best value of a single sample comparison as the basis for judgment, which could lead to incorrect recognition. Figure 4 As shown on the left, the feature vector of the face in the second detection box is closest to the feature vector A1 of a sample in category A of the face feature library. If only a single sample is considered, S will be incorrectly classified into category A. However, this disclosure, by considering all samples in each category, will classify S into category B. The first category is any category in the face feature library. P best This refers to the optimal feature probability of the face in the second detection frame. The purpose is to consider the recognition accuracy of a single sample and avoid misidentification caused by using only the overall sample's judgment result as the basis for judgment. Figure 4As shown on the right, the feature vector distances between S and all samples in category A are within the threshold, while the feature vector distance with a sample in category B exceeds the threshold. If only the whole sample is considered, S would be incorrectly classified as sample A. However, this disclosure classifies S into category B by considering the individual samples in each category. In summary, this disclosure considers both individual samples and the overall sample situation when performing classification, thus ensuring classification accuracy. α is a coefficient, generally chosen based on experience. A larger α indicates greater confidence in the group's judgment, while a smaller α indicates greater confidence in the individual's judgment.

[0127]

[0128] Where N is the total number of feature vectors of the first category, that is, the number of samples of the first category, Num is the number of samples less than the Euclidean distance threshold, d is the Euclidean distance, and threshold is the Euclidean distance threshold.

[0129]

[0130] Where, d min This refers to the minimum Euclidean distance between the feature vector of the face in the second detection frame and the feature vectors of all faces in the first category in the face feature database.

[0131] P best Function curves as follows Figure 5 As shown, it can be seen that P best In d min The closer to the two ends, the closer the probability is to 1 or 0, d min The curve is steepest near the threshold, which is consistent with reality.

[0132] When the recognition probability is lower than the set value, it is considered that the face in the second detection frame does not match any category in the face feature database, and an alarm can be issued at this time.

[0133] 211: Based on the recognition probabilities of the face in the second detection frame and each category in the face feature database, determine the person information corresponding to the category associated with the face in the second detection frame.

[0134] Here, the information about a person can include their name and other identity information, as well as the time information when the person was in that area.

[0135] The person information can be obtained from the database based on the face recognition results, while the time information of the person entering the area is obtained from the time of the first appearance of the first detection box corresponding to the second detection box in the video frame.

[0136] In this embodiment of the disclosure, the server can record the recognition and processing results of each frame of the image, including the newly added set of people and the set of matched people in each frame of the image. In this way, it is easy to determine the entry time based on the identifier of the person corresponding to the first detection box matched by the second detection box after face recognition.

[0137] 212: Determine the dwell time of each person captured by the video stream based on the person information detected in each frame of the video stream.

[0138] The duration of a character's presence can be determined by the time of their first and last appearance in the image.

[0139] The methods provided in this disclosure are illustrated below through examples:

[0140] Suppose that person A enters the video surveillance area at a certain moment. Every frame of the image is captured, and person A is detected by the object detection algorithm. The bounding box corresponding to person A and the original image are input into the pedestrian tracking algorithm. The pedestrian tracking algorithm considers pedestrian bounding box A to be a newly added pedestrian (initially "unknown person"). It immediately performs face detection and recognition on the frame image. By matching the face bounding box and the pedestrian bounding box, the corresponding attributes (such as name, entry time, etc.) are assigned to pedestrian bounding box A.

[0141] While person A remains within the area, they are tracked in real time. After each frame of tracking, person A is considered a matched tracking target, and their last visit time is updated. This process avoids performing face detection and recognition on every frame and solves the problem of continuous monitoring when A's back is turned and their face cannot be captured. To address the possibility of false recognition or failure to detect a face at certain times, face detection and recognition are performed periodically during subsequent tracking, and all recognition results are statistically accumulated. The result with the highest number of occurrences is the final recognition result. If a target is repeatedly identified as an "unknown person," an alarm is triggered.

[0142] After person A leaves the area, person A will be identified as an unmatched pedestrian by the tracking algorithm. When the number of unmatches reaches a certain number, it is considered that person A has left the area, the history of person A is saved, and A is deleted from the set of unmatched persons.

[0143] The method provided in this disclosure can be used to monitor key areas and locations, detect unauthorized entrants, and track the duration of their stay.

[0144] Figure 6 This is a schematic diagram of a dwell time detection device provided in an embodiment of this disclosure. See also... Figure 6The dwell time detection device includes: target detection algorithm module 301, tracking algorithm module 302, face detection algorithm module 303, matching module 304, face recognition module 305, and determination module 306.

[0145] The target detection algorithm module 301 is used to sequentially acquire each frame of the video stream, detect all people in the first image using the target detection algorithm, and generate a first detection box for the detected people. The first image is any frame of the video stream.

[0146] The tracking algorithm module 302 is used to process the first image and the corresponding first detection box using a tracking algorithm to obtain the set of newly added people, the set of matched people and the set of unmatched people in the first image;

[0147] The face detection algorithm module 303 is used to detect the first image using the face detection algorithm and generate a second detection box corresponding to the face if the newly added person set is not empty, or the matched person set is not empty and the number of tracking times reaches a threshold.

[0148] The matching module 304 is used to match the first detection box and the second detection box of the first image;

[0149] The face recognition module 305 is used to perform face recognition on the face in the second detection frame and obtain the person information of the face in the second detection frame;

[0150] The determining module 306 is used to determine the dwell time of each person captured by the video stream based on the person information detected in each frame of the video stream.

[0151] Optionally, the matching module 304 is configured to obtain the vertex coordinates of the first detection box, the vertex coordinates of the second detection box, and the coordinates of the facial feature points in the second detection box in the first image; determine the inclusion relationship between each second detection box and each first detection box based on the vertex coordinates of the first detection box and the coordinates of the facial feature points in the second detection box, wherein the inclusion relationship means that all facial feature points in the second detection box are located within the first detection box; if there is a one-to-one inclusion relationship between the first detection box and the second detection box, then it is determined that there is a one-to-one matching relationship between the first detection box and the second detection box; if there is a one-to-many or many-to-one inclusion relationship between the first detection box and the second detection box, then a one-to-one matching relationship between the first detection box and the second detection box is determined based on the area of ​​the first detection box and the second detection box with a one-to-many or many-to-one inclusion relationship.

[0152] Optionally, the matching module 304 is configured to calculate the correlation coefficient between the first detection box and the second detection box that have a one-to-many or many-to-one inclusion relationship according to the following formula:

[0153]

[0154] Among them, Face i Pedestrian represents the area of ​​the i-th second detection box. j Represents the area of ​​the j-th first detection box, Face i ∩Pedestrian j P represents the overlap area between the i-th second detection box and the j-th first detection box. ij This represents the correlation coefficient between the i-th second detection box and the j-th first detection box;

[0155] Based on the correlation coefficient between the first and second detection boxes, the one-to-one matching relationship between the first and second detection boxes is determined using the Hungarian algorithm or the KM algorithm.

[0156] Optionally, the face recognition module 305 is used to extract the feature vector of the face in the second detection frame; compare the extracted feature vector with the feature vectors of each category in the face feature database, and calculate the Euclidean distance; determine the recognition probability of the face in the second detection frame and each category in the face feature database based on the Euclidean distance between the extracted feature vector and the feature vectors of each category in the face feature database; and determine the person information corresponding to the category associated with the face in the second detection frame based on the recognition probability of the face in the second detection frame and each category in the face feature database.

[0157] Optionally, the face recognition module 305 is used to calculate the recognition probability of the face in the second detection frame and each category in the face feature database using the following formula:

[0158] P = α·P total +(1-α)·P best ;

[0159] Among them, P total P refers to the ratio of the number of feature vectors of a face in the second detection frame whose Euclidean distance to all feature vectors of the first category in the face feature database is less than the Euclidean distance threshold, to the total number of feature vectors in the first category, where the first category is any category in the face feature database; best The optimal feature probability of the face in the second detection frame; α is a coefficient;

[0160]

[0161] Where N is the total number of feature vectors of the first category, Num is the number of vectors less than the Euclidean distance threshold, d is the Euclidean distance, and threshold is the Euclidean distance threshold.

[0162]

[0163] Where, d min This refers to the minimum Euclidean distance between the feature vector of the face in the second detection frame and the feature vectors of all faces in the first category in the face feature database.

[0164] It should be noted that the dwell time detection device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the dwell time detection device and the dwell time detection method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0165] Figure 7 This is a structural block diagram of a server provided in an embodiment of this disclosure. Typically, a server includes a processor 601 and a memory 602.

[0166] Processor 601 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 601 may be implemented using at least one of the following hardware forms: DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0167] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 is used to store at least one instruction, which is executed by the processor 601 to implement the dwell time detection method executed by the server provided in the method embodiments of this application.

[0168] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0169] The above description is merely an optional embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for detecting dwell time, characterized in that, The method includes: Each frame of the video stream is acquired sequentially, and all people in the first image are detected using an object detection algorithm. A first detection box is generated for the detected people. The first image is any frame of the video stream. The first image and its corresponding first detection box are processed using a tracking algorithm to obtain a set of newly added people, a set of matched people, and a set of unmatched people in the first image. If the newly added set of people is not empty, or the set of matched people is not empty and the number of tracking times reaches a threshold, then the face detection algorithm is used to detect the first image and generate a second detection box corresponding to the face. Match the first detection box and the second detection box of the first image; Perform face recognition on the face in the second detection frame to obtain the person information of the face in the second detection frame; Based on the human information detected in each frame of the video stream, the dwell time of each human captured by the video stream is determined. The step of matching the first detection box and the second detection box of the first image includes: Obtain the vertex coordinates of the first detection box, the vertex coordinates of the second detection box, and the coordinates of the facial feature points in the second detection box in the first image; Based on the vertex coordinates of the first detection box and the coordinates of the facial feature points in the second detection box, the inclusion relationship between each second detection box and each first detection box is determined. The inclusion relationship means that the facial feature points in the second detection box are all located within the first detection box. If there is a one-to-one containment relationship between the first detection box and the second detection box, then it is determined that there is a one-to-one matching relationship between the first detection box and the second detection box. If there is a one-to-many or many-to-one inclusion relationship between the first detection box and the second detection box, then the one-to-one matching relationship between the first detection box and the second detection box is determined based on the area of ​​the first detection box and the second detection box that have a one-to-many or many-to-one inclusion relationship.

2. The method according to claim 1, characterized in that, Determining the one-to-one matching relationship between the first and second detection boxes based on their areas, where there is a one-to-many or many-to-one inclusion relationship, includes: The correlation coefficient between the first and second detection boxes that have a one-to-many or many-to-one inclusion relationship is calculated using the following formula: ； in, This represents the area of ​​the i-th second detection box. This represents the area of ​​the j-th first detection box. This represents the overlap area between the i-th second detection box and the j-th first detection box. This represents the correlation coefficient between the i-th second detection box and the j-th first detection box; Based on the correlation coefficient between the first and second detection boxes, the one-to-one matching relationship between the first and second detection boxes is determined using the Hungarian algorithm or the KM algorithm.

3. The method according to claim 1, characterized in that, The step of performing face recognition on the face in the second detection frame to obtain the person information of the face in the second detection frame includes: Extract the feature vector of the face in the second detection frame; The extracted feature vectors are compared with the feature vectors of each category in the face feature database, and the Euclidean distance is calculated. Based on the Euclidean distance between the extracted feature vector and the feature vectors of each category in the face feature database, the recognition probability of the face in the second detection frame and each category in the face feature database is determined. Based on the recognition probabilities of the face in the second detection frame and various categories in the face feature database, the person information corresponding to the category associated with the face in the second detection frame is determined.

4. The method according to claim 3, characterized in that, The step of determining the recognition probability of the face in the second detection frame relative to each category in the face feature database based on the Euclidean distance between the extracted feature vector and the feature vectors of each category in the face feature database includes: The recognition probability of the face in the second detection frame and each category in the face feature database is calculated using the following formula: ； in, The ratio of the number of feature vectors of a face in the second detection frame whose Euclidean distance to all feature vectors of the first category in the face feature library is less than the Euclidean distance threshold to the total number of feature vectors of the first category, where the first category is any category in the face feature library; The optimal feature probability of the face in the second detection frame; α is a coefficient; ； Where N is the total number of feature vectors in the first category, Num is the number of vectors less than the Euclidean distance threshold, and d is the Euclidean distance. The Euclidean distance threshold; ； in, This refers to the minimum Euclidean distance between the feature vector of the face in the second detection frame and the feature vectors of all faces in the first category in the face feature database.

5. A dwell time detection device, characterized in that, The device includes: The object detection algorithm module is used to sequentially acquire each frame of the video stream, detect all people in the first image using the object detection algorithm, and generate a first detection box for the detected people. The first image is any frame of the video stream. The tracking algorithm module is used to process the first image and the corresponding first detection box using a tracking algorithm to obtain the set of newly added people, the set of matched people, and the set of unmatched people in the first image. The face detection algorithm module is used to detect the first image using the face detection algorithm and generate a second detection box corresponding to the face if the newly added person set is not empty, or the already matched person set is not empty and the number of tracking times reaches a threshold. A matching module is used to match the first detection box and the second detection box of the first image; The face recognition module is used to perform face recognition on the face in the second detection frame and obtain the person information of the face in the second detection frame; The determination module is used to determine the dwell time of each person captured by the video stream based on the person information detected in each frame of the video stream. The matching module is used to obtain the vertex coordinates of the first detection box, the vertex coordinates of the second detection box, and the coordinates of the facial feature points in the second detection box in the first image; based on the vertex coordinates of the first detection box and the coordinates of the facial feature points in the second detection box, it determines the inclusion relationship between each second detection box and each first detection box, wherein the inclusion relationship means that all facial feature points in the second detection box are located within the first detection box; if there is a one-to-one inclusion relationship between the first detection box and the second detection box, it determines that there is a one-to-one matching relationship between the first detection box and the second detection box; if there is a one-to-many or many-to-one inclusion relationship between the first detection box and the second detection box, it determines the one-to-one matching relationship between the first detection box and the second detection box based on the area of ​​the first detection box and the second detection box with a one-to-many or many-to-one inclusion relationship.

6. The apparatus according to claim 5, characterized in that, The face recognition module is used to extract feature vectors of faces in the second detection frame; compare the extracted feature vectors with feature vectors of each category in the face feature database, and calculate the Euclidean distance; determine the recognition probability of the face in the second detection frame with each category in the face feature database based on the Euclidean distance between the extracted feature vectors and the feature vectors of each category in the face feature database; and determine the person information corresponding to the category associated with the face in the second detection frame based on the recognition probability of the face in the second detection frame with each category in the face feature database.

7. A server, characterized in that, The server includes a processor and a memory, the memory storing at least one line of program code, which is loaded and executed by the processor to implement the dwell time detection method as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the dwell time detection method as described in any one of claims 1 to 4.

Images