Method for tracking pedestrians, pedestrian tracking device, and pedestrian tracking system

Abstract

A method, device, and system for pedestrian tracking are provided, relating to the field of image processing, to reduce the probability of target tracking loss when using human detection methods for pedestrian tracking. The pedestrian tracking method includes: acquiring at least one first video data and at least one second video data; performing face recognition on at least a portion of the first images from multiple frames of first images to obtain tracking data for the first pedestrian including position information of the first pedestrian at N time points; performing human detection on at least a portion of the second images from multiple frames of second images to obtain tracking data for the second pedestrian including the movement trajectory of the second pedestrian; and determining the tracking data for a target pedestrian, which includes: the identity information and movement trajectory of the target pedestrian, provided that the deviation between the position information of the first pedestrian at N time points and the position information of the second pedestrian's movement trajectory at the same time point is within a preset range. The target pedestrian's tracking data includes the first pedestrian's identity information and movement trajectory, wherein the first pedestrian and the second pedestrian are both the target pedestrian.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a method, device, and system for pedestrian tracking. Background Technology

[0002] With the development of artificial intelligence, human detection methods have been widely applied in various fields such as security, video surveillance, and abnormal behavior analysis. Human detection methods can detect one or more human bodies in images captured by camera equipment and mark the position of each body, thereby obtaining the movement trajectory of each body.

[0003] However, in the current process of using human detection methods for pedestrian tracking, the target tracking may be lost when the human body is partially obscured or when the human body's posture changes significantly. Summary of the Invention

[0004] Embodiments of the present invention provide a method, device, and system for pedestrian tracking, which reduce the probability of target tracking loss when using human detection methods for pedestrian tracking.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0006] In a first aspect, a method for pedestrian tracking is provided. The method includes: acquiring at least one first video data and at least one second video data, the first video data comprising multiple frames of first images, and the second video data comprising multiple frames of second images; performing face recognition on at least a portion of the first images in the multiple first frames to obtain tracking data of a first pedestrian, the tracking data of the first pedestrian including: the identity information of the first pedestrian and the position information of the first pedestrian at N time points; N being greater than or equal to 1; performing human detection on at least a portion of the second images in the multiple second frames to obtain tracking data of a second pedestrian, the tracking data of the second pedestrian including: the movement trajectory of the second pedestrian; the movement trajectory of the second pedestrian including the position information of the second pedestrian at multiple time points; and determining tracking data of a target pedestrian if the deviation between the position information of the first pedestrian at N time points and the position information of the second pedestrian's movement trajectory at the same time point is within a preset range; the tracking data of the target pedestrian including: the identity information of the target pedestrian and the movement trajectory of the target pedestrian, wherein the first pedestrian and the second pedestrian are both target pedestrians, the identity information of the target pedestrian is determined at least based on the identity information of the first pedestrian, and the movement trajectory of the target pedestrian is determined at least based on the movement trajectory of the second pedestrian.

[0007] In some embodiments, face recognition is performed on at least a portion of the first images in multiple frames of first images to obtain tracking data of a first pedestrian, including: detecting whether each of the at least a portion of the first images has a pedestrian's face; if a first pedestrian's face is detected, acquiring the first pedestrian's face feature information and comparing the first pedestrian's face feature information with face feature information in a face database; the face database contains multiple face feature information and pedestrian identity information corresponding to the multiple face feature information respectively; if the comparison is successful, the pedestrian identity information in the face database that matches the first pedestrian's face feature information is used as the first pedestrian's identity information; or, if the comparison fails, the first pedestrian's face feature information is recorded.

[0008] In some embodiments, human detection is performed on at least a portion of the second images in a plurality of frames of second images to obtain tracking data of a second pedestrian, including: acquiring pedestrian bounding boxes in each of the second images in at least a portion of the second images in the second video data, and obtaining tracking data of the second pedestrian based on the acquired pedestrian bounding boxes, wherein the pedestrian bounding boxes are boxes labeled according to the human body detected in each second image.

[0009] In some embodiments, the tracking data of the second pedestrian is obtained based on the acquired pedestrian bounding boxes, including: processing the image blocks corresponding to each pedestrian bounding box using a trained neural network model to obtain the feature vectors of the image blocks corresponding to each pedestrian bounding box; determining the pedestrian trajectory of at least one pedestrian based on the feature vector of each pedestrian bounding box in each pedestrian bounding box, and using the pedestrian trajectory of at least one pedestrian as the tracking data of the second pedestrian, wherein each pedestrian in the at least one pedestrian corresponds to a pedestrian trajectory.

[0010] In some embodiments, when the deviation between the position information of the first pedestrian at N times and the position information of the second pedestrian's trajectory at the same time is within a preset range, the tracking data of the target pedestrian is determined, including: calculating the distance between the global coordinates of the first pedestrian and the global coordinates of the second pedestrian at each of the N times using the following formula; For each of the N time moments, if the global coordinates A of the first person are... i The global coordinates B of the second pedestrian i If the distance between them is less than or equal to a preset threshold, then the first pedestrian and the second pedestrian are determined to be the same target pedestrian; where, at the i-th time, the global coordinates of the first pedestrian are A. i (x1 i y1 i The global coordinates of the second person are B. i (x2 i y2 i), 1≤i≤N; the location information includes global coordinates, which are the coordinates of the pedestrian in the global coordinate system.

[0011] In some embodiments, the movement trajectory of the target pedestrian is the movement trajectory of the second pedestrian; or, the tracking data of the first pedestrian includes: the identity information of the first pedestrian and the movement trajectory of the first pedestrian, the movement trajectory of the first pedestrian includes the position information of the first pedestrian at N times; if the deviation between the position information of the first pedestrian at N times and the position information of the movement trajectory of the second pedestrian at the same time is within a preset range, the tracking data of the target pedestrian is determined, including: generating the movement trajectory of the target pedestrian based on the movement trajectory of the first pedestrian and the movement trajectory of the second pedestrian.

[0012] In some embodiments, the pedestrian tracking method further includes receiving first video data from a first camera device and receiving second video data from a second camera device, wherein the first camera device and the second camera device are different; or, receiving the first video data and the second video data from a third camera device.

[0013] In some embodiments, the pedestrian tracking method further includes receiving a first access request configured to access tracking data of a target pedestrian; and outputting access results or a first interface configured to display the access results, which include: the identity information of the target pedestrian and the movement trajectory of the target pedestrian.

[0014] In some embodiments, the pedestrian tracking method may further include obtaining the distribution area of ​​the target pedestrian and the dwell time of the target pedestrian in the distribution area based on the movement trajectory of the target pedestrian; and / or, extracting the image of the target pedestrian from at least a portion of the first images in a plurality of first images; and / or, extracting the image of the target pedestrian from at least a portion of the second images in a plurality of second images; and when the pedestrian tracking method includes outputting access results or a first interface, the access results may further include at least one of the following: the image of the target pedestrian, the distribution area of ​​the target pedestrian, and the dwell time of the target pedestrian in the distribution area.

[0015] In some embodiments, the pedestrian tracking method may further include receiving a second access instruction configured to access statistical results; outputting statistical results or a second interface configured to display statistical results; the statistical results include at least one of: first data, second data, third data, and fourth data; wherein, the first data is the total number of target pedestrians in at least one distribution area at the current time; the second data is the first number or first proportion of target pedestrians whose dwell time in at least one distribution area is in different preset time intervals at the current time, the first proportion being the ratio of the first number to the total number; the third data is the second number or second proportion of target pedestrians in each distribution area during a time period when there are multiple distribution areas, the second proportion being the ratio of the second number to the total number; and the fourth data is the total number of target pedestrians in at least one distribution area at different times during a historical time period.

[0016] Secondly, a pedestrian tracking device is provided, comprising: an acquisition module and a processing module; the acquisition module is configured to acquire at least one first video data and at least one second video data, each first video data comprising multiple frames of first images, and each second video data comprising multiple frames of second images; the processing module is configured to perform face recognition on at least a portion of the first images acquired by the acquisition module to obtain tracking data of a first pedestrian, the tracking data of the first pedestrian comprising: identity information of the first pedestrian and position information of the first pedestrian at N time points; N is greater than or equal to 1; the processing module is further configured to perform face recognition on at least a portion of the second images acquired by the acquisition module. The system detects the movement of a second pedestrian and obtains tracking data for that pedestrian. This tracking data includes the pedestrian's movement trajectory, which includes the pedestrian's location information at multiple times. The processing module is further configured to determine the tracking data for a target pedestrian if the deviation between the location information of the first pedestrian at N times and the location information of the second pedestrian's movement trajectory at the same time is within a preset range. The tracking data for the target pedestrian includes the target pedestrian's identity information and movement trajectory, wherein the first and second pedestrians are both target pedestrians, the target pedestrian's identity information is determined at least based on the first pedestrian's identity information, and the target pedestrian's movement trajectory is determined at least based on the second pedestrian's movement trajectory.

[0017] In some embodiments, the processing module is specifically configured to detect whether there is a pedestrian's face in each of at least a portion of the first images; if a pedestrian's face is detected, the module acquires the pedestrian's facial feature information and compares the pedestrian's facial feature information with facial feature information in a face database; the face database contains multiple facial feature information and pedestrian identity information corresponding to each of the multiple facial feature information; if the comparison is successful, the pedestrian identity information in the face database that matches the pedestrian's facial feature information is used as the pedestrian's identity information; or, if the comparison fails, the pedestrian's facial feature information is recorded.

[0018] In some embodiments, the processing module is specifically configured to acquire pedestrian bounding boxes in at least a portion of the second images in a plurality of frames of second images, and to obtain tracking data of the second pedestrians based on the acquired pedestrian bounding boxes, wherein the pedestrian bounding boxes are boxes labeled according to the human body detected in each second image.

[0019] In some embodiments, the processing module is specifically configured to process the image blocks corresponding to each pedestrian box using a trained neural network model to obtain the feature vectors of the image blocks corresponding to each pedestrian box; determine the pedestrian trajectory of at least one pedestrian based on the feature vector of each pedestrian box in each pedestrian box; and use the pedestrian trajectory of at least one pedestrian as the tracking data of a second pedestrian, wherein each pedestrian in the at least one pedestrian corresponds to a pedestrian trajectory.

[0020] In some embodiments, the processing module is specifically configured to calculate the distance between the global coordinates of the first person and the global coordinates of the second person at each of the N time points using the following formula; For each of the N time moments, if the global coordinates A of the first person are... i The global coordinates B of the second pedestrian i If the distance between them is less than or equal to a preset threshold, then the first pedestrian and the second pedestrian are determined to be the same target pedestrian; where, at the i-th time, the global coordinates of the first pedestrian are A. i (x1 i y1 i The global coordinates of the second person are B. i (x2 i y2 i ), 1≤i≤N; the location information includes the global coordinates, which are the coordinates of the pedestrian in the global coordinate system.

[0021] In some embodiments, the movement trajectory of the target pedestrian is the movement trajectory of the second pedestrian; or, the tracking data of the first pedestrian includes: the identity information of the first pedestrian and the movement trajectory of the first pedestrian, the movement trajectory of the first pedestrian includes the position information of the first pedestrian at N time points; the processing module is specifically configured to generate the movement trajectory of the target pedestrian based on the movement trajectory of the first pedestrian and the movement trajectory of the second pedestrian.

[0022] In some embodiments, the pedestrian tracking device may further include a receiving module configured to receive first video data from a first camera device and second video data from a second camera device, wherein the first camera device and the second camera device are different; or, the receiving module may be configured to receive the first video data and the second video data from a third camera device.

[0023] Thirdly, embodiments of the present invention provide a server, which includes a processor and a memory, the memory for storing computer instructions, and the processor for executing the computer instructions, so that the server implements the pedestrian tracking method described in any of the above aspects.

[0024] Fourthly, embodiments of the present invention provide a pedestrian tracking system, which includes a memory, a processor, a first camera device, and a second camera device. The memory is used to store computer instructions, and the processor is used to execute the computer instructions, so that the server implements the pedestrian tracking method described in any of the above embodiments. The first camera device is used to acquire at least one first video data, and the second camera device is used to acquire at least one second video data.

[0025] Fifthly, embodiments of the present invention provide a computer-readable storage medium for storing computer instructions, such that when a server executes the computer instructions, it implements the pedestrian tracking method described in any of the preceding aspects.

[0026] The pedestrian tracking method and apparatus provided in this invention obtain the movement trajectory of a second pedestrian through a human body detection method and obtain the identity information and position information of a first pedestrian at at least one moment through a face recognition method. If, at that at least one moment, the position of the first pedestrian is basically on the movement trajectory of the second pedestrian, that is, the deviation between the position of the first pedestrian and the movement trajectory of the second pedestrian is not large, then the first pedestrian and the second pedestrian can be considered to be the same pedestrian, i.e., the target pedestrian. Therefore, the movement trajectory of the target pedestrian can be determined based on the movement trajectory of the second pedestrian, and the identity information of the first pedestrian can be used as the identity information of the target pedestrian, thus obtaining the tracking data of the target pedestrian; that is, establishing a mapping relationship (also called a correspondence relationship) between the movement trajectory of the target pedestrian and the identity information of the target pedestrian. When situations arise, such as partial occlusion of a person or significant changes in posture, existing technologies may misinterpret the movement trajectories of a second pedestrian obtained through human detection methods before and after these situations, leading to target loss. However, in this application, by establishing a mapping relationship between the target pedestrian's movement trajectory and identity information, as long as the pedestrian's face is not completely occluded and their identity information can be identified, the movement trajectories of the two pedestrians before and after these situations can be correlated with the identity information of the same first pedestrian. This reduces the probability of target tracking loss when using human detection methods for pedestrian tracking. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A system architecture diagram provided for some embodiments of the present invention;

[0029] Figure 2 A structural diagram of a server provided for some embodiments of the present invention;

[0030] Figure 3 A flowchart illustrating a pedestrian tracking method provided in some embodiments of the present invention;

[0031] Figure 4 A model block diagram of an SSD algorithm provided for some embodiments of the present invention;

[0032] Figure 5 A flowchart illustrating another pedestrian tracking method provided in some embodiments of the present invention;

[0033] Figure 6 A statistical results interface diagram provided for some embodiments of the present invention;

[0034] Figure 7 This is a schematic diagram of the structure of a pedestrian tracking device provided in some embodiments of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] In the description of this invention, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0037] Unless the context otherwise requires, throughout the specification and claims, the term "comprise" and its other forms, such as the third-person singular "comprises" and the present participle "comprising," are interpreted as open-ended and encompassing, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiments," "example," "specific example," or "some examples," etc., are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples.

[0038] "At least one of A, B and C" has the same meaning as "at least one of A, B or C", both including the following combinations of A, B and C: only A, only B, only C, combinations of A and B, combinations of A and C, combinations of B and C, and combinations of A, B and C.

[0039] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.

[0040] In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0041] The use of “applies to” or “configured to” in this article implies an open and inclusive language that does not preclude applicability to or configuration to devices that perform additional tasks or steps.

[0042] In addition, the use of “based on” implies openness and inclusivity, because processes, steps, calculations or other actions “based on” one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated.

[0043] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0044] For example, to represent video data processed using different methods, this embodiment adds "first" and "second" before the video data to obtain "first video data" and "second video data"; wherein, the video data used for face recognition is called first video data, and the video data used for human detection is called second video data. Similarly, to distinguish pedestrians appearing in different video data, "first" and "second" are added before the pedestrian's name to obtain "first pedestrian" and "second pedestrian"; the pedestrian appearing in the first video data is called the first pedestrian, and the pedestrian appearing in the second video data is called the second pedestrian.

[0045] This invention provides a method for pedestrian tracking. It combines the identity information of a first pedestrian obtained through facial recognition with the movement trajectory of a second pedestrian obtained through human detection, establishing a mapping relationship between the movement trajectory of at least one second pedestrian and the same identity information. In this case, the first and second pedestrians are considered the same person, referred to as the target pedestrian. Therefore, when the movement trajectories of multiple second pedestrians are mapped to the same identity information, the movement trajectories of the multiple second pedestrians are associated and considered to be the movement trajectory of the same pedestrian, thus preventing target loss.

[0046] Figure 1A system architecture for implementing the above-described pedestrian tracking method is shown. (Reference) Figure 1 The system architecture 100 includes at least one (or more) image acquisition devices 110, a server 120, and a terminal 130. The image acquisition device 110 is configured to acquire video data; the server 120 is configured to process the video data to obtain the identity information and movement trajectory of the target pedestrian; and the terminal 130 is configured to access the server 120 to obtain the identity information and movement trajectory of the target pedestrian, and can also display the identity information and movement trajectory of the target pedestrian.

[0047] The system architecture 100 may further include a routing device 140, which can transmit video data acquired by the image acquisition device 110 to the server 120 by connecting the image acquisition device 110 and the server 120. The routing device 140 may include a switch or a router, etc., and this embodiment does not limit it.

[0048] In this embodiment, at least one image acquisition device 110 includes at least one image acquisition device 110_A and at least one image acquisition device 110_B; wherein, the video data acquired by image acquisition device 110_A is used for face recognition and not for human detection, and the video data acquired by image acquisition device 110_B is used for human detection and not for face recognition. Of course, at least a portion (partially or all) of the video data acquired by at least one image acquisition device 110 can be used for both face recognition and human detection. In this embodiment, the first video data used for face recognition and the second video data used for human detection can be the same video data acquired by the same image acquisition device 110, or they can be video data acquired by different image acquisition devices 110.

[0049] This embodiment does not limit the type of image acquisition device 110. For example, the image acquisition device 110 can be a camera device, such as a webcam. Those skilled in the art should understand that a webcam refers to a new generation of cameras that combine traditional cameras with network technology. In addition to the image capture function of a typical traditional camera, the camera also has a built-in digital compression controller and a network-based operating system, which enables video data to be compressed and encrypted before being transmitted to other devices (such as terminals or servers) through a network (e.g., a local area network or a wireless network).

[0050] This embodiment does not limit the type of terminal 130. For example, the terminal can be a mobile phone, desktop computer, tablet computer, laptop computer, handheld computer, vehicle terminal, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.

[0051] Figure 2 This is provided in this embodiment. Figure 1 Hardware structure diagram of the server. In some embodiments, Figure 2 The server 120 shown may include at least one processor 201 and memory 202.

[0052] The processor 201 may be one or more general-purpose central processing units (CPUs), microprocessors, application-specific integrated circuits (ASICs), or integrated circuits used to control the execution of programs in some embodiments of this disclosure; the CPU may be a single-core processor or a multi-core processor. A processor 201 may refer to one or more devices, circuits, or processing cores used to process data (e.g., computer program instructions).

[0053] The memory 202 may store the operating system and instructions (such as computer instructions), including but not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or optical memory. The memory 202 stores the operating system code.

[0054] For example, processor 201 causes server 120 to implement the pedestrian tracking method in the following embodiments by reading instructions stored in memory 202, thereby obtaining tracking data of the target pedestrian; or, processor 201 causes server 120 to implement the pedestrian tracking method in the following embodiments by internally stored instructions, thereby obtaining tracking data of the target pedestrian. When processor 201 implements the method in the following embodiments by reading instructions stored in memory 202, memory 202 stores instructions for implementing the pedestrian tracking method provided in this embodiment.

[0055] In other embodiments, Figure 2 The server 120 may also include a receiver 203 and a transmitter 204.

[0056] Receiver 203 is configured to receive video data acquired by at least one image acquisition device 110. For example, receiver 203 can be communicatively connected to router 140 via wired or wireless communication to receive video data transmitted through router 140.

[0057] The transmitter 204 can communicate with the terminal 130 via wired or wireless communication and is configured to send the tracking data of the target pedestrian (including the identity information and movement trajectory of the target pedestrian with the established mapping relationship) to the terminal 130. It can also be configured to send the aforementioned video data to the terminal 130 so that the terminal 130 can display the video data.

[0058] like Figure 3 As shown, based on Figure 1 The system architecture shown in this embodiment provides a method for pedestrian tracking. The execution subject of this method can be a server or a part of the device in the server (such as a processor). The pedestrian tracking method can include the following steps S301 to S304.

[0059] S301, acquire at least one first video data and at least one second video data.

[0060] Each of the aforementioned first video data sets is acquired by the image acquisition device 110 and contains multiple frames of first images. Each of the aforementioned second video data sets is acquired by the image acquisition device 110 and contains multiple frames of second images.

[0061] In some embodiments, at least one first video data is acquired by at least one image acquisition device 110_A and sent to server 120 via routing device 140. Upon receiving the first video data, server 120 can process each acquired first video data (e.g., image extraction) using processor 201 included in server 120 to obtain multiple frames of first images contained in all the first video data. At least one second video data is acquired by at least one image acquisition device 110_B and sent to server 120 via routing device 140. Similarly, server 120 can obtain multiple frames of second images contained in at least one second video data.

[0062] For example, when the image acquisition device 110 is a camera device, it can receive first video data from a first camera device and second video data from a second camera device, wherein the first camera device and the second camera device are different. That is, the first video data can be acquired by the first camera device and the second video data can be acquired by the second camera device.

[0063] In other embodiments, at least one first video data and at least one second video data are acquired by the same image acquisition device 110. In this case, when the server 120 acquires at least one first video data, it is equivalent to acquiring at least one second video data simultaneously, without needing to perform the step of acquiring the second video data again. Subsequently, the server 120 can process the acquired at least one first video data to obtain multiple frames of first images, which is equivalent to obtaining multiple frames of second images.

[0064] For example, when the image acquisition device 110 is a camera device, first video data and second video data can be received from the third camera device. That is, the first video data and second video data can be acquired by the same third camera device.

[0065] For example, the first multi-frame image can be stored only once on server 120, which is used for face recognition in subsequent S302 and also as a second multi-frame image for human detection in subsequent S303.

[0066] As another example, the first multi-frame image can be stored in two copies on server 120, one for face recognition in subsequent S302, and the other as the second multi-frame image for human detection in subsequent S303.

[0067] In some other embodiments, at least one first video data and at least one second video data contain a portion of video data acquired by the same image acquisition device 110, while the other video data are acquired by different image acquisition devices 110.

[0068] In summary, in this application, the video data used for face recognition in subsequent S302 is referred to as the first video data, and the video data used for human detection in subsequent S303 is referred to as the second video data. The two can be acquired by the same, partially the same, or completely different image acquisition devices 110.

[0069] S302. Perform face recognition on at least a portion of the first images in the multiple frames of the first video data to obtain the tracking data of the first pedestrian.

[0070] The tracking data for the first pedestrian includes: the pedestrian's identity information and the pedestrian's location information at N time points, where N is greater than or equal to 1.

[0071] In this embodiment, the location information of the first pedestrian can be the location coordinates of the first pedestrian. The location coordinates can be the coordinates of the pedestrian in the global coordinate system, or optionally, the coordinates in the world coordinate system, or the coordinates relative to a coordinate system with a specific location as the origin. For example, with a specific location indoors or outdoors as the origin, the pedestrian's location coordinates are the coordinates of the pedestrian relative to that specific location as the origin.

[0072] Those skilled in the art should understand that facial recognition is a method of identity verification based on a person's facial features.

[0073] In some embodiments, the above-mentioned S302 can be specifically implemented by the following S401 to S404.

[0074] S401. Detect whether there is a pedestrian's face in each of at least a portion of the first images.

[0075] In some embodiments, the SSD (Single Shot MultiBox Detector) algorithm can be used to detect whether an image (e.g., the first image) contains the face of a pedestrian (e.g., the first pedestrian).

[0076] like Figure 4 The diagram shows the model bounding box used in face detection using the SSD algorithm, where conv represents a convolutional layer and priorbox represents a prior bounding box. Those skilled in the art should understand that when using the SSD algorithm to detect whether an image contains a face, the image can be input first, and then processed through a trained model. That is, the image can be sequentially processed through multiple different convolutional layers (e.g., ...). Figure 4 The image is processed through five different convolutional layers (including Conv6 and Conv7), each outputting a feature map. These feature maps are then fed into a detection layer to output two sets of data: one set representing the probability of each class within the predicted bounding box, and the other set representing the location of the predicted bounding box. The detection layer contains two distinct 3x3 convolutional layers, used to output confidence for classification and localization for regression, respectively. This process yields the bounding boxes for the face.

[0077] It should be noted that since the SSD algorithm is a well-known algorithm in the art, a detailed explanation of this algorithm will not be repeated in this embodiment. S402, when the face of the first person is detected, the facial feature information of the first person is obtained, and the facial feature information of the first person is compared with the facial feature information in the face database.

[0078] This embodiment does not limit the method for obtaining the facial feature information of the first pedestrian; for example, a neural network method can be used.

[0079] In this embodiment, when using a neural network method to obtain facial feature information, an image f1 (which can be the entire first image or a portion of the first image containing the facial region) including a facial region (e.g., a bounding box of the face) can be input first. Then, this image f1 is passed through a trained classification network (e.g., a ResNet network) to extract features from the facial region, such as the facial features. Based on the color, texture, and other attribute vectors of each feature, a 1024-dimensional vector data for each feature is generated. Finally, the cosine distance between the generated 1024-dimensional vector data for each feature and the 1024-dimensional vector data for each facial feature in an image f2 in the face database is calculated. The feature vector x in image f1 is defined as (x1, x2, ..., x...). i ,…,x n ), x i Let f1 be the attribute vector, and f2 be the feature vector y = (y1, y2, ..., y2) in f2. i , ..., y n ), y i Let i be the attribute vector of image f2, where i is any integer between 1 and n, and n is the dimension of the vector data. For a 1024-dimensional vector, n is 1024. The cosine distance between the two images can be used as the similarity score (also called cosine similarity) between images f1 and f2. If the similarity score is greater than or equal to a threshold, then the face in image f1 and the face in image f2 in the face database are considered to be the same. Conversely, if the similarity score is less than the threshold, then the face in image f1 and the face in image f2 in the face database are not considered to be the same.

[0080] The formula for calculating the cosine distance of the included angle is:

[0081] Specifically, in the above formula for the cosine distance of the included angle, x i The attribute vector of image f1, y i This represents the attribute vector of image f2.

[0082] It should be noted that ResNet (Residual Neural Network) is well known to those skilled in the art, therefore, a detailed explanation of this network will not be repeated in this embodiment.

[0083] This embodiment does not limit the threshold of the similarity score. Those skilled in the art can determine the threshold of the similarity score according to actual needs. For example, the threshold of the similarity score can be 0.6 or 0.7.

[0084] In some embodiments, facial features can be, for example, the facial facial features themselves, and the facial feature information can include information such as the color and texture of the facial features. The face database contains facial feature information and the identity information of the pedestrian corresponding to that facial feature information. The server will compare the obtained facial feature information of the first pedestrian with each facial feature information stored in the face database in turn.

[0085] If the comparison is successful, meaning that there is a face feature in the face database that belongs to the same person as the first person, then S403 can be executed; if the comparison fails, meaning that there is no face feature in the face database that belongs to the same person as the first person, then S404 can be executed.

[0086] S403. Use the identity information of the pedestrian that matches the facial feature information of the first pedestrian in the face database as the identity information of the first pedestrian.

[0087] A face database can store facial feature information of multiple faces, such as Face1, Face2, etc., and corresponding identity information, such as ID1, ID2, etc. Assuming that the facial feature information of the first person in the group belongs to the same person as Face2, then ID2, corresponding to Face2, will be used as the identity information of the first person in the group.

[0088] S404. If the comparison fails, record the facial feature information of the first person in the row.

[0089] At this point, the server can issue a comparison failure notification, such as sending a message to the terminal indicating a failure. Additionally, after the face database is updated, the server can compare the recorded facial feature information of the first person in the row with the updated facial feature information in the database; if the comparison is successful, a success notification can be issued.

[0090] Based on S401 to S404 above, in some examples, assuming that at least one video data contains 10 frames of first images for face recognition, the server can sequentially detect whether there is a first person's face in these 10 first images. If a first person's face is detected in the first frame, the server will execute S402 to S404 above; if no face is detected in the first frame, the server will detect the second frame. If a first person's face is detected in the second frame, the server will execute S402 to S404 above; if not, the server will detect the third frame, and so on.

[0091] Furthermore, when the face of a first pedestrian is detected in a frame of the first image in step S401, the location information of the first pedestrian can be determined. The location information of the first pedestrian can be coordinates in the first image or coordinates in the global coordinate system, i.e., global coordinates. After detecting the face of the first pedestrian, the coordinates of the first pedestrian in the first image can be converted to obtain the coordinates in the global coordinate system, i.e., global coordinates. When the face of the first pedestrian is detected, a bounding box corresponding to the face can be obtained on the first image. The coordinates of a point in the bounding box can be calculated as the coordinates of the first pedestrian in the first image, such as an endpoint or midpoint. Since each image corresponds to a specific region in the real world, and each position in the image corresponds to a position in the real world, after establishing the global coordinate system, each position in the image corresponds to a coordinate in the global coordinate system. Therefore, the coordinates of an object in the image can be obtained in the global coordinate system. For example, the coordinates of the first pedestrian in the global coordinate system can be determined by the coordinates of one or more reference points (e.g., the upper left and upper right corners of the image) in a frame captured by the corresponding image acquisition device, both in the global coordinate system and within the image itself, as well as the position of the first pedestrian within that frame. The positions of the reference points in the frame captured by the image acquisition device can be obtained from the device's position, installation height, shooting angle, and shooting resolution. This embodiment does not limit the method used to determine the first pedestrian's position information.

[0092] S303. Perform human detection on at least a portion of the second images in the multiple frames of the second video data to obtain the tracking data of the second pedestrian.

[0093] The tracking data of the second pedestrian mentioned above includes the movement trajectory of the second pedestrian, which includes location information at multiple times. In other words, the location information of the second pedestrian at multiple times constitutes the movement trajectory of the second pedestrian.

[0094] In this embodiment, human detection refers to a technology that can distinguish different pedestrians and return the location information of each pedestrian. In some embodiments, the above-mentioned S303 can be specifically implemented by the following S3031.

[0095] S3031. Obtain pedestrian bounding boxes from at least a portion of the second images in the multiple frames of the second video data, and obtain tracking data of the second pedestrian based on the obtained pedestrian bounding boxes.

[0096] The pedestrian bounding box can be a bounding box labeled based on the human body detected in each second image. For example, the pedestrian bounding box can be a rectangle drawn based on the pedestrian's outline, and therefore can also be called a pedestrian rectangle. The pedestrian's position can be determined by the midpoint of the bottom edge of the rectangle. For example, in the direction perpendicular to the horizontal plane, the two long sides of the rectangle can be drawn based on the positions of the pedestrian's left side (e.g., left arm) and right side (e.g., right arm); in the direction parallel to the horizontal plane, the two short sides of the rectangle can be drawn based on the positions of the pedestrian's top (e.g., top of the head) and bottom (e.g., where the pedestrian's feet are). In this way, a rectangle can be drawn based on the pedestrian's outline, and the pedestrian's position can be determined by the midpoint of the bottom edge of the rectangle.

[0097] In this embodiment, the pedestrian bounding boxes in the second image can be obtained using algorithms such as SSD, YOLO (You Only Look Once), or Faster R-CNN. Since these algorithms are well-known to those skilled in the art, they will not be described in detail in this embodiment.

[0098] When a second pedestrian's body is detected in a second frame of the image in step S3031, the pedestrian's position information can be determined. This position information can be coordinates within the second image or coordinates in the global coordinate system (i.e., global coordinates). The calculation method for the second pedestrian's global coordinates is similar to that of the first pedestrian. When a second pedestrian is detected, a frame corresponding to the human body can be obtained in the second image. Optionally, the coordinates of a point within the frame can be calculated as the second pedestrian's coordinates in the second image, such as an endpoint or midpoint. Alternatively, the location of the pedestrian's feet can be identified from the image as the second pedestrian's coordinates in the second image. Then, the coordinates of the object in the image can be obtained in the global coordinate system. For example, the coordinates of one or more reference points (e.g., the upper left and upper right corners of the image) in a frame captured by the image acquisition device corresponding to that frame, both in the global coordinate system and in the image itself, along with the second pedestrian's position in that frame, can be used to determine the second pedestrian's coordinates in the global coordinate system.

[0099] In some embodiments, the tracking data of the second pedestrian based on the acquired pedestrian frames described in S3031 above can be obtained through the following two steps (i.e., step one and step two below).

[0100] Step 1: Use the trained neural network model to process the image blocks corresponding to each pedestrian box to obtain the feature vectors of the image blocks corresponding to each pedestrian box.

[0101] This embodiment does not impose any restrictions on the neural network model. For example, the neural network model described above can be a convolutional neural network model or a deep neural network model.

[0102] Those skilled in the art should understand that a Convolutional Neural Network (CNN) is a deep feedforward artificial neural network whose artificial neurons can respond to a portion of the surrounding units within their coverage area. It performs exceptionally well in large-scale image processing and is currently being applied in the field of image recognition.

[0103] In some embodiments, a convolutional neural network can be trained and used to process image patches contained in pedestrian bounding boxes to output feature vectors for the pedestrian bounding boxes. These feature vectors are then transmitted to a server so that the server can obtain the feature vectors of the pedestrian bounding boxes. For example, a VGG (Visual Geometry Group) 16 network or a ResNet50 network can be used to process the data and output 2048-dimensional feature vectors for each pedestrian bounding box.

[0104] It should be noted that VGG and ResNet50 networks are technologies well known to those skilled in the art, therefore, detailed explanations of these two networks will not be repeated in this embodiment.

[0105] In some embodiments, during the training of the convolutional neural network, a publicly available international dataset (mainly focusing on indoor scenes) can be used first. Based on the existing data model, the data model can be trained with new training samples to obtain a new data model. The feature vector output by the trained convolutional neural network can be used to distinguish whether two pedestrian bounding boxes belong to the same pedestrian.

[0106] Step 2: Determine the pedestrian trajectory of at least one pedestrian based on the feature vector of each pedestrian box in each pedestrian box, and use it as the tracking data of the second pedestrian. Each pedestrian in the at least one pedestrian box corresponds to a pedestrian trajectory.

[0107] In some embodiments, step two above can be specifically implemented by the following steps S501 to S507.

[0108] S501. Divide all pedestrian boxes into at least one (e.g., multiple) pedestrian trajectories to obtain a trajectory set.

[0109] Specifically, pedestrian matching can be performed on all pedestrian frames, and pedestrian frames belonging to the same pedestrian can be grouped into the same pedestrian trajectory.

[0110] In some embodiments, pedestrian matching can be performed by matching the similarity scores of the feature vectors of any two different pedestrian boxes. When the similarity score is greater than a first threshold, the pedestrian boxes are considered to be successfully matched, and the two different pedestrian boxes belong to the same pedestrian. When performing similarity matching, the calculation formula for the cosine distance of the included angle described above can be used, and the calculated cosine distance can be used as the similarity score between the feature vectors of the two different pedestrian boxes. It should be noted that this embodiment does not limit the value of the first threshold; for example, the first threshold can be 0.6 or 0.7.

[0111] In other embodiments, the pedestrian matching process can refer to a pedestrian re-identification (Person re-ID) algorithm. The pedestrian re-identification algorithm can predict the position and posture of pedestrians in the next frame image based on the position and posture of pedestrians in the previous frame image (which can be represented by the feature vector of the pedestrian bounding box). If the position and posture of the pedestrian detected in the next frame are consistent with the prediction or the similarity is greater than or equal to a second threshold, it can be confirmed that the pedestrians in these two frames are the same person (i.e., the pedestrian bounding box matching is successful). If they are inconsistent, it can be confirmed that they are not the same person. Here, the previous frame image and the next frame image are two adjacent frames. For example, assuming that 30 pedestrian bounding boxes are detected in 10 second images, in S501, these 30 pedestrian bounding boxes can be divided into 10 pedestrian trajectories through pedestrian matching, and these 10 pedestrian trajectories form a trajectory set.

[0112] For another example, if 30 pedestrian bounding boxes are detected in 10 second images, these 30 pedestrian bounding boxes can be directly divided into 30 pedestrian trajectories. That is, each pedestrian trajectory contains only one pedestrian bounding box, and these 30 pedestrian trajectories form a trajectory set.

[0113] It is understandable that if, in S501 above, all pedestrian boxes are assigned to the same pedestrian trajectory through pedestrian matching, then the subsequent steps can be omitted, and the obtained pedestrian trajectory is the required pedestrian movement trajectory. S502: For each pedestrian trajectory in the trajectory set, calculate the feature vector of the pedestrian trajectory based on the feature vector of the pedestrian box contained in the trajectory.

[0114] In some embodiments, for each pedestrian trajectory in the trajectory set, a weighted average or arithmetic average can be performed on the feature vectors of the pedestrian bounding boxes contained in the trajectory to obtain the feature vector of the pedestrian trajectory. When performing a weighted average, the weights can be set as needed; this embodiment does not limit this. For example, assuming a pedestrian trajectory contains 10 pedestrian bounding boxes, the feature vectors of these 10 bounding boxes can be weighted and averaged, and the average result can be used as the feature vector of the pedestrian trajectory.

[0115] S503. Determine the data matching degree of all pedestrian trajectories in the trajectory set based on the feature vector of the pedestrian trajectory.

[0116] The aforementioned data matching degree is obtained by combining motion matching degree and appearance feature matching degree.

[0117] In some embodiments, pedestrian trajectories in the trajectory set are compared pairwise to obtain the motion matching degree and appearance feature matching degree between the two pedestrian trajectories.

[0118] Motion matching degree is characterized by Mahalanobis distance; a larger Mahalanobis distance indicates a greater motion matching degree between the two pedestrian trajectories. The formula for calculating Mahalanobis distance is: Where Σ is the covariance matrix of the multidimensional random variables, and μ is the sample mean.

[0119] Apparent feature matching degree is characterized by the cosine distance between the feature vectors. The larger the cosine distance, the greater the apparent matching degree between the two pedestrian trajectories. The formula for the cosine distance can be found in the above embodiment, and will not be repeated here.

[0120] In other embodiments, the data matching degree can also be represented by the Euclidean distance between the two pedestrian trajectories. A larger Euclidean distance indicates a lower data matching degree; conversely, a smaller Euclidean distance indicates a higher data matching degree. The formula for calculating the Euclidean distance is as follows: Here, xi and yi both represent eigenvectors.

[0121] S504. Determine whether the maximum data matching degree among all the determined pedestrian trajectories is greater than the first matching threshold. If yes, proceed to S505; otherwise, proceed to S507.

[0122] It should be noted that the first matching threshold can be set as needed, and this embodiment does not limit it. For example, the first matching threshold can be 60% or 70%.

[0123] S505. Merge the two pedestrian trajectories with the highest data matching degree (i.e., the two pedestrian trajectories with the highest data matching degree) into one pedestrian trajectory to update the trajectory set.

[0124] Multiple pedestrian tracks belonging to the same pedestrian will be merged into one pedestrian track.

[0125] For example, if the set of pedestrian trajectories obtained by S501 at different times includes 10 pedestrian trajectories, the two pedestrian trajectories with the highest data matching degree can be merged into one pedestrian trajectory to update the trajectory set.

[0126] S506. Determine the feature vector of the merged pedestrian trajectory and return to S503. The method in S502 can be used here to calculate the feature vector of the merged pedestrian trajectory.

[0127] S507. Determine the trajectory of each pedestrian in the trajectory set as the tracking data of each second pedestrian.

[0128] For example, suppose the trajectory set obtained in S501 includes three pedestrian trajectories: trajectory 1, trajectory 2, and trajectory 3. The data matching degree between trajectory 1 and trajectory 2 is 90%, between trajectory 2 and trajectory 3 is 80%, and between trajectory 1 and trajectory 3 is 50%. The maximum data matching degree is the matching degree between trajectory 1 and trajectory 2, which is 90%. Assuming the first matching threshold is 75%, the maximum data matching degree of 90% satisfies the requirement of being greater than the first matching threshold of 75%, and trajectory 1 and trajectory 2 can be merged into one pedestrian trajectory. The merged trajectory is called trajectory 4, and the trajectory set now only includes trajectory 3 and trajectory 4. Subsequently, the feature vector of the merged trajectory 4 is determined, and the process returns to S503. Based on the determined feature vector of trajectory 4, the data matching degree between trajectory 3 and trajectory 4 is determined to determine whether they can be merged. If they can be merged, the merged trajectory is called trajectory 5, and the trajectory set now only includes trajectory 5. Trajectory 5 is the tracking data for the second pedestrian. Specifically, trajectory 5 is the movement trajectory of a second pedestrian. If the two cannot be merged, then trajectory 3 and trajectory 4 are the movement trajectories of two different second pedestrians.

[0129] It should be noted that S503 to S506 can be executed continuously in a loop until the maximum data matching degree is less than or equal to the first matching threshold.

[0130] S304. If the deviation between the position information of the first pedestrian at N times and the position information of the second pedestrian's trajectory at the same time is within a preset range, determine the tracking data of the target pedestrian.

[0131] The aforementioned location information includes global coordinates, which are the coordinates of the pedestrian in the global coordinate system; the aforementioned target pedestrian tracking data includes: the target pedestrian's identity information and the target pedestrian's movement trajectory, wherein the first pedestrian and the second pedestrian are both target pedestrians, the identity information of the target pedestrian is determined at least based on the identity information of the first pedestrian, and the movement trajectory of the target pedestrian is determined at least based on the movement trajectory of the second pedestrian.

[0132] It should be noted that the pre-range in this embodiment can be set as needed, and this embodiment does not impose any limitations on it.

[0133] In some embodiments, the following formula can be used Let 1 ≤ i ≤ N. Calculate the distance between the global coordinates of the first person and the global coordinates of the second person at each of the N time points. Specifically, for each of the N time points, if the global coordinates A of the first person are... i The global coordinates B of the second pedestrian i If the distance between them is less than or equal to a preset threshold, then the first pedestrian and the second pedestrian can be identified as the target pedestrians.

[0134] For example, when N is greater than or equal to 2 (i.e., at two moments, such as moment 1), facial recognition technology can be used to determine the location information of the first pedestrian at moment 1, and human detection technology can be used to determine the movement trajectory of the second pedestrian at moment 1 (where the movement trajectory contains the location information of the second pedestrian). The deviation between the two positions at moment 1 is then calculated to see if it is within a preset threshold, i.e., the distance between them is less than or equal to a preset threshold. If the distance is less than or equal to the preset threshold, it can be determined that both the first and second pedestrians are the target pedestrians. Optionally, the location information of the first and second pedestrians needs to be calculated relative to the same coordinate system. For example, both can be coordinates in an image, or both can be global coordinates relative to the same coordinate system.

[0135] For example, at time N, suppose the global coordinates of the first pedestrian determined by facial recognition technology are A. i (x1 i y1 i The global coordinates of the second pedestrian, determined by human detection technology, are B. i (x2 i y2 i ), through formula Calculate A i Point and B i If the distance between points is less than or equal to a preset threshold, then it can be determined that the first pedestrian and the second pedestrian are the same person, i.e., the target pedestrian.

[0136] For example, at multiple (at least two) time points, such as time points N-1, N, and N+1, the location information of the first pedestrian and the second pedestrian can be determined using facial recognition and human detection technologies at these three time points, and the distance between them can be calculated at each of these three time points. If the distance between them at each of these three time points is less than or equal to a preset threshold, then it can be determined that the first pedestrian and the second pedestrian are the same person, i.e., the target pedestrian. This can improve the accuracy of identifying the target pedestrian.

[0137] In some embodiments, the location information of the first pedestrian at N time points is pre-configured in the system. Specifically, when monitoring using facial recognition technology, if camera 1 is used to monitor area A, the location coordinates of area A will be extracted from camera 1 to obtain the location information of the first pedestrian at N time points.

[0138] The movement trajectory of the second pedestrian, i.e., the position information of the second pedestrian at N time points, changes in real time. Specifically, when monitoring using human detection technology, a monocular visual positioning algorithm can be used to convert pixel positions into coordinates, thereby obtaining the position information of the second pedestrian at N time points, and thus obtaining the movement trajectory of the second pedestrian.

[0139] Among them, the monocular vision positioning algorithm transforms the image pixel position of the tracked target (such as a second pedestrian) into two-dimensional coordinates projected onto the ground in the world coordinate system based on the installation position and internal parameters of the surveillance camera, thereby realizing the generation and display of the second pedestrian's movement trajectory.

[0140] In some embodiments, when the first pedestrian and the second pedestrian are both target pedestrians, since the movement trajectory of the second pedestrian is determined in S303, the movement trajectory of the target pedestrian is the movement trajectory of the second pedestrian.

[0141] Since combining the position coordinates of the first pedestrian at N times can form the movement trajectory of the first pedestrian, the tracking data of the first pedestrian can include the movement trajectory of the first pedestrian, wherein the movement trajectory of the first pedestrian includes the position information of the first pedestrian at N times.

[0142] Since the identity information of the first pedestrian was determined in S302 and the movement trajectory of the second pedestrian was determined in S303, and both the first and second pedestrians are the target pedestrians, it can be concluded that the determined identity information of the first pedestrian is the identity information of the target pedestrian; and the determined movement trajectory of the second pedestrian is the movement trajectory of the target pedestrian. Thus, the movement trajectory of the target pedestrian can be determined based on the movement trajectory of the second pedestrian, and the identity information of the first pedestrian can be used as the identity information of the target pedestrian, thereby obtaining the tracking data of the target pedestrian; that is, establishing a mapping relationship (also called a correspondence relationship) between the movement trajectory and the identity information of the target pedestrian. When situations such as partial occlusion or significant changes in posture occur, because a mapping relationship between the movement trajectory and identity information of the target pedestrian has been established, as long as the pedestrian's face is not completely obscured, the pedestrian's identity information can be identified. This allows the movement trajectories of the two pedestrians before and after the aforementioned situation to be matched with the identity information of the same first pedestrian, thereby reducing the probability of target tracking loss when using human detection methods for pedestrian tracking.

[0143] In some embodiments, S304 can be implemented by S304a.

[0144] S304a. Based on the movement trajectories of the first pedestrian and the second pedestrian, generate the movement trajectory of the target pedestrian. For example... Figure 5 As shown, the pedestrian tracking method provided in this embodiment may further include the following steps S305 to S306.

[0145] S305, Receive the first access command.

[0146] The first access instruction is configured to access the tracking data of the target pedestrian.

[0147] S306, Output access results or the first interface.

[0148] In this embodiment, a user interface is displayed on the terminal. This user interface can be presented by a browser or application installed on the terminal. For example, the user can click the view button displayed on the user interface to trigger the terminal to send a request to the server to access the tracking data of the target pedestrian.

[0149] In some embodiments, the server may directly output the access results. For example, after obtaining the access results, the server may output the obtained access results through an output device (e.g., a display screen) included in the server.

[0150] The above-mentioned access results include: the identity information of the target pedestrian and the movement trajectory of the target pedestrian.

[0151] In other embodiments, the server may output a first interface and transmit the first interface to a terminal so that the first interface is displayed on the terminal's screen, wherein the first interface may be configured to display access results.

[0152] In some embodiments, after determining the movement trajectory of the target pedestrian, the distribution area of ​​the target pedestrian and the dwell time of the target pedestrian in the distribution area can be obtained based on the movement trajectory of the target pedestrian. This allows the terminal's display screen to not only display the identity information and movement trajectory of the target pedestrian, but also the distribution area and the dwell time of the target pedestrian in the distribution area.

[0153] In other embodiments, the image of the target pedestrian is extracted from at least a portion of the first images in the first video data, or the image of the target pedestrian is extracted from at least a portion of the second images in the second video data, so that the display screen of the terminal can display not only the identity information and movement trajectory of the target pedestrian, but also the image of the target pedestrian.

[0154] In some other embodiments, the image of the target pedestrian is extracted from at least a portion of the first images in the first video data, and the image of the target pedestrian is extracted from at least a portion of the second images in the second video data, so that the display screen of the terminal can display not only the identity information and movement trajectory of the target pedestrian, but also the image of the target pedestrian.

[0155] like Figure 5 As shown, the pedestrian tracking method provided in this embodiment may further include the following steps S307 to S308.

[0156] S307, Receive the second access command.

[0157] The second access instruction mentioned above is configured as an access statistics result.

[0158] S308, Output statistical results or second interface.

[0159] The statistical results mentioned above include at least one of the first data, the second data, the third data, and the fourth data; the second interface is configured to display the statistical results.

[0160] The first data point is the total number of target pedestrians within at least one distribution area at the current time, such as... Figure 6 As shown, at the current time, there are a total of 3761 people in regions A, B, C, and D.

[0161] The second data is the number or proportion of pedestrians whose dwell time in at least one distribution area falls within different preset time intervals at the current time, where the first proportion is the ratio of the first number to the total number of people. For example... Figure 6 As shown, taking a preset time of 15 minutes as an example, within 0 to 15 minutes, the first proportion of staying in area A, area B, area C and area D is 99.97%.

[0162] The third data point, in the case of multiple distribution areas, represents the second number or proportion of target pedestrians within each distribution area over a given time period. This second proportion is the ratio of the second number of pedestrians to the total number of pedestrians. For example... Figure 6 As shown, in the case of four areas, namely area A, area B, area C and area D, taking a preset time of 15 minutes as an example, the proportion of people staying in area B within 0 to 15 minutes is 80%.

[0163] The fourth data point represents the total number of target pedestrians within at least one distribution area at different times during a historical time period. For example, taking the past 24 hours as the historical time period... Figure 6 As shown, this displays the total number of people at various times over the past 24 hours.

[0164] In some embodiments, the statistical results can be obtained by performing data statistics based on the movement trajectories of each target pedestrian in at least one distribution area.

[0165] It should be noted that, in this embodiment, the pedestrian tracking methods shown in the above figures are all illustrated by way of example with reference to one figure in this embodiment. In specific implementation, the pedestrian tracking methods shown in the above figures can also be implemented in conjunction with any other figures shown in the above embodiments, which will not be elaborated here.

[0166] like Figure 7As shown, this embodiment provides a pedestrian tracking device, which may include an acquisition module 701 and a processing module 702. The acquisition module 701 is configured to acquire at least one first video data and at least one second video data, each first video data containing multiple frames of first images, and each second video data containing multiple frames of second images. The processing module 702 is configured to perform face recognition on at least a portion of the first images acquired by the acquisition module 701 to obtain tracking data of a first pedestrian, the tracking data of the first pedestrian including: the identity information of the first pedestrian and the location information of the first pedestrian at N time points; N is greater than or equal to 1. The processing module 702 is also configured to perform human detection on at least a portion of the second images acquired by the acquisition module 701 to obtain tracking data of a second pedestrian. The tracking data for the second pedestrian includes: the movement trajectory of the second pedestrian; the movement trajectory of the second pedestrian includes the location information of the second pedestrian at multiple times; the processing module 702 is further configured to determine the tracking data of the target pedestrian when the deviation between the location information of the first pedestrian at N times and the location information of the second pedestrian's movement trajectory at the same time is within a preset range; the tracking data of the target pedestrian includes: the identity information of the target pedestrian and the movement trajectory of the target pedestrian, wherein the first pedestrian and the second pedestrian are both target pedestrians, the identity information of the target pedestrian is determined based on the identity information of the first pedestrian, and the movement trajectory of the target pedestrian is determined at least based on the movement trajectory of the second pedestrian.

[0167] In some embodiments, the processing module 702 is specifically configured to detect whether there is a pedestrian's face in each of at least a portion of the first images; if a pedestrian's face is detected, the processing module 702 acquires the pedestrian's facial feature information and compares the pedestrian's facial feature information with facial feature information in a face database; the face database contains multiple facial feature information and pedestrian identity information corresponding to the multiple facial feature information; if the comparison is successful, the pedestrian identity information in the face database that matches the pedestrian's facial feature information is used as the pedestrian's identity information; or, if the comparison fails, the pedestrian's facial feature information is recorded.

[0168] In some embodiments, the processing module 702 is specifically configured to acquire pedestrian bounding boxes in each of at least a portion of the second images in a plurality of frames of second images, and to obtain tracking data of a second pedestrian based on each of the acquired pedestrian bounding boxes, wherein the pedestrian bounding box is a box shape marked according to the human body detected in each second image.

[0169] In some embodiments, the processing module 702 is specifically configured to process the image blocks corresponding to each of the pedestrian boxes using a trained neural network model to obtain the feature vectors of the image blocks corresponding to each pedestrian box; determine the pedestrian trajectory of at least one pedestrian based on the feature vector of each pedestrian box in each pedestrian box; and use the pedestrian trajectory of at least one pedestrian as the tracking data of a second pedestrian, wherein each pedestrian in the at least one pedestrian corresponds to a pedestrian trajectory.

[0170] In some embodiments, the processing module 702 is specifically configured to calculate the distance between the global coordinates of the first pedestrian and the global coordinates of the second pedestrian at each of the N time points using the following formula; For each of the N time moments, if the global coordinates A of the first person are... i The global coordinates B of the second pedestrian i If the distance between them is less than or equal to a preset threshold, then the first pedestrian and the second pedestrian are determined to be the same target pedestrian; where, at the i-th time, the global coordinates of the first pedestrian are A. i (x1 i y1 i The global coordinates of the second person are B. i (x2 i y2 i ), 1≤i≤N; the location information includes global coordinates, which are the coordinates of the pedestrian in the global coordinate system.

[0171] In some embodiments, the movement trajectory of the target pedestrian is the movement trajectory of the second pedestrian; or, the tracking data of the first pedestrian includes: the identity information of the first pedestrian and the movement trajectory of the first pedestrian, the movement trajectory of the first pedestrian includes the location information of the first pedestrian at N time points; the processing module is specifically configured to generate the movement trajectory of the target pedestrian based on the movement trajectory of the first pedestrian and the movement trajectory of the second pedestrian.

[0172] In some embodiments, such as Figure 7 As shown, the pedestrian tracking device provided in this embodiment further includes a receiving module 703, which is configured to receive first video data from a first camera device and second video data from a second camera device, wherein the first camera device and the second camera device are different; or, the receiving module 703 is configured to receive the first video data and the second video data from a third camera device. In some embodiments, such as Figure 7As shown, the pedestrian tracking device provided in this embodiment also includes a first output module 704 and a receiving module 703, which is further configured to receive a first access instruction, which is configured to access the tracking data of the target pedestrian; the first output module 704 is configured to output the access result or a first interface, which is configured to display the access result, including: the identity information of the target pedestrian and the movement trajectory of the target pedestrian.

[0173] In some embodiments, the processing module 702 is further configured to obtain the distribution area of ​​the target pedestrian and the dwell time of the target pedestrian in the distribution area based on the movement trajectory of the target pedestrian; and / or, the processing module 702 is further configured to extract the image of the target pedestrian from at least a portion of the first images in multiple frames of first images; and / or, the processing module 702 is further configured to extract the image of the target pedestrian from at least a portion of the second images in multiple frames of second images; when the pedestrian tracking method includes outputting access results or a first interface, the access results further include at least one of the following: the image of the target pedestrian, the distribution area of ​​the target pedestrian, and the dwell time of the target pedestrian in the distribution area.

[0174] In some embodiments, the pedestrian tracking device provided in this embodiment further includes a second output module and a receiving module, which are further configured to receive a second access instruction, the second access instruction being configured to access statistical results; the second output module is configured to output statistical results or a second interface, the second interface being configured to display statistical results; the statistical results include at least one of: first data, second data, third data, and fourth data; wherein, the first data is the total number of target pedestrians in at least one distribution area at the current time; the second data is the first number or first proportion of target pedestrians whose dwell time in at least one distribution area is in different preset time intervals at the current time, the first proportion being the ratio of the first number to the total number; the third data is the second number or second proportion of target pedestrians in each distribution area during a time period when there are multiple distribution areas, the second proportion being the ratio of the second number to the total number; the fourth data is the total number of target pedestrians in at least one distribution area at different times during a historical time period.

[0175] like Figure 2 The diagram shown is a possible hardware structure diagram of a server provided in this embodiment. Figure 2 Therefore, the memory 202 includes an operating system and instructions (e.g., computer instructions), wherein the computer instructions may include four layers: an application layer, an algorithm layer, a base layer, and a user layer.

[0176] The system includes a video stream processing module in the application layer, which processes video data (e.g., first video data) to generate multiple frames of images (e.g., first image). A face recognition algorithm (used to implement S302 above) in the algorithm layer then processes the generated multiple frames of images to generate tracking data (e.g., identity information) of the pedestrian corresponding to the face (e.g., first pedestrian). A human detection and tracking algorithm (used to implement S303 above) in the algorithm layer processes the generated multiple frames of images to generate tracking data (e.g., motion trajectory) of a pedestrian (e.g., second pedestrian). In the process of generating the motion trajectory of the second pedestrian using the human detection and tracking algorithm, a visual positioning algorithm in the algorithm layer can be used to convert the pixel positions in the multiple frames of images into coordinates, thereby obtaining the position information of the second pedestrian at N time points, and thus obtaining the motion trajectory of the second pedestrian.

[0177] The base layer can include a database, stored data, and network components. The database can be a relational database, storing facial feature information of multiple faces and their corresponding identity information; the stored data can include pedestrian tracking data (e.g., the identity information of the first pedestrian); and the network components can be network interfaces that connect servers to terminals or other devices.

[0178] After the algorithm layer generates tracking data for the first pedestrian and the second pedestrian, these results can be input into the user layer. After the user layer receives these results, it can display them, such as at least one of the following: the identity information, movement trajectory, loitering area of ​​the first pedestrian, and a portrait of the first pedestrian.

[0179] Those skilled in the art should understand that a relational database is a database built on the relational model, which uses mathematical concepts and methods such as set algebra to process data in the database. It is a database composed of multiple interconnected two-dimensional row lists.

[0180] This invention also provides a server for implementing the pedestrian tracking method described in the above embodiments.

[0181] In some embodiments, the server may include a memory and a processor, wherein the memory is used to store computer instructions and the processor is used to execute the computer instructions, causing the server to implement the pedestrian tracking method described in the above embodiments.

[0182] This invention also provides a pedestrian tracking system, which includes a memory, a processor, a first camera device, and a second camera device. The memory is used to store computer instructions, and the processor is used to execute the computer instructions, enabling the server to implement the pedestrian tracking method described in any of the above embodiments. The first camera device is used to acquire at least one first video data, and the second camera device is used to acquire at least one second video data.

[0183] This invention also provides a computer-readable storage medium, such as a non-transient computer-readable storage medium, on which a computer program is stored. When executed by a processor, this computer program implements the various processes of the above-described pedestrian tracking method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for pedestrian tracking, characterized in that, include: Acquire at least one first video data and at least one second video data, wherein each first video data contains multiple frames of first images and each second video data contains multiple frames of second images; Face recognition is performed on at least a portion of the first images in the multiple frames of the first image to obtain tracking data of the first pedestrian. The tracking data of the first pedestrian includes: the identity information of the first pedestrian and the position information of the first pedestrian at N time points; N is greater than or equal to 1; the position information of the first pedestrian at the N time points is determined based on the position coordinates of the area where the first pedestrian is located. Human detection is performed on at least a portion of the second images in the multiple frames of second images to obtain tracking data of a second pedestrian. The tracking data of the second pedestrian includes: the motion trajectory of the second pedestrian; the motion trajectory of the second pedestrian includes the position information of the second pedestrian at multiple times; the position information of the second pedestrian at the multiple times is obtained by converting the pixel positions of the second images. The distance between the global coordinates of the first pedestrian and the global coordinates of the second pedestrian is calculated at each of the N time points using the following formula; For each of the N time moments, if the global coordinates A of the first pedestrian... i and the global coordinates B of the second pedestrian i If the distance between them is less than or equal to a preset threshold, then the first pedestrian and the second pedestrian are determined to be the same target pedestrian. Wherein, at the i-th time, the global coordinates of the first pedestrian are A. i (x1 i y1 i The global coordinates of the second pedestrian are B. i (x2 i y2 i ), 1≤i≤N; the location information includes the global coordinates, which are the coordinates of the pedestrian in the global coordinate system; The tracking data of the target pedestrian includes: the identity information of the target pedestrian and the movement trajectory of the target pedestrian, wherein the first pedestrian and the second pedestrian are both the target pedestrian, the identity information of the target pedestrian is determined at least based on the identity information of the first pedestrian, and the movement trajectory of the target pedestrian is determined at least based on the movement trajectory of the second pedestrian.

2. The pedestrian tracking method according to claim 1, characterized in that, Face recognition is performed on at least a portion of the first images in the multiple frames of the first image to obtain tracking data of the first pedestrian, including: Detect whether each of the at least a portion of the first images contains a pedestrian's face; Upon detecting the face of a first pedestrian, the facial feature information of the first pedestrian is acquired, and the facial feature information of the first pedestrian is compared with the facial feature information in the face database; the face database contains multiple facial feature information and the identity information of pedestrians corresponding to each of the multiple facial feature information. If the comparison is successful, the identity information of the pedestrian who matches the facial feature information of the first pedestrian in the face database is used as the identity information of the first pedestrian; or, if the comparison fails, the facial feature information of the first pedestrian is recorded.

3. The pedestrian tracking method according to claim 1, characterized in that, Human detection is performed on at least a portion of the second images in the multiple frames of the second images to obtain tracking data of the second pedestrian, including: Obtain pedestrian bounding boxes in at least a portion of the second images from the multiple frames of second images, and obtain tracking data for the second pedestrians based on the obtained pedestrian bounding boxes, wherein the pedestrian bounding boxes are boxes labeled according to the human body detected in each of the second images.

4. The pedestrian tracking method according to claim 3, characterized in that, Based on the acquired pedestrian bounding boxes, the tracking data of the second pedestrian is obtained, including: The trained neural network model is used to process the image blocks corresponding to each pedestrian frame to obtain the feature vector of the image block corresponding to each pedestrian frame; Based on the feature vector of each pedestrian box in each of the pedestrian boxes, the pedestrian trajectory of at least one pedestrian is determined, and the pedestrian trajectory of the at least one pedestrian is used as the tracking data of the second pedestrian, wherein each pedestrian in the at least one pedestrian corresponds to a pedestrian trajectory.

5. The pedestrian tracking method according to claim 1, characterized in that, The movement trajectory of the target pedestrian is the movement trajectory of the second pedestrian; or, The tracking data of the first pedestrian includes: the identity information of the first pedestrian and the movement trajectory of the first pedestrian, the movement trajectory of the first pedestrian includes the location information of the first pedestrian at N time points; If the deviation between the position information of the first pedestrian at N times and the position information of the second pedestrian's movement trajectory at the same time is within a preset range, the step of determining the tracking data of the target pedestrian includes: generating the movement trajectory of the target pedestrian based on the movement trajectory of the first pedestrian and the movement trajectory of the second pedestrian.

6. The pedestrian tracking method according to any one of claims 1 to 5, characterized in that, Also includes: The first video data is received from a first camera device, and the second video data is received from a second camera device, wherein the first camera device and the second camera device are different. Alternatively, the first video data and the second video data can be received from a third camera device.

7. The pedestrian tracking method according to any one of claims 1 to 5, characterized in that, Also includes: Receive a first access instruction, the first access instruction being configured to access the tracking data of the target pedestrian; Output access results or a first interface, the first interface being configured to display the access results, the access results including: the identity information of the target pedestrian and the movement trajectory of the target pedestrian.

8. The pedestrian tracking method according to any one of claims 1 to 5, characterized in that, Also includes: Based on the movement trajectory of the target pedestrian, the distribution area of ​​the target pedestrian and the dwell time of the target pedestrian in the distribution area are obtained; And / or, Extract the image of the target pedestrian from at least a portion of the first images in the multiple frames of the first images; And / or, Extract the image of the target pedestrian from at least a portion of the second images in the multiple frames of second images; When the pedestrian tracking method includes outputting access results or a first interface, the access results further include at least one of: an image of the target pedestrian, a distribution area of ​​the target pedestrian, and the dwell time of the target pedestrian in the distribution area.

9. The pedestrian tracking method according to claim 8, characterized in that, Also includes: Receive a second access instruction, which is configured as access statistics results; Output statistical results or a second interface, the second interface being configured to display the statistical results; the statistical results include at least one of: first data, second data, third data, and fourth data; Wherein, the first data is the total number of target pedestrians in at least one distribution area at the current time; The second data is the first number or first proportion of the target pedestrians whose dwell time in the at least one distribution area is in different preset time intervals at the current time, and the first proportion is the ratio of the first number to the total number of people; The third data is, in the case that there are multiple distribution areas, the second number or second proportion of target pedestrians in each distribution area during a time period, where the second proportion is the ratio of the second number to the total number of people. The fourth data is the total number of target pedestrians in the at least one distribution area at different times within a historical time period.

10. A pedestrian tracking device, characterized in that, Includes an acquisition module and a processing module; The acquisition module is configured to acquire at least one first video data and at least one second video data, each first video data containing multiple frames of first images, and each second video data containing multiple frames of second images; The processing module is configured to perform face recognition on at least a portion of the first images in the multiple frames of first images acquired by the acquisition module to obtain tracking data of a first pedestrian. The tracking data of the first pedestrian includes: the identity information of the first pedestrian and the position information of the first pedestrian at N times; N is greater than or equal to 1; the position information of the first pedestrian at the N times is determined based on the position coordinates of the area where the first pedestrian is located. The processing module is further configured to perform human detection on at least a portion of the second images in the multiple frames of second images acquired by the acquisition module to obtain tracking data of a second pedestrian. The tracking data of the second pedestrian includes: the motion trajectory of the second pedestrian; the motion trajectory of the second pedestrian includes the position information of the second pedestrian at multiple times; the position information of the second pedestrian at the multiple times is obtained by converting the pixel positions of the second images. The processing module is further configured to: calculate the distance between the global coordinates of the first pedestrian and the global coordinates of the second pedestrian at each of the N time points using the following formula; For each of the N time moments, if the global coordinates A of the first pedestrian... i and the global coordinates B of the second pedestrian i If the distance between them is less than or equal to a preset threshold, then the first pedestrian and the second pedestrian are determined to be the same target pedestrian. Wherein, at the i-th time, the global coordinates of the first pedestrian are A. i (x1 i y1 i The global coordinates of the second pedestrian are B. i (x2 i y2 i ), 1≤i≤N; the location information includes the global coordinates, which are the coordinates of the pedestrian in the global coordinate system; the tracking data of the target pedestrian includes: the identity information of the target pedestrian and the movement trajectory of the target pedestrian, wherein the first pedestrian and the second pedestrian are both the target pedestrian, the identity information of the target pedestrian is determined at least based on the identity information of the first pedestrian, and the movement trajectory of the target pedestrian is determined at least based on the movement trajectory of the second pedestrian.

11. The pedestrian tracking device according to claim 10, characterized in that, The processing module is specifically configured to detect whether each of the at least some of the first images contains a pedestrian's face; when a pedestrian's face is detected, the module acquires the pedestrian's facial feature information and compares the pedestrian's facial feature information with facial feature information in a face database; the face database contains multiple facial feature information and pedestrian identity information corresponding to the multiple facial feature information respectively. If the comparison is successful, the identity information of the pedestrian who matches the facial feature information of the first pedestrian in the face database will be used as the identity information of the first pedestrian. Alternatively, if the comparison fails, the facial feature information of the first pedestrian is recorded.

12. The pedestrian tracking device according to claim 10, characterized in that, The processing module is specifically configured to acquire pedestrian bounding boxes in at least a portion of the second images in the multi-frame second images, and to obtain tracking data of the second pedestrians based on the acquired pedestrian bounding boxes, wherein the pedestrian bounding boxes are boxes marked according to the human body detected in each of the second images.

13. The pedestrian tracking device according to claim 12, characterized in that, The processing module is specifically configured to use a trained neural network model to process the image blocks corresponding to each pedestrian frame in order to obtain the feature vectors of the image blocks corresponding to each pedestrian frame. Based on the feature vector of each pedestrian box in each of the pedestrian boxes, the pedestrian trajectory of at least one pedestrian is determined, and the pedestrian trajectory of the at least one pedestrian is used as the tracking data of the second pedestrian, wherein each pedestrian in the at least one pedestrian corresponds to a pedestrian trajectory.

14. The pedestrian tracking device according to claim 10, characterized in that, The movement trajectory of the target pedestrian is the movement trajectory of the second pedestrian; or, The tracking data of the first pedestrian includes: the identity information of the first pedestrian and the movement trajectory of the first pedestrian, the movement trajectory of the first pedestrian includes the location information of the first pedestrian at N time points; The processing module is specifically configured to generate the movement trajectory of the target pedestrian based on the movement trajectory of the first pedestrian and the movement trajectory of the second pedestrian.

15. The pedestrian tracking device according to any one of claims 10 to 14, characterized in that, It also includes a receiving module; The receiving module is configured to receive the first video data from a first camera device and the second video data from a second camera device, wherein the first camera device and the second camera device are different. Alternatively, the receiving module is configured to receive the first video data and the second video data from a third camera device.

16. A server, characterized in that, include: A memory and a processor, wherein the memory is used to store computer instructions and the processor is used to execute the computer instructions, causing the server to implement the pedestrian tracking method according to any one of claims 1 to 9.

17. A pedestrian tracking system, characterized in that, The system includes a memory, a processor, a first camera device, and a second camera device, wherein the memory is used to store computer instructions, the processor is used to execute the computer instructions, and the server implements the pedestrian tracking method according to any one of claims 1 to 9, the first camera device is used to acquire the at least one first video data, and the second camera device is used to acquire the at least one second video data.

18. A computer-readable storage medium, characterized in that, Used to store computer instructions, such that when the server executes the computer instructions, it implements the pedestrian tracking method according to any one of claims 1 to 9.

Images