Object tracking method, device, equipment, medium and program product

Abstract

The present disclosure provides an object tracking method, which relates to the field of security inspection, the field of artificial intelligence or other fields. The method includes: obtaining a current image frame of a target object in a monitoring area, and a current grid unit in which the target object is located in the monitoring area, wherein the monitoring area is pre-divided into M grid units, where M is an integer greater than or equal to 2; matching N tracked objects according to the feature vector of the target object extracted from the current image frame, where N is an integer greater than or equal to 1; obtaining the transition probability that each tracked object appears in the current grid unit; and determining the target object from the N tracked objects according to the size of the transition probability. The present disclosure also provides an object device, an apparatus, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of security inspection, artificial intelligence or other fields, and more specifically, to an object tracking method, device, equipment, medium and program product. Background Art

[0002] In some places, there is a need for security inspection. Deploying one or more cameras to obtain image information to track objects is an important function for tracking control targets in the field of security inspection. For example, the trajectory of the control target is tracked during the tracking process.

[0003] In related technologies, continuous multi-frame acquisition by a single camera can be used as the basis for tracking based on motion model prediction and simple features of the target object. Alternatively, multiple cameras can be deployed and a cross-camera re-identification method based on target object features can be used.

[0004] In the process of realizing the invention concept, the inventors found that in some places there are usually multiple similar targets (such as pedestrians, luggage or vehicles, etc.). In the process of re-identification of multiple targets in a large range, the tracking method based on a single camera or cross-camera in the related technology will have a high false alarm rate, affecting the accuracy and efficiency of tracking. Summary of the invention

[0005] In view of the above problems, the present disclosure provides an object tracking method, apparatus, device, medium and program product.

[0006] According to a first aspect of the present disclosure, an object tracking method is provided, characterized in that it includes: obtaining a current image frame of a target object in a monitoring area, and a current grid unit in which the target object is located in the monitoring area, wherein the monitoring area is pre-divided into M grid units, where M is an integer greater than or equal to 2; matching N tracked objects based on a feature vector of the target object extracted from the current image frame, where N is an integer greater than or equal to 1; obtaining a transition probability that each of the tracked objects appears in the current grid unit; and determining the target object from the N tracked objects based on the magnitude of the transition probability.

[0007] According to an embodiment of the present disclosure, obtaining the current grid unit in which the target object is located in the monitoring area includes: determining the current grid unit according to current spatiotemporal information of the target object.

[0008] According to an embodiment of the present disclosure, obtaining the transition probability of each tracked object appearing in the current grid unit includes: obtaining the transition probability of each tracked object transferring from the nearest grid unit to the current grid unit, and the nearest grid unit is determined from the M grid units based on the latest spatiotemporal information of each tracked object.

[0009] According to an embodiment of the present disclosure, obtaining the transfer probability of each tracked object from the nearest grid unit to the current grid unit includes: taking the nearest spatiotemporal information of each tracked object as the starting spatiotemporal information, and obtaining the spatiotemporal change between the starting spatiotemporal information and the current spatiotemporal information; obtaining the transfer probability from trajectory probability data based on the spatiotemporal change, wherein the trajectory probability data is obtained in advance through the historical trajectory of objects that have appeared in the monitoring area between the M grid units.

[0010] According to an embodiment of the present disclosure, obtaining the transition probability from the trajectory probability data based on the spatiotemporal variation includes: for each of the tracked objects, if its nearest grid unit is the same as the current grid unit, obtaining the transition probability within the same grid unit from the trajectory probability data based on the spatiotemporal variation; if its nearest grid unit is different from the current grid unit, obtaining the transition probability between different grid units from the trajectory probability data based on the spatiotemporal variation.

[0011] According to an embodiment of the present disclosure, the trajectory probability data is obtained in advance through the historical trajectory of the objects that have appeared in the monitoring area between the M grid units, including: obtaining the historical trajectory of S objects that have appeared in the monitoring area within a specific time period, where S is an integer greater than or equal to 1; in each of the grid units, based on the historical trajectory of each object that has appeared, the time difference and distance difference between the start and end times of each displacement are counted, and the starting and end points of the displacement are located in the same or different grid units; and the trajectory probability data is obtained based on the statistically obtained time difference and distance difference.

[0012] According to an embodiment of the present disclosure, obtaining the trajectory probability data based on the statistically obtained time difference and distance difference includes: discretizing the statistically obtained time difference and distance difference based on a preset unit duration and a preset unit distance to obtain a discretization result; and obtaining the trajectory probability data based on the discretization result.

[0013] According to an embodiment of the present disclosure, the discretization result includes a space-time coordinate system with a preset unit time length as a time scale and a preset unit distance as a space scale; the trajectory probability data is obtained based on the statistics of the discretization result, including: for each of the grid units, counting the first occurrence times of the S objects that have appeared in the grid unit corresponding to each space-time coordinate; based on the ratio of the first occurrence times corresponding to each space-time coordinate to the total number of samples S, the transfer probability within the same grid unit corresponding to the space-time coordinate is obtained.

[0014] According to an embodiment of the present disclosure, obtaining the trajectory probability data based on the statistics of the discretization results also includes: for each of the grid units, counting the second occurrence times of the S objects that have appeared in another grid unit corresponding to each space-time coordinate; based on the ratio of the second occurrence times corresponding to each space-time coordinate to the total number of samples S, obtaining the transition probability between different grid units corresponding to the space-time coordinate.

[0015] According to an embodiment of the present disclosure, the trajectory probability data includes a transition probability within the same grid unit corresponding to each space-time coordinate in the space-time coordinate system, or a transition probability between different grid units corresponding to each space-time coordinate.

[0016] According to an embodiment of the present disclosure, after determining the target object from the N tracked objects according to the size of the transition probability, the method further includes: generating a movement trajectory based on the current grid unit and the nearest grid unit of the target object; taking the target object as the S+1th previously appeared object and taking the movement trajectory as the historical trajectory to update the trajectory probability data.

[0017] According to an embodiment of the present disclosure, the feature vector of each tracked object is pre-stored in a tracking library, and matching N tracked objects based on the feature vector of the target object extracted from the current image frame includes: calculating the similarity between the feature vector of the target object and any feature vector in the tracking library; matching the N tracked objects based on a similarity threshold.

[0018] According to an embodiment of the present disclosure, determining the target object from the N tracked objects according to the size of the transition probability includes: correcting the corresponding similarity based on the size of the transition probability corresponding to each of the tracked objects; and determining the target object from the N tracked objects according to the corrected similarity corresponding to each of the tracked objects.

[0019] According to an embodiment of the present disclosure, when N is greater than or equal to 2, the N tracked objects are sorted based on the size of the similarity, wherein determining the target object from the N tracked objects according to the corrected similarity corresponding to each of the tracked objects includes: re-sorting the N tracked objects according to the size of the corrected similarity corresponding to each of the tracked objects; and determining the target object from the N tracked objects according to the result of the re-sorting.

[0020] According to an embodiment of the present disclosure, K cameras are pre-deployed in the monitoring area, K is an integer greater than or equal to 2, and the object tracking includes: performing cross-camera tracking of the target object in the monitoring area based on the K cameras.

[0021] According to an embodiment of the present disclosure, pre-dividing the monitoring area into M grid units includes: making the imaging range of the K cameras cover the monitoring area; performing pixel-level calibration on the imaging screens of the K cameras and the scene map of the monitoring area; dividing the scene map into the M grid units, wherein the M grid units are mapped to the imaging screens of the corresponding cameras based on the calibration results.

[0022] According to an embodiment of the present disclosure, after determining the target object from the N tracked objects based on the size of the transition probability, the method further includes: obtaining the current position of the target object based on the current image frame; and visually displaying the current position and the previous tracked position of the target object on the scene map.

[0023] According to an embodiment of the present disclosure, obtaining the current image frame of the target object in the monitoring area includes: obtaining a surveillance video containing the target object captured by at least one of the K cameras; parsing the surveillance video to obtain at least one of the current image frames; wherein the method further includes: in response to a playback operation on the current position in the scene map, playing the surveillance video containing the current image frame.

[0024] Another aspect of an embodiment of the present disclosure provides an object tracking device, characterized in that it includes: a target recognition module, used to obtain a current image frame of a target object in a monitoring area, and a current grid unit in which the target object is located in the monitoring area, wherein the monitoring area is pre-divided into M grid units, where M is an integer greater than or equal to 2; a vector matching module, used to match N tracked objects based on a feature vector of the target object extracted from the current image frame, where N is an integer greater than or equal to 1; a probability acquisition module, used to obtain a transition probability that each of the tracked objects appears in the current grid unit; and an object determination module, used to determine the target object from the N tracked objects based on the size of the transition probability.

[0025] Another aspect of an embodiment of the present disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method described above.

[0026] Another aspect of an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor executes the method as described above.

[0027] Another aspect of an embodiment of the present disclosure provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0028] One or more of the above embodiments have the following beneficial effects: the monitoring area is divided into multiple grid units, feature vectors are extracted based on the current image frame of the target object, and N tracked objects are matched, the movement rules of each tracked object between grid units are used as constraints, the transition probability appearing in the current grid unit is obtained, and the tracking and identification results are further optimized and corrected according to the size of the transition probability. This improves the accuracy of target object re-identification and improves the accuracy and continuity of target object identification on the entire trajectory path. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above contents and other purposes, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0030] Figure 1 The following schematically shows an application scenario diagram of object tracking according to an embodiment of the present disclosure;

[0031] Figure 2 A flowchart of an object tracking method according to an embodiment of the present disclosure is schematically shown;

[0032] Figure 3 A flowchart for obtaining a transition probability according to an embodiment of the present disclosure is schematically shown;

[0033] Figure 4 A flowchart of obtaining trajectory probability data in advance according to an embodiment of the present disclosure is schematically shown;

[0034] Figure 5 Schematically shows a flow chart of obtaining trajectory probability data according to another embodiment of the present disclosure;

[0035] Figure 6 A flowchart of an object tracking method according to another embodiment of the present disclosure is schematically shown;

[0036] Figure 7 A flowchart of an object tracking method according to another embodiment of the present disclosure is schematically shown;

[0037] Figure 8 An example of an engineering drawing of a scene map according to an embodiment of the present disclosure is schematically shown;

[0038] Fig. 9 An example of a camera deployment solution according to an embodiment of the present disclosure is schematically shown;

[0039] Fig.10The effect diagram of the world coordinate system mapping of the imaging plane according to the embodiment of the present disclosure is schematically shown;

[0040] Fig.11 An example of a target object re-identification trajectory map projection according to an embodiment of the present disclosure is schematically shown;

[0041] Fig.12 A block diagram schematically shows a structure of an object tracking device according to an embodiment of the present disclosure; and

[0042] Fig.13 A block diagram of an electronic device suitable for implementing an object tracking method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0043] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0044] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding prompts for users to choose to authorize or refuse.

[0045] In the related technology, based on the continuous multi-frame acquisition of a single camera, tracking is performed according to the motion model prediction and the simple features of the target object. The advantages of this method are low computational complexity, high real-time performance, and good continuity within the local tracking trajectory. The disadvantages are that the tracking range is small and can only be tracked within the current camera field of view. In addition, in the process of multi-target tracking, problems such as tracking trajectory interruption or tracking target error may occur.

[0046] The advantage of cross-camera re-identification based on target object features is that it is not limited by the camera position and does not require continuous recognition of multiple sequence frames. The target object can be re-identified based on the feature extraction of a single frame target and the retrieval of the target library. The visible light images collected by the surveillance camera in the cross-camera solution also have the following series of problems: different ambient lighting, different field of view angles, and different distances cause different clarity of the target object, which will reduce the consistency of target features and increase the rate of missed recognition and false recognition. Larger acquisition time and space ranges will increase the number of targets to be matched and the number of interference items, thereby increasing the false recognition rate.

[0047] At the same time, in the process of re-identifying large-scale multi-target objects, there are a large number of identical or similar appearance features between the multiple targets. In this case, relying solely on appearance features for large-scale target re-identification will result in a high false alarm rate, affecting the overall performance of the system.

[0048] Therefore, for multi-target monitoring scenes with similar appearance features, tracking within the range of a single camera based on pure vision or re-identification using multiple cameras has a certain probability of misidentification when the resolution of the monitoring targets is small, the appearance is highly similar, or the ambient light is very different. By projecting the results of the target object re-identification onto the scene map, it is found that the trajectory of the monitoring target will be interrupted, or the running trajectory will jump unreasonably, which makes it difficult to track and analyze the target trajectory.

[0049] Some embodiments of the present disclosure provide an object tracking method, which divides the monitoring area into multiple grid units, extracts feature vectors based on the current image frame of the target object, matches N tracked objects, uses the movement rules of each tracked object between grid units as a constraint, obtains the transition probability that appears in the current grid unit, and further optimizes and corrects the tracking and recognition results according to the size of the transition probability. This improves the accuracy of target object re-recognition and improves the accuracy and continuity of target object recognition on the entire trajectory path.

[0050] Figure 1 The following schematically shows an application scenario diagram of object tracking according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown are merely examples to which the embodiments of the present disclosure can be applied, so as to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0051] like Figure 1As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, a server 105, a first camera 106, a second camera 107, a turntable 108, luggage 109, and pedestrians (such as passenger A 110, passenger B 111, and passenger C 112). The network 104 is used to provide a medium for communication links between the first camera 106, the second camera 107, the terminal devices 101, 102, 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0052] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0053] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0054] The server 105 may be a server that provides various services, such as a background management server that provides support for websites browsed by users using the terminal devices 101, 102, and 103 (for example only). The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device. For example, the server 105 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud computing, network services, and middleware services.

[0055] For example, at the airport, passengers who get off the plane go to the carousel 108 to pick up their luggage. Before the luggage enters the carousel 108, each piece of luggage can be subjected to radiation scanning to detect whether the luggage contains items that are not allowed to be carried. If suspected luggage containing items that are not allowed to be carried is detected, it can be marked, and the first camera 106, the second camera 107 and other cameras monitor the carousel 108, the airport hall and places outside the airport, and send the monitoring images to the terminal devices 101, 102, 103 and the server 105 through the network 104. Therefore, at the carousel 108 and after the owner of the suspected luggage picks up the luggage, the object tracking method or object tracking device provided in some embodiments of the present disclosure is executed to continuously track the trajectory.

[0056] It should be noted that the object tracking method provided in the embodiments of the present disclosure can generally be executed by at least one of a terminal device or a server. Accordingly, the object tracking device provided in the embodiments of the present disclosure can generally be set in at least one of a terminal device or a server. In addition, the venues targeted by the present disclosure are not limited to airports, and the objects are not limited to luggage. For example, the venues may include indoors, squares, such as stations, scenic spots, etc., and the objects may also include pedestrians, vehicles, etc.

[0057] It should be understood that Figure 1 The number of terminal devices, cameras, networks and servers in the embodiment is only for illustration. Any number of terminal devices, cameras, networks and servers may be provided as required.

[0058] The following will be based on Figure 1 The scene described by Figure 2~Figure 11 The object tracking method of the embodiment of the present disclosure is described in detail.

[0059] Figure 2 The flowchart of the object tracking method according to the embodiment of the present disclosure is schematically shown.

[0060] like Figure 2 As shown, this embodiment includes:

[0061] In operation S210, a current image frame of a target object in a monitoring area and a current grid unit in which the target object is located in the monitoring area are obtained. The monitoring area is pre-divided into M grid units, where M is an integer greater than or equal to 2.

[0062] The monitoring area refers to an area that can be detected by a camera in one or more places. The monitoring area can be a partial or full area in a place, or cover multiple places. Target objects can include pedestrians, luggage, passengers, animals or objects, etc. The target object can be pre-marked or detected in real time by face recognition, gesture detection, radiation scanning, etc. Each grid unit refers to a local area in the monitoring area, which can be a regular or irregular shape, and one grid unit can have the same or different shape as any other grid unit. For example, one grid unit can be a rectangle, and the other grid unit can be a rectangle or an irregular polygon.

[0063] In some embodiments, the monitoring area is divided into a series of grid cells, each grid cell representing a local area. For example, according to the size of the monitoring area, the map is divided into U*V (U and V are both positive integers greater than 1) grid cells. More specifically, big data statistics can be performed within a specific time period. For example, in the past month, grid cells are divided according to dense areas and sparse areas where pedestrians and luggage are located in the monitoring area. For example, in the airport hall, the dense areas where pedestrians and luggage are located include the area near the turntable and the path area where you leave the airport hall after picking up your luggage from the turntable. In this way, detailed divisions can be performed, that is, the area of ​​the grid unit is relatively small. Other sparse areas can be divided into relatively large grid cells.

[0064] In some embodiments, K cameras are pre-deployed in the monitoring area, where K is an integer greater than or equal to 2, and object tracking includes: performing cross-camera tracking of a target object in the monitoring area based on the K cameras, so as to expand the monitoring range without being limited by the camera positions.

[0065] In some embodiments, pre-dividing the monitoring area into M grid units includes: making the imaging range of the K cameras cover the monitoring area. Performing pixel-level calibration on the imaging screens of the K cameras and the scene map of the monitoring area. Dividing the scene map into M grid units, wherein the M grid units are mapped to the imaging screens of the corresponding cameras based on the calibration results.

[0066] Calibration is used in image measurement and machine vision applications to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, and to establish a geometric model of camera imaging. These geometric model parameters are the camera parameters. Calibration of camera parameters is the process of solving the intrinsic and extrinsic parameters and distortion parameters of the camera through experiments and calculations. The role of calibration is that, on the one hand, since the degree of distortion of each lens is different, camera calibration can correct this lens distortion and generate a corrected image. On the other hand, the monitoring scene can be reconstructed based on the obtained image.

[0067] Reference Figure 1 Each camera has its own imaging range, and the entire monitoring area can be covered by reasonable camera deployment. Pixel-level calibration includes mapping the relationship between each pixel on the imaging screen and each position point in the scene map. Its role is to accurately obtain the position of the target object and the current grid unit through pixel-level granularity calibration, provide accurate data support for the determination of spatiotemporal information and the acquisition of transition probability, and allow the grid unit to be divided at a smaller granularity and accurately reflected in the image.

[0068] According to the embodiments of the present disclosure, the proposed camera full coverage scene deployment scheme and the method of achieving pixel-level point-to-point mapping with the actual scene map after internal and external parameter calibration of the imaging plane can solve the need for tracking target coordinate refinement and form an accurate and smooth action trajectory map.

[0069] In operation S220 , N tracked objects are matched according to the feature vector of the target object extracted from the current image frame, where N is an integer greater than or equal to 1.

[0070] In some embodiments, the feature vector of each tracked object is pre-stored in a tracking library, and matching the N tracked objects according to the feature vector of the target object extracted from the current image frame includes: calculating the similarity between the feature vector of the target object and any feature vector in the tracking library, and matching the N tracked objects based on a similarity threshold.

[0071] For example, the tracking library exists in the form of a database table. Each time a feature vector of a tracked object is determined, it is written into the database table. After the feature vector of the target object is extracted, the Euclidean distance between the feature vector of the target object and each feature vector in the database table is calculated to obtain the feature vector. The similarity threshold is used for screening. For example, the similarity range is between 0 and 1, and the similarity threshold is 0.9. That is, N tracked objects with a similarity greater than or equal to the similarity threshold are matched from the tracking library.

[0072] In some other embodiments, in addition to the similarity matching method, the feature vector of the target object can also be input into the neural network model to obtain the predicted identifier output by the neural network model, which can be an identity identifier. According to the predicted identifier, N tracked objects are retrieved from the database.

[0073] In operation S230, a transition probability of each tracked object appearing in the current grid unit is obtained.

[0074] When there are multiple tracked objects, multi-object tracking (MOT) is performed. Multi-object tracking is responsible for handling tracking and recognition tasks, and uses target detection and positioning tasks to help solve tasks such as behavior analysis. The purpose of the multi-object tracking task is to obtain the action trajectory of each target in the video. In addition to locating the specific position of each target in each frame, it is also necessary to ensure that the identity of each target remains unchanged. The action trajectory of each target obtained by the multi-object tracking task can provide a lot of valuable information. For example, the position, speed, acceleration, when it appears, when it disappears, and other information of each target can be used to process computer vision tasks such as behavior recognition and behavior prediction. In addition, multiple trajectories can be combined to process statistical traffic and analyze interactions between targets.

[0075] In some embodiments, a prediction method is used to predict or reversely deduce the trajectory of each tracked object's time, position, and distance-related spatiotemporal features. The transfer probability is calculated based on whether the movement of the same tracked object in the same time period conforms to the objective law, and then, for example, a weighted comprehensive judgment can be made on the time, position, and distance-related spatiotemporal features. When the comprehensive correlation coefficient is greater than a given threshold, the transfer probability is greater, and the probability of being the current target object is greater.

[0076] In other embodiments, the position, speed, acceleration, direction, posture and other information of the target object can be input into the machine learning model to obtain the predicted previous position. The most recent position, speed, acceleration, direction, posture and other information of the tracked object can be input into the machine learning model to obtain the predicted next position. The transition probability is obtained by comparing the position deviation between the two prediction results, that is, the smaller the deviation, the greater the transition probability.

[0077] In other embodiments, big data statistics can be performed in advance based on historical data, taking into account the moving distance and direction of objects at different times, and reflecting the probability of objects in each grid unit not leaving the unit within a period of time and the probability of objects transferring between any two grid units through group behavior. Furthermore, targeted statistics can be performed based on the type of target object, such as pedestrians, luggage, and vehicles. Pedestrians pulling luggage cause the movement patterns of the two to be roughly the same, while the movement patterns of vehicles are different from those of pedestrians and luggage. Therefore, the type of target object can be determined, and then the transfer probability can be queried from the pre-statisticated data.

[0078] In operation S240 , a target object is determined from the N tracked objects according to the magnitude of the transition probability.

[0079] Exemplarily, the tracked object with the largest transition probability can be directly taken as the target object. The similarity and transition probability of each tracked object can also be multiplied by different weight coefficients and summed, and the tracked object with the largest sum can be taken as the target object. The transition probability of each tracked object can also be used as a weight coefficient of the similarity to recalculate the similarity. Finally, a tracking trajectory is formed based on the tracked data. For example, determining the target object from N tracked objects according to the size of the transition probability includes: based on the size of the transition probability corresponding to each tracked object, correcting the corresponding similarity; and determining the target object from N tracked objects according to the corrected similarity corresponding to each tracked object.

[0080] In some preferred embodiments, the target object includes luggage. It can be understood that in the process of pedestrian target tracking and re-identification, a variety of obvious features such as clothing, body shape, gait, and carry-on luggage are extracted. In the process of vehicle re-identification, the license plate is unique, and the vehicle body target is large and the appearance features are obviously different. Luggage is an industrial product with little difference in style and size, and there is no unique identifier, so the misidentification rate is high. Therefore, it not only has a good recognition effect on pedestrians or vehicles with obvious features. It can also use the transfer probability to assist in luggage re-identification, which can greatly improve the recognition accuracy.

[0081] According to the embodiment of the present disclosure, the monitoring area is divided into multiple grid units, the feature vector is extracted based on the current image frame of the target object, and the N tracked objects are matched. The movement rules of each tracked object between grid units are used as constraints to obtain the transition probability that appears in the current grid unit, and the tracking and recognition results are further optimized and corrected according to the size of the transition probability. This improves the accuracy of target object re-identification and improves the accuracy and continuity of target object recognition on the entire trajectory path.

[0082] In some embodiments, obtaining the current grid unit where the target object is located in the monitoring area in operation S210 includes: determining the current grid unit according to current spatiotemporal information of the target object.

[0083] Spatiotemporal information can include time, camera number, and map coordinates. For example, a tracking library for cross-camera monitoring is established. When a target object is photographed, the appearance feature vector of the target object, the time node of acquisition, the camera number, and the absolute position coordinates projected on the map are recorded. The current grid unit ID is calculated based on the current coordinates.

[0084] In some embodiments, obtaining the transition probability of each tracked object appearing in the current grid unit in operation S230 includes: obtaining the transition probability of each tracked object transferring from the nearest grid unit to the current grid unit, where the nearest grid unit is determined from M grid units based on the latest spatiotemporal information of each tracked object.

[0085] For example, the current time is 10:10 am, and the most recent spatiotemporal information is the data recorded in the tracking library closest to 10:10 am, or the most recently updated data of each tracked object.

[0086] For example, the transfer probability can be determined by the distance between the nearest grid unit and the current grid unit. For example, if the latest spatiotemporal information of a tracked object includes the time 10:09:30 a.m., and its nearest grid unit is 500 meters away from the current grid unit, the transfer probability is relatively small.

[0087] In some embodiments, big data statistics can be performed based on historical data to determine the probability of a target object of each type moving from grid unit A to grid unit B within different time ranges through group behavior.

[0088] According to the embodiments of the present disclosure, with the help of non-visual information, such as the geographic coordinates of the camera and the object, the shooting time of each video frame, and the movement pattern of the target object in the area, single-camera tracking and cross-camera object detection re-identification results are optimized based on the space-time route constraints. This reduces the misidentification rate of the monitored object, improves the accuracy and continuity of the identification of the monitored object on the entire path, and provides a more accurate and continuous detection object trajectory identification and reconstruction method.

[0089] Figure 3 The flowchart of obtaining the transition probability according to the embodiment of the present disclosure is schematically shown.

[0090] like Figure 3 As shown, this embodiment is one of the embodiments of operation S230, including:

[0091] In operation S310, the most recent spatiotemporal information of each tracked object is used as the starting spatiotemporal information, and the spatiotemporal variation between the starting spatiotemporal information and the current spatiotemporal information is obtained.

[0092] Exemplarily, the spatiotemporal variation includes a time variation, a position coordinate variation, and may also include a change in a camera number.

[0093] In operation S320 , a transition probability is obtained from trajectory probability data based on the spatiotemporal variation, wherein the trajectory probability data is obtained in advance through historical trajectories of objects that have appeared between M grid cells in the monitoring area.

[0094] Exemplarily, the trajectory probability data may include a trajectory probability matrix or a trajectory probability coordinate system, which may be stored in a database table or other file. Specifically, the trajectory probability data includes a mapping relationship between different spatiotemporal variations and different transition probabilities. Different spatiotemporal variations include different spatial variations at the same time, different temporal variations at the same space, or different temporal variations corresponding to different spatial variations.

[0095] Since there are different types of target objects and different places, in order to further accommodate the random movement of various target objects in different places, this embodiment proposes to use the most recent spatiotemporal information of the tracked object as the starting time and starting place (i.e., the starting spatiotemporal information), evaluate the situation where the tracked object appears at different distances compared to the current spatiotemporal information within different time ranges, and perform probabilistic statistics. And analyze historical experience in the same way in advance to analyze the trajectory characteristics of the target object's movement. This is used as prior knowledge to improve the robustness of multi-tracking trajectory prediction for multiple targets.

[0096] According to an embodiment of the present disclosure, the spatiotemporal trajectory of historical behaviors is used as non-visual information to perform re-identification of the target object based on spatiotemporal information constraints, thereby improving the re-identification accuracy.

[0097] Figure 4 The flowchart of obtaining trajectory probability data in advance according to an embodiment of the present disclosure is schematically shown.

[0098] like Figure 4 As shown, the trajectory probability data obtained in advance in this embodiment includes:

[0099] In operation S410, historical trajectories of S objects that have appeared in a monitoring area within a specific time period are obtained, where S is an integer greater than or equal to 1.

[0100] Exemplarily, the actual scene map of the monitoring area is divided into grids, for example, in units of 20m*20m squares. Then, through manual labeling, all time and location information of objects with the same ID are obtained to form a historical track.

[0101] In operation S420, in each grid unit, the time difference and distance difference between the start and end times of each displacement are counted based on the historical trajectory of each object that has appeared, and the start and end points of the displacement are located in the same or different grid units.

[0102] In operation S430 , trajectory probability data is obtained based on the statistically obtained time difference and distance difference.

[0103] For example, the grid area where the current target position is located is determined in each grid unit. (i.e. the current grid unit), where m and n are the horizontal x-axis ID and y-axis ID of the grid after grid division.

[0104] The last time the target object appeared in its action trajectory is taken as time zero.

[0105] (Unit: milliseconds), with the location as the distance zero (Unit: meter), calculate the next time the target object appears (i.e. the current time and current location )'s relative time of action trajectory (in milliseconds) and relative distance (Unit: meter).

[0106]

[0107] in, That is, the time difference, That is the distance difference.

[0108] According to an embodiment of the present disclosure, the transition probability data between grid units can be statistically analyzed by time difference and distance difference, thereby introducing group behavior data for big data analysis based on historical experience as prior knowledge.

[0109] Figure 5 A flowchart for obtaining trajectory probability data according to another embodiment of the present disclosure is schematically shown.

[0110] like Figure 5 As shown, this embodiment is one of the embodiments of operation S430, including:

[0111] In operation S510, the time difference and the distance difference obtained by statistical discretization are obtained based on the preset unit duration and the preset unit distance to obtain a discretization result.

[0112] In operation S520 , trajectory probability data is obtained based on the discretization result statistics.

[0113] For example, Discretization (For example, discretize it into units of one minute), Discretization (For example, discretization is performed in units of ten meters).

[0114] The role of discretization is to unify the historical trajectories of different sample objects, making it easier to perform probability statistics in the same unit. It is also compatible with the spatiotemporal changes between the latest spatiotemporal information of different tracked objects and the target object. Regardless of the size of the spatiotemporal changes, it can be discretized into one or more ∆T and ∆D. When obtaining the transition probability in operation S230, the calculation speed during reasoning can be improved, and the acquisition of the transition probability has high real-time performance.

[0115] In some embodiments, the discretization result includes a space-time coordinate system with a preset unit time length as a time scale and a preset unit distance as a space scale. Obtaining trajectory probability data based on the discretization result includes: for each grid unit, counting the first occurrence times of the S objects that have appeared in the grid unit corresponding to each space-time coordinate. Based on the ratio of the first occurrence times corresponding to each space-time coordinate to the total number of samples S, the transition probability within the same grid unit corresponding to the space-time coordinate is obtained.

[0116] Exemplarily, the object that has appeared may be a historical object with authorization, or may be an object that is scheduled for simulation by a staff member.

[0117] For example, statistics of all labeled data in time distance Node under the grid area The first occurrence of . Each node corresponds to each space-time coordinate.

[0118] Step 5: By time distance The number of occurrences under a node Divide by the total number of samples S to get distance The target object under the node is in the area Transition probability .

[0119]

[0120] It should be noted that the space-time coordinate system is only one of the representation forms of the discretization results to facilitate the understanding of the technical solution. Therefore, the trajectory probability data can be represented in other forms, such as the transition probability matrix. In essence, the corresponding relationship between the transition probability is determined under the space-time dimension of the time scale and the space scale.

[0121] Taking the space-time coordinate system as an example, the time dimension is the horizontal axis and the space dimension is the vertical axis. As time zero, As the spatial zero point, with a preset unit duration (such as one minute) as the time scale, and a preset unit distance (such as 10 meters) as the space scale, each coordinate (t, s) between the horizontal axis and the vertical axis represents a time-space coordinate, and each time-space coordinate can correspond to a transition probability value. For example, it can be represented and stored in the form of (t1, s1, 65%), representing the first time scale t1 starting from the time zero point, the first space scale s1 starting from the distance zero point, and the corresponding grid area where the transition probability is the current target position. The transfer probability is 65%.

[0122] For example, if the time difference between the most recent spatiotemporal information and the current spatiotemporal information of a tracked object is 3 time scales and 5 space scales, then (t3, s5, 70%) can be queried in the spatiotemporal coordinate system.

[0123] In some embodiments, in order to prevent the occurrence of noise in the time dimension and the space dimension from deviating from the transition probability range, after executing Formula 3 to obtain the space-time coordinate system, filtering may be performed on at least one of the time dimension and the space dimension.

[0124] Will Perform filtering in the time dimension, for example, perform mean filtering on the five minutes before and after, as shown in Formula 4.

[0125]

[0126] Next, we use the value obtained from Formula 4 Filtering is performed in the spatial dimension, such as distance difference Rice is The mean filtering is performed within the range, as shown in Formula 5.

[0127]

[0128] It should be noted that the above method of filtering in the time dimension first and then in the space dimension is only an example. The space dimension can be filtered first and then in the time dimension, or filtering can be performed in only one dimension.

[0129] The trajectory between multiple grid cells includes movement within the same grid cell and movement within different grid cells. The trajectory across grid cells is further described below.

[0130] In some embodiments, obtaining trajectory probability data based on the discretization result statistics further includes: for each grid unit, counting the second appearance times of the S objects that have appeared in each spatiotemporal coordinate in another grid unit. Based on the ratio of the second appearance times corresponding to each spatiotemporal coordinate to the total number of samples S, obtaining the transition probability between different grid units corresponding to the spatiotemporal coordinate.

[0131] Exemplarily, the counting of the number of times across grid units includes not making further distinctions in another grid unit, and only counting the total number of times appearing in another grid unit.

[0132] In some embodiments, obtaining the transition probability between different grid units corresponding to the space-time coordinates includes: obtaining the transition probability between each grid unit and any adjacent grid unit.

[0133] That is, based on the total number of times, another grid unit is further distinguished, such as the specific statistical grid area arrive The number of times, but also count the grid area arrive For example, (t3, s6, 80%, → ) in the form of representation and storage. Among them, t3 is the third time scale from the time zero point, s6 is the sixth space scale relative to the zero point, and arrive The transfer probability is 80%.

[0134] In some embodiments, the trajectory probability data includes a transition probability within the same grid unit corresponding to each spatiotemporal coordinate in the spatiotemporal coordinate system, or a transition probability between different grid units corresponding to each spatiotemporal coordinate.

[0135] Obtaining the transition probability from the trajectory probability data based on the spatiotemporal variation includes: for each tracked object, if its nearest grid unit is the same as the current grid unit, obtaining the transition probability within the same grid unit from the trajectory probability data based on the spatiotemporal variation. If its nearest grid unit is different from the current grid unit, obtaining the transition probability between different grid units from the trajectory probability data based on the spatiotemporal variation.

[0136] For example, the time and location history of each tracked object's most recent detection is used as the time zero point and distance zero point to calculate the current time difference with the target object. and distance difference , discretize to get the time parameter and distance parameters , and obtain the transition probability from the trajectory probability data query .

[0137] For example, among multiple tracked objects, there may be some nearest grid cells that are the same as the current grid cell, while other nearest grid cells are different from the current grid cell. In this case, the transition probability is queried specifically.

[0138] According to an embodiment of the present disclosure, the transfer probability of the same grid unit or across grid units is preliminarily counted through historical data, and the relationship between the moving distance, moving direction and grid unit of the group within different time ranges is implicit, which can further improve the accuracy of tracking the target object.

[0139] The trajectory of the on-site monitoring object is constrained based on the historical probability distribution of the trajectory of the object that has appeared, and the conclusion can be optimized based on prior knowledge. This can improve the accuracy of multi-monitoring target and cross-camera tracking and re-identification.

[0140] In some embodiments, when N is greater than or equal to 2, the N tracked objects are sorted based on the magnitude of the similarity. Wherein, according to the corrected similarity corresponding to each of the tracked objects, determining the target object from the N tracked objects includes: re-sorting the N tracked objects according to the magnitude of the corrected similarity corresponding to each of the tracked objects. Determining the target object from the N tracked objects according to the result of the re-sorting.

[0141] For example, for each tracked object, the transition probability from the nearest grid cell to the current grid cell is , using Formula 6 to calculate the original similarity between the tracked object and the target object Perform weighted correction to obtain the corrected similarity .

[0142]

[0143] According to the corrected similarity , re-ranking is performed, and rank1 is selected as the target object re-identification retrieval result, that is, the tracked object of rank1 is determined as the target object to realize the re-identification process.

[0144] In some embodiments, after determining the target object from the N tracked objects according to the magnitude of the transition probability, the method further includes: generating a movement trajectory based on the current grid unit and the nearest grid unit of the target object, and taking the target object as the S+1th object that has appeared, and taking the movement trajectory as the historical trajectory to update the trajectory probability data.

[0145] For example, the current grid cell is the grid area

[0146] , then add the target trajectory conclusion to the grid area

[0147] Corresponding occurrence count matrix, obtain the new target occurrence count matrix And the total number of samples S+1, and recalculate the transfer probability according to formula 3, formula 4, and formula 5 If the current grid cell and the nearest grid cell are different, the transition probability between the cross-grid cells is calculated. According to the statistical matrix , and the state transition probability matrix It can reflect the passenger density and transfer speed in the current monitoring area in real time, and give early warning of stranded passengers based on the local historical weather patterns and passenger flow patterns of the season.

[0148] According to the embodiments of the present disclosure, during the object tracking process, continuous iterative updates can be performed to adaptively optimize the accuracy of the behavior trajectories of various objects, thereby obtaining optimal transition probability data at different times and places.

[0149] Figure 6 A flowchart of an object tracking method according to another embodiment of the present disclosure is schematically shown.

[0150] like Figure 6 As shown, this embodiment includes operations S210 to S240, which are not described in detail here. After determining the target object from the N tracked objects according to the size of the transition probability, this embodiment also includes operations S610 to S630, including:

[0151] In operation S610, a current position of a target object is obtained based on a current image frame. The current position of the target object may be determined according to a calibration result.

[0152] In operation S620, the current position and the previously tracked position of the target object are visually displayed on a scene map.

[0153] Exemplarily, a unique tag ID is assigned to each tracked object. After obtaining the unique tag ID for re-identification of the target object, a real-time target object behavior trajectory reproduction function with human-computer interaction is provided. The current target object map coordinates (i.e., current position) are displayed in real time on the scene map. Then, based on the re-identification, the unique tag ID is obtained, and the current map mark point is connected with the scene map mark point with the same ID at the previous moment (i.e., one or more previously tracked positions), and the same ID map trajectory is connected with a straight line of the same color. In this way, the target object's action trajectory can be monitored in real time on the map.

[0154] In some embodiments, obtaining a current image frame of a target object in a monitoring area includes: obtaining a surveillance video captured by at least one of K cameras and including the target object. Parsing the surveillance video to obtain at least one current image frame. For example, the surveillance video can be stored in a binding relationship with the current image frame.

[0155] In operation S630, in response to a playback operation on the current position in the scene map, a surveillance video including the current image frame is played.

[0156] If the user needs to know the scene and environment of the target object at each time point in history, he can click on the map track mark point to automatically play the surveillance video clip stored at that time for playback. This makes it convenient for users to trace back the movement trajectory of the monitored object through historical records, as well as the actual environment during the movement.

[0157] Based on accurate target object re-identification, the target object's behavior trajectory is projected onto the map in chronological order, so that users can quickly and accurately obtain video playback of the target object at a specific time and location according to their own needs.

[0158] Combined with the above Figure 1 to Figure 6 , Figure 7 A flowchart of an object tracking method according to another embodiment of the present disclosure is schematically shown. Figure 8 An example of an engineering drawing of a scene map according to an embodiment of the present disclosure is schematically shown. Fig. 9 An example of a camera deployment solution according to an embodiment of the present disclosure is schematically shown. Fig.10 The diagram schematically shows the mapping effect of the imaging plane to the world coordinate system according to the embodiment of the present disclosure. Fig.11 An example of target object re-identification trajectory map projection according to an embodiment of the present disclosure is schematically shown.

[0159] like Figure 7 As shown, this embodiment includes an engineering deployment process (operation S701 to operation S703), a spatiotemporal probability statistics process (operation S704 to operation S711), and a real-time monitoring target tracking and re-identification process (operation S712 to operation S719), specifically including:

[0160] In operation S701, the scene map is reconstructed. On-site engineers conduct on-site measurements and draw to complete the actual scene proportional engineering drawings, such as Figure 8 The coordinate base of the camera is defined, and multiple cameras are deployed according to one or more turntables.

[0161] In operation S702, full coverage deployment is performed according to the camera market angle.

[0162] In operation S703 , the camera imaging plane is aligned with the scene map.

[0163] The entire scene is calculated based on the actual site area, as well as the horizontal and vertical field of view of the camera. Through the design of relative camera shooting, the blind spots of the current camera can be supplemented by the opposite camera, thus ensuring full scene coverage without blind spots. For specific camera deployment plans, please refer to Fig. 9 .

[0164] Through the camera calibration principle, the internal and external parameters of each camera are calculated and solved. Based on the external parameters of the camera, the map of the imaging screen is mapped with the actual scene map as the unified coordinate system. This provides a basis for the map positioning of the target object. Since the position of the camera is fixed, a camera calibration can be used for a long time. If the camera position is adjusted, it needs to be recalibrated.

[0165] For example, a series of calibration plate images with different postures are taken through a standard chessboard calibration plate. The Zhang Zhengyou two-step method is used to calibrate the internal and external parameters of the camera to obtain the rotation and translation matrix of the camera imaging plane relative to the world coordinate system, and further optimize the translation matrix according to the position of the known points in the actual scene in the imaging plane. Make the imaging position of the known point coincide with the actual position in the world coordinate system. Through the optimized rotation and translation matrix, the imaging plane diagram is affine transformed to achieve a one-to-one correspondence between the imaging plane coordinates and the actual scene coordinates. After each camera is calibrated and optimized, a schematic diagram of the actual scene corresponding to the imaging plane can be obtained, such as Fig.10 .

[0166] In operation S704, the monitoring scene is divided into grids of 20m*20m squares. Then, in operations S705 to S711, trajectory probability data is obtained in advance by using historical trajectories of objects that have appeared in the monitoring area between M grid units.

[0167] In operation S705 , the time and location information of the same monitoring target captured by each camera is counted.

[0168] In operation S706, the grid area where the current target position is located is confirmed, that is, each grid unit where the object has appeared is located.

[0169] In operation S707 , the time difference and the distance difference between the previous moment and the current moment of each monitoring target in each grid unit are counted.

[0170] In operation S708 , the time is discretized into units of minutes, and the distance is discretized into units of 10 meters.

[0171] In operation S709, probability statistics are performed on the position transfer distance of the monitoring target from the previous moment to the current moment. For example, the probability statistics of the same grid unit or across grid units are obtained by dividing the number of occurrences by the total number of samples.

[0172] In operation S710, mean filtering is performed on the target transition probability from a time dimension and a space dimension.

[0173] In operation S711, a monitoring target spatiotemporal state probability distribution is obtained.

[0174] In operation S712, according to the correspondence between the camera and the scene map, the time and location information (ie, the current time and space information) of the current monitoring target and the current grid unit are obtained.

[0175] In operation S713, the features are compared with all target features in the tracking library to obtain a similarity value sim.

[0176] For example, in the process of real-time target object re-identification, the deep learning network is used to detect the target object in a single camera, and the real-time detection and tracking of multiple targets of the monocular camera is realized through the tracking method. The appearance features of the target object are extracted through network reasoning, and the current time, camera number, map coordinates are obtained, and the grid area ID is calculated according to the current coordinates. The comprehensive information of the monocular camera monitoring and tracking results is transmitted to the database query server to call the tracking library, and the cosine distance between the feature vector of the current target object and the feature vector of the target object in the tracking library is calculated as the original similarity Sim.

[0177] In operation S714, the time difference and distance difference (ie, the time-space variation) between the latest time of the N tracked objects and the current time of the current monitoring target are obtained and discretized.

[0178] In operation S715, the target spatiotemporal state probability is queried according to the discrete time distance parameter.

[0179] In operation S716, the similarity is corrected, and reference may be made to the above formula 6.

[0180] In operation S717, similarity re-ranking is performed as a target re-identification conclusion.

[0181] In operation S718, the target spatiotemporal information is added to the spatiotemporal statistical matrix to perform iterative optimization of the probability distribution.

[0182] In operation S719, on-site passenger, speed, and detention warnings, as well as trajectory visualization and replay functions are provided.

[0183] like Fig.11 , the target object can run the trajectory replay on the scene map. According to the unique mark ID obtained by re-identification, the current map mark point is connected with the scene map mark point with the same ID at the previous moment. The same ID map track is connected with the same color straight line, so that the target object's movement trajectory can be monitored in real time on the map.

[0184] Users can click on the map track mark point to play the associated stored surveillance video clip for playback. Fig.11 The surveillance video playback of the specific time and location of the controlled luggage is shown in the video.

[0185] With the continuous expansion and diversification of application scenarios, single target monitoring technology can no longer meet the needs. For various practical application product sites, tracking and re-identification of target objects is an important basic function. However, due to the uncertainty of the tracking target, there may be a large number of tracking targets with similar appearance in different scenarios. For example, tracked pedestrians, bags, motor vehicles, and bicycles all have similar appearance characteristics. In the process of tracking and re-identification of multiple targets and a large range of cameras, it is also possible that the similarity of the appearance features of approximate target objects is greater than the target true value due to different viewing angles, different lighting, and different distances.

[0186] This embodiment proposes to use the current location of the target object as the starting time and starting location, and to perform probability statistics on the situations in which the target object appears at different distances within different time ranges, and to use the historical probability distribution of the mobile trajectory to constrain the trajectory of the on-site monitoring target. On the basis of the network reasoning conclusion, the conclusion is optimized according to prior knowledge. The accuracy of multi-monitoring targets and cross-camera tracking and re-identification can be improved. At the same time, the behavioral trajectory of the monitoring target can be optimized to obtain a more reasonable and user-intuitive visual interactive interface. It provides users with more accurate, convenient and intuitive auxiliary tools for target monitoring, behavioral trajectory habit analysis and other tasks.

[0187] Based on the above object tracking method, the present disclosure also provides an object tracking device. Fig.12 The device is described in detail.

[0188] Fig.12 The structure block diagram of the object tracking device according to the embodiment of the present disclosure is schematically shown.

[0189] like Fig.12 As shown, the object tracking device 1200 of this embodiment includes a target recognition module 1210 , a vector matching module 1220 , a probability acquisition module 1230 and an object determination module 1240 .

[0190] The target recognition module 1210 can perform operation S210 to obtain a current image frame of a target object in a monitoring area and a current grid unit in which the target object is located in the monitoring area. The monitoring area is pre-divided into M grid units, where M is an integer greater than or equal to 2.

[0191] The vector matching module 1220 may perform operation S220 for matching N tracked objects according to the feature vector of the target object extracted from the current image frame, where N is an integer greater than or equal to 1.

[0192] The probability obtaining module 1230 may perform operation S230 to obtain a transition probability that each tracked object appears in the current grid unit.

[0193] The object determination module 1240 may perform operation S240 for determining a target object from the N tracked objects according to the magnitude of the transition probability.

[0194] In some embodiments, the target recognition module 1210 is used to determine the current grid unit according to the current spatiotemporal information of the target object. The probability acquisition module 1230 is used to obtain the transition probability of each tracked object from the nearest grid unit to the current grid unit, and the nearest grid unit is determined from M grid units according to the latest spatiotemporal information of each tracked object.

[0195] In some embodiments, the probability acquisition module 1230 may also perform operations S310 to S320, which will not be described in detail herein.

[0196] In some embodiments, the object tracking device 1200 may further include a statistical module, which may further perform operations S410 to S430 and operations S510 to S52, which will not be described in detail herein.

[0197] In some embodiments, the statistical module can be used to count the first occurrence times of the S objects that have appeared in each spatiotemporal coordinate in each grid unit. Based on the ratio of the first occurrence times corresponding to each spatiotemporal coordinate to the total number of samples S, the transition probability within the same grid unit corresponding to the spatiotemporal coordinate is obtained.

[0198] In some embodiments, the statistical module can be used to count the second occurrence times of the S objects that appeared in each spatiotemporal coordinate in another grid unit for each grid unit. Based on the ratio of the second occurrence times corresponding to each spatiotemporal coordinate to the total number of samples S, the transition probability between different grid units corresponding to the spatiotemporal coordinate is obtained.

[0199] In some embodiments, the statistical module may be used to obtain the transition probability between each grid unit and any adjacent grid unit.

[0200] In some embodiments, the probability acquisition module 1230 can be used for obtaining the transition probability within the same grid unit from the trajectory probability data based on the spatiotemporal variation for each tracked object if its nearest grid unit is the same as the current grid unit. If its nearest grid unit is different from the current grid unit, the transition probability between different grid units is obtained from the trajectory probability data based on the spatiotemporal variation.

[0201] In some embodiments, the object tracking device 1200 may include an updating module, which is used to generate a movement trajectory based on the current grid unit and the nearest grid unit of the target object after determining the target object from the N tracked objects according to the size of the transition probability. The target object is taken as the S+1th object that has appeared, and the statistics module is called to take the movement trajectory as the historical trajectory to update the trajectory probability data.

[0202] In some embodiments, the vector matching module 1220 may be used to calculate the similarity between the feature vector of the target object and any feature vector in the tracking library, and match N tracked objects based on a similarity threshold.

[0203] In some embodiments, the object determination module 1240 may be configured to sort the N tracked objects based on the magnitude of the similarity when N is greater than or equal to 2. The similarity corresponding to each tracked object is corrected based on the transition probability corresponding to each tracked object. The N tracked objects are reordered according to the magnitude of the corrected similarity. The target object is determined from the N tracked objects according to the reordered result.

[0204] In some embodiments, the object tracking device 1200 may include a partitioning module, which is used to make the imaging range of the K cameras cover the monitoring area. The imaging screens of the K cameras are respectively calibrated with the scene map of the monitoring area at the pixel level. The scene map is divided into M grid units, wherein the M grid units are mapped to the imaging screens of the corresponding cameras based on the calibration results.

[0205] In some embodiments, the object tracking device 1200 may include a visualization module, which may perform operations S610 to S620, which will not be described in detail herein.

[0206] In some embodiments, the object tracking device 1200 may include a playback module, which may perform operation S630, which will not be described in detail herein.

[0207] The present disclosure also provides an object tracking system, which may include multiple cameras deployed in a monitoring area and an object tracking device 1200, which may track an object based on images captured by the cameras.

[0208] For the parts not mentioned in the device part, they can be understood with reference to the various embodiments of the above method. That is, the device part includes modules for executing the various steps of any one of the method embodiments described above. In addition, the implementation methods, technical problems solved, functions implemented, and technical effects achieved of each module / unit / subunit, etc. in the device part embodiment are respectively the same or similar to the implementation methods, technical problems solved, functions implemented, and technical effects achieved of each corresponding step in the method part embodiment, and will not be repeated here.

[0209] According to an embodiment of the present disclosure, any multiple modules among the target recognition module 1210, the vector matching module 1220, the probability acquisition module 1230 and the object determination module 1240 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0210] According to an embodiment of the present disclosure, at least one of the target recognition module 1210, the vector matching module 1220, the probability acquisition module 1230 and the object determination module 1240 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or may be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by an appropriate combination of any of them. Alternatively, at least one of the target recognition module 1210, the vector matching module 1220, the probability acquisition module 1230 and the object determination module 1240 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0211] Fig.13 A block diagram of an electronic device suitable for implementing an object tracking method according to an embodiment of the present disclosure is schematically shown.

[0212] like Fig.13 As shown, the electronic device 1300 according to an embodiment of the present disclosure includes a processor 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage part 1308 to a random access memory (RAM) 1303. The processor 1301 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1301 may also include an onboard memory for caching purposes. The processor 1301 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0213] In RAM 1303, various programs and data required for the operation of electronic device 1300 are stored. Processor 1301, ROM 1302 and RAM 1303 are connected to each other via bus 1304. Processor 1301 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1302 and / or RAM 1303. It should be noted that the program can also be stored in one or more memories other than ROM 1302 and RAM 1303. Processor 1301 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0214] According to an embodiment of the present disclosure, the electronic device 1300 may further include an input / output (I / O) interface 1305, which is also connected to the bus 1304. The electronic device 1300 may further include one or more of the following components connected to the I / O interface 1305: an input portion 1306 including a keyboard, a mouse, etc. An output portion 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc. A storage portion 1308 including a hard disk, etc. And a communication portion 1309 including a network interface card such as a LAN card, a modem, etc. The communication portion 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed, so that the computer program read therefrom is installed into the storage portion 1308 as needed.

[0215] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments. It may also exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0216] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1302 and / or RAM 1303 described above and / or one or more memories other than ROM 1302 and RAM 1303.

[0217] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0218] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1301. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0219] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1309, and / or installed from the removable medium 1311. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0220] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1309, and / or installed from the removable medium 1311. When the computer program is executed by the processor 1301, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0221] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0222] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0223] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. An object tracking method, characterized in that: include: Obtaining a current image frame of a target object in a monitoring area and a current grid unit in which the target object is located in the monitoring area, wherein the monitoring area is pre-divided into M grid units, where M is an integer greater than or equal to 2; Matching N tracked objects according to the feature vector of the target object extracted from the current image frame, where N is an integer greater than or equal to 1; Taking the most recent spatiotemporal information of each tracked object as the starting spatiotemporal information, and obtaining the spatiotemporal variation between the starting spatiotemporal information and the current spatiotemporal information of the target object; Calculate the nearest grid unit where the tracked object is located according to the position coordinates in the latest spatiotemporal information; the latest spatiotemporal information includes the recorded spatiotemporal information closest to the current time; Obtaining a transfer probability of each tracked object from the nearest grid unit to the current grid unit based on the spatiotemporal variation from the trajectory probability data, wherein the transfer probability includes a probability of not leaving the current grid unit within a period of time, or a probability of transferring between different grid units; Determine the target object from the N tracked objects according to the magnitude of the transition probability; The trajectory probability data is obtained by the following operations: Obtain the historical trajectories of S objects that have appeared in the monitoring area within a specific time period, where S is an integer greater than or equal to 1; In each of the grid units, based on the historical trajectory of each of the objects that have appeared, the time difference and distance difference between the start and end moments of each displacement are counted, and the start and end points of the displacement are located in the same or different grid units; The trajectory probability data is obtained based on the statistically obtained time difference and distance difference.

2. The method according to claim 1, characterized in that Obtaining the current grid unit where the target object is located in the monitoring area includes: The current grid unit is determined according to current spatiotemporal information of the target object.

3. The method according to claim 1, characterized in that For each of the tracked objects, If the nearest grid unit is the same as the current grid unit, obtaining the transition probability within the same grid unit from the trajectory probability data based on the spatiotemporal variation; If the nearest grid unit is different from the current grid unit, a transition probability between different grid units is obtained from the trajectory probability data based on the spatiotemporal variation.

4. The method according to claim 1, characterized in that: The trajectory probability data obtained based on the statistically obtained time difference and distance difference includes: Discretize the statistically obtained time difference and distance difference based on a preset unit duration and a preset unit distance to obtain a discretization result; The trajectory probability data is obtained based on the discretization result statistics.

5. The method according to claim 4, characterized in that The discretization result includes a space-time coordinate system with a preset unit time length as a time scale and a preset unit distance as a space scale; Obtaining the trajectory probability data based on the discretization result statistics includes: for each of the grid cells, Counting the first occurrence times of the S objects that have appeared in the grid unit corresponding to each spatiotemporal coordinate; Based on the ratio of the first number of occurrences corresponding to each space-time coordinate to the total number of samples S, the transition probability within the same grid unit corresponding to the space-time coordinate is obtained.

6. The method according to claim 5, characterized in that Obtaining the trajectory probability data based on the discretization result statistics further includes: for each of the grid cells, Counting the second occurrence times of the S objects that appeared previously in another grid unit corresponding to each spatiotemporal coordinate; Based on the ratio of the second number of occurrences corresponding to each space-time coordinate to the total number of samples S, the transition probability between different grid units corresponding to the space-time coordinate is obtained.

7. The method according to claim 6, characterized in that The trajectory probability data includes the transition probability within the same grid unit corresponding to each space-time coordinate in the space-time coordinate system, or the transition probability between different grid units corresponding to each space-time coordinate.

8. The method according to any one of claims 4 to 7, characterized in that: After determining the target object from the N tracked objects according to the magnitude of the transition probability, the method further includes: Generate a movement trajectory based on a current grid cell and a nearest grid cell of the target object; The target object is taken as the S+1th previously appeared object, and the moving trajectory is taken as the historical trajectory to update the trajectory probability data.

9. The method according to claim 1, characterized in that: The feature vector of each tracked object is pre-stored in the tracking library. According to the feature vector of the target object extracted from the current image frame, matching N tracked objects includes: Calculating the similarity between the feature vector of the target object and any feature vector in the tracking library; The N tracked objects are matched based on a similarity threshold.

10. The method according to claim 9, characterized in that Determining the target object from the N tracked objects according to the magnitude of the transition probability includes: Based on the size of the transition probability corresponding to each of the tracked objects, correcting the corresponding similarity; The target object is determined from the N tracked objects according to the corrected similarity corresponding to each of the tracked objects.

11. The method according to claim 10, characterized in that When N is greater than or equal to 2, the N tracked objects are sorted based on the magnitude of the similarity, wherein, according to the corrected similarity corresponding to each of the tracked objects, determining the target object from the N tracked objects comprises: Reordering the N tracked objects according to the magnitude of the corrected similarity corresponding to each of the tracked objects; The target object is determined from the N tracked objects according to the reordering result.

12. The method according to claim 1, characterized in that K cameras are pre-deployed in the monitoring area, where K is an integer greater than or equal to 2, and the object tracking includes: The target object in the monitoring area is tracked across cameras based on the K cameras.

13. The method according to claim 12, characterized in that Dividing the monitoring area into M grid units in advance includes: Make the imaging ranges of the K cameras cover the monitoring area; Perform pixel-level calibration on the imaging images of each of the K cameras and the scene map of the monitoring area; The scene map is divided into the M grid units, wherein the M grid units are correspondingly mapped to the imaging screens of the corresponding cameras based on the calibration results.

14. The method according to claim 13, characterized in that After determining the target object from the N tracked objects according to the magnitude of the transition probability, the method further includes: Based on the current image frame, obtaining a current position of the target object; The current position and the previously tracked positions of the target object are visually displayed on the scene map.

15. The method according to claim 14, characterized in that The obtaining of the current image frame of the target object in the monitoring area comprises: Obtaining a surveillance video containing the target object captured by at least one of the K cameras; Parsing the surveillance video to obtain at least one current image frame; Wherein, the method further comprises: In response to a playback operation on the current position in the scene map, a surveillance video including the current image frame is played.

16. An object tracking device, characterized in that: include: A target recognition module, used to obtain a current image frame of a target object in a monitoring area and a current grid unit in which the target object is located in the monitoring area, wherein the monitoring area is pre-divided into M grid units, where M is an integer greater than or equal to 2; A vector matching module, configured to match N tracked objects according to a feature vector of the target object extracted from the current image frame, where N is an integer greater than or equal to 1; A probability acquisition module, used to use the most recent spatiotemporal information of each tracked object as the starting spatiotemporal information, and obtain the spatiotemporal variation between the starting spatiotemporal information and the current spatiotemporal information of the target object; Calculate the nearest grid unit where the tracked object is located according to the position coordinates in the latest spatiotemporal information; the latest spatiotemporal information includes the recorded spatiotemporal information closest to the current time; Obtaining a transition probability of each tracked object transferring from the nearest grid unit to the current grid unit from the trajectory probability data based on the spatiotemporal variation, and obtaining a transition probability of each tracked object appearing in the current grid unit, wherein the transition probability includes a probability of not leaving the current grid unit within a period of time, or a probability of transferring between different grid units; An object determination module, configured to determine the target object from the N tracked objects according to the magnitude of the transition probability; The trajectory probability data is obtained by the following operations: Obtain the historical trajectories of S objects that have appeared in the monitoring area within a specific time period, where S is an integer greater than or equal to 1; In each of the grid units, based on the historical trajectory of each of the objects that have appeared, the time difference and distance difference between the start and end moments of each displacement are counted, and the start and end points of the displacement are located in the same or different grid units; The trajectory probability data is obtained based on the statistically obtained time difference and distance difference.

17. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 15.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images