Vision-based cross-lens target trajectory prediction and tracking method

Abstract

The invention discloses a vision-based cross-lens target trajectory prediction and tracking method, and belongs to the technical field of target tracking. Firstly, a tracking object is selected manually, so that the geographic position corresponding to a camera where the tracking object is located can be determined; creating an analysis task according to a task decision module; an analysis result is obtained through a video stream pulling module and a video analysis module; and finally, displaying the analysis state through an event display module. According to the method, not all cameras in the whole area start analysis tasks, but unnecessary video analysis tasks are cut off, so that unnecessary computing power distribution is reduced. The analysis tasks are decided according to the position and direction of the target, geographic information distribution and camera distribution information, so that unnecessary analysis tasks are reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target tracking, and particularly relates to a visual-based cross-lens target trajectory prediction and tracking method. BACKGROUND

[0002] With the development of IT infrastructure and AI technology, the public security management department gradually builds an AI-based intelligent identification, detection and tracking system to improve emergency decision-making efficiency by monitoring equipment and AI algorithms to monitor important targets (such as criminal suspects, missing persons, specific vehicles, etc.) in real time. However, AI technology relies on deep learning algorithms, and detection algorithms require a large amount of CPU and GPU resources for video decoding and identification inference. At the same time, due to limited budget and computing resources, the public security monitoring system is often in a state of limited hardware performance. In limited hardware devices, the monitoring system cannot perform at its best, resulting in low efficiency in detecting emergency events and delaying decision-making, thereby creating potential safety hazards.

[0003] Today, surveillance cameras have spread to every corner of the city, as if there is an invisible guardian at every intersection and hidden place. These surveillance cameras are responsible for different areas, but unfortunately, despite their large number, there is still a lack of a systematic mechanism, method or tool to effectively integrate these scattered monitoring information. If the monitoring data of each area can be integrated, it not only provides valuable clues for police to crack cases, but also significantly improves the efficiency of traffic management departments in handling events.

[0004] In case investigation, whether it is to find missing persons or track the activity trajectory of suspects, it often depends on the familiarity of the area and spends a lot of time retrieving historical recordings of surveillance cameras one by one to carefully investigate the action trajectory of the suspect. This traditional method not only consumes time and effort, but also easily misses key clues due to human negligence, thereby increasing the difficulty of case investigation. If an automated, multi-dimensional monitoring information integration system can be developed, it will greatly improve the efficiency of investigation and reduce omissions, providing strong support for case investigation and traffic incident handling. SUMMARY

[0005] In view of the above technical problems, the application provides a visual-based cross-lens target trajectory prediction and tracking scheme.

[0006] The first aspect of the application provides a visual-based cross-lens target trajectory prediction and tracking method, in which a map-camera graph construction module, a task analysis scheduling module, and an event display module are configured, and the task analysis scheduling module includes a task decision module, a video streaming module, and a video analysis module. The method specifically includes:

[0007] Call the map-camera atlas construction module, combine the map information and the camera distribution, map the position of each camera to the map, and construct the map-camera atlas;

[0008] The atlas is used to query the camera distribution of a street, the streets connected by an intersection, and the adjacent intersection camera information of the current intersection.

[0009] Call the task decision module to obtain the orientation and location of the target to be tracked, query the cameras near the target using the atlas, and create corresponding analysis tasks, and assign corresponding computing power to the analysis tasks.

[0010] Call the video streaming module, calculate the video transcoding and decoding corresponding to each camera near the target, and dynamically pull the video stream of the corresponding camera.

[0011] Call the video analysis module to decode the pulled video stream and analyze the decoded image to obtain the analysis result.

[0012] Call the event display module to display on the corresponding map according to the located target and location.

[0013] According to the method of the first aspect of the application, for the map-camera atlas construction module:

[0014] An OSM map is used, which is an OpenStreetMap open street map, a building distribution API is constructed, OSM map open source data is analyzed and mined, and a building distribution map with direction is formed.

[0015] An API for querying intersection information is constructed, including: a forward query next intersection API, a reverse query previous intersection API, and a query all intersection information API.

[0016] Combine the map geographic data, camera information, and road information to map the positions of regional cameras to the map to form a map-camera distribution atlas.

[0017] According to the method of the first aspect of the application, for the task analysis and scheduling module:

[0018] Obtain the target to be tracked and the initial position information of the target, determine the on-site location, road, and direction of the target in the initial state.

[0019] According to the position and direction of the target, the search direction and search camera determined by the atlas, create corresponding search task processes according to the number of search cameras and assign corresponding computing power.

[0020] According to the direction of the target, the priority search path and search camera source are determined; specifically including:

[0021] preferentially search a video source covering an area in a direction same as the direction of the target;

[0022] when the position of the target is an intersection, determine the search direction according to the order in which all video sources at the intersection discover the target to leave the target.

[0023] According to the method of the first aspect of the application, for the task analysis scheduling module: the video stream pulling module uses the ZLMediaKit media service to uniformly transcode various video streams into H264 rtsp video streams for subsequent video analysis modules to analyze; the video analysis module uses yolov5 for target detection and deepsort for tracking.

[0024] The second aspect of the application provides a visual-based cross-lens target trajectory prediction and tracking system, which comprises a map-camera atlas construction module, a task analysis scheduling module and an event display module; the task analysis scheduling module comprises a task decision module, a video stream pulling module and a video analysis module; wherein:

[0025] The map-camera atlas construction module is configured to: combine map information and camera distribution, map the positions of various cameras to a map, and construct a map-camera atlas;

[0026] The atlas is used to query the camera distribution of a street, the streets connected by an intersection, and the adjacent intersection camera information of the current intersection.

[0027] The task decision module is configured to: obtain the orientation and position of a target to be tracked, query the cameras near the target using the atlas, and create corresponding analysis tasks, and allocate corresponding computing power to the analysis tasks.

[0028] The video stream pulling module is configured to: calculate the video transcoding and decoding corresponding to each camera near the target, and dynamically pull the video stream of the corresponding camera.

[0029] The video analysis module is configured to: decode the pulled video stream, analyze the decoded image, and obtain an analysis result.

[0030] The event display module is configured to: display on the corresponding map according to the located target and position.

[0031] According to the system of the second aspect of the application, for the map-camera atlas construction module:

[0032] An OSM map, which is an OpenStreetMap open street map, is used to construct a building distribution API, analyze and mine the open source data of the OSM map, and form a building distribution map with directions.

[0033] The API for constructing intersection information includes: forward query next intersection API, reverse query last intersection API, and query all intersection information of a road API.

[0034] The positions of the regional cameras are mapped to the map by combining the map geographic data, camera information and road information, to form a map-camera distribution atlas.

[0035] According to the system of the second aspect of the present application, for the task analysis and scheduling module:

[0036] The target to be tracked and initial position information of the target are acquired, and the on-site position, road and direction of the target in the initial state are determined.

[0037] According to the search direction and search camera determined by the atlas, corresponding search task processes are created according to the number of search cameras and the corresponding computing power is allocated.

[0038] The priority search path and search camera source are determined according to the direction of the target, and specifically include:

[0039] The video source of the area covered in the same direction as the direction of the target is preferentially searched.

[0040] When the target is located at an intersection, the search direction is determined according to the order of the target found and the target left by all video sources of the intersection.

[0041] According to the system of the second aspect of the present application, for the task analysis and scheduling module: the video stream pulling module uses ZLMediaKit media service to uniformly transcode various video streams into H264 rtsp video streams for subsequent video analysis module analysis; the video analysis module uses yolov5 for target detection and deepsort for target tracking.

[0042] The third aspect of the present application discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the first aspect of the present application, a visual-based cross-lens target trajectory prediction and tracking method.

[0043] The fourth aspect of the present application discloses a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the first aspect of the present application, a visual-based cross-lens target trajectory prediction and tracking method.

[0044] In summary, the application does not analyze all tasks of the full-area camera, but cuts off unnecessary video analysis tasks, thereby reducing unnecessary allocation of computing power. According to the location, direction, geographical information distribution and camera distribution information of the target, the analysis task is decided, thereby reducing unnecessary analysis tasks. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings required to be used in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1 A schematic diagram of a visual-based cross-lens target trajectory prediction and tracking system according to an embodiment of the present application.

[0047] Figure 2 A schematic diagram of a map-camera graph construction module according to an embodiment of the present application.

[0048] Figure 3 A directed osm map according to an embodiment of the present application.

[0049] Figure 4 A directed map-camera mapping diagram according to an embodiment of the present application.

[0050] Figure 5 A schematic diagram of a task analysis scheduling module according to an embodiment of the present application.

[0051] Figure 6 A schematic diagram of a target search strategy according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0053] The present application first selects a tracking object by hand, so as to determine the camera where the tracking target is located and the corresponding geographical position. Then, an analysis task is created according to a task decision module. Subsequently, an analysis result is obtained through a video streaming module and a video analysis module. Finally, an analysis state is displayed through an event display module.

[0054] AsFigure 1 As shown, the map-camera atlas construction module mainly combines map information and camera distribution to map each camera to the map to construct a map-camera atlas, through which the camera distribution of any street or any road can be queried; the connected streets or intersections of any intersection can be queried; the next intersection camera information or the previous intersection camera information of any intersection can be queried, etc.

[0055] The task decision module is configured to analyze the target reasoning result to obtain a target direction and a target position, query a camera closest to the target and having a relatively large probability of appearing from the map-camera atlas, and create a corresponding analysis task and assign a corresponding computing power to the analysis task.

[0056] The video streaming module is configured to calculate video transcoding and decoding corresponding to each camera near the target and dynamically pull a corresponding camera video stream.

[0057] The video analysis module is configured to analyze and reason a decoded image according to the pulled video stream to obtain a reasoning result.

[0058] The event display module is configured to draw the located target and position on a corresponding map for intuitive display.

[0059] As shown in Figure 2 OSM is OpenStreetMap, an open street map, which is a free and open source editable map service jointly created by the public. The osm data is open source and can be freely downloaded and used. The free global map database can be mined and utilized.

[0060] The building distribution API in the above figure is an analysis and mining of the osm open source data, and finally forms a building distribution map with directions, as shown in Figure 3

[0061] By deeply mining the osm open source data, an API for querying intersection information is constructed. The API constructed by us includes a forward query next intersection API (find_next_forward_crossing), a reverse query previous intersection API (find_pre_crossing), and a query all intersection information API of a road (find_road_info).

[0062] Figure 2 The regional camera data is all camera data in a region. The camera data includes intersection information, installation longitude and latitude, height, focal length, etc.

[0063] Figure 2 ​The building-camera distribution map is a combination of map geographic data, camera information, and road information to map regional cameras to the map (such as Figure 4 As shown in the figure, the red points are road monitoring cameras), and finally a map-camera distribution map is formed.

[0064] like Figure 5 As shown in Figure 1, the initial target position information in the task analysis and scheduling module is manually selected from the on-site video data to track the target, so that the initial target's on-site position, the target's road, and the target's walking direction (if it is a car, it is the target's driving direction) can be determined.

[0065] The task decision module in the task analysis and scheduling module is the key invention technology of this invention. It mainly determines the search direction and search camera through the target position, target orientation and map-camera map. It then creates corresponding search task processes and allocates corresponding computing power based on the number of search cameras (if the number of cameras in the search area is 3, then 3 processes are created to analyze 3 video sources).

[0066] like Figure 6 As shown, if the target is found in the middle of the road but not at an intersection, such as near Xiangqifang, the priority search path and search camera source are determined according to the target orientation.

[0067] Specifically, if the target is facing right, the search prioritizes video sources in direction B and area B. If the camera is facing left, the search prioritizes video sources in direction A and area A. If the target is located at an intersection, such as area B, the search direction is determined based on the order in which all video sources at the intersection discover the target and then leave it. If the last area left was Jianghan Road, the next search direction is Jianghan Road. If the last area entered was Binkang Road, the search direction is Binkang Road.

[0068] OpenStreetMap (OSM) is a highly detailed map database and a collaborative online map project that creates free content. Its goal is to create a world map that is free and editable by everyone, and to provide convenient navigation solutions for general mobile devices.

[0069] The video streaming module within the Task Analysis and Scheduling module utilizes the mature ZLMediaKit media service. ZLMediaKit is an open-source streaming media server developed in C++. It provides high-performance audio and video processing, supports common streaming protocols such as RTSP, RTMP, HLS, and HTTP-FLV, and boasts low latency and high concurrency. ZLMediaKit can uniformly transcode various streams into H264 RTSP video streams for subsequent analysis by the video analysis module.

[0070] The video analysis module in the task analysis and scheduling module is used for target detection by using yolov5 and target tracking by using deepsort.

[0071] The present application solves the problem of insufficient computing resources by constructing a map-camera mapping diagram through map information and camera distribution mapping.

[0072] The present application solves the problem of insufficient computing resources and greatly improves the search and tracking efficiency by constructing a target search strategy through the map-camera mapping diagram.

[0073] The present application constructs the API required for map-camera mapping by mining OSM offline data, and through the API call, the current target location, nearby camera distribution and nearby intersection distribution can be quickly obtained, so as to be quickly used by the subsequent task decision module.

[0074] Please note that the technical features of the above embodiments can be combined in any way, and in order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application. The above examples only express several embodiments of the present application, and the description is more specific and detailed, but it should not be construed as limiting the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A vision-based cross-shot target trajectory prediction and tracking method, characterized in that: In the method, a map-camera graph construction module, a task analysis and scheduling module, and an event display module are configured. The task analysis and scheduling module includes a task decision module, a video streaming module, and a video analysis module. The method specifically includes: Call the map-camera atlas construction module, combine the map information with the camera distribution, map the location of each camera onto the map, and construct a map-camera atlas; The map is used to query the distribution of cameras on streets, the streets connected to intersections, and the camera information of adjacent intersections of the current intersection; Call the task decision module to obtain the direction and location of the target to be tracked, use the map to query the cameras near the target, create a corresponding analysis task, and allocate corresponding computing power to the analysis task; Call the video stream pulling module to calculate the video transcoding and decoding corresponding to each camera near the target, and dynamically pull the video stream of the corresponding camera; Call the video analysis module, decode the pulled video stream, analyze the decoded image, and obtain the analysis results; Call the event display module and display it on the corresponding map according to the located target and position.

2. The method for predicting and tracking target trajectories across shots based on vision according to claim 1, wherein: For the Map-Camera Graph building block: Use OSM maps, which are OpenStreetMap open street maps, to build a building distribution API, analyze and mine OSM map open source data to form a building distribution map with directions; Build an API for querying intersection information, including: forward query API for the next intersection, reverse query API for the previous intersection, and API for querying information about all intersections on a road; By combining map geographic data, camera information, and road information, the locations of regional cameras are mapped onto the map to form a map-camera distribution map.

3. The method for predicting and tracking target trajectories across shots based on vision according to claim 2, wherein: For the task analysis and scheduling module: Obtain the target to be tracked and its initial position information, and determine the target's initial location, road, and direction; Based on the position and direction of the target, the search direction and search cameras determined by the map, a corresponding search task process is created and corresponding computing power is allocated according to the number of search cameras; The method of determining a priority search path and a search camera source according to the direction of the target specifically includes: Prioritizing searching for video sources in an area covered by the same direction as the target; When the target is located at an intersection, the search direction is determined according to the order in which all video sources at the intersection discover the target and then leave the target.

4. The method for predicting and tracking target trajectories across shots based on vision according to claim 3, wherein: For the task analysis and scheduling module: the video streaming module uses the ZLMediaKit media service to uniformly transcode various video streams into H264 rtsp video streams for analysis by the subsequent video analysis module; the video analysis module uses yolov5 for target detection and deepsort for target tracking.

5. A vision-based cross-shot target trajectory prediction and tracking system, characterized by: The system includes: a map-camera graph construction module, a task analysis and scheduling module, and an event display module; the task analysis and scheduling module includes a task decision module, a video streaming module, and a video analysis module; wherein: The map-camera atlas construction module is configured to: combine map information and camera distribution, map the location of each camera onto the map, and construct a map-camera atlas; The map is used to query the distribution of cameras on streets, the streets connected to intersections, and the camera information of adjacent intersections of the current intersection; The task decision module is configured to: obtain the direction and location of the target to be tracked, use the atlas to query cameras near the target, create a corresponding analysis task, and allocate corresponding computing power to the analysis task; The video stream pulling module is configured to: calculate the video transcoding and decoding corresponding to each camera near the target, and dynamically pull the video stream of the corresponding camera; The video analysis module is configured to: decode the pulled video stream, analyze the decoded image, and obtain analysis results; The event display module is configured to display the event on a corresponding map according to the located target and position.

6. The vision-based cross-shot target trajectory prediction and tracking system according to claim 5, characterized in that: For the Map-Camera Graph building block: Using OSM maps, which are OpenStreetMap open street maps, we built a building distribution API, analyzed and mined OSM map open source data, and formed a building distribution map with directions; Build an API for querying intersection information, including: forward query API for the next intersection, reverse query API for the previous intersection, and API for querying information about all intersections on a road; By combining map geographic data, camera information, and road information, the locations of regional cameras are mapped onto the map to form a map-camera distribution map.

7. The vision-based cross-shot target trajectory prediction and tracking system according to claim 6, characterized in that: For the task analysis and scheduling module: Obtain the target to be tracked and its initial position information, and determine the target's initial location, road, and direction; Based on the position and direction of the target, the search direction and search cameras determined by the map, a corresponding search task process is created and corresponding computing power is allocated according to the number of search cameras; The method of determining a priority search path and a search camera source according to the direction of the target specifically includes: Prioritizing searching for video sources in an area covered by the same direction as the target; When the target is located at an intersection, the search direction is determined according to the order in which all video sources at the intersection discover the target and then leave the target.

8. The vision-based cross-shot target trajectory prediction and tracking system according to claim 7, characterized in that: For the task analysis and scheduling module: the video streaming module uses the ZLMediaKit media service to uniformly transcode various video streams into H264 rtsp video streams for analysis by the subsequent video analysis module; the video analysis module uses yolov5 for target detection and deepsort for target tracking.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the vision-based cross-shot target trajectory prediction and tracking method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the computer program implements the vision-based cross-shot target trajectory prediction and tracking method according to any one of claims 1 to 4.

Images