A target trajectory generation method based on multi-path security video

Abstract

The present application relates to a kind of target trajectory generation method based on multi-path security video, belong to technical field.The present application is oriented to the target trajectory generation problem in security video, in view of the current situation that target loss caused by single target detection and re-identification, matching degree is not high, provide a kind of target trajectory generation method based on multi-path security video.For multi-path security video, there are four important features:one is that there is security area overlap and target appears repeatedly in multiple places in multi-path video;Second, there is a connection or adjacent relationship in the security area of multi-path video;Third, video is stream data, not video clip, and the matching range is wider;Fourth, the application feasibility of target space-time association is stronger.Therefore, the present application makes full use of the four features of security video and auxiliary information, based on multi-path video, multi-method fusion, multi-target correlation, to improve the accuracy of target tracking and trajectory generation, improve the re-identification precision.

Description

Technical Field

[0001] The present invention belongs to the technical field, and in particular relates to a target trajectory generation method based on multi-channel security videos. Background Art

[0002] Security video technology is used to detect and monitor protected areas, displaying and recording on-site images in real time, and retrieving and displaying historical images. Captured security video is unstructured data. Structured data is irregular or incomplete, lacking a predefined data model, and is not easily represented using a two-dimensional database logical table. Security video requires structured processing and analysis to extract target information for more convenient and efficient security objectives. Security video analysis begins with target detection, which then uses this information for subsequent analysis and processing. Target trajectory generation is one such application. Unlike target detection in a single image or video segment, security video is typically collected from multiple cameras.

[0003] Target trajectories are usually generated using target tracking and target re-identification methods. Target tracking is a common computer vision task that requires detecting targets in consecutive video frames and assigning a track id to each target. This id is unique in the video sequence. Target re-identification aims to determine whether the same target exists in images or videos from different camera perspectives. The main technologies can be divided into feature matching-based methods and learning-based methods. Among them, learning-based methods have received widespread attention in recent years. By training deep neural network models, more robust and effective feature representations can be learned to improve the performance of re-identification. The process of trajectory generation is to first use the detection model to locate the bounding box position of the target in the image, then use the association model to extract re-identification features for each bounding box, and link the bounding box to an existing tracking result based on the specific metrics defined by these features.

[0004] The method of simply using target tracking and target re-identification has the following problems: First, target detection may result in missed targets, lost targets, or the entire image needs to be traversed, which is time-consuming; second, targets in different scenes need to be re-identified, that is, feature matching is performed, but this matching is a holistic traversal, which is time-consuming and has low matching accuracy; third, existing methods mainly target single image segments and do not consider the correlation between multiple videos. Summary of the Invention

[0005] (1) Technical issues to be resolved

[0006] The technical problem to be solved by the present invention is: facing the problem of target trajectory generation in security videos, and aiming at the current situation where target loss and low matching degree are caused by the single use of target detection and re-identification, a target trajectory generation method based on multi-channel security videos is provided.

[0007] (2) Technical solution

[0008] In order to solve the above technical problems, the present invention provides a target trajectory generation method based on multi-channel security video. First, the following definitions are made:

[0009] The detection or tracking result of the target in a certain image frame is expressed as a rectangular frame. Let the coordinate of the upper left corner of the rectangular frame be (x lu ,y lu ), the coordinate of the lower right corner is (x rb ,y rb ), where lu represents the upper left corner of the rectangular box, rb represents the lower right corner of the rectangular box, and the center position of the rectangular box is The above two coordinates take the upper left corner of the image frame as the origin, and also record the time of the image frame, that is, the time when the target appears at that position;

[0010] Then, follow these steps:

[0011] Step 1: Establishing a regional mapping relationship: Based on the characteristics of overlapping areas in multiple video channels, a mapping relationship is established between the target pixel position and the regional position;

[0012] Step 1 is as follows: the image is divided into regions according to the resolution, divided into m regions by row and n regions by column, that is, the image frame is divided into m×n regions; let the region in a video channel A be represented as The area in a video channel B is represented as

[0013] Based on the coordinate system and the divided area, several aspects of mapping are completed:

[0014] 1) Mapping of the target area, i.e., the area within which the center coordinates of the i-th target are currently located; which areas the target rectangle covers, and the proportion of each area;

[0015] 2) The correspondence between the single overlapping area of ​​video A and video B, that is, Indicates area and Correspondingly, IOU is the region intersection-union ratio;

[0016] 3) Correspondence between overlapping regions of the target rectangles: Calculate the correspondence between each region and merge the regions.

[0017] 4) According to the camera position and security range, establish the connection area of ​​a camera boundary in other cameras, that is,

[0018] Step 2: Recapture the target trajectory using the overlapping area in the security screen: Based on step 1, re-detect the target at the overlapping position when the target is lost and continue the target trajectory.

[0019] Preferably, step 2 specifically includes the following steps:

[0020] Step 21: Start the target detection algorithm yolov8 and the tracking algorithm deepsort to detect and track the target in a single frame image;

[0021] Step 22: Record the position and time (t i ,x i ,y i ), where t i is the time when the target appears, accurate to milliseconds, (x i ,y i ) is the relative image position of the target in the image frame;

[0022] Step 23: If the target is not lost, repeat step 22 to obtain the position sequence of the target in the current image sequence and form the target trajectory in chronological order; if the target is lost, determine whether there is an overlapping area between the video of this channel and other videos. If there is no overlapping area, jump to step 24; if there is an overlapping area, jump to step 25;

[0023] Step 24: Lower the confidence screening threshold of the current target detection algorithm, obtain the target that appears in the neighborhood where the target appeared in the previous frame, and perform feature vector matching. If the target in the current frame is detected again, continue to generate the target trajectory; if the target is not detected, use another method to detect in the neighboring area and perform feature vector matching. If the match is successful, continue to generate the trajectory; otherwise, the target is considered lost and the trajectory generation process ends.

[0024] Step 25: Use the same detection method to detect in the overlapping area and perform feature matching. If the target is detected and matched successfully, continue to generate the trajectory. If the target cannot be detected, lower the confidence screening threshold of the current target detection algorithm and continue detection. If the target still cannot be detected or is not matched, switch to another method to perform target detection and matching at the original position. If the trajectory is matched, continue to generate the trajectory using the current method. If the target is not detected or is not matched, it is considered that the target is lost and the trajectory generation process ends.

[0025] The present invention provides another target trajectory generation method based on multi-channel security video. First, the following definitions are made:

[0026] The detection or tracking result of the target in a certain image frame is expressed as a rectangular frame. Let the coordinate of the upper left corner of the rectangular frame be (x lu ,y lu ), the coordinate of the lower right corner is (x rb ,y rb ), where lu represents the upper left corner of the rectangular box, rb represents the lower right corner of the rectangular box, and the center position of the rectangular box is The above two coordinates take the upper left corner of the image frame as the origin, and also record the time of the image frame, that is, the time when the target appears at that position;

[0027] Then, follow these steps:

[0028] Step 1: Establishing a regional mapping relationship: Based on the characteristics of overlapping areas in multiple video channels, a mapping relationship is established between the target pixel position and the regional position;

[0029] Step 1 is as follows: the image is divided into regions according to the resolution, divided into m regions by row and n regions by column, that is, the image frame is divided into m×n regions; let the region in a video channel A be represented as The area in a video channel B is represented as

[0030] Based on the coordinate system and the divided area, several aspects of mapping are completed:

[0031] 1) Mapping of the target area, i.e., the area within which the center coordinates of the i-th target are currently located; which areas the target rectangle covers, and the proportion of each area;

[0032] 2) The correspondence between the single overlapping area of ​​video A and video B, that is, Indicates area and Correspondingly, IOU is the region intersection-union ratio;

[0033] 3) Correspondence between overlapping regions of the target rectangles: Calculate the correspondence between each region and merge the regions.

[0034] 4) According to the camera location and security range, establish the connection area of ​​a camera boundary to other cameras, that is,

[0035] Step 2: Based on step 1, when the target leaves the current security screen, re-detect the target in the adjacent area and continue the target trajectory.

[0036] Preferably, step 2 is based on step 1, and according to the position of the security camera, the speed and direction of travel of the target, a time series prediction method is used to estimate the position and time when the target appears in another video, and target detection and re-identification are performed in a targeted manner.

[0037] Preferably, step 2 specifically includes the following steps:

[0038] Step 31: Based on the position and direction changes of the existing target in the previous and next frames, calculate the target's speed and direction of movement, and use the LSTM method to estimate the position and time of the target's appearance in the other video.

[0039] Step 32: At the predicted target location, use the yolov8 method to detect the target and perform feature vector matching. If the target is detected and matched successfully, continue tracking the target. If the target is not detected or not matched, lower the confidence screening threshold of the target detection algorithm, or change the FastRCNN method for target detection. If the target is still not detected or not matched, jump to step 33.

[0040] Step 33: Determine whether there are other docking areas. If so, perform target detection in the docking area; otherwise, determine that the target is lost.

[0041] The present invention provides another target trajectory generation method based on multi-channel security video. First, the following definitions are made:

[0042] The detection or tracking result of the target in a certain image frame is expressed as a rectangular frame. Let the coordinate of the upper left corner of the rectangular frame be (x lu ,y lu ), the coordinate of the lower right corner is (x rb ,y rb ), where lu represents the upper left corner of the rectangular box, rb represents the lower right corner of the rectangular box, and the center position of the rectangular box is The above two coordinates take the upper left corner of the image frame as the origin, and also record the time of the image frame, that is, the time when the target appears at that position;

[0043] Then, follow these steps:

[0044] Step 1: Establishing a regional mapping relationship: Based on the characteristics of overlapping areas in multiple video channels, a mapping relationship is established between the target pixel position and the regional position;

[0045] Step 1 is as follows: the image is divided into regions according to the resolution, divided into m regions by row and n regions by column, that is, the image frame is divided into m×n regions; let the region in a video channel A be represented as The area in a video channel B is represented as

[0046] Based on the coordinate system and the divided area, several aspects of mapping are completed:

[0047] 1) Mapping of the target area, i.e., the area within which the center coordinates of the i-th target are currently located; which areas the target rectangle covers, and the proportion of each area;

[0048] 2) The correspondence between the single overlapping area of ​​video A and video B, that is, Indicates area and Correspondingly, IOU is the region intersection-union ratio;

[0049] 3) Correspondence between overlapping regions of the target rectangles: Calculate the correspondence between each region and merge the regions.

[0050] 4) According to the camera location and security range, establish the connection area of ​​a camera boundary to other cameras, that is,

[0051] Step 2: Based on step 1, the target trajectory is obtained by fusing the results of multiple tracking methods.

[0052] Preferably, step 2 uses two tracking methods to track in parallel, matches the position trajectory of the target, and continues the target trajectory: simultaneously uses sort and deepsort tracking algorithms to record the target position and trajectory detected by each; when both tracking algorithms obtain tracking results, calculate the center of mass of the target and delineate the adjacent area; the center of mass of the target is the center position of the detected rectangular box; the adjacent area is the area obtained by expanding the length and width of the rectangular box by 1.2 times with the center of mass as the center; traverse and match the targets obtained by the current two tracking algorithms, calculate the distance between the center of mass and the IOU of the adjacent area, if the difference in the center of mass distance between the two targets is less than 5 pixels and the IOU is greater than 70%, it is judged that the two targets to be matched are the same target, and the target trajectory is merged; when a tracking algorithm loses the target, the tracking result of the other tracking algorithm is used to supplement it and continue to generate the target trajectory; when both tracking algorithms lose the target, the confidence screening threshold of the tracking algorithm is lowered, feature matching is performed, and the target position is retrieved.

[0053] Preferably, the method further includes step 3: for the situation where multiple targets accompany each other in time and space, when a target is lost, the position and trajectory of the lost target are continuously acquired based on the information of the accompanying target, so as to retrieve the position of the lost target and continue the target trajectory.

[0054] (3) Beneficial effects

[0055] The present application is directed to the problem of target trajectory generation in security video, and in view of the current situation of target loss and low matching degree caused by single use of target detection and re-identification, a target trajectory generation method based on multi-path security video is provided. For multi-path security video, there are four important features: first, there are overlapping security areas and repeated appearance of targets in multiple places in multi-path video; second, there is a connection or adjacency relationship between the security areas of multi-path video; third, the video is stream data rather than video segments, and the matching range is wider; fourth, the application feasibility of the target space-time association is stronger. Therefore, the present application fully utilizes the four features and auxiliary information of security video, and improves the accuracy of target tracking and trajectory generation based on multi-path video, multi-method fusion and multi-target association, and improves the re-identification accuracy. DETAILED DESCRIPTION

[0056] In order to make the purpose, content and advantages of the present application clearer, the specific embodiments of the present application will be further described in combination with examples.

[0057] The existing target tracking and target re-identification algorithm is generally applied to video segments, and has the problems of easy target loss and low target re-identification accuracy. Although the performance of single target tracking and re-identification algorithm can be improved, it is difficult to completely avoid the above problems. For multi-path security video, there are four important features: first, there are overlapping security areas and repeated appearance of targets in multiple places in multi-path video; second, there is a connection or adjacency relationship between the security areas of multi-path video; third, the video is stream data rather than video segments, and the matching range is wider; fourth, the application feasibility of the target space-time association is stronger. Therefore, the present application fully utilizes the four features, and improves the accuracy of target tracking and trajectory generation based on multi-path video, multi-method fusion and multi-target association.

[0058] The present application is directed to the problem of target trajectory generation in security video, and in view of the current situation of target loss and low matching degree caused by single use of target detection and re-identification, a target trajectory generation method based on multi-path security video is provided, which fully utilizes the features and auxiliary information of security video and improves the re-identification accuracy.

[0059] Firstly, the following definitions are made:

[0060] The detection or tracking result of a target in an image frame is in the form of a rectangular box, and the left upper corner coordinates of the rectangular box are (x lu ,y lu ), and the right lower corner coordinates are (x rb ,y rb ), wherein lu is the abbreviation of left-up, indicating the left upper corner of the rectangular box, and rb is the abbreviation of right-bottom, indicating the right lower corner of the rectangular box. The center position of the rectangular box is The above coordinates take the top left corner of the image frame as the origin. At the same time, the time of the image frame is recorded, i.e., the time when the target appears at the position.

[0061] Then the following steps are performed:

[0062] Step 1: Establishing a region mapping relationship: according to the characteristics of the overlapping regions of multiple videos, a mapping relationship between the target pixel position and the region position is established.

[0063] Considering the overlapping or crossing of security video deployment, it is necessary to establish a mapping relationship of the same region in different security pictures. It is relatively complex and unnecessary to establish a mapping relationship between image pixels. Therefore, it is considered to establish a mapping relationship on the concept of a region with a coarse granularity than a pixel. The image is divided into regions according to the resolution, divided into m regions by rows and divided into n regions by columns, i.e., the image frame is divided into m x n regions. Let the region in video A be represented as The region in video B is represented as

[0064] On the basis of the coordinate system and the divided regions, several mappings can be completed:

[0065] 1). Mapping of the target belonging region, i.e., which region range the center coordinate of the i-th target is currently located in; which regions are covered by the target rectangular frame, and the proportion of each region is recorded.

[0066] 2). Corresponding relationship of a single overlapping region between video A and video B, i.e., represents the region corresponds to , and the IOU is the region intersection ratio (Intersection over Union).

[0067] 3). Corresponding relationship of the overlapping region of the target rectangular frame: the corresponding relationship of each region is calculated, and the regions are merged.

[0068] 4). Connection position: according to the camera position and the security range, the connection (adjacent) region of the boundary of a certain camera in other cameras is established, i.e.,

[0069] Step 2: Re-capturing the target trajectory in the security picture using the overlapping region: on the basis of step 1, the target is re-detected at the overlapping position when the target is lost, and the target trajectory is continued.

[0070] In a single security video, after detecting a target, target detection is performed on adjacent targets in the next frame or keyframe to generate the target trajectory within the single security video, and the target's speed and direction of travel are estimated or predicted. To address the issue of overlapping security video frames, target detection and feature matching can be performed in the same area. This includes the following steps:

[0071] Step 21: Start the target detection algorithm yolov8 and the tracking algorithm deepsort to detect and track the target in a single frame image;

[0072] Step 22: Record the position and time (t i ,x i ,y i ), where t i is the time when the target appears, accurate to milliseconds, (x i ,y i ) is the relative image position of the target in the image frame;

[0073] Step 23: If the target is not lost, repeat step 22 to obtain the position sequence of the target in the current image sequence and form the target trajectory in chronological order; if the target is lost, determine whether there is an overlapping area between the video of this channel and other videos. If there is no overlapping area, jump to step 24; if there is an overlapping area, jump to step 25;

[0074] Step 24: Lower the confidence screening threshold of the current target detection algorithm, obtain the target that appears in the neighborhood where the target appeared in the previous frame, and perform feature vector matching. If the target in the current frame is detected again, continue to generate the target trajectory; if the target is not detected, use another method to detect in the neighboring area and perform feature vector matching. If the match is successful, continue to generate the trajectory; otherwise, the target is considered lost and the trajectory generation process ends.

[0075] Step 25: Use the same detection method to detect in the overlapping area and perform feature matching. If the target is detected and matched successfully, continue to generate the trajectory. If the target cannot be detected, lower the confidence screening threshold of the current target detection algorithm and continue detection. If the target still cannot be detected or is not matched, switch to another method to perform target detection and matching at the original position. If the trajectory is matched, continue to generate the trajectory using the current method. If the target is not detected or is not matched, it is considered that the target is lost and the trajectory generation process ends.

[0076] Step 3: Based on step 1, when the target leaves the current security screen, re-detect the target in the adjacent area and continue the target trajectory.

[0077] This step is based on step 1, according to the position of security camera, the speed and direction of the target, using time series prediction method to estimate the position and time of the target in another video, and carry out target detection and re-identification, the specific steps are as follows:

[0078] Step 31: According to the position change and direction change of the existing target before and after the frame, the speed and direction of change of the target are calculated, and the LSTM method is used to estimate the position and time of the target in another video;

[0079] Step 32: In the predicted position of the target, the yolov8 method is used for target detection, and the feature vector matching is carried out, if the target is detected and matched successfully, the target is continued to be tracked, if the target is not detected or not matched, the confidence threshold of the target detection algorithm is reduced, or the FastRCNN method is replaced for target detection, if the target is still not detected or not matched, jump to step 33;

[0080] Step 33: Determine whether there is other connection (adjacent) area, if yes, carry out target detection in the connection area; otherwise, determine that the target is lost.

[0081] Step 4: Based on step 1, the target trajectory is obtained by using the result fusion method of multiple tracking methods

[0082] In view of the problem that single algorithm is easy to lose target or the re-identification precision is not high, two tracking methods are used to track in parallel, the position trajectory of the target is matched, the precision of target detection is improved, and the target trajectory is continued.

[0083] At the same time, sort and deepsort two tracking algorithms are run, and the positions of the detected targets and the trajectories are recorded;

[0084] When both tracking algorithms get tracking results, the center of mass of the target is calculated and the adjacent area is determined; the center of mass of the target is the center position of the detected rectangular frame; the adjacent area is the area obtained by expanding the length and width of the rectangular frame by 1.2 times with the center as the center; the targets obtained by the current two tracking algorithms are traversed and matched, the distance between the centers of mass and the IOU of the adjacent areas are calculated, if the distance between the centers of mass of the two targets is less than 5 pixels and the IOU is greater than 70%, the two targets to be matched are judged as the same target, and the target trajectory is merged;

[0085] When a tracking algorithm loses the target, the tracking results of the other tracking algorithm are used to supplement and continue to generate the target trajectory;

[0086] When both tracking algorithms lose the target, the threshold of the confidence screening of the tracking algorithm is reduced, the feature matching is carried out, and the target position is reacquired.

[0087] The steps 2, 3, and 4 above respectively solve different technical problems on the basis of step 1, and steps 1 and 2 constitute scheme 1, steps 1 and 3 constitute scheme 2, and steps 1 and 4 constitute scheme 3.

[0088] Step 5: Obtain the position and trajectory of the target for the multi-target spatiotemporal accompanying case

[0089] On the basis of steps 2, 3, and 4, the spatiotemporal accompanying case is considered, and when the target is lost, the position and trajectory of the lost target are obtained according to the information of the accompanying target, so as to find the position of the lost target and continue the trajectory of the target, thereby perfecting the trajectory results of schemes 1, 2, and 3.

[0090] Taking the spatiotemporal accompanying of two targets as an example. Suppose that the sequences of the time and position information of two targets a and b are respectively are the time of target a and target b respectively, are the position coordinates of target a and target b respectively, and the correlation degree is 0, i.e. i = 1, and then the following steps are executed:

[0091] Step 51: Sort the trajectories of target a and target b in chronological order,

[0092] Step 52: Start from the time point , find the time point with the shortest interval in the trajectory of target b from , and find the time point with the shortest interval in the trajectory of target a from the time point, if the two time points are still , it is considered that the two time points found in target b and target a are adjacent time points, if not , jump to step 53;

[0093] Step 53: i = i + 1, if the updated i < n, jump to step 52, until i = n, jump to step 54;

[0094] Step 54: Calculate the distance between the positions of the two targets at the i-th pair of adjacent time points obtained in step 53, if the distance is less than a certain preset threshold or is contained in the adjacent region, it is considered that the two adjacent time points are a pair of accompanying time points, and the correlation degree is increased by 1;

[0095] Step 56: If the ratio of the correlation degree to the number of pairs of accompanying time points is greater than a threshold value of 0.8, it is considered that the two targets exist in the spatiotemporal accompanying phenomenon;

[0096] Step 57: When the target b is lost, the threshold of the confidence screening of the target detection algorithm is lowered in the area where the target a appears, the target detection algorithm is used for detection, and the feature vector of the detected target b is matched;

[0097] Step 58: If the feature vector can be successfully matched, the detection and tracking of the target b are continued; otherwise, it is judged that the target b is lost.

[0098] Step 6: On the basis of obtaining the target trajectory information in steps 2, 3, 4, and 5, a trajectory video is generated

[0099] On the basis of obtaining the target trajectory, the target trajectory is completely reproduced in a visualized manner, and a trajectory video of the target is generated. Usually, a method of generating in time sequence is adopted, but the situation of the trajectory in the overall region is not intuitive. Here, based on multi-path image splicing, the trajectory video is generated, and a method of superimposing the target image and information on the static image is adopted to generate the trajectory video. The specific steps are as follows:

[0100] Step 61: A complete image frame is generated by image splicing:

[0101] Step 611: Obtain the frames of the same time in the daytime of the current multi-path security video;

[0102] Step 612: Arrange the static frames in the order of left to right and top to bottom according to the actual security region;

[0103] Step 613: Use opencv based on SIFT (Scale-Invariant Feature Transform) features to traverse the image matrix in the order of left to right and then top to bottom for image splicing;

[0104] Step 614: Obtain a complete image of a security region;

[0105] Step 62: Obtain a target trajectory sequence information {(t1, x1, y1), (t2, x2, y2), … (t n ,x n ,y n )} and arrange it in time sequence; the coordinates (x i ,x i ) are relative coordinates in a video frame;

[0106] Step 63: According to the time and position of the target appearing, the corresponding target rectangular frame and background rectangular frame are cropped from the frame in the video;

[0107] Step 64: Use the background rectangular frame to perform pixel matching in the complete frame image to obtain the target (x i ,xi ) the position in the complete picture, and replacing the corresponding background rectangular frame with the target rectangular frame;

[0108] Step 65: generating the trajectory video of the target in time sequence according to the frame rate of 25 frames per second; wherein, if there is a time interval of the trajectory of the target, then the position of the target is interpolated by using the cubic spline method according to the time interval and the position of the target.

[0109] The above only describes the preferred embodiments of the present application, and it should be noted that for those skilled in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A target trajectory generation method based on multi-channel security video, characterized in that: First, define the following: The detection or tracking result of the target in a certain image frame is expressed as a rectangular frame. Let the coordinate of the upper left corner of the rectangular frame be (x lu ,y lu ), the coordinate of the lower right corner is (x rb ,y rb ), where lu represents the upper left corner of the rectangular box, rb represents the lower right corner of the rectangular box, and the center position of the rectangular box is The above two coordinates take the upper left corner of the image frame as the origin, and also record the time of the image frame, that is, the time when the target appears at that position; Then, follow these steps: Step 1: Establishing a region mapping relationship: Based on the overlapping characteristics of multiple video channels, a mapping relationship is established between the target pixel position and the region position. Specifically, the image is divided into regions according to the resolution, with the rows divided into m regions and the columns divided into n regions. That is, the image frame is divided into m×n regions. Let the region in a video A be represented as The area in a video channel B is represented as Based on the coordinate system and the divided area, several aspects of mapping are completed: 1) Mapping of the target area, i.e., the area within which the center coordinates of the i-th target are currently located; which areas the target rectangle covers, and the proportion of each area; 2) The correspondence between the single overlapping area of ​​video A and video B, that is, Indicates area and Correspondingly, IOU is the region intersection-union ratio; 3) Correspondence between overlapping regions of the target rectangles: Calculate the correspondence between each region and merge the regions. 4) According to the camera location and security range, establish the connection area of ​​a camera boundary to other cameras, that is, Step 2: Recapture the target trajectory using the overlapping area in the security screen: Based on step 1, re-detect the target at the overlapping position when the target is lost and continue the target trajectory.

2. The method according to claim 1, wherein Step 2 specifically includes the following steps: Step 21: Start the target detection algorithm yolov8 and the tracking algorithm deepsort to detect and track the target in a single frame image; Step 22: Record the position and time (t i ,x i ,y i ), where t i is the time when the target appears, accurate to milliseconds, (x i ,y i ) is the relative image position of the target in the image frame; Step 23: If the target is not lost, repeat step 22 to obtain the position sequence of the target in the current image sequence and form the target trajectory in chronological order; if the target is lost, determine whether there is an overlapping area between the video of this channel and other videos. If there is no overlapping area, jump to step 24; if there is an overlapping area, jump to step 25; Step 24: Lower the confidence screening threshold of the current target detection algorithm, obtain the target that appears in the neighborhood where the target appeared in the previous frame, and perform feature vector matching. If the target in the current frame is detected again, continue to generate the target trajectory; if the target is not detected, use another method to detect in the neighboring area and perform feature vector matching. If the match is successful, continue to generate the trajectory; otherwise, the target is considered lost and the trajectory generation process ends. Step 25: Use the same detection method to detect in the overlapping area and perform feature matching. If the target is detected and matched successfully, continue to generate the trajectory. If the target cannot be detected, lower the confidence screening threshold of the current target detection algorithm and continue detection. If the target still cannot be detected or is not matched, switch to another method to perform target detection and matching at the original position. If the trajectory is matched, continue to generate the trajectory using the current method. If the target is not detected or is not matched, it is considered that the target is lost and the trajectory generation process ends.

3. A target trajectory generation method based on multi-channel security video, characterized in that: First, define the following: The detection or tracking result of the target in a certain image frame is expressed as a rectangular frame. Let the coordinate of the upper left corner of the rectangular frame be (x lu ,y lu ), the coordinate of the lower right corner is (x rb ,y rb ), where lu represents the upper left corner of the rectangular box, rb represents the lower right corner of the rectangular box, and the center position of the rectangular box is The above two coordinates take the upper left corner of the image frame as the origin, and also record the time of the image frame, that is, the time when the target appears at that position; Then, follow these steps: Step 1: Establishing a regional mapping relationship: Based on the characteristics of overlapping areas in multiple video channels, a mapping relationship is established between the target pixel position and the regional position; Step 1 is as follows: the image is divided into regions according to the resolution, divided into m regions by row and n regions by column, that is, the image frame is divided into m×n regions; Let the region in a video A be represented as The area in a video channel B is represented as Based on the coordinate system and the divided area, several aspects of mapping are completed: 1) Mapping of the target area, i.e., the area within which the center coordinates of the i-th target are currently located; which areas the target rectangle covers, and the proportion of each area; 2) The correspondence between the single overlapping area of ​​video A and video B, that is, Indicates area and Correspondingly, IOU is the region intersection-union ratio; 3) Correspondence between overlapping regions of the target rectangles: Calculate the correspondence between each region and merge the regions. 4) According to the camera location and security range, establish the connection area of ​​a camera boundary to other cameras, that is, Step 2: Based on step 1, when the target leaves the current security screen, re-detect the target in the adjacent area and continue the target trajectory.

4. The method according to claim 3, wherein Step 2 is based on step 1. According to the position of the security camera, the speed and direction of the target, the time series prediction method is used to estimate the position and time of the target in another video, and the target is detected and re-identified in a targeted manner.

5. The method according to claim 3, wherein The specific steps of step 2 are as follows: Step 21: Based on the position and direction changes of the existing target in the previous and next frames, calculate the target's speed and direction of movement, and use the LSTM method to estimate the position and time of the target's appearance in the other video. Step 22: At the predicted target location, use the yolov8 method to detect the target and perform feature vector matching. If the target is detected and matched successfully, continue tracking the target. If the target is not detected or not matched, lower the confidence screening threshold of the target detection algorithm, or change the FastRCNN method for target detection. If the target is still not detected or not matched, jump to step 23. Step 23: Determine whether there are other docking areas. If so, perform target detection in the docking area; otherwise, determine that the target is lost.

6. A target trajectory generation method based on multi-channel security video, characterized in that: First, define the following: The detection or tracking result of the target in a certain image frame is expressed as a rectangular frame. Let the coordinate of the upper left corner of the rectangular frame be (x lu ,y lu ), the coordinate of the lower right corner is (x rb ,y rb ), where lu represents the upper left corner of the rectangular box, rb represents the lower right corner of the rectangular box, and the center position of the rectangular box is The above two coordinates take the upper left corner of the image frame as the origin, and also record the time of the image frame, that is, the time when the target appears at that position; Then, follow these steps: Step 1: Establishing a regional mapping relationship: Based on the characteristics of overlapping areas in multiple video channels, a mapping relationship is established between the target pixel position and the regional position; Step 1 is as follows: the image is divided into regions according to the resolution, divided into m regions by row and n regions by column, that is, the image frame is divided into m×n regions; Let the region in a video A be represented as The area in a video channel B is represented as Based on the coordinate system and the divided area, several aspects of mapping are completed: 1) Mapping of the target area, i.e., the area within which the center coordinates of the i-th target are currently located; which areas the target rectangle covers, and the proportion of each area; 2) The correspondence between the single overlapping area of ​​video A and video B, that is, Indicates area and Correspondingly, IOU is the region intersection-union ratio; 3) Correspondence between overlapping regions of the target rectangles: Calculate the correspondence between each region and merge the regions. 4) According to the camera location and security range, establish the connection area of ​​a camera boundary to other cameras, that is, Step 2: Based on step 1, the target trajectory is obtained by fusing the results of multiple tracking methods.

7. The method according to claim 6, wherein In step 2, two tracking methods are used in parallel for tracking to match the position trajectory of the target and continue the target trajectory: both the sort and deepsort tracking algorithms are adopted to record the detected target positions and trajectories respectively; when the tracking results are obtained by both tracking algorithms, the centroid of the target is calculated and the adjacent area is defined; the centroid of the target is the central position of the detected rectangular box; the adjacent area is an area obtained by expanding the length and width of the rectangular box by 1.2 times with the centroid as the center; the targets obtained by the two current tracking algorithms are traversed and matched, the distance between the centroids and the IOU of the adjacent area are calculated, if the difference in the centroid distances of the two targets is less than 5 pixels and the IOU is greater than 70%, it is determined that the two targets to be matched are the same target, and the target trajectories are merged; when a certain tracking algorithm loses the target, the tracking result of the other tracking algorithm is used for supplementation to continue generating the target trajectory; when both tracking algorithms lose the target, the threshold of the confidence screening of the tracking algorithm is reduced for feature matching to re-obtain the target position.

8. The method according to claim 2, 5 or 7, wherein: This method further includes step 3: for the multi-target spatio-temporal co-occurrence situation, when the target is lost, according to the information of the co-occurring target, the position and trajectory of the lost target are continuously obtained to retrieve the position of the lost target and continue the target trajectory.

9. The method according to claim 8, wherein In step 3, it is assumed that the sequences of time and position information of two targets a and b are are the time for target a and b respectively, Let a and b be the position coordinates of targets a and b respectively, set the correlation degree to 0, set i = 1, and then perform the following steps: Step 31: Sort the trajectories of targets a and b in chronological order. Step 32: From the time point Start by searching for the target b in its trajectory. The time point with the shortest interval; and find the time point with the shortest interval with this time point in the trajectory of target a, if it is still The two time points found in targets b and a are considered to be adjacent time points. Then jump to step 33; Step 33: Let i = i + 1. If the updated i < n, jump to step 32 until i = n and then jump to step 34; Step 34: Calculate the position distance between the two targets at the i-th pair of adjacent time points among the obtained appearance time points in step 33. If the distance is less than a certain preset threshold or is included in the adjacent area, it is considered that these two adjacent time points are a pair of co-occurring time points and the correlation degree is incremented by 1; Step 36: If the ratio of the correlation degree to the number of co-occurring time point pairs is greater than the threshold, it is considered that there is a spatio-temporal co-occurrence phenomenon between the two targets; Step 37: When target b is lost, reduce the threshold of the confidence screening of the target detection algorithm at the area where target a appears, use the target detection algorithm for detection, and match the feature vectors of the detected target b; Step 38: If the feature vectors can be successfully matched, continue to detect and track target b; otherwise, it is determined that target b is lost.

10. The method according to claim 9, wherein This method further includes step 4: on the basis of obtaining the target trajectory, the target trajectory is completely reproduced in a visual way to generate a trajectory video of the target; among them, based on multi-channel image stitching, a trajectory video is generated, and a trajectory video is generated by the method of superimposing the target image and information on the static image. The specific steps are as follows: Step 41: Generate a complete image frame through image stitching: Step 411: Obtain the frames of the current multi-channel security videos at the same moment during the day; Step 412: Arrange the static frames in the order from left to right and from top to bottom according to the actual security area; Step 413: Use opencv to perform image stitching on the image matrix based on SIFT features by traversing in the order from left to right and then from top to bottom; Step 414: Obtain a complete image of the security area; Step 42: Get a target trajectory sequence information {(t1,x1,y1),(t2,x2,y2),…(t n ,x n ,y n )} and arranged in time series; the coordinates (x i ,x i ) is the relative coordinate in a certain video image; Step 43: Based on the time and location of the target, a corresponding target rectangle and a corresponding background rectangle are cropped from the image in the video. Step 44: Use the background rectangle to perform pixel matching in the complete image to obtain the target (x i ,x i ) in the complete picture, and replace the corresponding background rectangle with the target rectangle; Step 45: Generate a target trajectory video in chronological order at a certain frame rate; if there is a time interval between the target trajectory, interpolate the target position using a cubic spline method according to the time interval and the target position.

Images