Target trajectory generation method based on multi-channel security video

Abstract

The invention relates to a target trajectory generation method based on multiple paths of security and protection videos, and belongs to the technical field. The invention provides a target trajectory generation method based on multiple paths of security and protection videos for the problem of target trajectory generation in the security and protection videos and for the current situations of target loss and low matching degree caused by single adoption of target detection and re-identification. For multi-channel security and protection videos, four important characteristics exist as follows: 1, the multi-channel videos have conditions of security and protection area overlapping and repeated occurrence of targets at multiple positions; 2, the multi-channel video security protection area has a connection or adjacency relationship; thirdly, the video is stream data instead of video clips, so that the matching range is wider; and fourthly, the application feasibility of time-space accompanying of the target is higher. Therefore, the four characteristics and auxiliary information of the security video are fully utilized, the accuracy of target tracking and trajectory generation is improved based on multi-channel video, multi-method fusion and multi-target relevance, and the re-recognition precision is improved.

Description

Technical Field

[0001] The invention belongs to the technical field, and in particular relates to a target trajectory generation method based on multi-channel security videos. Background Art

[0002] Security video technology is used to detect and monitor the defense area, and on-site images can be displayed and recorded in real time, and historical images can be retrieved and displayed. The acquired security video is a kind of unstructured data. Structured data is data with irregular or incomplete data structure, no predefined data model, and is not convenient to be represented by a two-dimensional logical table in a database. Security video needs to be structured and analyzed to obtain the target information of concern in order to achieve the purpose of security more conveniently and quickly. Security video analysis must first detect the target, and then use its information for subsequent analysis and processing. Target trajectory generation is one of its applications. Unlike target detection in a single image or a single video segment, security video is generally collected through multiple cameras.

[0003] Target trajectories are usually generated using target tracking and target re-identification methods. Target tracking is a common computer vision task that requires detecting targets in consecutive video frames and assigning a track id to each target, which is unique in the video sequence. Target re-identification aims to determine whether the same target exists in images or videos from different camera perspectives. The main technologies can be divided into feature matching-based methods and learning-based methods. Among them, learning-based methods have received widespread attention in recent years. By training deep neural network models, more robust and effective feature representations can be learned to improve the performance of re-identification. The process of trajectory generation is to first use the detection model to locate the bounding box position of the target in the image, then use the association model to extract re-identification features for each bounding box, and then connect the bounding box with an existing tracking result based on the specific metrics defined by these features.

[0004] The method of simply using target tracking and target re-identification has the following problems: first, target detection may result in missed targets, lost targets, or the entire image needs to be traversed, which is time-consuming; second, targets in different scenes need to be re-identified, that is, feature matching is performed, but this matching is a whole traversal, which is time-consuming and has low matching accuracy; third, the existing methods mainly target single image fragments and do not consider the correlation between multiple videos. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] The technical problem to be solved by the present invention is: to provide a target trajectory generation method based on multi-channel security videos in view of the target trajectory generation problem in security videos and the current situation that the target is lost and the matching degree is not high due to the single use of target detection and re-identification.

[0007] (II) Technical solution

[0008] In order to solve the above technical problems, the present invention provides a target trajectory generation method based on multi-channel security videos. First, the following definitions are made:

[0009] The detection or tracking result of the target in a certain image frame is expressed as a rectangular frame. Let the coordinates of the upper left corner of the rectangular frame be (x lu ,y lu ), the coordinate of the lower right corner is (x rb ,y rb ), where lu represents the upper left corner of the rectangular box, rb represents the lower right corner of the rectangular box, and the center position of the rectangular box is The above two coordinates take the upper left corner of the image frame as the origin, and record the time of the image frame, that is, the time when the target appears at that position;

[0010] Then, follow these steps:

[0011] Step 1: Establishing a regional mapping relationship: Based on the characteristics of overlapping areas in multiple video channels, a mapping relationship is established between the target pixel position and the regional position;

[0012] Step 1 is as follows: the image is divided into regions according to the resolution, divided into m regions by row and n regions by column, that is, the image frame is divided into m×n regions; let the region in a video A be represented as The area in a video channel B is represented as

[0013] Based on the coordinate system and the divided area, several aspects of mapping are completed:

[0014] 1) Mapping of the target area, that is, the area in which the center coordinates of the i-th target are currently located; which areas the target rectangle box covers, and record the proportion of each area;

[0015] 2). The correspondence between the single overlapping area of ​​video A and video B, that is, Indicates area and Correspondingly, IOU is the area intersection-union ratio;

[0016] 3) Correspondence between overlapping areas of target rectangular frames: Calculate the correspondence between each area and merge the areas;

[0017] 4). According to the camera location and security range, establish the connection area of ​​a camera boundary in other cameras, that is,

[0018] Step 2: Recapture the target trajectory using the overlapping area in the security screen: Based on step 1, re-detect the target at the overlapping position when the target is lost and continue the target trajectory.

[0019] Preferably, step 2 specifically includes the following steps:

[0020] Step 21: Start the target detection algorithm yolov8 and the tracking algorithm deepsort to detect and track the target in a single frame image;

[0021] Step 22: Record the position and time (t i ,x i ,y i ), where t i is the time when the target appears, accurate to milliseconds, (x i ,y i ) is the relative image position of the target in the image frame;

[0022] Step 23: If the target is not lost, repeat step 22 to obtain the position sequence of the target in the current image sequence, and form the target trajectory in chronological order; when the target is lost, determine whether there is an overlapping area between the video of this channel and other videos, if there is no overlapping area, jump to step 24, if there is an overlapping area, jump to step 25;

[0023] Step 24: Lower the confidence screening threshold of the current target detection algorithm, obtain the target that appears in the neighborhood where the target appeared in the previous frame, and perform feature vector matching. If the target in the current frame is detected again, continue to generate the target trajectory; if the target is not detected, use other methods to detect in the neighboring area and perform feature vector matching. If the match is successful, continue to generate the trajectory, otherwise it is considered that the target is lost and the trajectory generation process ends;

[0024] Step 25: Use the same detection method to perform detection in the overlapping area and perform feature matching; if the target can be detected and matched successfully, continue to generate the trajectory; if the target cannot be detected, lower the confidence screening threshold of the current target detection algorithm and continue to detect; if the target still cannot be detected or is not matched, change other methods to perform target detection and matching at the original position; if the trajectory can be matched, continue to generate the trajectory using the current method; if the target cannot be detected or is not matched, it is considered that the target is lost and the trajectory generation process ends.

[0025] The present invention provides another target trajectory generation method based on multi-channel security videos. First, the following definitions are made:

[0026] The detection or tracking result of the target in a certain image frame is expressed as a rectangular frame. Let the coordinates of the upper left corner of the rectangular frame be (x lu ,y lu ), the coordinate of the lower right corner is (x rb ,y rb ), where lu represents the upper left corner of the rectangular box, rb represents the lower right corner of the rectangular box, and the center position of the rectangular box is The above two coordinates take the upper left corner of the image frame as the origin, and record the time of the image frame, that is, the time when the target appears at that position;

[0027] Then, follow these steps:

[0028] Step 1: Establishing a regional mapping relationship: Based on the characteristics of overlapping areas in multiple video channels, a mapping relationship is established between the target pixel position and the regional position;

[0029] Step 1 is as follows: the image is divided into regions according to the resolution, divided into m regions by row and n regions by column, that is, the image frame is divided into m×n regions; let the region in a video A be represented as The area in a video channel B is represented as

[0030] Based on the coordinate system and the divided area, several aspects of mapping are completed:

[0031] 1) Mapping of the target area, that is, the area in which the center coordinates of the i-th target are currently located; which areas the target rectangle box covers, and record the proportion of each area;

[0032] 2). The correspondence between the single overlapping area of ​​video A and video B, that is, Indicates area and Correspondingly, IOU is the area intersection-union ratio;

[0033] 3) Correspondence between overlapping areas of target rectangular frames: Calculate the correspondence between each area and merge the areas;

[0034] 4). According to the camera location and security range, establish the connection area of ​​a camera boundary in other cameras, that is,

[0035] Step 2: Based on step 1, when the target leaves the current security screen, re-detect the target in the adjacent area and continue the target trajectory.

[0036] Preferably, step 2 is based on step 1, and according to the position of the security camera, the speed and direction of travel of the target, a time series prediction method is used to estimate the position and time when the target appears in another video, and target detection and re-identification are performed in a targeted manner.

[0037] Preferably, step 2 specifically includes the following steps:

[0038] Step 31: Calculate the speed and direction of the target change based on the position change and direction change of the previous and next frames of the existing target, and use the LSTM method to estimate the position and time of the target appearing in another video.

[0039] Step 32: At the predicted position of the target, use the yolov8 method to detect the target and perform feature vector matching. If the target is detected and matched successfully, continue to track the target. If the target cannot be detected or is not matched, reduce the confidence screening threshold of the target detection algorithm, or replace the FastRCNN method for target detection. If the target is still not detected or not matched, jump to step 33;

[0040] Step 33: Determine whether there are other docking areas. If so, perform target detection in the docking area; otherwise, determine that the target is lost.

[0041] The present invention provides another target trajectory generation method based on multi-channel security video. First, the following definitions are made:

[0042] The detection or tracking result of the target in a certain image frame is expressed as a rectangular frame. Let the coordinates of the upper left corner of the rectangular frame be (x lu ,y lu ), the coordinate of the lower right corner is (x rb ,y rb ), where lu represents the upper left corner of the rectangular box, rb represents the lower right corner of the rectangular box, and the center position of the rectangular box is The above two coordinates take the upper left corner of the image frame as the origin, and record the time of the image frame, that is, the time when the target appears at that position;

[0043] Then, follow these steps:

[0044] Step 1: Establishing a regional mapping relationship: Based on the characteristics of overlapping areas in multiple video channels, a mapping relationship is established between the target pixel position and the regional position;

[0045] Step 1 is as follows: the image is divided into regions according to the resolution, divided into m regions by row and n regions by column, that is, the image frame is divided into m×n regions; let the region in a video A be represented as The area in a video channel B is represented as

[0046] Based on the coordinate system and the divided area, several aspects of mapping are completed:

[0047] 1) Mapping of the target area, that is, the area in which the center coordinates of the i-th target are currently located; which areas the target rectangle box covers, and record the proportion of each area;

[0048] 2). The correspondence between the single overlapping area of ​​video A and video B, that is, Indicates area and Correspondingly, IOU is the area intersection-union ratio;

[0049] 3) Correspondence between overlapping areas of target rectangular frames: Calculate the correspondence between each area and merge the areas;

[0050] 4). According to the camera location and security range, establish the connection area of ​​a camera boundary in other cameras, that is,

[0051] Step 2: Based on step 1, the target trajectory is obtained by fusing the results of multiple tracking methods.

[0052] Preferably, step 2 uses two tracking methods to track in parallel, matches the position trajectory of the target, and continues the target trajectory: simultaneously uses two tracking algorithms, sort and deepsort, to record the target position and trajectory detected by each; when both tracking algorithms obtain tracking results, calculate the center of mass of the target and delineate the adjacent area; wherein the center of mass of the target is the center position of the detected rectangular frame; the adjacent area is the area obtained by expanding the length and width of the rectangular frame by 1.2 times with the center of mass as the center; traverse and match the targets obtained by the current two tracking algorithms, calculate the distance of the center of mass and the IOU of the adjacent area, if the difference in the center of mass distance between the two targets is less than 5 pixels, and the IOU is greater than 70%, it is judged that the two targets to be matched are the same target, and the target trajectory is merged; when a tracking algorithm loses the target, the tracking result of another tracking algorithm is used to supplement it, and the target trajectory is continuously generated; when both tracking algorithms lose the target, the threshold of the confidence screening of the tracking algorithm is lowered, feature matching is performed, and the target position is obtained again.

[0053] Preferably, the method further includes step 3: for the situation of multi-target spatiotemporal accompanying, when the target is lost, continue to obtain the position and trajectory of the lost target based on the information of the accompanying target, so as to find the position of the lost target and continue the target trajectory.

[0054] (III) Beneficial effects

[0055] The present invention is aimed at the problem of target trajectory generation in security videos. Aiming at the current situation that the single use of target detection and re-identification leads to target loss and low matching degree, a target trajectory generation method based on multi-channel security videos is provided. For multi-channel security videos, there are four important characteristics: first, there are overlapping security areas and repeated targets in multiple places in multi-channel videos; second, there are connection or adjacency relationships between security areas in multiple channels of video; third, the video is streaming data rather than video clips, and the matching range is wider; fourth, the application feasibility of target spatiotemporal accompaniment is stronger. To this end, the present invention makes full use of these four characteristics of security videos and auxiliary information, and improves the accuracy of target tracking and trajectory generation based on multi-channel videos, multi-method fusion, and multi-target correlation, and improves the accuracy of re-identification. DETAILED DESCRIPTION

[0056] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with embodiments.

[0057] Existing target tracking and target re-identification algorithms are generally applied to video clips, and there are problems such as easy loss of targets and low accuracy of target re-identification. Although it can be improved by improving the performance of single target tracking and re-identification algorithms, it is difficult to completely avoid the above problems. For multi-channel security videos, there are four important characteristics: first, there are overlapping security areas in multi-channel videos and targets are repeated in multiple places; second, there are connection or adjacency relationships between security areas in multiple channels of video; third, the video is streaming data, not video clips, and the matching range is wider; fourth, the application feasibility of target spatiotemporal accompaniment is stronger. To this end, the present invention makes full use of these four characteristics to improve the accuracy of target tracking and trajectory generation based on multi-channel videos, multi-method fusion, and multi-target correlation.

[0058] The present invention aims at the problem of target trajectory generation in security videos. Aiming at the current situation that target loss and low matching degree are caused by single target detection and re-identification, a target trajectory generation method based on multi-channel security videos is provided, which makes full use of the characteristics of security videos and auxiliary information to improve the re-identification accuracy.

[0059] First, define the following:

[0060] The detection or tracking result of the target in a certain image frame is expressed as a rectangular frame. Let the coordinates of the upper left corner of the rectangular frame be (x lu ,y lu ), the coordinate of the lower right corner is (x rb ,y rb ), where lu is the abbreviation of left-up, indicating the upper left corner of the rectangular box, and rb is the abbreviation of right-bottom, indicating the lower right corner of the rectangular box. The center position of the rectangular box is The above coordinates take the upper left corner of the image frame as the origin. At the same time, the time of the image frame is recorded, that is, the time when the target appears at that position.

[0061] Then follow these steps:

[0062] Step 1: Establishing a regional mapping relationship: Based on the characteristics of overlapping areas in multiple video channels, a mapping relationship is established between the target pixel position and the regional position.

[0063] Considering the overlapping or cross-cutting nature of security video deployment, it is necessary to establish a mapping relationship between the same area in different security images. It is more complicated and unnecessary to establish a mapping relationship between image pixels. Therefore, consider establishing a mapping relationship based on the concept of regions that are coarser than pixels. The image is divided into regions according to resolution, divided into m regions by row and n regions by column, that is, the image frame is divided into m×n regions. Let the region in a certain video A be represented as The area in a video channel B is represented as

[0064] Based on the coordinate system and the divided area, several aspects of mapping can be completed:

[0065] 1) Mapping of the target area, that is, the area where the center coordinates of the i-th target are currently located; which areas the target rectangle covers, and record the proportion of each area.

[0066] 2). The correspondence between the single overlapping area of ​​video A and video B, that is, Indicates area and Correspondingly, IOU is the Intersection over Union.

[0067] 3). Correspondence between overlapping areas of target rectangular frames: Calculate the correspondence between each area separately and merge the areas.

[0068] 4) Connection location: According to the camera location and security range, establish the connection (adjacent) area of ​​a camera boundary to other cameras, that is,

[0069] Step 2: Recapture the target trajectory using the overlapping area in the security screen: Based on step 1, re-detect the target at the overlapping position when the target is lost and continue the target trajectory.

[0070] In a single security video, after a target is detected, target detection is performed on the neighboring targets in the next frame or the next key frame to generate the target trajectory in the single security video, and estimate or predict the target's speed and direction of travel. Considering the problem of cross-screen security video, target detection can be performed in the same area and feature matching can be performed. The specific steps include:

[0071] Step 21: Start the target detection algorithm yolov8 and the tracking algorithm deepsort to detect and track the target in a single frame image;

[0072] Step 22: Record the position and time (t i ,x i ,y i ), where t i is the time when the target appears, accurate to milliseconds, (x i ,y i ) is the relative image position of the target in the image frame;

[0073] Step 23: If the target is not lost, repeat step 22 to obtain the position sequence of the target in the current image sequence, and form the target trajectory in chronological order; when the target is lost, determine whether there is an overlapping area between the video of this channel and other videos, if there is no overlapping area, jump to step 24, if there is an overlapping area, jump to step 25;

[0074] Step 24: Lower the confidence screening threshold of the current target detection algorithm, obtain the target that appears in the neighborhood where the target appeared in the previous frame, and perform feature vector matching. If the target in the current frame is detected again, continue to generate the target trajectory; if the target is not detected, use other methods to detect in the neighboring area and perform feature vector matching. If the match is successful, continue to generate the trajectory, otherwise it is considered that the target is lost and the trajectory generation process ends;

[0075] Step 25: Use the same detection method to perform detection in the overlapping area and perform feature matching; if the target can be detected and matched successfully, continue to generate the trajectory; if the target cannot be detected, lower the confidence screening threshold of the current target detection algorithm and continue to detect; if the target still cannot be detected or is not matched, change other methods to perform target detection and matching at the original position; if the trajectory can be matched, continue to generate the trajectory using the current method; if the target cannot be detected or is not matched, it is considered that the target is lost and the trajectory generation process ends.

[0076] Step 3: Based on step 1, when the target leaves the current security screen, re-detect the target in the adjacent area and continue the target trajectory.

[0077] This step is based on step 1. According to the location of the security camera, the speed and direction of the target, the time series prediction method is used to estimate the location and time of the target in another video, and the target detection and re-identification are carried out in a targeted manner. The specific steps are as follows:

[0078] Step 31: Calculate the speed and direction of the target change based on the position change and direction change of the previous and next frames of the existing target, and use the LSTM method to estimate the position and time of the target appearing in another video.

[0079] Step 32: At the predicted position of the target, use the yolov8 method to detect the target and perform feature vector matching. If the target is detected and matched successfully, continue to track the target. If the target cannot be detected or is not matched, reduce the confidence screening threshold of the target detection algorithm, or replace the FastRCNN method for target detection. If the target is still not detected or not matched, jump to step 33;

[0080] Step 33: Determine whether there are other connected (adjacent) areas. If so, perform target detection in the connected area; otherwise, determine that the target is lost.

[0081] Step 4: Based on step 1, obtain the target trajectory by fusing the results of multiple tracking methods

[0082] In order to solve the problem that a single algorithm is prone to losing the target or the re-identification accuracy is low, two tracking methods are used in parallel to track the target's position trajectory, improve the accuracy of target detection, and continue the target trajectory.

[0083] Run the sort and deepsort tracking algorithms simultaneously, and record the target positions and trajectories detected by each.

[0084] When both tracking algorithms obtain tracking results, the centroid of the target is calculated and the adjacent area is delineated; the centroid of the target is the center of the detected rectangular box; the adjacent area is the area obtained by expanding the length and width of the rectangular box by 1.2 times with the centroid as the center; the targets obtained by the current two tracking algorithms are traversed and matched, and the distance of the centroid and the IOU of the adjacent area are calculated. If the difference in the centroid distance between the two targets is less than 5 pixels and the IOU is greater than 70%, the two targets to be matched are judged to be the same target and the target tracks are merged;

[0085] When a tracking algorithm loses the target, the tracking result of another tracking algorithm is used to supplement and continue to generate the target trajectory;

[0086] When both tracking algorithms lose the target, the confidence screening threshold of the tracking algorithm is lowered, feature matching is performed, and the target position is retrieved.

[0087] The above steps 2, 3, and 4 solve different technical problems based on step 1 respectively. Steps 1 and 2 form Solution 1, steps 1 and 3 form Solution 2, and steps 1 and 4 form Solution 3.

[0088] Step 5: For the multi-target spatio-temporal co-occurrence situation, obtain the target positions and trajectories

[0089] Based on steps 2, 3, and 4, considering the multi-target spatio-temporal co-occurrence situation, when a target is lost, according to the information of the co-occurring target, continue to obtain the position and trajectory of the lost target to retrieve the position of the lost target and continue the target trajectory, thereby improving the trajectory results of Solutions 1, 2, and 3.

[0090] Take the two-target spatio-temporal co-occurrence as an example. Suppose the sequences composed of the time and position information of the two targets a and b are respectively are the times of targets a and b respectively, are the position coordinates of targets a and b respectively, and let the correlation degree be 0. Let i = 1, and then execute the following steps:

[0091] Step 51: Sort the trajectories of targets a and b in chronological order.

[0092] Step 52: Starting from the time point in the trajectory of target b, find the time point with the shortest interval from ; and in the trajectory of target a, find the time point with the shortest interval from this time point. If it is still then these two time points found in targets b and a are considered adjacent time points. If not then jump to step 53;

[0093] Step 53: Let i = i + 1. If the updated i < n, then jump to step 52. When i = n, jump to step 54;

[0094] Step 54: Calculate the position distance between the two targets at the i-th pair of adjacent time points obtained in step 53. If the distance is less than a certain preset threshold or is included in an adjacent area, then these two adjacent time points are considered a pair of co-occurrence time points, and the correlation degree is incremented by 1;

[0095] Step 56: If the ratio of the correlation degree to the number of co-occurrence time point pairs is greater than the threshold 0.8, then it is considered that there is a spatio-temporal co-occurrence phenomenon between the two targets;

[0096] Step 57: When target b is lost, the confidence screening threshold of the target detection algorithm is lowered at the area where target a appears, the target detection algorithm is used for detection, and the feature vector of the detected target b is matched;

[0097] Step 58: If the feature vector can be successfully matched, continue to detect and track the target b; otherwise, it is determined that the target b is lost.

[0098] Step 6: Generate a trajectory video based on the target trajectory information obtained in steps 2, 3, 4, and 5

[0099] On the basis of obtaining the target trajectory, the target trajectory is fully reproduced in a visual way to generate a target trajectory video. Usually, the method of generating one by one in chronological order is adopted, but the display of the trajectory in the overall area is not intuitive. Here, based on the splicing of multiple images, the trajectory video is generated, and the trajectory video is generated by superimposing the target image and information on the static image. The specific steps are as follows:

[0100] Step 61: Generate a complete image by stitching the images:

[0101] Step 611: Obtain the images of the current multiple security videos at the same time during the day;

[0102] Step 612: Arrange the static images in order from left to right and from top to bottom according to the actual security areas;

[0103] Step 613: Using opencv based on SIFT (Scale-Invariant Feature Transform) features, the image matrix is ​​traversed from left to right and then from top to bottom to perform image stitching;

[0104] Step 614: Acquire a complete image of the security area;

[0105] Step 62: Obtain a target trajectory sequence information {(t 1 ,x 1 ,y 1 ),(t 2 ,x 2 ,y 2 ),…(t n ,x n ,y n )} and arranged in time series; the coordinates (x i ,x i ) is the relative coordinate in a video screen;

[0106] Step 63: According to the time and position of the target appearance, a corresponding target rectangular frame and a corresponding background rectangular frame are cut out from the picture in the video;

[0107] Step 64: Use the background rectangle to perform pixel matching in the complete image to obtain the target (x i ,x i ) in the complete picture, and replace the corresponding background rectangle with the target rectangle;

[0108] Step 65: Generate a target trajectory video in chronological order at a frame rate of 25 frames per second; if there is a gap in the target trajectory time, interpolate the target position using a cubic spline method according to the time interval and the target position.

[0109] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating target trajectories based on multi-channel security videos, characterized in that: First, define the following: The detection or tracking result of the target in a certain image frame is expressed as a rectangular frame. Let the coordinates of the upper left corner of the rectangular frame be (x lu ,y lu ), the coordinate of the lower right corner is (x rb ,y rb ), where lu represents the upper left corner of the rectangular box, rb represents the lower right corner of the rectangular box, and the center position of the rectangular box is The above two coordinates take the upper left corner of the image frame as the origin, and record the time of the image frame, that is, the time when the target appears at that position; Then, follow these steps: Step 1: Establishing a regional mapping relationship: According to the characteristics of overlapping areas in multiple video channels, a mapping relationship is established between the target pixel position and the regional position; specifically, the image is divided into regions according to the resolution, divided into m regions by row, and divided into n regions by column, that is, the image frame is divided into m×n regions; Let the region in a video A be represented as The area in a video channel B is represented as Based on the coordinate system and the divided area, several aspects of mapping are completed: 1) Mapping of the target area, that is, the area in which the center coordinates of the i-th target are currently located; which areas the target rectangle box covers, and record the proportion of each area; 2). The correspondence between the single overlapping area of ​​video A and video B, that is, Indicates area and Correspondingly, IOU is the area intersection-union ratio; 3) Correspondence between overlapping areas of target rectangular frames: Calculate the correspondence between each area and merge the areas; 4) According to the camera location and security range, establish the connection area of ​​a camera boundary in other cameras, that is, Step 2: Recapture the target trajectory using the overlapping area in the security screen: Based on step 1, re-detect the target at the overlapping position when the target is lost and continue the target trajectory.

2. The method according to claim 1, characterized in that Step 2 specifically includes the following steps: Step 21: Start the target detection algorithm yolov8 and the tracking algorithm deepsort to detect and track the target in a single frame image; Step 22: Record the position and time (t i ,x i ,y i ), where t i is the time when the target appears, accurate to milliseconds, (x i ,y i ) is the relative image position of the target in the image frame; Step 23: If the target is not lost, repeat step 22 to obtain the position sequence of the target in the current image sequence, and form the target trajectory in chronological order; when the target is lost, determine whether there is an overlapping area between the video of this channel and other videos, if there is no overlapping area, jump to step 24, if there is an overlapping area, jump to step 25; Step 24: Lower the confidence screening threshold of the current target detection algorithm, obtain the target that appears in the neighborhood where the target appeared in the previous frame, and perform feature vector matching. If the target in the current frame is detected again, continue to generate the target trajectory; if the target is not detected, use other methods to detect in the neighboring area and perform feature vector matching. If the match is successful, continue to generate the trajectory, otherwise it is considered that the target is lost and the trajectory generation process ends; Step 25: Use the same detection method to perform detection in the overlapping area and perform feature matching; if the target can be detected and matched successfully, continue to generate the trajectory; if the target cannot be detected, lower the confidence screening threshold of the current target detection algorithm and continue to detect; if the target still cannot be detected or is not matched, change other methods to perform target detection and matching at the original position; if the trajectory can be matched, continue to generate the trajectory using the current method; if the target cannot be detected or is not matched, it is considered that the target is lost and the trajectory generation process ends.

3. A method for generating target trajectories based on multi-channel security videos, characterized in that: First, define the following: The detection or tracking result of the target in a certain image frame is expressed as a rectangular frame. Let the coordinates of the upper left corner of the rectangular frame be (x lu ,y lu ), the coordinate of the lower right corner is (x rb ,y rb ), where lu represents the upper left corner of the rectangular box, rb represents the lower right corner of the rectangular box, and the center position of the rectangular box is The above two coordinates take the upper left corner of the image frame as the origin, and record the time of the image frame, that is, the time when the target appears at that position; Then, follow these steps: Step 1: Establishing a regional mapping relationship: Based on the characteristics of overlapping areas in multiple video channels, a mapping relationship is established between the target pixel position and the regional position; Step 1 is specifically as follows: the image is divided into regions according to the resolution, divided into m regions by row and n regions by column, that is, the image frame is divided into m×n regions; Let the region in a video A be represented as The area in a video channel B is represented as Based on the coordinate system and the divided area, several aspects of mapping are completed: 1) Mapping of the target area, that is, the area in which the center coordinates of the i-th target are currently located; which areas the target rectangle box covers, and record the proportion of each area; 2). The correspondence between the single overlapping area of ​​video A and video B, that is, Indicates area and Correspondingly, IOU is the area intersection-union ratio; 3) Correspondence between overlapping areas of target rectangular frames: Calculate the correspondence between each area and merge the areas; 4) According to the camera location and security range, establish the connection area of ​​a camera boundary in other cameras, that is, Step 2: Based on step 1, when the target leaves the current security screen, re-detect the target in the adjacent area and continue the target trajectory.

4. The method according to claim 3, characterized in that Step 2 is based on step 1. According to the position of the security camera, the speed and direction of the target, the time series prediction method is used to estimate the position and time of the target in another video, and the target detection and re-identification are carried out in a targeted manner.

5. The method according to claim 3, characterized in that The specific steps of step 2 are as follows: Step 21: Calculate the speed and direction of the target change based on the position change and direction change of the previous and next frames of the existing target, and use the LSTM method to estimate the position and time of the target appearing in another video. Step 22: At the predicted position of the target, use the yolov8 method to detect the target and perform feature vector matching. If the target is detected and matched successfully, continue to track the target. If the target cannot be detected or matched, lower the confidence screening threshold of the target detection algorithm, or replace the FastRCNN method for target detection. If the target is still not detected or matched, jump to step 23; Step 23: Determine whether there are other docking areas. If so, perform target detection in the docking area; otherwise, determine that the target is lost.

6. A method for generating target trajectories based on multi-channel security videos, characterized in that: First, define the following: The detection or tracking result of the target in a certain image frame is expressed as a rectangular frame. Let the coordinates of the upper left corner of the rectangular frame be (x lu ,y lu ), the coordinate of the lower right corner is (x rb ,y rb ), where lu represents the upper left corner of the rectangular box, rb represents the lower right corner of the rectangular box, and the center position of the rectangular box is The above two coordinates take the upper left corner of the image frame as the origin, and record the time of the image frame, that is, the time when the target appears at that position; Then, follow these steps: Step 1: Establishing a regional mapping relationship: Based on the characteristics of overlapping areas in multiple video channels, a mapping relationship is established between the target pixel position and the regional position; Step 1 is specifically as follows: the image is divided into regions according to the resolution, divided into m regions by row and n regions by column, that is, the image frame is divided into m×n regions; Let the region in a video A be represented as The area in a video channel B is represented as Based on the coordinate system and the divided area, several aspects of mapping are completed: 1) Mapping of the target area, that is, the area in which the center coordinates of the i-th target are currently located; which areas the target rectangle box covers, and record the proportion of each area; 2). The correspondence between the single overlapping area of ​​video A and video B, that is, Indicates area and Correspondingly, IOU is the area intersection-union ratio; 3) Correspondence between overlapping areas of target rectangular frames: Calculate the correspondence between each area and merge the areas; 4) According to the camera location and security range, establish the connection area of ​​a camera boundary in other cameras, that is, Step 2: Based on step 1, the target trajectory is obtained by fusing the results of multiple tracking methods.

7. The method according to claim 6, characterized in that In step 2, two tracking methods are used in parallel for tracking to match the position trajectory of the target and continue the target trajectory: both the sort and deepsort tracking algorithms are adopted to record the detected target positions and trajectories respectively; when the tracking results are obtained by both tracking algorithms, the centroid of the target is calculated and the adjacent area is delimited; the centroid of the target is the center position of the detected rectangular box; the adjacent area is the area obtained by expanding the length and width of the rectangular box by 1.2 times with this centroid as the center; the targets obtained by the current two tracking algorithms are traversed and matched, the distance between the centroids and the IOU of the adjacent area are calculated, if the difference in the centroid distances of the two targets is less than 5 pixels and the IOU is greater than 70%, it is determined that the two targets to be matched are the same target and the target trajectories are merged; when a certain tracking algorithm loses the target, the tracking result of the other tracking algorithm is used for supplementation to continue generating the target trajectory; when both tracking algorithms lose the target, the threshold of the confidence screening of the tracking algorithm is reduced for feature matching to re-obtain the target position.

8. The method according to claim 2, 5 or 7, characterized in that: This method further includes step 3: for the multi-target spatio-temporal accompanying situation, when the target is lost, based on the information of the accompanying target, the position and trajectory of the lost target are continuously obtained to retrieve the position of the lost target and continue the target trajectory.

9. The method according to claim 8, characterized in that In step 3, it is assumed that the sequences composed of the time and position information of the two targets a and b are are the time for target a and b respectively, are the position coordinates of targets a and b respectively, and set the correlation degree to 0, set i = 1, and then perform the following steps: Step 31: Sort the trajectories of targets a and b in chronological order. Step 32: From the time point Start by looking for the target b in its trajectory. The time point with the shortest interval; and find the time point with the shortest interval with this time point in the trajectory of target a. If it is still The two time points found in targets b and a are considered to be adjacent time points. Then jump to step 33; Step 33: Let i = i + 1. If the updated i < n, jump to step 32 until i = n and then jump to step 34; Step 34: Calculate the position distance between the two targets at the i-th pair of adjacent time points among the obtained appearance time points in step 33. If the distance is less than a certain preset threshold or is included in the adjacent area, these two adjacent time points are considered as a pair of accompanying time points and the correlation degree is incremented by 1; Step 36: If the ratio of the correlation degree to the number of pairs of accompanying time points is greater than the threshold, it is considered that there is a spatio-temporal accompanying phenomenon between the two targets; Step 37: When target b is lost, lower the threshold of the confidence screening of the target detection algorithm at the area where target a appears, use the target detection algorithm for detection, and match the feature vectors of the detected target b; Step 38: If the feature vectors can be successfully matched, continue to detect and track target b; otherwise, it is determined that target b is lost.

10. The method according to claim 9, characterized in that This method further includes step 4: on the basis of obtaining the target trajectory, the target trajectory is completely reproduced in a visual way to generate a trajectory video of the target; among them, based on multi-channel image stitching, a trajectory video is generated, and a trajectory video is generated by the method of superimposing the target image and information on the static image. The specific steps are as follows: Step 41: Generate a complete image frame through image stitching: Step 411: Obtain the frames of the current multi-channel security videos at the same moment during the day; Step 412: Arrange the static frames in the order from left to right and from top to bottom according to the actual security area; Step 413: Use opencv to perform image stitching on the image matrix based on SIFT features by traversing in the order from left to right first and then from top to bottom; Step 414: Obtain a complete image of the security area; Step 42: Get a target trajectory sequence information {(t1,x1,y1),(t2,x2,y2),…(t n ,x n ,y n )} and arranged in time series; the coordinates (x i ,x i ) is the relative coordinate in a video screen; Step 43: According to the time and position of the target appearance, a corresponding target rectangular frame and a corresponding background rectangular frame are cut out from the picture in the video; Step 44: Use the background rectangle to perform pixel matching in the complete image to obtain the target (x i ,x i ) in the complete picture, and replace the corresponding background rectangle with the target rectangle; Step 45: Generate a target trajectory video in chronological order at a certain frame rate; if there is a time interval between the target trajectory, interpolate the target position using a cubic spline method according to the time interval and the target position.

Images