Monitoring video splicing method and device, equipment and storage medium

Abstract

The invention relates to the technical field of video stitching, and discloses a monitoring video stitching method, device and equipment and a storage medium, and the method comprises the steps: carrying out the video framing of a monitoring video collected by a monitoring terminal, and obtaining a plurality of target video frames; performing human shape detection to obtain a human shape detection result; aligning and cutting a face region in the human shape detection result to obtain a standardized face image, and extracting a face feature vector; performing motion prediction and trajectory extrapolation to obtain a multi-target human shape tracking result; inputting the multi-target human shape tracking result into a distributed computing framework for task allocation and cross-node feature matching to obtain a global human shape tracking result; and performing double-flow network processing on the plurality of target video frames according to the global human shape tracking result, calculating to obtain video stitching parameters, and performing stitching operation on the plurality of target video frames to obtain a stitched video, thereby effectively improving the quality and visual effect of the stitched video.

Description

Technical Field

[0001] The present invention relates to the technical field of video splicing, and in particular to a monitoring video splicing method, device, equipment and storage medium. Background Art

[0002] With the widespread application of video surveillance systems in public safety, traffic management, smart cities and other fields, higher requirements are placed on high-quality, large-scale video surveillance. Traditional single cameras are difficult to meet the needs of large-scale scene monitoring, and multi-camera collaborative work has become an inevitable trend. However, multi-camera systems face a series of technical challenges such as video splicing, target tracking and real-time processing.

[0003] Existing video stitching methods often ignore the continuity of dynamic targets, resulting in target fragmentation, duplication or loss in the stitching results. In addition, the massive data generated by large-scale video surveillance systems poses a severe test to real-time processing capabilities, and traditional centralized computing architectures are difficult to meet the requirements of low latency and high throughput. At the same time, target detection and tracking in complex scenes still face many difficulties, such as occlusion, lighting changes, posture diversity and other factors that affect the stability and accuracy of the algorithm. Summary of the invention

[0004] The present invention provides a monitoring video splicing method, device, equipment and storage medium, and the present invention effectively improves the quality and visual effect of the spliced ​​video.

[0005] In a first aspect, the present invention provides a method for splicing surveillance videos, the method comprising:

[0006] Perform video framing on the surveillance video collected by the surveillance terminal to obtain multiple target video frames;

[0007] Input the multiple target video frames into the improved YOLOv5 model to perform human figure detection, and obtain human figure detection results;

[0008] Aligning and cropping the face area in the human figure detection result to obtain a standardized face image, and extracting a face feature vector;

[0009] Perform motion prediction and trajectory extrapolation based on the face feature vector and the human figure detection result to obtain a multi-target human figure tracking result;

[0010] Inputting the multi-target humanoid tracking results into a distributed computing framework for task allocation and cross-node feature matching to obtain a global humanoid tracking result;

[0011] According to the global human tracking result, the multiple target video frames are processed by a dual-stream network, video stitching parameters are calculated, and a stitching operation is performed on the multiple target video frames to obtain a stitched video.

[0012] In a second aspect, the present invention provides a monitoring video splicing device, the monitoring video splicing device comprising:

[0013] A framing module is used to perform video framing on the surveillance video collected by the surveillance terminal to obtain multiple target video frames;

[0014] A detection module, used for inputting the multiple target video frames into an improved YOLOv5 model to perform human shape detection and obtain a human shape detection result;

[0015] An extraction module, used for aligning and cropping the face area in the human figure detection result to obtain a standardized face image and extract a face feature vector;

[0016] A prediction module, used for performing motion prediction and trajectory extrapolation based on the face feature vector and the human figure detection result to obtain a multi-target human figure tracking result;

[0017] A tracking module, used for inputting the multi-target humanoid tracking results into a distributed computing framework for task allocation and cross-node feature matching to obtain a global humanoid tracking result;

[0018] The calculation module is used to perform dual-stream network processing on the multiple target video frames according to the global human tracking result, calculate the video splicing parameters, and perform a splicing operation on the multiple target video frames to obtain a spliced ​​video.

[0019] The third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the above-mentioned surveillance video splicing method.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned method for splicing surveillance videos.

[0021] In the technical solution provided by the present invention, human detection is performed by an improved YOLOv5 model, and spatial attention and channel attention mechanisms are introduced to improve the accuracy and robustness of target detection in complex scenes. The improved FaceNet network with SE-ResNet-50 backbone network and self-attention mechanism is used to extract facial features, which enhances the discriminability of facial features and is conducive to subsequent target tracking and identity recognition. The Kalman filter with an adaptive noise covariance matrix is ​​used for motion prediction and trajectory extrapolation, which improves the accuracy and stability of multi-target tracking, especially the performance under short-term occlusion of the target. The introduction of a distributed computing framework and a dynamic task allocation mechanism realizes efficient processing of large-scale video data, and significantly improves the real-time performance and scalability of the system. Through the dual-stream network processing method, the spatial semantic features and temporal residual noise information are comprehensively considered, the accuracy and coherence of video splicing are improved, and the problems of breakage and repetition of dynamic targets in the splicing process are effectively solved. The present invention effectively improves the quality and visual effect of spliced ​​videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0023] Figure 1 Schematic diagram of the steps of the method for splicing surveillance videos in an embodiment of the present invention;

[0024] Figure 2 Schematic diagram of the structure of the monitoring video splicing device in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Embodiments of the present invention provide a method, device, equipment and storage medium for splicing surveillance videos. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the method for splicing surveillance videos in an embodiment of the present invention includes:

[0027] Step S1, performing video framing on the monitoring video collected by the monitoring terminal to obtain multiple target video frames;

[0028] It is understandable that the execution subject of the present invention may be a monitoring video splicing device, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0029] Specifically, the monitoring video data is collected through the monitoring terminal, including continuous video streams in different scenes. The collected video stream is sampled according to the preset frame rate, and multiple initial video frames are extracted from the video. The frame rate is set according to actual needs, such as extracting 30 frames or 60 frames per second to ensure that the extracted initial video frames can fully reflect the dynamic information in the monitoring video. After the sampling is completed, the series of initial video frames obtained may have different resolutions or quality problems. The resolution of the initial video frames is standardized one by one, and the resolution of all frames is unified to a predetermined standard resolution, such as 1920x1080 or 1280x720, so that each frame image is consistent during subsequent processing, thereby reducing the calculation deviation or processing error caused by different resolutions in the processing process. The standardized initial video frame is subjected to bilateral filtering. Bilateral filtering is a smoothing filtering method that retains the edge features of the image. By considering the information of the spatial domain and the pixel value domain at the same time, it effectively removes the noise in the image while retaining the edge details. In bilateral filtering, two key parameters are set: the radius of the spatial domain kernel function and the standard deviation of the value domain kernel function. The radius of the spatial domain kernel function is set to 5, indicating that the influence range of a pixel in the filtering process is a radius area of ​​5 pixels; and the standard deviation of the range kernel function is set to 50, indicating the smoothness of the pixel value change considered in the filtering process. After processing, multiple video frames with preliminary denoising are obtained. The video frames after preliminary denoising are subjected to edge-preserving filtering to enhance the edge features in the image. Edge-preserving filtering is a technology that can effectively enhance the edge of an image. It can highlight the edge structure in the image while smoothing the image noise. The edge-enhanced video frame is divided into 8x8 non-overlapping blocks. For each small block, its cumulative distribution function (CDF) is calculated to obtain a local CDF matrix. The cumulative distribution function can describe the distribution of pixel values ​​in an image and help enhance the local contrast of the image. Based on the local CDF matrix, the mapping function of each pixel is calculated using bilinear interpolation. Bilinear interpolation is an image processing technique that can calculate the pixel value of an unknown point on a given pixel grid, making the image processing result smoother and more refined. Through the interpolation method, the local cumulative distribution function information is mapped to the entire image to obtain a global mapping matrix that describes the mapping relationship of each pixel in the entire image. Based on the global mapping matrix, the grayscale value of each video frame after edge enhancement is remapped at the pixel level to enhance the contrast of the entire image and obtain multiple contrast-enhanced video frames. The color space of the contrast-enhanced video frames is converted. The image is converted from the RGB color space to the YUV color space. The YUV color space can better separate the brightness information and color information of the image. In the YUV color space, only the Y channel (brightness channel) is subjected to histogram equalization. Histogram equalization makes the contrast of the image more uniform and clear by adjusting the brightness distribution of the image.After processing, the YUV image is converted back to the RGB color space, retaining the original color information of the image while improving the brightness distribution of the image. The resulting multiple video frames have better contrast and can maintain the original effect in color.

[0030] Step S2: input multiple target video frames into the improved YOLOv5 model to perform human shape detection and obtain human shape detection results;

[0031] Specifically, the input video frame is transformed into multiple scales to enhance the model's ability to detect objects of different sizes. Three feature maps of different scales are generated by bilinear interpolation, corresponding to 1 / 8, 1 / 16 and 1 / 32 of the original input size. The multi-scale input feature maps are sequentially input into the backbone network of the improved YOLOv5 model. The improved backbone network is different from the original YOLOv5 model in that a spatial attention module is added after each convolution block. The spatial attention module emphasizes the important spatial regions in the image by generating a two-dimensional attention weight map and ignores irrelevant or unimportant background information. The spatial attention module calculates the weight of each pixel on the feature map and adjusts the feature values ​​at different positions so that the features of the important regions are more emphasized in the network to obtain a preliminary feature map. Channel attention is performed on the preliminary feature map containing spatial attention information. The channel attention mechanism is implemented by calculating the global average pooling value of each channel, and the pooling value represents the importance of each channel on the entire feature map. The pooling result is input into a multi-layer perceptron consisting of two fully connected layers. In the multi-layer perceptron, the number of neurons in the first layer is set to 1 / 16 of the number of channels, which effectively reduces the number of parameters and increases the nonlinear representation ability of the model. The second layer restores the number of neurons to the original number of channels to match the initial feature map. The output channel weight is mapped to between 0 and 1 through the Sigmoid activation function, which effectively controls the importance of each channel. The channel weight is multiplied element by element with the preliminary feature map to obtain an enhanced feature map containing both spatial and channel attention information. The enhanced feature map containing spatial and channel attention information is input into the feature pyramid network for processing. The feature pyramid network performs feature fusion through a top-down path and lateral connections. The top-down path uses 1x1 convolution to adjust the number of channels, and each layer performs a 2-fold upsampling operation, so that the lower resolution feature map is gradually upsampled to a higher resolution. At the same time, the lateral connection uses 1x1 convolution to process the same-layer features, so that feature information from different levels can be fused. The design of the feature pyramid network enables the network to use feature maps of different scales at the same time, so that the model has better performance when detecting human-shaped targets of different sizes. Through feature fusion, a multi-scale fused feature map is obtained. A 3x3 convolution kernel is applied to the multi-scale fused feature map for feature extraction, effectively capturing local spatial information and extracting richer local features without changing the size of the feature map. A 1x1 convolution kernel is used to compress the number of channels to the preset number of anchor boxes multiplied by (5 + number of categories). 5 represents the 4 coordinate values ​​and 1 confidence value of the bounding box, and the number of categories represents the different possible types of human detection, such as target objects such as people and cars. After processing, the original prediction result contains data such as the location information, confidence, and category probability of each anchor box.The original prediction result is decoded, and the predicted bounding box coordinates are converted from the offset relative to the grid to the absolute coordinates of the image. The predicted relative coordinates are multiplied by the step size of the feature map and added to the coordinates of the upper left corner of the corresponding grid. The relative position of the anchor box is converted to the absolute position on the image to accurately locate the position of the target. At the same time, the Sigmoid function is applied to process the confidence and category probability, mapping them between 0 and 1. The prediction results obtained after decoding clearly reflect the absolute position and category probability of the target. Based on the decoded prediction results, the non-maximum suppression algorithm is used to filter the overlapping detection boxes. The non-maximum suppression algorithm arranges all detection boxes in descending order according to the confidence, and prioritizes the detection results with the highest confidence. For each detection box, compare its overlap with other detection boxes one by one, and judge by calculating the intersection and union ratio of the overlapping boxes. When the intersection-over-union ratio is greater than the preset threshold (set to 0.5), only the box with the highest confidence is retained, and all other overlapping detection boxes are deleted, thereby effectively removing duplicate detections or overlapping detections, avoiding false detections and missed detections caused by the overlap of multiple boxes, and finally obtaining the human detection result.

[0032] Step S3, aligning and cropping the face area in the human shape detection result to obtain a standardized face image, and extracting a face feature vector;

[0033] Specifically, the MTCNN algorithm is applied to the face area in the human shape detection result for key point detection. MTCNN is a multi-task learning model that can detect faces and locate the key points of faces at the same time. The MTCNN algorithm consists of three cascade networks, namely P-Net (Proposal Network), R-Net (Refine Network), and O-Net (Output Network). During the detection process, P-Net generates candidate face frames, R-Net screens and refines the candidate frames, and O-Net further adjusts and outputs the final face bounding box and five key point coordinates. Through the MTCNN algorithm, the coordinates of the left eye center, right eye center, nose tip, left mouth corner and right mouth corner in each face image are obtained. Based on the coordinates of the five key points, the affine transformation matrix is ​​calculated. Affine transformation is a geometric transformation that can translate, rotate, scale and shear the image to transform the face image to a standard position. The face is aligned. The face image is adjusted to a predetermined standard position through affine transformation, so that the centers of the two eyes are located on the horizontal line, and the relative positions of the eyes and mouth conform to the regular distribution of face geometry. The calculation of the affine transformation matrix is ​​based on the relationship between the coordinates of the five key points and the coordinates of the key points in the standard position, and is solved by the least squares method. The original face image is multiplied by the affine transformation matrix to obtain the aligned face image. The aligned face image is cropped and scaled. The face image is cropped to only contain the face area, and the cropped face image is scaled to a standard size of 112x112 pixels. Standardized size can ensure that different face images have the same scale when input into the neural network, avoiding the problem of poor feature extraction due to inconsistent face sizes. The pixel values ​​are normalized and scaled to the [-1,1] interval to eliminate the influence of external factors such as lighting and contrast on the image, improve the robustness and stability of the model, and obtain a standardized face image. The standardized face image is input into the SE-ResNet-50 backbone network for feature extraction. SE-ResNet-50 is a deep convolutional neural network that adds a Squeeze-and-Excitation (SE) module to the traditional ResNet-50 network. ResNet-50 contains 50 convolutional layers. By introducing a residual structure, it effectively alleviates the problem of gradient vanishing in deep networks, allowing the network to better learn deeper features. The introduction of the SE module improves the network's expressiveness, compresses the features of each convolutional layer in space through global average pooling, and then learns the importance weight of each channel through two fully connected layers.The number of neurons in the first fully connected layer is 1 / 16 of the number of channels, which is used to reduce the computational complexity and introduce nonlinear relationships; the second layer restores the number of neurons to the original number of channels, and maps the weights to the [0,1] interval through the Sigmoid function, thereby assigning a weight value to each channel. The self-attention mechanism is applied to the face feature map. By calculating the correlation between each position in the feature map and all other positions, an attention weight matrix is ​​generated to capture the long-range dependencies between different regions in the image. The self-attention mechanism calculates the correlation between each position in the face feature map and all other positions to obtain the attention weight matrix, and then multiplies the matrix with the original feature map to obtain the enhanced feature map. The enhanced feature map is input into the global average pooling layer for processing. Global average pooling reduces the spatial dimension of the enhanced feature map to 1x1, retaining only the channel dimension. The spatial information of each channel is compressed into a global information, eliminating the influence of spatial position on the feature, while reducing the number of parameters and avoiding overfitting. After global average pooling, a 2048-dimensional feature vector is obtained. The 2048-dimensional feature vector is input into the fully connected layer and mapped to a 512-dimensional space to obtain the face feature vector.

[0034] Step S4: performing motion prediction and trajectory extrapolation based on the face feature vector and the human shape detection result to obtain a multi-target human shape tracking result;

[0035] Specifically, a state vector is established for each target in the human detection result, which includes information such as the target's center coordinates, width, height, speed and acceleration, and describes the target's position, size and motion state in space. At the same time, the state vector and covariance matrix of the Kalman filter are initialized. The Kalman filter is a recursive filtering algorithm that estimates the system state by prediction and update, and can provide relatively accurate state estimation results in the presence of noise. The state vector at the time of initialization is set by the target position and size information extracted from the human detection result, and the covariance matrix is ​​used to describe the uncertainty of the state vector. After initialization, the initial state estimate is obtained. Based on the initial state estimate, the state transfer matrix is ​​used to predict the next frame position of the target. The state transfer matrix is ​​a linear model that describes how the system transitions from the current state to the next state, and considers factors such as the target's speed and acceleration to make the prediction more accurate. The target state is predicted by the state transfer matrix to obtain a priori state estimate. The priori state estimate is an estimated result of the position, size and motion state of the target that may appear in the next frame based on the information of the current frame. According to the prior state estimation and the human detection result of the current frame, the matching cost matrix is ​​calculated. The matching cost matrix is ​​a two-dimensional matrix, in which each element represents the matching cost between a predicted target and a detected target. The matching cost consists of two parts: one is the difference in position, that is, the Euclidean distance of the target position, and the other is the difference in size, that is, the difference in target width and height. In this way, each predicted target is matched with the detected target in the current frame. The Hungarian algorithm is used for data association. The Hungarian algorithm is a method for solving the bipartite graph matching problem, which can find a global optimal matching solution in the many-to-many case. Through the Hungarian algorithm, the matching pairs of the target and the detection result are obtained. For each target in the matching pair, the cosine similarity between the facial feature vectors is calculated. Cosine similarity is an indicator to measure the similarity between two vectors. The value range is between [-1,1]. The closer to 1, the more similar the two vectors are. Cosine similarity effectively measures whether the faces in two different detection frames belong to the same person. Combining cosine similarity with position information can more accurately determine the identity of the target and obtain the target identity confirmation result. Based on the target identity confirmation result and the current observation value, the state estimate of the Kalman filter is updated to obtain the posterior state estimate. The observation noise and process noise in the posterior state estimate are adaptively adjusted. The observation noise and process noise describe the uncertainty in the measurement process and the system motion process respectively. The adaptive noise covariance matrix is ​​obtained by calculating the sample covariance of the observation residual sequence of the most recent A frame and updating the noise covariance matrix using the exponential weighted average method.The sample covariance can reflect the fluctuation of the observation residual, while the exponential weighted average method can respond to the latest change trend faster while retaining historical information, so that the Kalman filter can better adapt to the changes in the actual scene. The adaptive noise covariance matrix is ​​substituted into the prediction and update equations of the Kalman filter to recalculate the Kalman gain and state estimation. The Kalman gain describes the weight distribution of the observation information in the state update. The adaptively adjusted Kalman gain can better balance the relationship between prediction and observation, and obtain a more optimized state estimation result. For unmatched detection results, a new trajectory is initialized. For each new detection result, a new trajectory is initialized according to its position, speed and other information, and it is tracked. At the same time, for the unmatched trajectories of consecutive A frames, the long short-term memory network (LSTM) is used for trajectory extrapolation. LSTM predicts the position in the future A frame based on the historical motion trajectory of the target. Through trajectory extrapolation, the blank areas in the trajectory are effectively filled, and a more continuous and complete multi-target humanoid tracking result is obtained.

[0036] Step S5: input the multi-target humanoid tracking results into the distributed computing framework for task allocation and cross-node feature matching to obtain a global humanoid tracking result;

[0037] Specifically, the multi-target humanoid tracking results are segmented in time and space, and the continuous B-frame tracking results are divided into independent task units, which are convenient for parallel processing and scheduling management in the distributed computing framework. Each task unit represents the tracking data and its corresponding spatial information within a period of time (i.e., B-frame), so that it can be better allocated to different computing nodes for processing. The future load of each node is predicted based on the CPU usage, memory occupancy, and network bandwidth of each computing node. The resource usage of each node is analyzed and modeled through historical data and trend analysis models to obtain the node load prediction value. The load prediction value reflects the possible resource occupancy of each node at a future moment. The task units are preliminarily allocated based on the node load prediction value and the complexity of the task unit. The complexity of the task unit is determined by factors such as the number of tracking targets, the complexity of the motion trajectory, and the amount of data. For task units with higher complexity, try to allocate them to computing nodes with lower load and sufficient resources to ensure their processing efficiency and stability. The preliminarily allocated task scheme is load balanced and optimized. Through task migration and splitting operations, the load difference of each node is controlled within the preset threshold. Task migration refers to transferring task units on certain nodes to other nodes with lighter loads, while task splitting refers to decomposing task units with higher complexity into multiple smaller task units so that they can be processed in parallel on different nodes. Through optimization operations, a more reasonable and balanced task allocation scheme is obtained to ensure load balancing of all nodes, thereby improving the processing efficiency and stability of the entire distributed computing framework. The optimized task allocation scheme is sent to each computing node, and a local feature index is built on each node. Local feature index refers to using efficient data structures such as hash tables on each node to store facial feature vectors and corresponding trajectory IDs. Hash tables can quickly access feature vectors, thereby improving matching efficiency in subsequent feature matching processes. The local feature index structure on each node ensures fast retrieval and comparison of facial features within the node. A global feature index is built on the central control node. The global feature index uses a hierarchical tree structure to organize the feature information of each node, and each leaf node corresponds to a local index of a computing node. This structure can effectively manage and organize feature data in the entire distributed system, making it easy to quickly locate the target node when performing cross-node matching. For cross-node face matching requests, a rough match is first performed in the global feature index. The global feature index is used to quickly filter out the K leaf nodes that are most likely to contain the target feature. The rough match uses hash mapping, KD tree and other methods to quickly locate the node range where the target feature is located. An exact match is performed in the local index of the K leaf nodes. The exact match calculates the similarity between the target feature and all feature vectors in the local feature index to find the best match and obtain the cross-node matching result.Based on the cross-node matching results, the trajectory segments on different nodes are associated and fused to eliminate duplicate trajectory records. The trajectory fusion process includes comparing the start and end times of the trajectory segments, merging the overlapping trajectory segments, and removing duplicate trajectory data. The trajectory ID is processed for global consistency, and a unique trajectory ID is assigned to each target globally to ensure that the same target has one and only one unique identifier in the entire system, and finally obtain a global humanoid tracking result.

[0038] Step S6: perform dual-stream network processing on multiple target video frames according to the global human tracking result, calculate the video stitching parameters, and perform a stitching operation on the multiple target video frames to obtain a stitched video.

[0039] Specifically, multiple target video frames are reasonably grouped according to the global human tracking results. The same human target that is continuously tracked is divided into the same video clip, and the frame sequence in each video clip is regarded as an independent analysis unit. The entire video stream is divided into multiple video clips, each of which contains E frame images. The ResNext-101 network is applied to each frame of the multiple segmented video clips for feature extraction. ResNext-101 is a deep convolutional neural network that contains 32 residual blocks with a base of 4. These residual blocks can extract multi-level and rich image features. Each residual block outputs a 256-dimensional feature representation, which is stacked and reduced to a 2048-dimensional spatial feature vector through a global average pooling layer. All frames of each video clip are represented as an E×2048-dimensional spatial feature sequence, where E represents the number of frames contained in the clip. The self-attention mechanism is applied to the spatial feature sequence, and the correlation between video frames is measured by calculating the E×E attention weight matrix. Each element in the attention weight matrix represents the correlation between two frames in the sequence. After multiplying the attention weight matrix with the original spatial feature sequence, a weighted spatial feature sequence is obtained, making the originally relatively independent frame sequence more coherent in spatial features. At the same time, the difference image is calculated between each video frame in the video clip and its previous frame, and the inter-frame difference method is used to obtain E-1 residual noise images. Inter-frame difference can effectively reflect the motion changes and differences between two consecutive frames, thereby extracting dynamic information in the video. The residual noise images are stacked into a residual noise sequence of E-1×H×W×3, where H and W are the height and width of the image, respectively. The residual noise sequence is used to capture the motion characteristics and change patterns between frames in the video. The residual noise sequence is input into a 3D convolutional neural network. The 3D convolutional neural network contains 4 3D convolutional layers, each with a convolution kernel size of 3×3×3, a step size of 1×2×2, and a number of channels of 64, 128, 256, and 512, respectively. The 3D convolutional network can extract features in both spatial and temporal dimensions and capture dynamic change information in the video. After being processed by four 3D convolutional layers, the feature map is reduced to a 512-dimensional temporal feature vector through 3D global average pooling. Each video clip is represented by an E×512-dimensional temporal feature sequence. The temporal feature sequence describes the dynamic pattern of the video clip that changes over time. The weighted spatial feature sequence and temporal feature sequence are bilinearly pooled to obtain an E×d-dimensional fused feature sequence, where d is the dimension of the fused feature. Based on the fused feature sequence, a two-layer fully connected neural network is used to regress and calculate the relative position offset, rotation angle, and scaling factor between adjacent video clips. The first layer of the fully connected network is used to perform a preliminary linear transformation on the fused features and extract features that are helpful for the regression of splicing parameters; the second layer of the fully connected network maps the features to specific splicing parameters.Through the trained regression model, the relative position relationship between adjacent video clips is calculated to obtain the video stitching parameters, including information such as translation, rotation and scaling between video clips. According to the video stitching parameters, multiple video clips are affine transformed. Affine transformation is a geometric transformation method that can simultaneously translate, rotate and scale, aligning different video clips into a global coordinate system. The transformed images are stitched according to their relative positions, connecting all clips into a complete stitched video, and obtaining a spatiotemporally coherent stitched video with good visual effects.

[0040] In the embodiment of the present invention, human detection is performed by an improved YOLOv5 model, and spatial attention and channel attention mechanisms are introduced to improve the accuracy and robustness of target detection in complex scenes. The improved FaceNet network with SE-ResNet-50 backbone network and self-attention mechanism is used to extract facial features, which enhances the discriminability of facial features and is conducive to subsequent target tracking and identity recognition. The Kalman filter with an adaptive noise covariance matrix is ​​used for motion prediction and trajectory extrapolation, which improves the accuracy and stability of multi-target tracking, especially the performance under short-term occlusion of the target. The introduction of a distributed computing framework and a dynamic task allocation mechanism realizes efficient processing of large-scale video data, significantly improving the real-time performance and scalability of the system. Through the dual-stream network processing method, the spatial semantic features and temporal residual noise information are comprehensively considered, the accuracy and coherence of video splicing are improved, and the problems of breakage and repetition of dynamic targets in the splicing process are effectively solved. The present invention effectively improves the quality and visual effect of spliced ​​video.

[0041] In a specific embodiment, the process of executing step S1 may specifically include the following steps:

[0042] The monitoring video is collected through the monitoring terminal, and the monitoring video is sampled according to a preset frame rate to obtain multiple initial video frames, and each frame in the multiple initial video frames is subjected to resolution standardization processing to obtain multiple standardized video frames; the multiple standardized video frames are subjected to bilateral filtering processing, wherein the radius of the spatial domain kernel function is set to 5, and the standard deviation of the value domain kernel function is set to 50, to obtain multiple video frames after preliminary denoising; the multiple video frames after preliminary denoising are subjected to edge-preserving filtering to obtain multiple edge-enhanced video frames, and the edge-enhanced video frames are divided into 8x8 non-overlapping blocks, and the cumulative distribution function is calculated for each block to obtain a local CDF matrix; based on the local CDF matrix, the mapping function of each pixel is calculated using a bilinear interpolation method to obtain a global mapping matrix, and the edge-enhanced video frames are subjected to pixel-level grayscale value remapping according to the global mapping matrix to obtain multiple contrast-enhanced video frames; the multiple contrast-enhanced video frames are subjected to color space conversion, the RGB color space is converted to the YUV color space, histogram equalization is applied only to the Y channel, and then the processed YUV image is converted back to the RGB color space to obtain multiple target video frames.

[0043] Specifically, the collected original surveillance video is sampled according to a preset frame rate to obtain multiple initial video frames. Assume that the frame rate of the original surveillance video is R frames per second, and the preset frame rate is R s frames per second, downsample the video. Extract a video frame from the frame to obtain multiple initial video frames. Perform resolution standardization on the initial video frames. Assume that the original resolution of each video frame is H orig ×W orig′ The target resolution is H target ×W target The purpose of resolution normalization is to adjust all initial video frames to the same resolution to ensure that frames have consistent spatial characteristics during subsequent processing. The normalization process is implemented using bilinear interpolation, and the interpolation formula is:

[0044]

[0045] Among them, I target (x, y) represents the pixel value with coordinates (x, y) in the normalized frame, I orig (x′, y′) represents the pixel value at the corresponding position in the original frame. W orig and H orig are the width and height of the original frame, W target and H targetis the width and height of the target frame. The pixel values ​​in the original frame are mapped to the target resolution frame through this formula to obtain multiple video frames with standardized resolution. Bilateral filtering is performed on the standardized video frames. Bilateral filtering is a filtering method that can remove noise while retaining image edge details. The image is smoothed by considering the weights of both the spatial domain and the value domain. The mathematical expression of bilateral filtering is:

[0046]

[0047] Among them, I filtered (x, y) represents the pixel value at the coordinate (x, y) after bilateral filtering, I norm (x,y represents the pixel value at (x,y) in the frame after resolution normalization, Ω is the filter window range, usually a square area. d The standard deviation of the spatial domain kernel function determines the size of the filter window and is set to 5; while σ r is the standard deviation of the range kernel function, which controls the weight of color similarity and is set to 50. W(x,y) is a normalization factor used to ensure that the pixel values ​​of the filtering results are within the correct range. By calculating with this formula, it is possible to remove noise while maintaining the integrity of the edge area of ​​the image, and obtain multiple video frames after preliminary denoising. The video frames after preliminary denoising are processed by edge-preserving filtering to enhance the edge features in the image. Edge-preserving filtering suppresses the filtering effect in flat areas by assigning higher weights to areas with large gradient changes. The edge-enhanced video frame is divided into 8x8 non-overlapping blocks, and the cumulative distribution function (CDF) of each block is calculated to capture the pixel distribution characteristics of the local area in the image. The cumulative distribution function is defined as:

[0048]

[0049] Among them, CDF(x,y,k) represents the cumulative probability of the pixel point with the pixel value less than or equal to k at the coordinate (x,y) in block B. is the indicator function, when I enhanced When (x′, y′)≤k, the indicator function is 1, otherwise it is 0. By calculating the cumulative distribution function of each non-overlapping block, the local CDF matrix can be obtained to describe the distribution of pixel values ​​in each block. Based on the local CDF matrix, the mapping function of each pixel is calculated using bilinear interpolation to generate a global mapping matrix. The bilinear interpolation method can smoothly extend the local CDF information to the entire image, so that each pixel value can be adjusted according to the cumulative distribution characteristics of the block in which it is located. The formula for calculating the mapping function of each pixel is:

[0050] I mapped (x,y)=I enhanced(x,y)·(1+α·(CDF(x,y,I enhanced (x,y))-0.5));

[0051] Among them, I mapped (x, y) is the pixel value after mapping, α is the adjustment coefficient used to control the mapping intensity, CDF(x, y, I enhanced (x,y)) is the cumulative probability of the original pixel value in its local block. The grayscale value of each pixel is remapped by this formula, which effectively enhances the contrast of the image and obtains multiple contrast-enhanced video frames. The contrast-enhanced video frames are converted into color space, and the RGB color space is converted into YUV color space. The three channels of the RGB color space represent the red, green, and blue color components respectively, while in the YUV color space, the Y channel represents brightness information, and the U and V channels represent chrominance information. The color space conversion formula is:

[0052]

[0053] Among them, Y, U and V are brightness and chrominance components respectively, and R, G and B are red, green and blue components respectively. After conversion to YUV color space, histogram equalization is applied only to the Y channel (brightness channel) to even out the brightness distribution and enhance the brightness contrast of the image. Histogram equalization stretches the brightness value distribution of the original image to the entire brightness range, making the brightness distribution more uniform, thereby improving the visual effect of the image. After processing, the YUV image is converted back to RGB color space to restore the color information and obtain multiple target video frames.

[0054] In a specific embodiment, the process of executing step S2 may specifically include the following steps:

[0055] Multi-scale transformation is performed on multiple target video frames, and three feature maps of different scales are generated by bilinear interpolation method, which are 1 / 8, 1 / 16 and 1 / 32 of the original input size, respectively, to obtain multi-scale input feature maps; the multi-scale input feature maps are sequentially input into the backbone network of the improved YOLOv5 model, and the backbone network adds a spatial attention module after each convolution block. The spatial attention module emphasizes important spatial areas by generating a two-dimensional attention weight map to obtain a preliminary feature map containing spatial attention information; channel attention is performed on the preliminary feature map, firstly the global average pooling value of each channel is calculated, and then the pooling result is input into a multi-layer perceptron composed of two layers of fully connected layers, where the number of neurons in the first layer is 1 / 16 of the number of channels, and the second layer is restored to the original number of channels. Finally, the channel weight is obtained by the Sigmoid activation function, and the channel weight is multiplied element by element with the preliminary feature map to obtain an enhanced feature map containing both spatial and channel attention information; the enhanced feature map is input into the feature pyramid network, and feature fusion is performed through a top-down path and a lateral connection, where the top-down path uses 1x 1 convolution is used to adjust the number of channels and upsample by 2 times, and the horizontal connection uses 1x1 convolution to process the features of the same layer, and finally a multi-scale fusion feature map is obtained; a 3x3 convolution kernel is applied to the multi-scale fusion feature map for feature extraction, keeping the feature map size unchanged, and then a 1x1 convolution kernel is used to compress the number of channels to the preset number of anchor boxes multiplied by (5+number of categories), where 5 represents the 4 coordinate values ​​of the bounding box and 1 confidence value, to obtain the original prediction result; the original prediction result is decoded, and the predicted bounding box coordinates are converted from the offset relative to the grid to the absolute value of the image. Coordinates are calculated by multiplying the predicted relative coordinates by the step size of the feature map and adding the coordinates of the upper left corner of the corresponding grid. At the same time, the Sigmoid function is applied to process the confidence and category probability, and they are mapped between 0 and 1 to obtain the decoded prediction results. Based on the decoded prediction results, the non-maximum suppression algorithm is used to filter the overlapping detection frames. First, all detection frames are arranged in descending order by confidence, and then compared one by one to calculate the intersection-and-union ratio of the overlapping frames. When the intersection-and-union ratio is greater than the preset threshold of 0.5, the frame with the highest confidence is retained and other overlapping frames are deleted to obtain the human detection result.

[0056] Specifically, the target video frame is subjected to multi-scale transformation. Assuming that the size of the original input image is H×W, the bilinear interpolation method is used to generate feature maps of 1 / 8, 1 / 16 and 1 / 32 of the original size, respectively, to obtain feature maps of three different scales. For the generation of each feature map, the formula of the bilinear interpolation method is:

[0057]

[0058] Among them, I scaled (x,y) indicates the scale after transformation is W scaled ×Hscaled The pixel value of the coordinate (x, y) in the feature map of orig (x′, y′) represents the pixel value at coordinate (x′, y′) in the original input image, W scaled and H scaled Represent the width and height of the scaled image respectively. Through this formula, the original input image is scaled to different scales in order to capture different features of the target at different resolutions. The three feature maps of different scales correspond to 1 / 8, 1 / 16 and 1 / 32 of the original input size, respectively. The purpose is to extract multi-level features of the target at different scales to cope with complex scenes such as target size and perspective changes. The generated multi-scale input feature maps are input into the backbone network of the improved YOLOv5 model in sequence. The backbone network adds a spatial attention module after each convolution block. The spatial attention module emphasizes important spatial areas in the image by generating a two-dimensional attention weight map. The spatial attention module generates weights according to the importance of different positions of the input feature map. The calculation formula is:

[0059]

[0060] Among them, Attention spatial (x,y) represents the weight of the spatial attention weight map at (x,y), Conv 3×3 represents a 3x3 convolution operation, I input is the input feature map, and σ is the Sigmoid activation function. This module captures the local information of the input feature map through convolution operations, and maps the weights to between 0 and 1 through the Sigmoid activation function to adjust the importance of each position. Channel attention processing is performed on the preliminary feature map containing spatial attention information, and different weights are assigned to each channel according to its importance. The average value of each channel is calculated by global average pooling, and the pooling formula is:

[0061]

[0062] Among them, Pool avg (c) represents the global average pooling value of channel c, I input (x, y, c) is the pixel value of the input feature map in channel c and coordinate (x, y). The pooling result is input into a multi-layer perceptron consisting of two fully connected layers. The number of neurons in the first layer is 1 / 16 of the original number of channels, which effectively reduces the number of parameters and increases the nonlinear expression ability of the model; the second layer restores the number of neurons to the original number of channels. The channel weight is calculated using the Sigmoid activation function:

[0063] Attention channel (c) = σ9MLP(Pool avg (c)));

[0064] Among them, Attention channel (c) is the weight of channel c, and MLP represents the multi-layer perceptron operation. The weight is element-wise multiplied with the preliminary feature map to obtain an enhanced feature map that contains both spatial and channel attention information. The enhanced feature map is input into the feature pyramid network for feature fusion. The feature pyramid network fuses feature information of different scales through a top-down path and lateral connections. In the top-down path, 1x1 convolution is used to adjust the number of channels and upsample by 2; in the lateral connection, 1x1 convolution is used to process the feature map from the backbone network. The role of the feature pyramid network is to fuse high-resolution detail information with low-resolution semantic information to obtain a multi-scale fused feature map. A 3x3 convolution kernel is applied to the multi-scale fused feature map for feature extraction to keep the spatial size of the feature map unchanged. The 3x3 convolution kernel can effectively capture local spatial information and extract richer features without changing the size of the feature map. A 1x1 convolution kernel is used to compress the number of channels to the preset number of anchor boxes multiplied by 5 + the number of categories, where 5 represents the 4 coordinate values ​​and 1 confidence value of the bounding box. The four coordinate values ​​of the bounding box represent the center coordinates, width, and height of the target box, and the confidence value represents the model's confidence in the existence of the target. After the operation, the original prediction result is obtained. The original prediction result is decoded, and the predicted bounding box coordinates are converted from the offset relative to the grid to the absolute coordinates of the image. The decoding calculation formula is:

[0065]

[0066] Where (x, y) is the absolute coordinate of the center of the bounding box, w and h are the width and height of the bounding box respectively, and (t x ,t y ) is the relative offset predicted by the model, (c x ,c y ) is the coordinate of the upper left corner of the grid, s is the step size of the feature map, (p w ,p h ) is the width and height of the anchor box σ(·) is the Sigmoid function. Through this formula, the relative coordinates are converted to absolute coordinates in the image, and through and Convert the width and height to real values, and apply the Sigmoid function to map the confidence and category probability to between 0 and 1 to obtain the decoded prediction results. Based on the decoded prediction results, use the non-maximum suppression algorithm to filter overlapping detection boxes. Non-maximum suppression arranges all detection boxes in descending order of confidence, then compares the overlaps between them one by one and calculates the intersection over union (IoU). When the IoU is greater than the preset threshold (for example, 0.5), keep the box with the highest confidence and delete other overlapping boxes. This effectively removes duplicate detections and obtains the final human detection result.

[0067] In a specific embodiment, the process of executing step S3 may specifically include the following steps:

[0068] A facial key point detection algorithm based on a multi-task cascade convolutional network is applied to the face area in the human figure detection result to locate 5 key points, including the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth and the right corner of the mouth, and obtain the coordinates of the facial key points; based on the coordinates of the facial key points, the affine transformation matrix is ​​calculated to transform the face image to the standard position so that the centers of the two eyes are located on the horizontal line and at a fixed position in the image to obtain the aligned face image; the aligned face image is cropped and scaled to adjust the aligned face image to a size of 112x112 pixels, and the pixel values ​​are normalized to scale the pixel values ​​to the interval [-1,1] to obtain a standardized face image; the standardized face image is input into the SE-ResNet-50 backbone network, and the SE-ResNet- The 50 backbone network contains 50 convolutional layers. A Squeeze-and-Excitation module is added after each residual block. The correlation between channels is learned through global average pooling and two fully connected layers to obtain a face feature map. The self-attention mechanism is applied to the face feature map to calculate the correlation between each position in the face feature map and all other positions, generate an attention weight matrix, and multiply the attention weight matrix with the face feature map to obtain an enhanced feature map. The enhanced feature map is input into the global average pooling layer, the spatial dimension of the enhanced feature map is reduced to 1x1, the channel dimension is retained, and a 2048-dimensional feature vector is obtained. A fully connected layer is applied to the 2048-dimensional feature vector to map the 2048-dimensional feature vector to a 512-dimensional space to obtain a face feature vector.

[0069] Specifically, the face region in the human shape detection result is cropped and preprocessed. Assume that the bounding box coordinates (x1, y1, x2, y2) of the face region are obtained from the human shape detection result, where (x1, y1) represents the upper left corner coordinate of the face bounding box, and (x2, y2) represents the lower right corner coordinate. The face region is cropped from the original image to obtain a sub-image containing the face. faceIn order to avoid cropping due to uneven boundaries or part of the head area, the bounding box is appropriately enlarged during cropping, for example, the width and height are increased by 10% respectively. face The input is fed into the face key point detection model based on the multi-task cascade convolutional network. The multi-task cascade convolutional network is a model with a three-level network structure, namely the proposal network, the refinement network and the output network, which successively perform step-by-step refinement detection and key point location on the face area. The proposal network performs a preliminary scan on the input image to generate a face candidate box and rough coordinates of five key points; the refinement network further modifies and screens the candidate box, and refines the coordinates of the five key points; the output network performs precise regression of the final face bounding box and key point coordinates on the screened candidate. Through the step-by-step refinement of these three-level networks, the multi-task cascade convolutional network can accurately locate the five key points of the face in complex scenes: the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth and the right corner of the mouth. The face image is aligned to transform the face image to a standardized position and posture. The alignment process is achieved by calculating the affine transformation matrix. Affine transformation is a linear transformation that maintains the parallelism of lines, including geometric operations such as translation, rotation, scaling and shearing. In order to calculate the affine transformation matrix, a standard key point template is defined. Assume that the coordinates of the center of the two eyes in the standard template are (x le ,y le ) and (x re ,y re ), the coordinates of the nose tip, left corner of the mouth, and right corner of the mouth are (x′ nose ,y′ nose )、(x′ lm ,y′ lm ) and (x′ rm ,y′ rm ). According to the actual key point coordinates in the target image and the key point coordinates in the standard template, the affine transformation matrix M is calculated. The calculation of the affine transformation matrix uses the least squares method to solve the linear equations. Assume that the key point coordinates in the target image are (x i ,y i ), the corresponding coordinates in the standard template are (x′ i ,y′ i ), the transformation matrix M can be expressed as:

[0070]

[0071] Among them, m 11 ,m 12 ,m 13 ,m 21 ,m 22 ,m 23is the parameter of the affine transformation matrix. By substituting the actual coordinates and the standard coordinates into the equations and solving them using the least squares method, we get the affine transformation matrix M. face Perform an affine transformation to align the face image to the horizontal position of the center of the two eyes in the standard template. Crop and scale the aligned face image. Crop it to only the part containing the face area and resize it to the standard size of 112x112 pixels. Normalize the pixel values ​​and scale them to the [-1,1] interval to eliminate the influence of lighting conditions on the image and improve the numerical stability of the model during training and inference. The formula for normalizing pixel values ​​is:

[0072]

[0073] Among them, I norm (x, y) is the normalized pixel value, I resized (x, y) is the pixel value of the resized standardized face image at (x, y). Through the normalization formula, the image with the original pixel value in the range of [0, 255] is mapped to the interval of "-1, 1" to obtain a standardized face image. The standardized face image is input into the SE-ResNet-50 backbone network for feature extraction. SE-ResNet-50 is an improved residual neural network structure, which adds a Squeeze-and-Excitation (SE) module after each residual block. The SE module learns the correlation between different channels by modeling the global information of each channel. The implementation of the SE module is divided into two steps: Squeeze and Excitation. The Squeeze operation compresses the spatial information of each channel into a scalar through global average pooling, that is:

[0074]

[0075] Among them, z c represents the pooling result of the cth channel, I feat (i,j,c) is the pixel value of the cth channel at position i,j) in the feature map, H and W are the height and width of the feature map respectively. The Excitation operation learns the importance weight of each channel through two fully connected layers:

[0076] s c =σ(W2·ReLU(W1·z c ));

[0077] Among them, s crepresents the weight of the cth channel, W1 and W2 are the weight matrices of the fully connected layer, and σ represents the Sigmoid activation function. The enhanced feature map is obtained by multiplying the weights with the original feature map channel by channel. The SE module can improve the model's attention allocation between channels, thereby enhancing the feature expression ability. The self-attention mechanism is applied to the face feature map. By calculating the correlation between each position in the feature map and all other positions, an attention weight matrix is ​​generated to obtain an enhanced feature map containing global context information. The enhanced feature map is input into the global average pooling layer to reduce its spatial dimension to 1x1 and only retain the channel dimension. Through the global average pooling operation, the spatial information of the feature map is compressed into global information while retaining the feature information in the channel dimension. Assuming that after global average pooling, a 2048-dimensional feature vector is obtained, a fully connected layer is applied to it to map the 2048-dimensional feature vector to a 512-dimensional space to obtain a face feature vector.

[0078] In a specific embodiment, the process of executing step S4 may specifically include the following steps:

[0079] For each target in the human detection result, a state vector is established, including the target's center coordinates, width, height, speed and acceleration, and the state vector and covariance matrix of the Kalman filter are initialized to obtain the initial state estimate; based on the initial state estimate, the state transfer matrix is ​​used to predict the next frame position of the target to obtain the prior state estimate, and the matching cost matrix is ​​calculated based on the prior state estimate and the human detection result of the current frame, and the Hungarian algorithm is used for data association to obtain a matching pair of the target and the detection result; for each target in the matching pair, the cosine similarity between the facial feature vectors is calculated, and the position information is combined to obtain the target identity confirmation result, and based on the target identity confirmation result and the current The observation value is used to update the state estimate of the Kalman filter to obtain the posterior state estimate; the observation noise and process noise in the posterior state estimate are adaptively adjusted, and the sample covariance of the observation residual sequence of the most recent A frame is calculated, and the noise covariance matrix is ​​updated using the exponential weighted average method to obtain an adaptive noise covariance matrix; the adaptive noise covariance matrix is ​​substituted into the prediction and update equations of the Kalman filter, the Kalman gain and state estimate are recalculated to obtain the optimized state estimate; for unmatched detection results, new trajectories are initialized; for unmatched trajectories of consecutive A frames, the long short-term memory network is used to perform trajectory extrapolation, and the position of future A frames is predicted to obtain multi-target humanoid tracking results.

[0080] Specifically, extract the current detection information of each target. Assume that the detection result of a target in the tth frame includes the center coordinates (x t ,y t ), width w t and height h t, its state vector is expressed as:

[0081]

[0082] in, and They represent the velocity components of the target center coordinates, and Represent the rate of change of the target width and height respectively. The initial state estimate of the Kalman filter includes the state vector x t and the covariance matrix P t The state vector describes the motion state of the target and the covariance matrix is ​​used to describe the uncertainty of the state. The initial covariance matrix P0 is set to a diagonal matrix whose diagonal elements represent the initial uncertainty of each state component:

[0083]

[0084] in, represent the uncertainty of position and size respectively, and Represent the uncertainty of the velocity component respectively. The motion state of each target is modeled by initialization. Based on the initial state estimation, the state transfer matrix is ​​used to predict the next frame position of the target. The state transfer model of the Kalman filter is expressed as:

[0085] x t+1 =Fx t +Bu t +w t ;

[0086] Where F is the state transfer matrix, u t is the control input vector, B is the control input matrix, w t is the process noise, usually assuming that w t ~N(0,Q t ). In the absence of control input, the control input matrix B can be ignored. The state transfer matrix F defines the dynamic evolution law of the state vector. For example, in a simple uniform motion model, the state transfer matrix is ​​set to:

[0087]

[0088] Among them, Δt represents the time interval between two adjacent frames. The prior state estimate of the next frame is obtained through the state transfer matrix:

[0089]

[0090] At the same time, the prediction covariance matrix is:

[0091]

[0092] Among them, Q t is the process noise covariance matrix. The prior state estimate describes the predicted state of the target without the current observation. Based on the prior state estimate and the human detection result of the current frame, the matching cost matrix is ​​calculated. The element d of the matching cost matrix D is ij represents the matching cost between the i-th predicted target and the j-th detected target, which is determined by the position deviation and size difference:

[0093]

[0094] Among them, (x i ,y i ) and (x j ,y j ) represent the center coordinates of the predicted target and the detected target, respectively, w i ,h i and w j ,h j Represent the width and height of the predicted target and the detected target respectively, and α is the importance weight used to balance the position and size differences. Through the formula, the prior state estimate and the current observation are combined to calculate the matching cost between each pair of targets. The matching cost matrix is ​​data associated using the Hungarian algorithm. The Hungarian algorithm is an optimization algorithm for solving the bipartite graph matching problem, which can find the combination with the lowest cost among all possible matching pairs. The matching pairs of targets and detection results are obtained through the Hungarian algorithm, that is, which detection results correspond to which tracking targets. For each target in the matching pair, the cosine similarity between their facial feature vectors is calculated. Assume that the feature vector of the i-th target is f i , the feature vector of the jth detection result is f j , the cosine similarity is expressed as:

[0095]

[0096] Among them, cos(θ ij ) represents the cosine similarity between two feature vectors, f i ·f j represents their inner product, ||f i || and ||f j || respectively represent their modulus lengths. The value range of cosine similarity is between [-1,1]. The closer to 1, the more similar their directions and features are. Combining the position information and cosine similarity, the identity confirmation result of the target is obtained. Based on the target identity confirmation result and the current observation value, the state estimate of the Kalman filter is updated to obtain the posterior state estimate. The updated calculation formula is:

[0097]

[0098] Among them, z t+1 is the current observation value, H is the observation matrix, usually the unit matrix, K t+1 is the Kalman gain, and the calculation formula is:

[0099]

[0100] Among them, R t+1 is the observation noise covariance matrix. The updated covariance matrix is:

[0101]

[0102] The observation value is integrated into the predicted state through the update formula to obtain a more accurate posterior state estimate. The observation noise and process noise in the posterior state estimate are adaptively adjusted. The sample covariance of the observation residual sequence of the most recent A frame is calculated, and the noise covariance matrix is ​​updated using the exponential weighted average method. Suppose the observation residual sequence of the most recent A frame is r t , its sample covariance is:

[0103]

[0104] in, is the mean of the observed residual sequence. The observed noise covariance matrix R is calculated using the exponential weighted average method. t To update:

[0105] R t =βR t-1 +(1-β)S t ;

[0106] Where β is the smoothing coefficient, which is usually between 0.8 and 0.95. This method enables the noise covariance matrix to be continuously adaptively adjusted during the tracking process, so as to better adapt to the noise changes in different environments. Substitute the adaptive noise covariance matrix into the prediction and update equations of the Kalman filter, recalculate the Kalman gain and state estimation, and obtain the optimized state estimation. The method of dynamically adjusting the noise covariance can effectively improve the stability and tracking accuracy of the Kalman filter in complex environments. For unmatched detection results, a new trajectory is initialized; for those unmatched trajectories in consecutive A frames, the target may disappear or be blocked temporarily. In order to continue tracking when the target reappears, a long short-term memory network (LSTM) is used for trajectory extrapolation. LSTM is a neural network model that can process time series data. It predicts the position of the future time step by memorizing the historical information of the previous time step. The target position of the future A frame is predicted by the trained LSTM model to obtain a more complete multi-target humanoid tracking result.

[0107] In a specific embodiment, the process of executing step S5 may specifically include the following steps:

[0108] The multi-target humanoid tracking results are segmented in time and space, and the tracking results of continuous B frames are divided into a task unit to obtain multiple task units. Based on the CPU usage, memory occupancy and network bandwidth of each computing node, the future load of each node is predicted to obtain the node load prediction value; according to the node load prediction value and the complexity of the task unit, the task units are preliminarily allocated to obtain the initial task allocation plan, and the initial task allocation plan is load-balanced optimized. Through task migration and splitting operations, the load difference of each node is controlled within the preset threshold to obtain the optimized task allocation plan; the optimized task allocation plan is sent to each computing node, and a local feature index is built on each node. A hash table is used to store facial feature vectors and corresponding trajectory IDs to obtain a local feature index structure. A global feature index is constructed at the central control node, and a hierarchical tree structure is used to organize the feature information of each node. Each leaf node corresponds to a local index of a computing node to obtain a global feature index structure. For cross-node face matching requests, a rough match is first performed in the global feature index to locate the most likely K leaf nodes, and then an exact match is performed in the local index of these K leaf nodes to obtain a cross-node matching result. Based on the cross-node matching results, the trajectory segments on different nodes are associated and fused to eliminate duplicate trajectories, and the trajectory ID is globally consistent to obtain a global human tracking result.

[0109] Specifically, the multi-target humanoid tracking data is processed in blocks according to the time dimension. Assume that in the entire tracking data, the tracking result of each target contains T frames, where the tracking information of each frame is represented by t i ={x i ,y i ,w i ,h i ,id i}, where x i and i are the center coordinates of the target, w i and h i The width and height of the target, id i is the unique identifier of the target. In order to divide the tracking data into multiple task units, each task unit should contain continuous B-frame tracking results to ensure that the task unit has good temporal continuity. The divided task unit is represented by T k ={t kB ,t kB+1 ,…,t( k+1 ) B-1}, where k represents the number of the task unit, and the value range of k is For example, for a segment of 300-frame human tracking data, if B=50 is selected, it will be divided into 6 task units, each of which contains 50 frames of tracking results. Based on the resource information such as CPU usage, memory occupancy, and network bandwidth of each computing node, the future load of each node is predicted to obtain the load prediction value of the node. Suppose the CPU usage, memory occupancy, and network bandwidth of the j-th node at the current time t are CPU j (t), Mem j (t) and Net j (t). Use the linear regression model to predict the load condition at the future time t+Δt. The expression of the linear regression model is:

[0110]

[0111] in, represents the load prediction value of the jth node at the future moment, α1, α2, α3 are regression coefficients, and β is the bias term. By fitting the historical data, the values ​​of the regression coefficients and bias terms are estimated to obtain the load prediction value of each node, reflecting the resource occupancy of each node at the future moment. The task units are preliminarily allocated according to the node load prediction value and the complexity of the task unit. The complexity of the task unit is determined by factors such as the number of targets it contains, the movement complexity of the targets, and the amount of data. Assume that the complexity of the kth task unit is C k , calculated by the following formula:

[0112]

[0113] Among them, N k is the number of targets in the task unit, (x i ,y i ) is the center coordinate of the target in each frame, indicating the motion complexity of the target, D k is the amount of data of the task unit, which is related to factors such as image resolution and number of frames. γ1, γ2, and γ3 are weight coefficients. The complexity of each task unit is measured by the formula to provide a basis for task allocation. The preliminary allocation plan is implemented by a greedy algorithm, which preferentially allocates task units with greater complexity to nodes with lighter loads. Suppose the preliminary allocation plan is M={(k,j)}, which means that the kth task unit is assigned to the jth computing node. In order to optimize the preliminary allocation plan, load balancing optimization is performed. Through task migration and task splitting operations, the load difference of each node is controlled within the preset threshold. Suppose the load of each node is L j The goal is to make the load of all nodes satisfy:

[0114]

[0115] Among them, δ is the preset load difference threshold. Task migration is to transfer some task units from nodes with heavier loads to nodes with lighter loads, and task splitting is to decompose a task unit with higher complexity into multiple sub-task units with lower complexity so that they can be executed in parallel on different nodes. After task migration and splitting operations, a load-balanced optimized task allocation scheme M is obtained. opt . After the optimized task allocation scheme is sent to each computing node, a local feature index is constructed on each node. The local feature index is used to store the face feature vector and the corresponding trajectory ID so as to perform efficient face matching and retrieval within the node. Assume that there are N different trajectories on each node, and each trajectory contains M feature vectors. A hash table is used to store the mapping relationship between feature vectors and trajectory IDs. The hash table storage method is used to achieve fast retrieval and matching of feature vectors. On the central control node, a global feature index structure is constructed. The global feature index uses a hierarchical tree structure to organize the feature information of each node, and each leaf node corresponds to a local index of a computing node. The root node of the tree structure represents the feature information of the entire distributed system, the internal nodes represent the feature information of the sub-region or sub-node, and the leaf nodes store the actual feature data. The construction of the tree structure is achieved through a bottom-up clustering algorithm, which aggregates the feature information of each computing node into a leaf node, then merges the adjacent leaf nodes into an internal node, and so on, until the entire tree structure is constructed. When processing cross-node face matching requests, a rough match is performed in the global feature index to quickly locate the K leaf nodes that are most likely to contain the target features. Let the input feature vector be f, and the similarity between each leaf node and the feature vector is calculated using the following formula:

[0116]

[0117] Among them, S j represents the maximum cosine similarity between the input feature vector f and all feature vectors in the jth leaf node, θ ij Represents the angle between the input feature vector and the i-th feature vector in the leaf node. j The size of the leaf node is used to select the first K leaf nodes for exact matching. When performing exact matching in the local index of the selected K leaf nodes, the cosine similarity between the input feature vector and each local feature vector is calculated one by one to find the matching result with the highest similarity, effectively reducing the search space and improving the matching efficiency. The trajectory segments on different nodes are associated and fused based on the cross-node matching results. The association of trajectory segments is achieved by comparing their spatiotemporal overlap and feature similarity. Assuming that the temporal overlap of two trajectory segments is Δt, the spatial overlap is Δs, and the feature similarity is cos(θ), the association of trajectory segments is expressed as:

[0118] R=α·Δt+β·Δs+γ·cos(θ);

[0119] Among them, α, β, and γ are the weight coefficients of time overlap, spatial overlap, and feature similarity, respectively. According to the size of the correlation R, it is judged whether two trajectory segments belong to the same target. For trajectory segments determined to be the same target, trajectory fusion is performed to merge the trajectory points of different segments into a complete trajectory. The trajectory ID is processed for global consistency, and a unique global ID is assigned to each target to ensure that the same target on different nodes has a unique identifier in the entire system, and a global human tracking result is obtained.

[0120] In a specific embodiment, the process of executing step S6 may specifically include the following steps:

[0121] Multiple target video frames are grouped according to the global human tracking results, and the continuously tracked human targets are divided into the same video clip to obtain multiple video clips, each of which contains E frame images; the ResNext-101 network is applied to each video frame in the multiple video clips, and the ResNext-101 network contains 32 residual blocks with a base of 4, each residual block outputs 256-dimensional features, and finally a 2048-dimensional spatial feature vector is obtained through global average pooling, and then an E×2048-dimensional spatial feature sequence is obtained; the self-attention mechanism is applied to the spatial feature sequence, and an E×E attention weight matrix is ​​calculated, in which each element represents the correlation between frames, and then the attention weight matrix is ​​multiplied by the original feature sequence to obtain a weighted spatial feature sequence; for each video frame in the multiple video clips, a differential image with the previous frame is calculated, and an inter-frame difference method is used to obtain E-1 residual noise images, and these images are stacked into E-1×H× W×3 residual noise sequence, where H and W are the height and width of the image respectively; the residual noise sequence is input into a 3D convolutional neural network, which contains 4 3D convolutional layers, each of which has a convolution kernel size of 3×3×3, a step size of 1×2×2, and the number of channels is 64, 128, 256, and 512, respectively. Finally, a 512-dimensional temporal feature vector is obtained through 3D global average pooling, and then an E×512-dimensional temporal feature sequence is obtained; a bilinear pooling operation is performed on the weighted spatial feature sequence and the temporal feature sequence to obtain an E×d-dimensional fused feature sequence; based on the fused feature sequence, a two-layer fully connected neural network is used to regress and calculate the relative position offset, rotation angle and scaling factor between adjacent video clips to obtain video splicing parameters; according to the video splicing parameters, multiple video clips are affine transformed, and the image is resampled using bilinear interpolation, and then the transformed images are spliced ​​according to the relative position to obtain a spliced ​​video.

[0122] Specifically, the global human tracking results are analyzed. The global human tracking results usually include the trajectory data of the target, where each trajectory consists of a series of frame target detection boxes and corresponding timestamps. The continuous frames are divided into video segments, and a fixed number of frames E is set as the length of each video segment to obtain multiple continuous video segments containing E frames. In order to extract the spatial features of each frame in each video segment, the ResNext-101 network is used. ResNext-101 is a deep convolutional neural network containing 32 residual blocks with a base of 4, and each residual block outputs 256-dimensional features. The expression of each residual block can be expressed as:

[0123]

[0124] Among them, y is the output of the residual block, x is the input feature, F(x) represents the convolution operation, and each residual block contains multiple parallel branches, each branch uses a different convolution kernel. By stacking residual blocks, ResNext-101 can extract richer image features. Through the global average pooling operation, the features of each frame are mapped to a 2048-dimensional feature vector. The formula for global average pooling is:

[0125]

[0126] Among them, f i is the 2048-dimensional feature vector of the i-th frame, f ijk represents the eigenvalue of the feature map at the i-th frame and position (j, k), H and W are the height and width of the feature map respectively. Through this formula, the feature map of each frame is compressed into a global feature vector to obtain a spatial feature sequence of E×2048 dimensions. In order to capture the temporal correlation between frames in the video clip, the self-attention mechanism is applied to the spatial feature sequence. The self-attention mechanism can capture the dependency between elements in the sequence and represent the correlation between frames by calculating the E×E attention weight matrix. The calculation formula of the attention weight matrix is:

[0127]

[0128] Among them, α ij represents the attention weight between the i-th frame and the j-th frame, q i =W q f i and k j =W k f j denote the query vector and key vector of the i-th frame and the j-th frame respectively, W q and W kis a learnable linear transformation matrix. Through this formula, each frame in the original feature sequence is associated with other frames to obtain a weighted spatial feature sequence. In order to capture the dynamic information in the video clip, the differential image between each video frame and the previous frame is calculated. The differential image reflects the motion changes between frames in the video, and the inter-frame difference method is used to obtain E-1 residual noise images. The formula for the inter-frame difference method is:

[0129] ΔI t =I t -I t-1 ;

[0130] Among them, ΔI t represents the difference image between the t-th frame and the t-1-th frame, I t and I t-1 Represent the images of the tth frame and the t-1th frame respectively. The differential images are stacked into a residual noise sequence of (E-1)×H×W×3, where H and W are the height and width of the image, respectively. The residual noise sequence is input into a 3D convolutional neural network for temporal feature extraction. The 3D convolutional neural network can extract features in both spatial and temporal dimensions, and is suitable for processing spatiotemporal information in videos. Assume that the 3D convolutional neural network contains 4 3D convolutional layers, the convolution kernel size of each layer is 3×3×3, the step size is 1×2×2, and the number of channels is 64, 128, 256, and 512 respectively. The calculation formula for 3D convolution is:

[0131]

[0132] Among them, f i,j,k represents the feature value of the 3D convolutional layer output feature map at position (i, j, k), W d,m,n represents the weight of the 3D convolution kernel at depth d, height m and width n, I i+d-1,j+m-1,k+n-1 Represents the value of the input feature map at the corresponding position, and D, M, and N represent the dimensions of the convolution kernel in depth, height, and width, respectively. By stacking 3D convolutional layers, rich temporal features are extracted from the video clip. Finally, through 3D global average pooling, the output feature map is reduced to a 512-dimensional temporal feature vector, and an E×512-dimensional temporal feature sequence is obtained. In order to fuse the spatial features and temporal features of the video clip, bilinear pooling operations are performed on the weighted spatial feature sequence and the temporal feature sequence. Bilinear pooling is a feature fusion method that can capture the interaction information between two different feature vectors. The formula for bilinear pooling is:

[0133]

[0134] Among them, f fusion,t represents the fusion feature of the tth frame, f space,t and f time,tRepresent the spatial and temporal features of the tth frame respectively, Represents an element-level multiplication operation. After the bilinear pooling operation, a fused feature sequence of E×d dimensions is obtained, where d is the fused feature dimension. Based on the fused feature sequence, a two-layer fully connected neural network is used to regress the relative position offset, rotation angle, and scaling factor between adjacent video clips. Assume that the relative position offset between adjacent video clips P1 and P2 is (Δx, Δy), the rotation angle is θ, and the scaling factor is s. The input of the fully connected neural network is the fused feature vector, and the output is the splicing parameter (Δx, Δy, θ, s). The calculation formula of the regression network is:

[0135]

[0136] Among them, W fcl and W fc2 denote the weight matrices of the first and second fully connected layers, respectively, and b fc1 and b fc2 Respectively represent the bias term, and σ(·) represents the nonlinear activation function. Through a two-layer fully connected network, the fused features are mapped to the splicing parameter space to obtain the splicing parameters of adjacent video clips. Affine transformation is performed on multiple video clips according to the video splicing parameters. Affine transformation is a linear transformation that performs translation, rotation, and scaling at the same time. Assume that the coordinates of the input image are (x, y), and the coordinates after transformation are (x′, y′). The formula of affine transformation is:

[0137]

[0138] Among them, s represents the scaling factor, θ represents the rotation angle, and Δx and Δy represent the position offset respectively. Through this formula, each pixel in the image is mapped to the new coordinates after the transformation. In order to maintain the details of the image during the transformation process, the image is resampled using bilinear interpolation. The formula for bilinear interpolation is: I′(x′,y′)=(1-Δx)(1-Δy)I(x1,y1)+Δx(1-Δy)I(x2,y1)+(1-Δx)ΔyI(x1,y2)+ΔxΔyI(x2,y2). Among them, I′(x′,y′) represents the pixel value after resampling, I(x1,y1), I(x2,y1), I(x1,y2) and I(x2,y2) represent the four adjacent pixel values ​​respectively, and Δx and Δy represent the coordinate offset of the decimal part respectively. The image is smoothed by bilinear interpolation, and the transformed images are spliced ​​according to their relative positions to obtain the final spliced ​​video.

[0139] The above describes the method for splicing surveillance videos in the embodiment of the present invention. The following describes the apparatus for splicing surveillance videos in the embodiment of the present invention. Figure 2, an embodiment of the splicing device of the monitoring video in the embodiment of the present invention includes:

[0140] A framing module is used to perform video framing on the surveillance video collected by the surveillance terminal to obtain multiple target video frames;

[0141] A detection module is used to input multiple target video frames into the improved YOLOv5 model to perform human shape detection and obtain human shape detection results;

[0142] An extraction module is used to align and crop the face area in the human detection result to obtain a standardized face image and extract a face feature vector;

[0143] The prediction module is used to perform motion prediction and trajectory extrapolation based on the face feature vector and human shape detection results to obtain multi-target human shape tracking results;

[0144] The tracking module is used to input the multi-target humanoid tracking results into the distributed computing framework for task allocation and cross-node feature matching to obtain the global humanoid tracking results;

[0145] The calculation module is used to perform dual-stream network processing on multiple target video frames according to the global human tracking results, calculate the video splicing parameters, and perform splicing operations on the multiple target video frames to obtain spliced ​​videos.

[0146] Through the synergy of the above components, the improved YOLOv5 model is used for human detection, and the spatial attention and channel attention mechanisms are introduced to improve the accuracy and robustness of target detection in complex scenes. The improved FaceNet network with SE-ResNet-50 backbone network and self-attention mechanism is used to extract facial features, which enhances the discriminability of facial features and is conducive to subsequent target tracking and identity recognition. The Kalman filter with adaptive noise covariance matrix is ​​used for motion prediction and trajectory extrapolation, which improves the accuracy and stability of multi-target tracking, especially the performance under short-term occlusion of the target. The introduction of distributed computing framework and dynamic task allocation mechanism realizes the efficient processing of large-scale video data, and significantly improves the real-time performance and scalability of the system. Through the dual-stream network processing method, the spatial semantic features and temporal residual noise information are comprehensively considered, the accuracy and coherence of video splicing are improved, and the problems of breakage and repetition of dynamic targets in the splicing process are effectively solved. The present invention effectively improves the quality and visual effect of spliced ​​video.

[0147] The present invention also provides a computer device, which includes a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor executes the steps of the surveillance video splicing method in the above-mentioned embodiments.

[0148] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the surveillance video splicing method.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0151] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for splicing surveillance videos, characterized in that: The method comprises: Perform video framing on the surveillance video collected by the surveillance terminal to obtain multiple target video frames; Input the multiple target video frames into the improved YOLOv5 model to perform human figure detection, and obtain human figure detection results; Aligning and cropping the face area in the human figure detection result to obtain a standardized face image, and extracting a face feature vector; Perform motion prediction and trajectory extrapolation based on the face feature vector and the human figure detection result to obtain a multi-target human figure tracking result; Inputting the multi-target humanoid tracking results into a distributed computing framework for task allocation and cross-node feature matching to obtain a global humanoid tracking result; According to the global human tracking result, the multiple target video frames are processed by a dual-stream network, video stitching parameters are calculated, and a stitching operation is performed on the multiple target video frames to obtain a stitched video.

2. The method for splicing surveillance videos according to claim 1, characterized in that: The video frame division is performed on the monitoring video collected by the monitoring terminal to obtain multiple target video frames, including: The monitoring video is collected by the monitoring terminal, and the monitoring video is sampled according to a preset frame rate to obtain a plurality of initial video frames, and each of the plurality of initial video frames is subjected to resolution standardization processing to obtain a plurality of standardized video frames; Performing bilateral filtering on the multiple standardized video frames, wherein the radius of the spatial domain kernel function is set to 5 and the standard deviation of the range kernel function is set to 50, to obtain multiple video frames after preliminary denoising; Performing edge-preserving filtering on the multiple preliminarily denoised video frames to obtain multiple edge-enhanced video frames, dividing the edge-enhanced video frames into 8x8 non-overlapping blocks, calculating the cumulative distribution function for each block, and obtaining a local CDF matrix; Based on the local CDF matrix, a mapping function of each pixel is calculated using a bilinear interpolation method to obtain a global mapping matrix, and the edge-enhanced video frame is remapped at a pixel level grayscale value according to the global mapping matrix to obtain a plurality of contrast-enhanced video frames; The plurality of contrast-enhanced video frames are subjected to color space conversion, the RGB color space is converted into the YUV color space, histogram equalization is applied only to the Y channel, and then the processed YUV image is converted back into the RGB color space to obtain a plurality of target video frames.

3. The method for splicing surveillance videos according to claim 1, characterized in that: The step of inputting the plurality of target video frames into the improved YOLOv5 model for human shape detection to obtain human shape detection results includes: Performing multi-scale transformation on the multiple target video frames, generating feature maps of three different scales by a bilinear interpolation method, which are 1 / 8, 1 / 16 and 1 / 32 of the original input size, respectively, to obtain a multi-scale input feature map; The multi-scale input feature map is sequentially input into the backbone network of the improved YOLOv5 model, and the backbone network adds a spatial attention module after each convolution block. The spatial attention module emphasizes important spatial areas by generating a two-dimensional attention weight map to obtain a preliminary feature map containing spatial attention information; Performing channel attention processing on the preliminary feature map, first calculating the global average pooling value of each channel, then inputting the pooling result into a multilayer perceptron consisting of two fully connected layers, wherein the number of neurons in the first layer is 1 / 16 of the number of channels, and the second layer is restored to the original number of channels, and finally obtaining the channel weight through the Sigmoid activation function, and multiplying the channel weight by the preliminary feature map element by element to obtain an enhanced feature map containing both spatial and channel attention information; Input the enhanced feature map into the feature pyramid network, and perform feature fusion through a top-down path and a lateral connection, wherein the top-down path uses a 1x1 convolution to adjust the number of channels and upsample by a factor of 2, and the lateral connection uses a 1x1 convolution to process the features of the same layer, and finally obtain a multi-scale fused feature map; Apply a 3x3 convolution kernel to the multi-scale fusion feature map for feature extraction, keep the size of the feature map unchanged, and then use a 1x1 convolution kernel to compress the number of channels to the preset number of anchor boxes multiplied by (5+number of categories), where 5 represents the 4 coordinate values ​​of the bounding box and 1 confidence value, to obtain the original prediction result; Decoding the original prediction result, converting the predicted bounding box coordinates from the offset relative to the grid to the absolute coordinates of the image, specifically by multiplying the predicted relative coordinates by the step size of the feature map and adding the coordinates of the upper left corner of the corresponding grid, and applying the Sigmoid function to process the confidence and category probability, mapping them between 0 and 1, to obtain the decoded prediction result; Based on the prediction results after decoding, the overlapping detection frames are screened using the non-maximum suppression algorithm. First, all detection frames are arranged in descending order by confidence, and then compared one by one to calculate the intersection-and-union ratio of the overlapping frames. When the intersection-and-union ratio is greater than the preset threshold of 0.5, the frame with the highest confidence is retained and other overlapping frames are deleted to obtain the human detection result.

4. The method for splicing surveillance videos according to claim 1, characterized in that: The step of aligning and cropping the face region in the human shape detection result to obtain a standardized face image and extracting a face feature vector includes: Applying a facial key point detection algorithm based on a multi-task cascade convolutional network to the face area in the human figure detection result, locating five key points, including the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth, and the right corner of the mouth, to obtain the coordinates of the facial key points; Based on the coordinates of the facial key points, an affine transformation matrix is ​​calculated to transform the facial image to a standard position so that the centers of the two eyes are located on a horizontal line and at a fixed position in the image, thereby obtaining an aligned facial image; The aligned face image is cropped and scaled to adjust the aligned face image to a size of 112×112 pixels, and pixel values ​​are normalized to scale the pixel values ​​to a [-1, 1] interval to obtain a standardized face image; Input the standardized face image into the SE-ResNet-50 backbone network, wherein the SE-ResNet-50 backbone network includes 50 convolutional layers, and a Squeeze-and-Excitation module is added after each residual block, and the correlation between channels is learned through global average pooling and two fully connected layers to obtain a face feature map; Applying a self-attention mechanism to the face feature map, calculating the correlation between each position and all other positions in the face feature map, generating an attention weight matrix, and multiplying the attention weight matrix by the face feature map to obtain an enhanced feature map; The enhanced feature map is input into the global average pooling layer, the spatial dimension of the enhanced feature map is reduced to 1x1, the channel dimension is retained, and a 2048-dimensional feature vector is obtained. A fully connected layer is applied to the 2048-dimensional feature vector, and the 2048-dimensional feature vector is mapped to a 512-dimensional space to obtain a face feature vector.

5. The method for splicing surveillance videos according to claim 1, characterized in that: The performing motion prediction and trajectory extrapolation based on the face feature vector and the human shape detection result to obtain a multi-target human shape tracking result includes: Establishing a state vector for each target in the human detection result, including the center coordinates, width, height, speed and acceleration of the target, initializing the state vector and covariance matrix of the Kalman filter, and obtaining an initial state estimate; Based on the initial state estimation, the next frame position of the target is predicted using the state transfer matrix to obtain a priori state estimation, and a matching cost matrix is ​​calculated according to the priori state estimation and the human detection result of the current frame, and the Hungarian algorithm is used to perform data association to obtain a matching pair of the target and the detection result; For each target in the matching pair, the cosine similarity between the facial feature vectors is calculated, and the target identity confirmation result is obtained by combining the position information, and the state estimation of the Kalman filter is updated based on the target identity confirmation result and the current observation value to obtain the posterior state estimation; Adaptively adjusting the observation noise and process noise in the posterior state estimation, by calculating the sample covariance of the observation residual sequence of the most recent A frames and updating the noise covariance matrix using an exponentially weighted average method, to obtain an adaptive noise covariance matrix; Substituting the adaptive noise covariance matrix into the prediction and update equations of the Kalman filter, recalculating the Kalman gain and state estimation, and obtaining an optimized state estimation; For unmatched detection results, new trajectories are initialized; for unmatched trajectories in consecutive A frames, the long short-term memory network is used to extrapolate the trajectories and predict the positions of future A frames to obtain multi-target humanoid tracking results.

6. The method for splicing surveillance videos according to claim 1, characterized in that: The step of inputting the multi-target humanoid tracking results into a distributed computing framework for task allocation and cross-node feature matching to obtain a global humanoid tracking result includes: Performing spatiotemporal segmentation on the multi-target humanoid tracking results, dividing the tracking results of consecutive B frames into a task unit, obtaining a plurality of task units, and predicting the future load of each node based on the CPU usage, memory occupancy, and network bandwidth of each computing node to obtain a node load prediction value; According to the predicted value of the node load and the complexity of the task unit, the task units are preliminarily allocated to obtain an initial task allocation plan, and the initial task allocation plan is load balanced and optimized. Through task migration and splitting operations, the load difference of each node is controlled within a preset threshold, and an optimized task allocation plan is obtained; The optimized task allocation scheme is sent to each computing node, and a local feature index is constructed on each node, and a hash table is used to store the face feature vector and the corresponding trajectory ID to obtain a local feature index structure; A global feature index is constructed at the central control node, and the feature information of each node is organized using a hierarchical tree structure. Each leaf node corresponds to a local index of a computing node, and a global feature index structure is obtained. For cross-node face matching requests, we first perform a rough match in the global feature index to locate the most likely K leaf nodes, and then perform an exact match in the local index of these K leaf nodes to obtain the cross-node matching results. Based on the cross-node matching result, the trajectory segments on different nodes are associated and fused, duplicate trajectories are eliminated, and the trajectory ID is processed for global consistency to obtain a global human tracking result.

7. The method for splicing surveillance videos according to claim 1, characterized in that: The method of performing dual-stream network processing on the multiple target video frames according to the global human tracking result, calculating video stitching parameters, and performing a stitching operation on the multiple target video frames to obtain a stitched video includes: The plurality of target video frames are grouped according to the global human figure tracking result, and the continuously tracked human figure targets are divided into the same video segment to obtain a plurality of video segments, each of which contains an E frame image; Applying a ResNext-101 network to each video frame in the multiple video clips, the ResNext-101 network comprising 32 residual blocks with a base number of 4, each residual block outputting a 256-dimensional feature, and finally obtaining a 2048-dimensional spatial feature vector through global average pooling, thereby obtaining an E×2048-dimensional spatial feature sequence; Applying a self-attention mechanism to the spatial feature sequence, calculating an E×E attention weight matrix, where each element represents the correlation between frames, and then multiplying the attention weight matrix by the original feature sequence to obtain a weighted spatial feature sequence; For each video frame in the multiple video clips, a differential image is calculated from a previous frame, an inter-frame difference method is used to obtain E-1 residual noise images, and these images are stacked into a residual noise sequence of E-1×H×W×3, where H and W are the height and width of the image respectively; The residual noise sequence is input into a 3D convolutional neural network, which includes 4 3D convolutional layers, each of which has a convolution kernel size of 3×3×3, a step size of 1×2×2, and channel numbers of 64, 128, 256, and 512, respectively. Finally, a 512-dimensional time feature vector is obtained through 3D global average pooling, thereby obtaining an E×512-dimensional time feature sequence; Performing a bilinear pooling operation on the weighted spatial feature sequence and the temporal feature sequence to obtain an E×d-dimensional fused feature sequence; Based on the fusion feature sequence, a two-layer fully connected neural network is used to regress and calculate the relative position offset, rotation angle and scaling factor between adjacent video clips to obtain video splicing parameters; According to the video stitching parameters, the multiple video clips are affine transformed, and the images are resampled using a bilinear interpolation method. The transformed images are then stitched according to relative positions to obtain a stitched video.

8. A surveillance video splicing device, characterized in that: The method for splicing surveillance videos according to any one of claims 1 to 7 is used to perform the splicing method of surveillance videos, wherein the splicing device of surveillance videos comprises: A framing module is used to perform video framing on the surveillance video collected by the surveillance terminal to obtain multiple target video frames; A detection module, used for inputting the multiple target video frames into an improved YOLOv5 model to perform human shape detection and obtain a human shape detection result; An extraction module, used for aligning and cropping the face area in the human figure detection result to obtain a standardized face image and extract a face feature vector; A prediction module, used for performing motion prediction and trajectory extrapolation based on the face feature vector and the human figure detection result to obtain a multi-target human figure tracking result; A tracking module, used for inputting the multi-target humanoid tracking results into a distributed computing framework for task allocation and cross-node feature matching to obtain a global humanoid tracking result; The calculation module is used to perform dual-stream network processing on the multiple target video frames according to the global human tracking result, calculate the video splicing parameters, and perform a splicing operation on the multiple target video frames to obtain a spliced ​​video.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program executable on the processor, and wherein the processor implements the surveillance video splicing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is enabled to execute the monitoring video splicing method according to any one of claims 1 to 7.

Images