A method, device and equipment for tracking moving targets in video surveillance

By adding SE modules to UNet networks and using DeepSORT model, the problem of boundary inconsistency when semantic segmentation and edge detection is solved, the clarity and tracking accuracy of the motion target boundary are enhanced, and the scene changes are adapted to achieve more accurate motion Target tracking.

CN118864537BActive Publication Date: 2024-12-03SHANDONG JIANZHU UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411338083.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-25
Publication Date
2024-12-03
Estimated Expiration
2044-09-25

AI Technical Summary

Technical Problem

In the prior art, the boundary processing methods are different when semantic segmentation and edge detection are combined, resulting in inconsistent boundaries of moving targets, and the target motion causes the boundaries to mix with the background, which is difficult to distinguish, resulting in inaccurate tracking effects.

Method used

Add SE modules to the original UNet network, form an improved UNet network through the connection between the encoder and the decoder, enhance the edge characteristics of the semantic segmentation model, combine it with motion masks, and track it using the DeepSORT target tracking model.

Benefits of technology

By enhancing the fusion of edge features and motion masks of the semantic segmentation model, the clarity and tracking accuracy of the motion target boundaries are improved, adapting to scene changes, and achieving more accurate motion target tracking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118864537B_ABST
    Figure CN118864537B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, device and equipment for tracking a moving target in video surveillance, and relates to the technical field of target tracking. The method comprises: extracting each frame of the video surveillance; obtaining an edge-enhanced semantic segmentation mask and a motion mask for each frame of the image; fusing the semantic segmentation mask and the motion mask to obtain the moving target in each frame of the image; and tracking the moving target using a target tracking model. In order to solve the problem of blurred boundaries of moving targets during tracking, the present invention obtains an edge-enhanced semantic segmentation mask by introducing a compression and excitation attention mechanism module in semantic segmentation, and by using channel weights that respond to the edges of moving targets; obtains a motion mask for each frame of the image by adaptively updating the background in each frame of the image; combines the edge-enhanced semantic segmentation mask and the motion mask to obtain a more accurate moving target segmentation result, thereby obtaining a more accurate tracking result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular to a method, device and equipment for tracking a moving target in video surveillance. Background Art

[0002] Video surveillance technology is a technology that uses cameras to record, detect, and track objects. It is widely used in public safety, transportation, smart cities, and other fields. By analyzing the movement trajectory of objects and the traffic environment, video surveillance technology can provide users with timely warnings to deal with complex situations and reduce risks.

[0003] In the prior art, Chinese patent CN117876428B discloses a target tracking method, device, computer equipment and medium based on image processing, including: inputting video image data into an edge detection model to perform an edge detection task, generating an edge feature image, inputting a region growth model to perform a region growth task, generating a pixel feature image, inputting a semantic segmentation model to perform a semantic segmentation task, generating a semantic feature image; performing image fusion on the edge feature image and the semantic feature image, performing image fusion on the pixel feature image and the semantic feature image, and generating a final feature image by adding feature images; determining a contour image of a target object to be detected according to image features in the final feature image; and performing a target tracking task on the target object to be detected in the video image data based on the contour image.

[0004] However, when the above problem is combined with semantic segmentation and edge detection, the different ways of processing boundaries between the two will lead to inconsistency in the boundary area of ​​​​the moving target; and the movement of the target may cause its boundary to blend with the background, making the boundary area difficult to distinguish; it may be difficult to accurately capture these dynamic changes based on the semantic segmentation model alone, and the boundary of the moving target is blurred, resulting in inaccurate tracking effect. Summary of the invention

[0005] Based on this, in order to solve the technical problem of inaccurate moving target tracking in the prior art, the present invention provides a method, device and equipment for tracking a moving target in video surveillance.

[0006] The present invention provides a method for tracking a moving target in video surveillance, comprising:

[0007] Extract each frame of image from the video surveillance data to be processed;

[0008] An improved UNet network is formed by adding an SE module to the connection part of the encoder and the decoder in the original UNet network; the improved UNet network is trained to obtain a semantic segmentation model for segmenting moving targets in an image; each frame of the image is input into the semantic segmentation model, and the encoder is used to extract features of the input image to obtain a multi-channel semantic feature map; the SE module is used to enhance the feature channels in the multi-channel semantic feature map that respond to the edge of the moving target to obtain an edge-enhanced semantic feature map; the edge-enhanced semantic feature map is decoded by the decoder to obtain a semantic segmentation mask that enhances the edge of the moving target in each frame of the image;

[0009] An initial background model is constructed based on a first frame image extracted from the video surveillance data to be processed, and the initial background model is updated according to pixel changes in the remaining frame images caused by scene changes to obtain motion masks corresponding to the remaining frame images;

[0010] The semantic segmentation mask and the motion mask are fused to obtain the moving target in each frame image; the moving target is tracked using the target tracking model.

[0011] Furthermore, the SE module is added to the connection part of the encoder and the decoder in the original UNet network, and the SE module is added to the output end of each downsampling layer included in the encoder in the original UNet network, and the output end of each downsampling layer included in the encoder is connected to the input end of the subsequent sampling layer and the input end of the corresponding upsampling layer in the decoder; the SE module is used to gradually enhance the edge features during the downsampling process.

[0012] Furthermore, the adding of the SE module to the connection part of the encoder and the decoder in the original UNet network is to add the SE module to the input end of each upsampling layer included in the decoder in the original UNet network, and the input of each upsampling layer included in the decoder is a spliced ​​feature map obtained by splicing the output of the previous upsampling layer and the output of the corresponding downsampling layer in the encoder; the SE module is used to enhance the edge features in the spliced ​​feature map during the upsampling process.

[0013] Furthermore, the constructing of the initial background model based on the first frame image extracted from the video surveillance data to be processed specifically includes:

[0014] The first frame image is filtered using a Gaussian blur method to remove noise in the first frame image;

[0015] For each pixel point in the filtered first frame image, a plurality of pixels points are randomly selected from the surrounding area to form an initial background model of the corresponding pixel point.

[0016] Further, the updating of the initial background model according to the pixel changes in the remaining frame images caused by the scene changes specifically includes:

[0017] Perform foreground detection on the pixels in the remaining frame images, and output the foreground detection results in the form of a binary motion mask, where the foreground pixels are marked as white and the background pixels are marked as black;

[0018] New pixels are extracted from the background pixels to update the initial background model. During the updating process, the initial background model is updated to adapt to the scene changes through the adaptive learning rate. The adaptive learning rate is :

[0019]

[0020] in, is the binary motion mask corresponding to the current frame image, The mean used to compute the binary motion mask.

[0021] Furthermore, the foreground detection of pixels in the remaining frame images includes:

[0022] Calculate the distance between the pixel value of each pixel point in the remaining frame image and the pixel value of each pixel point in the corresponding initial background sample;

[0023] If the distance between the pixel value of a certain pixel in the remaining frame image and the pixel value of at least one pixel in the corresponding initial background sample is less than the adaptive threshold, the pixel is judged to be a background pixel; otherwise, it is classified as a foreground pixel; the adaptive threshold for:

[0024]

[0025] in, is the average intensity of the current frame image, Indicates rounding.

[0026] Furthermore, the fusion of the semantic segmentation mask and the motion mask is to fuse the semantic segmentation mask and the motion mask through an AND operation, and the AND operation is used to obtain pixels marked by both the semantic segmentation mask and the motion mask.

[0027] Furthermore, the target tracking model is a DeepSORT target tracking model composed of an unscented Kalman filter and a MobileNetV3 network; wherein the unscented Kalman filter is used to capture the nonlinear characteristics of the moving target in each frame image; and the MobileNetV3 network is used to extract the deep characteristics of the moving target in each frame image;

[0028] The DeepSORT target tracking model introduces adaptive matching thresholds and adaptive sampling to dynamically adjust parameters according to feature changes of the tracked object;

[0029] The adaptive matching threshold first sets a reference matching threshold for determining whether the existing motion trajectory and the newly detected motion target trajectory are similar, obtains the confidence of the existing motion trajectory and the newly detected motion target trajectory respectively, calculates the adaptive matching threshold according to the two obtained confidences and the reference matching threshold, and performs association detection on the newly detected motion target trajectory and the existing trajectory based on the adaptive matching threshold;

[0030] The adaptive sampling sets a base sampling rate, obtains the trajectory speed and the density information of the current frame respectively, calculates the adaptive sampling rate according to the obtained trajectory speed, density information and the base sampling rate, and performs feature extraction based on the adaptive sampling rate.

[0031] The present invention provides a tracking device for a moving target in video surveillance, comprising:

[0032] A data acquisition module, used for extracting each frame of image from the video surveillance data to be processed;

[0033] A semantic segmentation mask acquisition module is used to form an improved UNet network by adding an SE module to the connection part of the encoder and the decoder in the original UNet network; training the improved UNet network to obtain a semantic segmentation model for segmenting moving targets in an image; inputting each frame of the image into the semantic segmentation model respectively, extracting features of the input image through the encoder to obtain a multi-channel semantic feature map; enhancing feature channels in the multi-channel semantic feature map that respond to the edge of the moving target through the SE module to obtain an edge-enhanced semantic feature map; decoding the edge-enhanced semantic feature map through the decoder to obtain a semantic segmentation mask that enhances the edge of the moving target in each frame of the image;

[0034] A motion mask acquisition module is used to construct an initial background model based on a first frame image extracted from the video surveillance data to be processed, and to update the initial background model according to pixel changes in the remaining frame images caused by scene changes to obtain motion masks corresponding to the remaining frame images;

[0035] The tracking module is used to fuse the semantic segmentation mask and the motion mask to obtain the moving target in each frame image; and use the target tracking model to track the moving target.

[0036] The present invention provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for tracking a moving target in video surveillance when executing the program.

[0037] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0038] The present invention firstly constructs a semantic segmentation model by adding an SE module to an original UNet network. The SE module enables the semantic segmentation model to fuse image semantic features and edge features by expanding the channel weights corresponding to the edges of image moving targets in a multi-channel feature map, thereby avoiding errors caused by inconsistent boundaries, and obtaining a semantic segmentation mask that enhances the edges of each frame of the image. In addition, the present invention updates the initial background samples according to pixel changes in the remaining frame images caused by scene changes, thereby being able to adapt to scene changes caused by motion, and obtaining a moving target motion mask that can correspond to the scene changes. The semantic segmentation mask and the motion mask are fused, so that even if the semantic segmentation model blurs the segmentation boundary of the moving target, the dynamic scene adaptability of the motion mask can supplement it, so as to obtain a clearer moving target boundary, thereby obtaining a more accurate moving target tracking result. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0040] Figure 1 A schematic diagram of a flow chart of a method for tracking a moving target in video surveillance provided by the present invention;

[0041] Figure 2 A schematic diagram of a process for obtaining a semantic segmentation mask provided by the present invention;

[0042] Figure 3 A schematic diagram of the framework of the DeepSORT target tracking model provided by the present invention;

[0043] Figure 4 A schematic diagram of the framework of a method for tracking a moving target in video surveillance provided by the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in combination with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Current methods for object detection and tracking in video surveillance still have flaws. Motion detection is not sensitive and robust enough to environmental conditions, and tracking accuracy needs to be improved. Current semantic segmentation methods cannot effectively utilize contextual information and have low positioning accuracy. The motion detection method ViBe (Visual Background Extraction) is not robust enough to sudden changes in lighting. These challenges have brought difficulties to object detection and tracking.

[0046] Based on this, the present invention provides a method for tracking moving targets in video surveillance, which combines the semantic segmentation architecture U-Net with the SE (Squeeze and Excitation) model, uses U-Net to capture image context information and precise positioning, and enhances the model's feature representation ability and channel dependency through compression and excitation attention mechanism modules. At the same time, Gaussian blur is used to improve the computational efficiency of ViBe motion detection, and the obtained motion mask is fused with the semantic segmentation mask of SE-UNet. In the DeepSORT (Deep Simple Online and Realtime Tracking) model, image features after semantic segmentation are extracted, and the state of the target is estimated by UKF (Unscented Kalman Filte). In addition, adaptive technology is introduced for adaptive matching, and the sampling rate is dynamically adjusted according to the object speed and scene density. The detected target trajectory is matched with the predicted trajectory, and the successfully matched trajectory is fused with the predicted frame, and the state of the target is updated.

[0047] Figure 1 The flow chart of the method for tracking moving targets in video surveillance of this embodiment is shown. Figure 1 The method is described in detail and specifically comprises the following steps:

[0048] S1: Extract each frame of image from the video surveillance data to be processed. Use a video surveillance device to obtain a video sequence and extract each frame of image from it.

[0049] S2: An improved UNet network is formed by adding an SE module to the connection part of the encoder and the decoder in the original UNet network; the improved UNet network is trained to obtain a semantic segmentation model for segmenting moving targets in an image; each frame of the image is input into the semantic segmentation model respectively, and the encoder is used to extract features of the input image to obtain a multi-channel semantic feature map; the SE module is used to enhance the feature channels in the multi-channel semantic feature map that respond to the edges of the moving targets to obtain an edge-enhanced semantic feature map; the edge-enhanced semantic feature map is decoded by the decoder to obtain a semantic segmentation mask that enhances the edges of the moving targets in each frame of the image.

[0050] Semantic segmentation is usually based on pixel-level classification, which divides images into regions with semantic information, but cannot accurately describe the details of the boundaries. Edge detection focuses on detecting significant edges or transition areas in images, which can provide more accurate boundary locations; so when the two are combined, the regions generated by semantic segmentation may not completely match the edges generated by edge detection, which will lead to blurred boundaries. Therefore, a semantic segmentation model that introduces a compression and excitation attention mechanism module is constructed to use the compression and excitation attention mechanism module to expand the channel weights that respond to the edges of the moving targets in the multi-channel feature map; and obtain the edge-enhanced semantic segmentation mask for each frame of the image.

[0051] Figure 2 The flowchart of obtaining the semantic segmentation mask provided in this embodiment specifically includes the following steps:

[0052] S201: Build a semantic segmentation model that introduces compression and incentive attention mechanism modules.

[0053] The semantic segmentation model is implemented by the trained SE-UNet network; it includes a feature encoding module, a compression and excitation attention mechanism module, a feature decoding module and an output module, wherein the feature encoding module extracts the features of each frame of the image by downsampling, and uses a pooling layer to reduce the resolution of each frame of the image while increasing the depth of the feature map to capture contextual information; the compression and excitation attention mechanism module calculates the attention score of the channel in the feature map to enhance the dependency of the channels in the feature map, which can be located at the output of each downsampling layer included in the encoder to gradually enhance the edge features during the downsampling process; it can also be located at the input of each upsampling layer included in the decoder, and the SE module is used to enhance the edge features in the spliced ​​feature map during the upsampling process; the feature decoding module increases the resolution of the feature map by upsampling, and fuses the feature map output by the feature encoding module with the upsampled feature map through jump connections; the output module classifies each pixel in the feature map output by the feature decoding module through a 1×1 convolutional layer, and outputs a semantic segmentation mask. Specifically:

[0054] The feature encoding module is composed of a ResNet 34 network, which extracts the features of each frame of the input video sequence through a series of convolutional layers, and uses a pooling layer to reduce the resolution and increase the depth of the feature map to capture contextual information.

[0055] The compression and incentive attention mechanism module calculates the channel attention score and enhances the channel dependency of the model. The compression operation is:

[0056]

[0057] in X Indicates the size B×H×W×C The input feature map is B , H , W and C Represents batch size, height, width, and number of channels respectively.

[0058] The incentive operations are:

[0059]

[0060] in, is the ReLU activation function, W 1 and W 2 are learnable weights, is a sigmoid activation function, is the output of the compression operation.

[0061] The feature decoding module increases the resolution of the feature map by upsampling, and introduces scaled cross entropy attention in the upsampling process to retain important features during the upsampling process; and fuses the feature map of the contraction path with the upsampled feature map through jump connections to achieve precise positioning.

[0062] Finally, the obtained effective feature layer is used through a 1×1 convolutional layer to classify each point on each frame image, output the segmentation map and generate a semantic segmentation mask.

[0063] S202: Train the semantic segmentation model by minimizing the pixel-wise binary cross entropy loss. The pixel-wise binary cross entropy loss is:

[0064]

[0065] in X represents the input image, Y Represents the corresponding ground truth segmentation map; It's pixels The predicted probability of belonging to the prospect,N represents the total number of samples, η is log The base number of .

[0066] S3: constructing an initial background model based on the first frame image extracted from the video surveillance data to be processed, and updating the initial background model according to pixel changes in the remaining frame images caused by scene changes to obtain motion masks corresponding to the remaining frame images.

[0067] ViBe is a background subtraction algorithm for video surveillance and motion detection, which aims to separate static background and moving targets from video sequences. When tracking moving targets, the ViBe algorithm first uses the first frame of the video sequence as the starting point, and constructs and initializes a background model by selecting the pixel values ​​of the surrounding area of ​​each pixel in the first frame. Then, for each new frame in the video, each pixel value in the current frame is compared with the sample set in its corresponding background model: if the current pixel value is within the preset similarity threshold range with a value in the sample set, then the pixel is identified as part of the background, and according to the dynamic update strategy of the algorithm, some values ​​in the sample set are randomly selected for replacement to maintain the adaptability and accuracy of the background model to new environmental changes; conversely, if the current pixel value does not meet the similarity condition with all values ​​in the background sample set, then the pixel is considered to be part of the moving target. Finally, in order to cope with the continuous changes in the scene, the ViBe algorithm continuously updates and adjusts the background sample set. Specifically, it includes:

[0068] Background model initialization. Use the first frame to initialize the background model, use Gaussian blur to extract the average color information of the surrounding area, and calculate each coordinate point ( x,y ) and generate the initial background sample. Gaussian blur can improve the computational efficiency and effectively reduce image noise, making subsequent motion detection more accurate. The pixel average calculation formula is as follows:

[0069]

[0070] x and y represents the distance relative to the horizontal and vertical center of the kernel, and represents the standard deviation of the Gaussian kernel.

[0071] Background sample update. To account for changing background features, background samples are updated periodically. An adaptive learning rate controls how quickly the background model adapts to scene changes, especially changes in motion intensity. The more motion, the lower the learning rate. New background samples are extracted from areas that are not classified as foreground in the refined motion mask. The adaptive mechanism ensures that the background model is updated more slowly when there is significant motion, preventing it from merging moving objects into the background. Adaptive learning rate The calculation is as follows:

[0072]

[0073] in is the binary mask of motion in the current frame, The average value used to calculate the motion mask, indicating the intensity of motion.

[0074] Adaptive threshold. Adjusted according to the average intensity of the image, it determines the minimum number of matches required to classify a pixel as background. The threshold becomes more stringent as the average intensity of the image increases. If the distance between the pixel value of a certain pixel in the remaining frame image and the pixel value of at least one pixel in the corresponding initial background sample is less than the adaptive threshold, the pixel is judged to be a background pixel. Adaptive threshold The calculation is as follows:

[0075]

[0076] in is the average intensity of the current frame. A baseline value of 4 is added to the calculation to ensure a minimum number of matches even in relatively bright scenes.

[0077] Foreground detection. Convert the background model to grayscale, calculate the absolute difference between the sample and the current frame, and compare the matching items with the adaptive threshold to determine the foreground, that is, the pixels in the current frame image that belong to the moving target.

[0078] Shadow removal. Calculate the average hue saturation of the HSV image. If the pixel is lower than the threshold and the hue saturation is the same as the average value, it is a shadow. And the foreground and HSV image are operated “AND” to remove the shadow.

[0079] Edge detection. Find the difference between the current frame and the background frame, apply Canny edge detection to the difference image, and obtain a binary motion mask sequence of the moving object containing only black and white pixels and removing shadows.

[0080] S4: Fuse the semantic segmentation mask and the motion mask to obtain the moving target in each frame; use the target tracking model to track the moving target. Specifically:

[0081] First, the motion mask obtained by ViBe is fused with the semantic segmentation mask obtained by SE-UNet through an “AND” operation to generate a motion detection image with fused semantic information. This enhances the motion detection capability and improves the background distinction capability. Then DeepSORT is used to track the moving target. Figure 3The framework diagram of the DeepSORT target tracking model provided for this embodiment first captures the nonlinear features of the moving target in each frame through UKF; then extracts the deep features of the moving target in each frame through the MobileNetV3 network; finally, the output of UKF and the output of the MobileNetV3 network are dynamically matched to output the state estimate of the tracking result. DeepSORT improves tracking accuracy and robustness by combining deep neural networks. DeepSORT integrates SORT's Kalman filter with a deep learning model for association metrics. This fusion enables DeepSORT to handle challenges such as occlusion and viewpoint changes while maintaining efficiency for real-time applications.

[0082] Using UKF can effectively improve the tracking accuracy of objects under nonlinear motion. There are two stages in predicting the position of the object in the current frame through UKF: state prediction and state update.

[0083] State prediction: UKF uses sigma points propagated by a nonlinear motion model to capture the state variable distribution, and then estimates the mean and covariance of the prediction from these points:

[0084]

[0085]

[0086] in is the predicted state, is the forecast covariance matrix, is a nonlinear motion model. is the process noise covariance; yes The Jacobian matrix of T Indicates transpose.

[0087] State update: Transform the predicted sigma point to get the final estimated state:

[0088]

[0089]

[0090]

[0091] in is the Kalman gain, is the Jacobian matrix of the measurement model, R is the measurement noise covariance, is the actual measurement, It is a nonlinear measurement model.

[0092] In order to extract deep features of the image and make better use of global context information, MobileNetV3 is used for feature extraction. Each detected object has bounding box coordinates. The region of interest (ROI) is extracted from the frame based on the detection coordinates, and then the preprocessed ROI is passed through MobileNetV3 to obtain feature embedding. The obtained feature embedding is used for cross-frame association detection. DeepSORT uses metrics such as cosine distance to determine the similarity between feature embeddings. Given two feature embeddings a and b , the cosine distance is calculated as:

[0093] .

[0094] In order to improve the tracking performance, adaptive matching threshold and adaptive sampling are introduced to dynamically adjust parameters according to the feature changes of the tracked object. The adaptive matching threshold first sets a baseline matching threshold, and then obtains the confidence of the existing trajectory and the new detection. The confidence of the existing trajectory is the degree of trust that the model has in the continued existence of the existing trajectory; the confidence of the new detection is the degree of trust that the model has in the newly detected target, which comes from the output of MobileNetV3; the adaptive matching threshold is calculated based on the two confidences and the basic threshold. The adaptive threshold can be a linear combination of the two confidences and the basic threshold. The weight of the linear combination is learned through training, and then the adaptive threshold is used in the associated detection. Adaptive sampling first sets the baseline sampling rate, and then obtains the trajectory speed and the density information of the current frame. The adaptive sampling rate is calculated based on the obtained trajectory speed, density information and the baseline sampling rate. The adaptive sampling rate can be a linear combination of the trajectory speed, density information and the baseline sampling rate. The weight of the linear combination is learned through training. After the sampling rate is obtained, it can be used in feature extraction. Adaptive tracking allows the system to adapt to changes in object behavior and changes in complex scenes, improving the robustness of tracking. The detection process is adjusted through adaptive sampling, and then the detected target trajectory is adaptively matched with the predicted trajectory, and the successfully matched trajectory is fused with the predicted box, and the state of the target is updated.

[0095] On the motion detection image fused with semantic information output by ViBe segmentation integration, optimized DeepSORT is used for tracking, unscented Kalman filter is used for state estimation, MobileNetV3 is used for enhanced feature extraction, and adaptive sampling and matching threshold are used to adjust parameters. Semantic segmentation, motion detection, and target tracking are integrated into a unified monitoring system.

[0096] The present invention proposes a video surveillance method that integrates motion detection and target tracking. Figure 4As shown in the figure, the method first collects the video sequence, and then each frame image in the video sequence is input into the semantic segmentation model and the motion detection model for semantic segmentation and motion detection; then the semantic segmentation mask and the motion mask are subjected to feature fusion; state prediction and further feature extraction are performed based on the fused features, trajectory matching is performed based on the outputs of the two, and the state prediction is updated according to the trajectory matching results, and finally the tracking result is output. In the semantic segmentation process, feature extraction is performed by introducing context extraction and channel dependency. In the motion detection process, the background in each frame is obtained by background initialization and background update, and then edge detection, foreground detection and shadow removal are performed based on the adaptive threshold. When extracting features from the fused features, deep features and global information are introduced.

[0097] The original video sequence containing the target is obtained through video surveillance. The semantic information and motion state information in the deep features of the image are fully utilized. In order to improve the ability of semantic segmentation to capture contextual information and channel-dependent features, semantic segmentation and motion detection are improved and optimized respectively. U-Net is used to capture contextual information and accurately locate, and channel dependency is modeled through squeezing and excitation modules. At the same time, Gaussian blur is used to improve the computational efficiency of ViBe motion detection and reduce noise, and adaptive thresholds and learning rates are used to improve the model's sensitivity to dynamic environmental changes, especially lighting changes. The obtained semantic segmentation image is then fused into motion detection, which enhances the ability of motion detection and distinguishing moving targets from backgrounds. Finally, DeepSORT is used to track the motion detection image fused with semantic information. In practical applications, the target may not be detected in multiple frames due to rapid deformation and disappearance of the tracked target in consecutive frames, or the predicted target trajectory may differ too much from the actual trajectory due to changes in complex scenes. In order to solve these problems and improve the robustness of DeepSORT tracking in complex scenes, the unscented Kalman filter is used for state estimation to better capture nonlinear features. MobileNetV3 is used to extract the deep features of the image and make better use of global context information. Finally, through the adaptive sampling and adaptive threshold adjustment detection process, the extracted features are used for target detection, the detected target trajectory is adaptively matched with the trajectory predicted by UKF, and the successfully matched trajectory is fused with the predicted frame, and the target state is updated. This method can adapt to the changes in target behavior and complex scenes, and has excellent tracking accuracy and robustness in complex scenes, and can improve computing efficiency and resource utilization, with high adaptability.

[0098] The advantages and positive effects of the above-mentioned method for tracking moving targets in video surveillance include:

[0099] 1. The present invention combines the semantic segmentation architecture U-Net with the SE model, and captures the contextual information and channel dependencies of the scene by segmenting the scene into semantic regions and identifying the object boundaries, thereby improving the accuracy of the semantic segmentation model. 2. The present invention uses Gaussian blur to improve the computational efficiency of motion detection, and effectively reduces image noise, making subsequent motion detection more accurate. At the same time, adaptive learning rates and thresholds are used to update the background and process changes in motion and illumination, thereby enhancing the sensitivity and robustness to dynamic environmental conditions. 3. The method of the present invention optimizes DeepSORT and uses UKF for state estimation, which can effectively improve the tracking accuracy of objects under nonlinear motion. MobileNetV3 is used to extract the deep features of the image to better utilize global contextual information. In addition, adaptive technology is also used for tracking, so that the system can be more adaptable to the behavioral changes of the target and the changes in complex scenes, improve the robustness of tracking, and optimize computing resources to adapt to the changing needs of the system. 4. The present invention integrates semantic segmentation, motion detection, and target tracking into a unified monitoring system, achieving high precision and high efficiency, and ensuring comprehensive coverage and reliable detection capabilities in different monitoring environments.

[0100] The above is a method for tracking a moving target in video surveillance provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding device for tracking a moving target in video surveillance, including:

[0101] The data acquisition module is used to extract each frame of image from the video surveillance data to be processed.

[0102] A semantic segmentation mask acquisition module is used to form an improved UNet network by adding an SE module to the connection part of the encoder and the decoder in the original UNet network; training the improved UNet network to obtain a semantic segmentation model for segmenting moving targets in an image; inputting each frame of the image into the semantic segmentation model respectively, extracting features of the input image through the encoder to obtain a multi-channel semantic feature map; enhancing the feature channels in the multi-channel semantic feature map that respond to the edges of the moving targets through the SE module to obtain an edge-enhanced semantic feature map; decoding the edge-enhanced semantic feature map through the decoder to obtain a semantic segmentation mask that enhances the edges of the moving targets in each frame of the image.

[0103] The motion mask acquisition module is used to construct an initial background model based on the first frame image extracted from the video surveillance data to be processed, and update the initial background model according to the pixel changes in the remaining frame images caused by the scene changes to obtain the motion mask corresponding to the remaining frame images.

[0104] The tracking module is used to fuse the semantic segmentation mask and the motion mask to obtain the moving target in each frame image; and use the target tracking model to track the moving target.

[0105] For the specific definition of the moving target tracking device in video surveillance, please refer to the definition of the moving target tracking method in video surveillance above, which will not be repeated here. Each module in the above-mentioned moving target tracking device in video surveillance can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0106] The present invention also provides a computer device structure. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 A method for tracking moving targets in video surveillance is provided.

[0107] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0108] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A method for tracking a moving target in video surveillance, characterized in that: include: Extract each frame of image from the video surveillance data to be processed; An improved UNet network is formed by adding a SE module to the connection part of the encoder and decoder in the original UNet network; The improved UNet network is trained to obtain a semantic segmentation model for segmenting moving objects in an image; each frame of the image is input into the semantic segmentation model, and features of the input image are extracted by an encoder to obtain a multi-channel semantic feature map; feature channels corresponding to the edges of the moving objects in the multi-channel semantic feature map are enhanced by an SE module to obtain an edge-enhanced semantic feature map; The edge-enhanced semantic feature map is decoded by a decoder to obtain a semantic segmentation mask that enhances the edges of moving objects in each frame of the image; An initial background model is constructed based on a first frame image extracted from the video surveillance data to be processed, and the initial background model is updated according to pixel changes in the remaining frame images caused by scene changes to obtain motion masks corresponding to the remaining frame images; The step of constructing an initial background model based on a first frame image extracted from the video surveillance data to be processed specifically includes: Using a Gaussian blur method to filter the first frame image to remove noise in the first frame image; For each pixel in the filtered first frame image, randomly select multiple pixels from its surrounding area to form an initial background model of the corresponding pixel; The updating of the initial background model according to the pixel changes in the remaining frame images caused by the scene change specifically includes: Perform foreground detection on the pixels in the remaining frame images, and output the foreground detection results in the form of a binary motion mask, where foreground pixels are marked as white and background pixels are marked as black; New pixels are extracted from the background pixels to update the initial background model. During the update process, the update of the initial background model is controlled by an adaptive learning rate to adapt to scene changes. The adaptive learning rate is α: Among them, motion_mask is the binary motion mask corresponding to the current frame image, and mean() is used to calculate the average value of the binary motion mask; The performing foreground detection on the pixels in the remaining frame images includes: Calculate the distance between the pixel value of each pixel in the remaining frame image and the pixel value of each pixel in the corresponding initial background sample; If the distance between the pixel value of a certain pixel in the remaining frame image and the pixel value of at least one pixel in the corresponding initial background sample is less than the adaptive threshold, the pixel is judged to be a background pixel; otherwise, it is classified as a foreground pixel. The adaptive threshold min_matches is: Among them, frame.mean() is the average intensity of the current frame image, and int() means rounding; The semantic segmentation mask and motion mask are fused to obtain the moving target in each frame image; the moving target is tracked using the target tracking model.

2. The method for tracking a moving target in video surveillance according to claim 1, wherein: The SE module is added to the connection part of the encoder and decoder in the original UNet network, and the SE module is added to the output end of each downsampling layer included in the encoder in the original UNet network. The output end of each downsampling layer included in the encoder is connected to the input end of the subsequent sampling layer and the input end of the corresponding upsampling layer in the decoder; the SE module is used to gradually enhance the edge features during the downsampling process.

3. The method for tracking a moving target in video surveillance according to claim 1, wherein: The SE module is added to the connection part of the encoder and the decoder in the original UNet network, and the SE module is added to the input end of each upsampling layer included in the decoder in the original UNet network. The input of each upsampling layer included in the decoder is a spliced ​​feature map obtained by splicing the output of the previous upsampling layer and the output of the corresponding downsampling layer in the encoder; the SE module is used to enhance the edge features in the spliced ​​feature map during the upsampling process.

4. The method for tracking a moving target in video surveillance according to claim 1, wherein: The fusion of the semantic segmentation mask and the motion mask is to fuse the semantic segmentation mask and the motion mask through an AND operation, wherein the AND operation is used to obtain pixels marked by both the semantic segmentation mask and the motion mask.

5. The method for tracking a moving target in video surveillance according to claim 1, wherein: The target tracking model is a DeepSORT target tracking model composed of an unscented Kalman filter and a MobileNetV3 network; wherein the unscented Kalman filter is used to capture the nonlinear characteristics of the moving target in each frame image; and the MobileNetV3 network is used to extract the deep characteristics of the moving target in each frame image; The DeepSORT target tracking model introduces adaptive matching thresholds and adaptive sampling to dynamically adjust parameters according to feature changes of the tracked object; The adaptive matching threshold first sets a reference matching threshold for determining whether an existing motion trajectory and a newly detected motion target trajectory are similar, obtains confidence levels for the existing motion trajectory and the newly detected motion target trajectory, calculates an adaptive matching threshold based on the two obtained confidence levels and the reference matching threshold, and performs association detection on the newly detected motion target trajectory and the existing trajectory based on the adaptive matching threshold; The adaptive sampling sets a base sampling rate, obtains the trajectory speed and density information of the current frame respectively, calculates the adaptive sampling rate according to the obtained trajectory speed, density information and the base sampling rate, and performs feature extraction based on the adaptive sampling rate.

6. A device for tracking a moving target in video surveillance, characterized in that: include: A data acquisition module, used for extracting each frame of image from the video surveillance data to be processed; A semantic segmentation mask acquisition module is configured to form an improved UNet network by adding an SE module to the connection between the encoder and the decoder in the original UNet network; train the improved UNet network to obtain a semantic segmentation model for segmenting moving objects in an image; input each frame of the image into the semantic segmentation model, perform feature extraction on the input image through the encoder, and obtain a multi-channel semantic feature map; and enhance the feature channels in the multi-channel semantic feature map that respond to the edges of the moving objects through the SE module to obtain an edge-enhanced semantic feature map. The edge-enhanced semantic feature map is decoded by a decoder to obtain a semantic segmentation mask that enhances the edges of moving objects in each frame of the image; A motion mask acquisition module is configured to construct an initial background model based on a first frame image extracted from the video surveillance data to be processed, and to update the initial background model according to pixel changes in the remaining frame images caused by scene changes to obtain motion masks corresponding to the remaining frame images; The step of constructing an initial background model based on a first frame image extracted from the video surveillance data to be processed specifically includes: Using a Gaussian blur method to filter the first frame image to remove noise in the first frame image; For each pixel in the filtered first frame image, randomly select multiple pixels from its surrounding area to form an initial background model of the corresponding pixel; The updating of the initial background model according to the pixel changes in the remaining frame images caused by the scene change specifically includes: Perform foreground detection on the pixels in the remaining frame images, and output the foreground detection results in the form of a binary motion mask, where foreground pixels are marked as white and background pixels are marked as black; New pixels are extracted from the background pixels to update the initial background model. During the update process, the update of the initial background model is controlled by an adaptive learning rate to adapt to scene changes. The adaptive learning rate is α: Among them, motion_mask is the binary motion mask corresponding to the current frame image, and mean() is used to calculate the average value of the binary motion mask; The performing foreground detection on the pixels in the remaining frame images includes: Calculate the distance between the pixel value of each pixel in the remaining frame image and the pixel value of each pixel in the corresponding initial background sample; If the distance between the pixel value of a certain pixel in the remaining frame image and the pixel value of at least one pixel in the corresponding initial background sample is less than the adaptive threshold, the pixel is judged to be a background pixel; otherwise, it is classified as a foreground pixel. The adaptive threshold min_matches is: Among them, frame.mean() is the average intensity of the current frame image, and int() means rounding; The tracking module is used to fuse the semantic segmentation mask and the motion mask to obtain the moving target in each frame image; and the moving target is tracked using the target tracking model.

7. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 5 is implemented.