Moving target detection method and system for indoor and outdoor complex scene monitoring video

Abstract

The invention relates to the technical field of image communication processing, in particular to a moving target detection method and system for an indoor and outdoor complex scene surveillance video, and the method comprises the following steps: randomly selecting two frames of original images from the surveillance video to generate corresponding prompt information, and inputting the prompt information into a visual large model; determining a moving target area and a background area of each frame of original image; obtaining current frame image target information; obtaining a previous frame of image after background compensation; obtaining motion information of the current frame image; obtaining current frame image depth target information and current frame image depth motion information; obtaining feature information of the current frame image after fusion; and outputting the final segmentation mask of the current frame image as a moving target detection result. The method and the system provided by the invention have less dependence on training samples, and have very high detection precision under the condition of using less sample training and under various complex scenes.

Description

Technical Field

[0001] The present invention relates to the field of image communication technology, and in particular to a moving target detection method and system for monitoring videos of complex indoor and outdoor scenes. Background Art

[0002] Motion target detection is a fundamental task in computer vision and plays a crucial role in video analysis and processing. Its core task is to process each frame of a video, classify each pixel as a moving target or background, and ultimately output a mask image of the target. Driven by concepts such as smart transportation, smart logistics, and smart cities, a large number of surveillance devices are being deployed in various complex scenarios. Detecting moving targets of interest in surveillance videos can remove a large amount of redundant background information and accurately extract valuable information from the video, facilitating more efficient processing and analysis of subsequent advanced tasks. This places higher demands on the robustness and accuracy of motion target detection algorithms.

[0003] Traditional moving target detection methods include optical flow, inter-frame difference, and background modeling. Optical flow is computationally complex and computationally intensive, making it incapable of meeting real-time requirements. Inter-frame difference is simple and fast, but the determination of pixel differences between consecutive frames can easily lead to missed detections and incomplete targets. Background modeling methods, such as SuBSENSE, can dynamically update the background and accurately detect moving targets in real time, but their modeling complexity is high and they are not adaptable to complex and changing application scenarios.

[0004] Deep learning-based moving object detection models have achieved high detection accuracy in complex scenarios. However, these deep learning models mostly extract image features through deep networks and then perform object detection, without comprehensively considering the target's motion information. In principle, they cannot guarantee that all segmented moving objects are moving objects. For example, GraphMOS, based on instance segmentation and graph learning, can misclassify moving objects and background in some scenarios. Furthermore, most of the most advanced deep learning-based moving object detection methods are supervised methods, requiring a large number of high-quality training samples for model training. For example, the FgSegNet-v2 model experienced a certain degree of performance degradation when trained with fewer frames. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for detecting moving targets in surveillance videos of complex indoor and outdoor scenes. The method uses the output of a large visual model as model prior information and uses a motion information module to perform background compensation on the previous frame of the original image of the current frame to obtain the motion information of the current frame image. The model's dependence on training samples is significantly reduced, and it can be widely used in surveillance videos in various complex scenarios, especially in scenarios where it is difficult to obtain a large number of high-quality training samples. It also effectively avoids the interference of dynamic background on motion information extraction in complex scenarios, thereby improving the accuracy of moving target detection.

[0006] The present invention is achieved through the following technical solutions: A moving target detection method for monitoring videos of complex indoor and outdoor scenes includes the following steps: S1: Manually label two frames of original images from the surveillance video, generate corresponding prompt information, and input it into the visual model; S2: The large visual model segments all N frames of original images in the surveillance video according to the prompt information of the selected two frames of original images, obtains a preliminary segmentation mask, and determines the moving target area and background area of each frame of original image; S3: The visual large model performs local contrast enhancement on the original image of the current frame to be processed to obtain an image with local contrast enhancement of the current frame. The image with local contrast enhancement of the current frame is superimposed on the original image of the current frame according to the preliminary segmentation mask to obtain the target information of the image of the current frame. S4: The motion information module performs background compensation on the previous frame of the original image of the current frame to obtain the previous frame image after background compensation; S5: The motion information module calculates the current frame original image and the previous frame image after background compensation using the frame difference method, and determines whether the image in the relevant area belongs to the background or the moving target based on the calculation result, thereby obtaining the motion information of the current frame image; S6: Input the current frame image target information and the current frame image motion information into corresponding encoders for depth feature extraction, thereby obtaining the current frame image depth target information and the current frame image depth motion information; S7: Inputting the depth target information of the current frame image and the depth motion information of the current frame image into the corresponding feature fusion module, so that the features of the depth target information of the current frame image and the depth motion information of the current frame image are fused at different scales to obtain the fused feature information of the current frame image; S8: Input the fused feature information of the current frame image into the decoder for decoding, and output the final segmentation mask of the current frame image as the moving target detection result.

[0007] Optimally, the prompt information in step S1 includes original image information and corresponding annotated true value image information.

[0008] Furthermore, in step S2, the visual large model segments all N frames of original images in the surveillance video according to the prompt information of the selected two frames of original images to obtain a preliminary segmentation mask as follows: S21: Input all N frames of original images of the surveillance video into the visual big model in sequence; S22: The large visual model is trained using random masking and reconstruction based on the prompt information provided in step S1; S23: After training is completed, the large visual model outputs a preliminary segmentation mask to segment each frame image into the moving target area and the background area.

[0009] Furthermore, in step S3, the visual model performs local contrast enhancement on the original image of the current frame to be processed using the following method: S31: Divide the current frame original image to be processed into multiple local blocks of equal size, and calculate the grayscale frequency of each local block according to formula (1); (1); in: Indicates the gray level is The gray level frequency, Indicates the gray level is The number of gray levels, Represents a local patch The total number of gray levels; S32: Calculate the cumulative probability of the gray level based on the gray level frequency of each local block according to formula (2): (2); in: represents the cumulative probability of gray level, Indicates the total number of gray levels; S33: normalize the cumulative probability of each gray level to obtain the normalized cumulative probability of the gray level, calculate the processed gray value according to formula (3) based on the normalized cumulative probability of the gray level, and perform local contrast enhancement on each local block according to the calculated processed gray value: (3); in: Represents the grayscale value after processing, represents the minimum value of the cumulative probability of the gray level, Indicates the maximum threshold of the set gray value; S34: Splicing all the local small blocks after local contrast enhancement into an image after local contrast enhancement of the current frame.

[0010] Optimized, in step S3, the following method is used to superimpose the image with local contrast enhancement of the current frame onto the original image of the current frame according to the preliminary segmentation mask to obtain the target information of the current frame image: If the area of the original image of the current frame belongs to the moving target area, the corresponding area of the image after local contrast enhancement of the current frame is used to replace the moving target area of the original image of the current frame. If the area of the original image of the current frame belongs to the background area, the background area of the original image of the current frame is retained. The final image information is the target information of the current frame image.

[0011] Furthermore, in step S4, background compensation is performed on the previous frame image of the current frame original image using the following method to obtain the previous frame image after background compensation: S41: The motion information module performs Gaussian blurring of different scales on all original images in the surveillance video and uses a Gaussian difference kernel to build a Gaussian difference pyramid; S42: Take 3×3 adjacent grayscale points in the central layer of the Gaussian difference pyramid and the upper and lower layers adjacent to the central layer, then compare the central grayscale point of the central layer with all adjacent grayscale points to determine whether the central grayscale point of the central layer is an extreme point in the scale space. If the central grayscale point of the central layer is an extreme point in the scale space, the central grayscale point of the central layer is used as a candidate key point to proceed to the next step. If the central grayscale point of the central layer is not an extreme point in the scale space, the central grayscale point of the central layer is screened out. S43: Obtain accurate scales and positions of candidate key points by fitting a three-dimensional quadratic function, and remove edge candidate key points and low-contrast candidate key points to obtain key points; S44: Calculate the gradient magnitude and direction of the grayscale of the adjacent area of each key point according to formula (4), and select the direction with the largest gradient magnitude of the grayscale of the adjacent area of the key point as the main direction of the key point: (4); in: Represents key points The gradient component in the axial direction, represents the coordinates of the key points, Represents the grayscale value of the key point, Represents key points The gradient component in the axial direction, Represents the gradient amplitude of the grayscale of the adjacent area of the key point, Indicates the gradient direction of the grayscale of the adjacent area of the key point; S45: Construct a 16×16 grayscale area with the key point as the center, and rotate the constructed grayscale area to make the grayscale area The axis direction is aligned with the main direction of the key point, and then the constructed grayscale area is divided into 16 sub-areas, each sub-area is 4×4 grayscale, the gradient amplitude and direction of each grayscale are calculated in each sub-area, and an 8-direction gradient histogram is constructed. The gradient histograms of all sub-areas are merged to obtain a 128-dimensional key point feature vector; S46: Calculate the Euclidean distance based on the key point feature vector of the current frame and the key point feature vector of the previous frame according to formula (5), and select the key point with the smallest Euclidean distance as the initial feature point matching pair: (5); in: represents the Euclidean distance, The first key point feature vector of the current frame dimensional component, The first key point feature vector of the previous frame Dimensional component; S47: Divide the current frame image and the previous frame image into set grids, retain the feature points with the highest contrast in each grid, filter out other feature points, and obtain filtered feature point matching pairs; S48: Randomly select four pairs of feature points from the filtered feature point matching pairs and calculate the projection error under the current estimated transmission transformation matrix according to formula (6). The feature point pairs whose projection error under the current estimated transmission transformation matrix is less than the set threshold are regarded as inliers. Multiple rounds of iterations are performed until all feature point pairs are selected. The transmission transformation matrix with the most inliers is selected as the final estimation result: (6); in: Represents the projection error under the currently estimated transmission transformation matrix, represents the two-norm, represents the currently estimated transmission transformation matrix, Indicates the Keypoints in the current frame of the matching pair The coordinates of Indicates the Keypoints from the previous frame in the matching pair coordinates of S49: Based on the final estimation result, the key point coordinates of the previous frame image after background compensation are calculated according to formula (7), thereby obtaining the previous frame image after background compensation: (7); in: Represents the coordinate matrix of the previous frame image after background compensation, Represents the original image coordinate matrix before transformation.

[0012] Furthermore, in step S5, the following method is used to calculate the motion information of the current frame image by using the frame difference method between the current frame original image and the previous frame image after background compensation: S51: The motion information module performs a differential calculation on the grayscale of the current frame original image and the grayscale of the previous frame image after background compensation according to formula (8), and obtains the grayscale difference between the current frame original image and the previous frame image after background compensation: (8); in: Indicates the coordinates of the current frame original image and the previous frame image after background compensation The grayscale difference at Indicates the original image of the current frame at the coordinate point The grayscale at Indicates the coordinate point of the previous frame image after background compensation The grayscale at Indicates time; S52: The original image of the current frame and the image of the previous frame after background compensation are combined at the coordinate point The grayscale difference at the coordinate point is compared with the set threshold. If the original image of the current frame and the previous frame image after background compensation are at the coordinate point The grayscale difference at is less than the set threshold, then the original image of the current frame is placed at the coordinate point The image at is set as the background. If the original image of the current frame and the image of the previous frame after background compensation are at the coordinate point The grayscale difference at is greater than or equal to the set threshold, then the original image of the current frame is placed at the coordinate point The image at is set as the moving target: S53: Output the mask image of the moving target to obtain the motion information of the current frame image.

[0013] Furthermore, the feature fusion module includes a target information feature fusion module and a motion information feature fusion module. In step S7, the following method is used to obtain the fused feature information of the current frame image: S71: The dilated convolution layers of the target information feature fusion module and the motion information feature fusion module respectively receive the current frame image depth target information and the current frame image depth motion information input by the corresponding encoders. The first dilated convolution layer performs 2×2 maximum pooling downsampling on the input current frame image depth target information and the current frame image depth motion information, and then performs 1×1 convolution to obtain feature information after convolution by the first dilated convolution layer. S72: The second atrous convolution layer performs a 3×3 convolution on the input current frame image depth target information and the current frame image depth motion information to obtain feature information after the convolution of the second atrous convolution layer; S73: The third dilated convolution layer concatenates the input current frame image depth target information and the current frame image depth motion information with the feature information after convolution by the second dilated convolution layer, and then performs a 3×3 convolution with a dilation rate of 4 to obtain the feature information after convolution by the third dilated convolution layer; S74: The fourth atrous convolution layer concatenates the input current frame image depth target information and the current frame image depth motion information with the feature information after convolution by the third atrous convolution layer, and then performs a 3×3 convolution with a dilation rate of 8 to obtain the feature information after convolution by the fourth atrous convolution layer; S75: The fifth atrous convolution layer concatenates the input current frame image depth target information and the current frame image depth motion information with the feature information after convolution by the fourth atrous convolution layer, and then performs a 3×3 convolution with a dilation rate of 16 to obtain the feature information after convolution by the fifth atrous convolution layer; S76: Connecting the feature information after convolution of the first dilated convolution layer, the feature information after convolution of the second dilated convolution layer, the feature information after convolution of the third dilated convolution layer, the feature information after convolution of the fourth dilated convolution layer, and the feature information after convolution of the fifth dilated convolution layer along the depth dimension to obtain multi-scale feature information; S77: Process the multi-scale feature information through an instance normalization layer and a spatial Dropout layer to obtain fused feature information of the current frame image.

[0014] Furthermore, in step S8, the feature information after fusion of the current frame image is input into the decoder for decoding using the following method, and the final segmentation mask of the current frame image is output as the result of moving target detection: S81: The fused feature information of the current frame image is input to the decoder; S82: The decoder is provided with a first 3×3 convolutional layer, a first instance normalization layer, a first ReLU activation function, a second 3×3 convolutional layer, a second instance normalization layer, and a second ReLU activation function in sequence, and a first feature attention layer is provided after the first instance normalization layer, and a second feature attention layer is provided after the second instance normalization layer, the first feature attention layer receives the output of the second convolutional layer of the target information encoder and the output of the second convolutional layer of the motion information encoder, and the second feature attention layer receives the output from the first convolutional layer of the target information encoder and the output from the first convolutional layer of the motion information encoder, and fuses the low-level features of the encoder with the high-level features of the decoder; S83: Setting a third 3×3 convolutional layer after the second ReLU activation function to perform feature processing on the fused encoder low-level features and decoder high-level features; S84: A 1×1 convolution layer is set after the third 3×3 convolution layer to perform mapping from the feature space to the image space, thereby outputting a final segmentation mask of the current frame image as a result of moving target detection.

[0015] A motion target detection system for surveillance videos of complex indoor and outdoor scenes, used to execute a motion target detection method for surveillance videos of complex indoor and outdoor scenes as described in any one of the above, which comprises a surveillance video, a large visual model, a motion information module, an encoder, a feature fusion module and a decoder in sequence, the encoder comprising a target information encoder and a motion information encoder, the feature fusion module comprising a target information feature fusion module and a motion information feature fusion module, the surveillance video is used to provide video frames, the large visual model is used to obtain target information of the current frame image, the motion information module is used to obtain motion information of the current frame image, the encoder is used to obtain depth target information and depth motion information of the current frame image, the feature fusion module is used to obtain feature information after fusion of the current frame image, and the decoder is used to decode the fused feature information of the current frame image and output the final segmentation mask of the current frame image as the result of motion target detection.

[0016] Beneficial effects of the invention: The moving target detection method and system for monitoring videos of complex indoor and outdoor scenes provided by the present invention have the following advantages: 1. The large visual model is used to segment all N frames of original images in the surveillance video, determine the moving target area and background area of each original image, and then perform local contrast enhancement on the current frame to be processed. The image with local contrast enhancement is then superimposed on the current frame original image to obtain the target information of the current frame image. This significantly reduces the model's dependence on training samples and can be widely applied to surveillance videos in various complex scenarios, especially scenarios where it is difficult to obtain a large number of high-quality training samples.

[0017] 2. The motion information module performs background compensation on the previous frame of the original image of the current frame to obtain the motion information of the current frame image, thereby avoiding the large amount of noise generated in the motion information calculation due to background movement, effectively avoiding the interference of dynamic background on motion information extraction in complex scenes, and improving the accuracy of moving target detection.

[0018] 3. A feature fusion module is set up between the encoder and decoder, which can perform multi-scale feature fusion of the two parts of features belonging to target information and motion information at different scales and then input them into the decoder, making it easier for the decoder to process the two parts of features at the same time, thereby improving the decoding efficiency and effect of the decoder. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1It is a schematic flow chart of the present invention.

[0020] Figure 2 It is a schematic diagram of the system of the present invention.

[0021] In the figure: 1. Monitoring video; 2. Motion information module; 3. Visual large model; 4. Motion information encoder; 5. Target information encoder; 6. Motion information feature fusion module; 7. Target information feature fusion module; 8. Decoder. DETAILED DESCRIPTION

[0022] A moving target detection method for monitoring videos of complex indoor and outdoor scenes includes the following steps, and its flow chart is as follows: Figure 1 As shown: S1: Manually label two frames of original images from the surveillance video, generate corresponding prompt information, and input it into the visual model; Specifically, the prompt information includes original image information and corresponding annotated true value image information.

[0023] S2: The large visual model segments all N frames of original images in the surveillance video according to the prompt information of the two selected original images, obtains a preliminary segmentation mask, and determines the moving target area and background area of each frame of the original image; The visual large model here can choose the general segmentation large model (SegGPT), which unifies various segmentation tasks into a general context learning framework for segmenting all things in the context.

[0024] SegGPT training is formulated as a contextual colorization problem, where a color map is randomly assigned to each data sample. The goal is to complete different segmentation tasks based on the context, rather than relying on specific colors. After training, SegGPT can perform arbitrary segmentation tasks in images or videos through contextual reasoning.

[0025] Furthermore, in step S2, the visual large model segments all N frames of original images in the surveillance video according to the prompt information of the selected two frames of original images to obtain a preliminary segmentation mask as follows: S21: Input all N frames of original images of the surveillance video into the visual big model in sequence; S22: The large visual model is trained using random masking and reconstruction based on the prompt information provided in step S1; S23: After training is completed, the large visual model outputs a preliminary segmentation mask to segment each frame image into the moving target area and the background area.

[0026] S3: The visual large model performs local contrast enhancement on the original image of the current frame to be processed to obtain an image with local contrast enhancement of the current frame. The image with local contrast enhancement of the current frame is superimposed on the original image of the current frame according to the preliminary segmentation mask to obtain the target information of the image of the current frame. Specifically, the visual large model can use the following method to perform local contrast enhancement on the original image of the current frame to be processed: S31: Divide the current frame original image to be processed into multiple local blocks of equal size, and calculate the grayscale frequency of each local block according to formula (1); (1); in: Indicates the gray level is The gray level frequency, Indicates the gray level is The number of gray levels, Represents a local patch The total number of gray levels; Assuming the image size is M×N, it is divided into P×P blocks. After division, the size of each block is P×P, and the image is divided into (M / P)×(N / P) blocks in total, which allows different areas in the image to have different degrees of contrast enhancement.

[0027] S32: Calculate the cumulative probability of the gray level based on the gray level frequency of each local block according to formula (2): (2); in: represents the cumulative probability of gray level, Indicates the total number of gray levels; The frequency of occurrence of each grayscale level in each local block in the image block can be plotted as a grayscale histogram. A histogram is a statistical representation of the grayscale value distribution, showing the frequency of occurrence of each grayscale level in the image block. Assume that the grayscale value range of the grayscale image is [0, L-1], where L is the maximum grayscale value threshold, generally 256.

[0028] The cumulative probability of the gray level is calculated and a gray level histogram is drawn. The gray level is redistributed by equalizing the block histogram so that the distribution of gray values is more uniform, thereby improving the contrast of the image.

[0029] S33: normalize the cumulative probability of each gray level to obtain the normalized cumulative probability of the gray level, calculate the processed gray value according to formula (3) based on the normalized cumulative probability of the gray level, and perform local contrast enhancement on each local block according to the calculated processed gray value: (3); in: Represents the grayscale value after processing, represents the minimum value of the cumulative probability of the gray level, Indicates the maximum threshold of the set gray value; Through normalization processing, the cumulative probability of gray levels can be normalized to a value within the range of [0, 255], making the contrast enhancement effect of the image more uniform.

[0030] S34: Splicing all the local small blocks after local contrast enhancement into an image after local contrast enhancement of the current frame.

[0031] The following method can be used to superimpose the image with local contrast enhancement of the current frame onto the original image of the current frame to obtain the target information of the current frame image: If the area of the original image of the current frame belongs to the moving target area, the corresponding area of the image after local contrast enhancement of the current frame is used to replace the moving target area of the original image of the current frame. If the area of the original image of the current frame belongs to the background area, the background area of the original image of the current frame is retained. The final image information is the target information of the current frame image.

[0032] The above-mentioned image superposition method can effectively reduce the model's dependence on training samples, because the preliminary segmentation mask output by the large visual model provides rich prior information for the subsequent deep neural network, making it easier for the model to learn the characteristics of moving targets. The evaluation indicators of the detection accuracy of the final output segmentation mask will be greatly improved.

[0033] S4: The motion information module performs background compensation on the previous frame of the original image of the current frame to obtain the previous frame image after background compensation; Specifically, the following method can be used to perform background compensation on the previous frame image of the current frame original image to obtain the previous frame image after background compensation: S41: The motion information module performs Gaussian blurring of different scales on all original images in the surveillance video. Perform Gaussian blurring at different scales, i.e., create a Gaussian pyramid to accurately detect feature points in the scale space. Then, downsample the image and use the Gaussian difference kernel to build a Gaussian difference pyramid to extract key feature points in the image. S42: Take 3×3 adjacent grayscale points in the central layer of the Gaussian difference pyramid and the upper and lower layers adjacent to the central layer, then compare the central grayscale point of the central layer with all adjacent grayscale points to determine whether the central grayscale point of the central layer is an extreme point in the scale space. If the central grayscale point of the central layer is an extreme point in the scale space, the central grayscale point of the central layer is used as a candidate key point to proceed to the next step. If the central grayscale point of the central layer is not an extreme point in the scale space, the central grayscale point of the central layer is screened out. S43: Obtain accurate scales and positions of candidate key points by fitting a three-dimensional quadratic function, and remove edge candidate key points and low-contrast candidate key points to obtain key points; Fitting through a three-dimensional quadratic function can more accurately locate the extreme points in the dimension space, remove edge candidate key points and low-contrast candidate key points, and improve the stability and noise resistance of the key points.

[0034] S44: Calculate the gradient magnitude and direction of the grayscale of the adjacent area of each key point according to formula (4), and select the direction with the largest gradient magnitude of the grayscale of the adjacent area of the key point as the main direction of the key point: (4); in: Indicates key points The gradient component in the axial direction, represents the coordinates of the key points, Represents the grayscale value of the key point, Represents key points The gradient component in the axial direction, Represents the gradient amplitude of the grayscale of the adjacent area of the key point, Indicates the gradient direction of the grayscale of the adjacent area of the key point; S45: Construct a 16×16 grayscale area with the key point as the center, and rotate the constructed grayscale area to make the grayscale area The axis direction is aligned with the main direction of the key point, and then the constructed grayscale area is divided into 16 sub-areas, each sub-area is 4×4 grayscale, the gradient amplitude and direction of each grayscale are calculated in each sub-area, and an 8-direction gradient histogram is constructed. The gradient histograms of all sub-areas are merged to obtain a 128-dimensional key point feature vector; S46: Calculate the Euclidean distance based on the key point feature vector of the current frame and the key point feature vector of the previous frame according to formula (5), and select the key point with the smallest Euclidean distance as the initial feature point matching pair: (5); in: represents the Euclidean distance, The first key point feature vector of the current frame dimensional component, The first key point feature vector of the previous frame Dimensional component; The above method is used to obtain the initial feature point matching pairs, which can make the distribution of feature points as uniform as possible while ensuring that the number of feature points is sufficient.

[0035] S47: Divide the current frame image and the previous frame image into set grids, retain the feature points with the highest contrast in each grid, filter out other feature points, and obtain filtered feature point matching pairs; S48: Randomly select four pairs of feature points from the filtered feature point matching pairs and calculate the projection error under the current estimated transmission transformation matrix according to formula (6). The feature point pairs whose projection error under the current estimated transmission transformation matrix is less than the set threshold are regarded as inliers. Multiple rounds of iterations are performed until all feature point pairs are selected. The transmission transformation matrix with the most inliers is selected as the final estimation result: (6); in: Represents the projection error under the currently estimated transmission transformation matrix, represents the two-norm, represents the currently estimated transmission transformation matrix, Indicates the Keypoints in the current frame of the matching pair The coordinates of Indicates the Match the key points of the previous frame coordinates of S49: Based on the final estimation result, the key point coordinates of the previous frame image after background compensation are calculated according to formula (7), thereby obtaining the previous frame image after background compensation: (7); in: Represents the coordinate matrix of the previous frame image after background compensation, Represents the original image coordinate matrix before transformation.

[0036] The above-mentioned background compensation method can effectively avoid the interference of dynamic background on motion information extraction in complex scenes. Compared with using the frame difference method alone to extract motion information, using background compensation and then using the frame difference method to extract motion information greatly improves the evaluation index (F-Measure) of detection accuracy.

[0037] S5: The motion information module calculates the current frame original image and the previous frame image after background compensation using the frame difference method, and determines whether the image in the relevant area belongs to the background or the moving target based on the calculation result, thereby obtaining the motion information of the current frame image; Furthermore, in step S5, the following method is used to calculate the motion information of the current frame image by using the frame difference method between the current frame original image and the previous frame image after background compensation: S51: The motion information module performs a differential calculation on the grayscale of the current frame original image and the grayscale of the previous frame image after background compensation according to formula (8), and obtains the grayscale difference between the current frame original image and the previous frame image after background compensation: (8); in: Indicates the coordinates of the current frame original image and the previous frame image after background compensation The grayscale difference at Indicates the original image of the current frame at the coordinate point The grayscale at Indicates the coordinate point of the previous frame image after background compensation The grayscale at Indicates time; S52: The original image of the current frame and the image of the previous frame after background compensation are combined at the coordinate point The grayscale difference at the coordinate point is compared with the set threshold. If the original image of the current frame and the previous frame image after background compensation are at the coordinate point The grayscale difference at is less than the set threshold, then the original image of the current frame is placed at the coordinate point The image at is set as the background. If the original image of the current frame and the image of the previous frame after background compensation are at the coordinate point The grayscale difference at is greater than or equal to the set threshold, then the original image of the current frame is placed at the coordinate point The image at is set as the moving target: The grayscale difference between the current frame original image and the previous frame image after background compensation can be binarized using the following formula: ; in: It represents the result of binarization of the grayscale difference between the original image of the current frame and the previous frame image after background compensation. Indicates setting the threshold. Indicates the movement target, Indicates background.

[0038] S53: Output the mask image of the moving target to obtain the motion information of the current frame image.

[0039] The motion information module is used to perform background compensation on the previous frame of the current frame original image to obtain the motion information of the current frame image, thereby avoiding the large amount of noise generated in the motion information calculation due to background movement, effectively avoiding the interference of dynamic background on motion information extraction in complex scenes, and improving the accuracy of moving target detection.

[0040] S6: Input the current frame image target information and the current frame image motion information into corresponding encoders for depth feature extraction, thereby obtaining the current frame image depth target information and the current frame image depth motion information; The encoder is designed based on the pretrained VGG16 network and includes a target information encoder and a motion information encoder. The target information encoder compresses the VGG16 encoder size, using only the first four blocks of the VGG16 encoder and removing the fifth block. Pooling layers are only set in the third and fourth blocks, and dropout layers are inserted between each layer of the fourth block. Pooling layers in the third and fourth blocks balance the semantic information and resolution of feature maps and ensure that the feature fusion module can integrate sufficient multi-channel and multi-scale information. Dropout layers are inserted between each layer of the fourth block to prevent overfitting. Because the target information encoder compresses the encoder size, processing efficiency is improved.

[0041] Since the motion information encoder only needs to extract features from simple grayscale images, it can only retain the first three blocks of the VGG16 encoder, further compressing the encoder size and improving the processing efficiency of the encoder.

[0042] S7: Inputting the depth target information of the current frame image and the depth motion information of the current frame image into the corresponding feature fusion module, so that the features of the depth target information of the current frame image and the depth motion information of the current frame image are fused at different scales to obtain the fused feature information of the current frame image; Specifically, the following method can be used to obtain the feature information after the current frame image is fused: S71: The dilated convolution layers of the target information feature fusion module and the motion information feature fusion module respectively receive the current frame image depth target information and the current frame image depth motion information input by the corresponding encoders. The first dilated convolution layer performs 2×2 maximum pooling downsampling on the input current frame image depth target information and the current frame image depth motion information, and then performs 1×1 convolution to obtain feature information after convolution by the first dilated convolution layer. S72: The second atrous convolution layer performs a 3×3 convolution on the input current frame image depth target information and the current frame image depth motion information to obtain feature information after the convolution of the second atrous convolution layer; S73: The third dilated convolution layer concatenates the input current frame image depth target information and the current frame image depth motion information with the feature information after convolution by the second dilated convolution layer, and then performs a 3×3 convolution with a dilation rate of 4 to obtain the feature information after convolution by the third dilated convolution layer; S74: The fourth atrous convolution layer concatenates the input current frame image depth target information and the current frame image depth motion information with the feature information after convolution by the third atrous convolution layer, and then performs a 3×3 convolution with a dilation rate of 8 to obtain the feature information after convolution by the fourth atrous convolution layer; S75: The fifth atrous convolution layer concatenates the input current frame image depth target information and the current frame image depth motion information with the feature information after convolution by the fourth atrous convolution layer, and then performs a 3×3 convolution with a dilation rate of 16 to obtain the feature information after convolution by the fifth atrous convolution layer; S76: Connecting the feature information after convolution of the first dilated convolution layer, the feature information after convolution of the second dilated convolution layer, the feature information after convolution of the third dilated convolution layer, the feature information after convolution of the fourth dilated convolution layer, and the feature information after convolution of the fifth dilated convolution layer along the depth dimension to obtain multi-scale feature information; S77: Processing the multi-scale feature information through an instance normalization layer and a spatial Dropout layer to obtain feature information after fusion of the current frame image. Processing the multi-scale feature information through an instance normalization layer and a spatial Dropout layer can improve segmentation performance and prevent overfitting.

[0043] The feature fusion module uses dilated convolutional layers of different resolutions to extract multi-scale information, and then connects the corresponding features of the same scale along the channel dimension to obtain multi-scale feature information, which makes it easier for the decoder to process target information features and motion information features together, thereby improving the decoding efficiency and decoding effect of subsequent decoders.

[0044] S8: Input the fused feature information of the current frame image into the decoder for decoding, and output the final segmentation mask of the current frame image as the moving target detection result.

[0045] Specifically, the following method can be used to input the fused feature information of the current frame image into the decoder for decoding, and output the final segmentation mask of the current frame image as the result of moving target detection: S81: The fused feature information of the current frame image is input to the decoder; S82: The decoder is provided with a first 3×3 convolutional layer, a first instance normalization layer, a first ReLU activation function, a second 3×3 convolutional layer, a second instance normalization layer, and a second ReLU activation function in sequence, and a first feature attention layer is provided after the first instance normalization layer, and a second feature attention layer is provided after the second instance normalization layer, the first feature attention layer receives the output of the second convolutional layer of the target information encoder and the output of the second convolutional layer of the motion information encoder, and the second feature attention layer receives the output from the first convolutional layer of the target information encoder and the output from the first convolutional layer of the motion information encoder, and fuses the low-level features of the encoder with the high-level features of the decoder; Two feature attention layers fuse the decoder and encoder features together through a dense attention process, where high-level features provide global information and guide the feature attention layer to weight appropriate low-level features, which helps to predict the encoder features in the input image. The encoder features are reweighted and connected to the decoder layer at the pixel level.

[0046] The decoding process is from the previous decoding layer to the next decoding layer. The input part is the output of the corresponding layers of the target information encoder and the motion information encoder and the output of the previous layer of the decoder. The output of the feature attention layer is downsampled and output to the next layer of the decoder.

[0047] Specifically, the corresponding layers of the target information encoder and the motion information encoder output the target information feature map tensor and the motion information feature map tensor respectively, and then connect the target information feature map tensor and the motion information feature map tensor together; after the connection, the number of channels of the feature map becomes twice as large as before, and then the feature map is obtained through the ReLU activation function and the convolution layer, that is, the feature map is obtained by the following formula: ; in: represents the feature map, Indicates that the convolution kernel operation is performed on the contents in the brackets. Indicates that the contents in brackets are Activation processing, Indicates that the contents in the brackets are connected. Indicates the current frame target information, Indicates the current frame motion information; The output of the previous layer of the decoder passes through the same convolutional layer and Sigmoid layer to obtain the weight coefficient mapping , then Multiply pixel by pixel with the feature map to get the weighted feature map , that is, the weighted feature map is obtained by the following formula: in: Indicates normalization of the content instances in the brackets. Indicates that the contents in brackets are Activation processing, represents pixel-by-pixel multiplication, Represents the output of the previous layer of the decoder, Connect the weighted feature map and the output of the previous layer of the decoder, and get the output of the next layer of the decoder after upsampling, which is the following formula: ; in: Represents the next layer output of the decoder; Then use the following formula to calculate the binary cross entropy function and compare the true value with the predicted value: ; in: represents the binary cross entropy function, For the In the example The true label of the pixel at each position, For the In the example The predicted probability value of the pixel at position Indicates the total number of instances.

[0048] The feature attention layer uses information from the encoder to assist decoding in the above manner, which is beneficial to the final output of an accurate segmentation mask.

[0049] S83: Setting a third 3×3 convolutional layer after the second ReLU activation function to perform feature processing on the fused encoder low-level features and decoder high-level features; S84: A 1×1 convolution layer is set after the third 3×3 convolution layer to perform mapping from the feature space to the image space, thereby outputting a final segmentation mask of the current frame image as a result of moving target detection.

[0050] A moving target detection system for complex indoor and outdoor scene surveillance videos Figure 2As shown, a motion target detection method for monitoring videos of complex indoor and outdoor scenes as described in any one of the above is used to execute, which includes a monitoring video 1, a visual large model 3, a motion information module 2, an encoder, a feature fusion module and a decoder 8 in sequence, wherein the encoder includes a target information encoder 5 and a motion information encoder 4, the feature fusion module includes a target information feature fusion module 7 and a motion information feature fusion module 6, the monitoring video is used to provide video frames, the visual large model is used to obtain the target information of the current frame image, the motion information module is used to obtain the motion information of the current frame image, the encoder is used to obtain the depth target information and the depth motion information of the current frame image, the feature fusion module is used to obtain the fused feature information of the current frame image, and the decoder is used to decode the fused feature information of the current frame image and output the final segmentation mask of the current frame image as the result of motion target detection.

[0051] In summary, the motion target detection method and system for monitoring videos of complex indoor and outdoor scenes provided by the present invention have low dependence on training samples and have high detection accuracy under conditions of using fewer samples for training and in various complex scenes.

[0052] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A moving target detection method for monitoring videos of complex indoor and outdoor scenes, characterized by: The steps include: S1: Manually label two frames of original images from the surveillance video, generate corresponding prompt information, and input it into the visual model; S2: The large visual model segments all N frames of original images in the surveillance video according to the prompt information of the two selected original images, obtains a preliminary segmentation mask, and determines the moving target area and background area of each frame of the original image; S3: The visual large model performs local contrast enhancement on the original image of the current frame to be processed to obtain an image with local contrast enhancement of the current frame. The image with local contrast enhancement of the current frame is superimposed on the original image of the current frame according to the preliminary segmentation mask to obtain the target information of the image of the current frame. S4: The motion information module performs background compensation on the previous frame of the original image of the current frame to obtain the previous frame image after background compensation; S5: The motion information module calculates the current frame original image and the previous frame image after background compensation using the frame difference method, and determines whether the image in the relevant area belongs to the background or the moving target based on the calculation result, thereby obtaining the motion information of the current frame image; S6: Input the current frame image target information and the current frame image motion information into the corresponding encoder respectively to extract the depth feature, and obtain the current frame image depth target information and the current frame image depth motion information; S7: Inputting the depth target information of the current frame image and the depth motion information of the current frame image into the corresponding feature fusion module, so that the features of the depth target information of the current frame image and the depth motion information of the current frame image are fused at different scales to obtain fused feature information of the current frame image; S8: Input the fused feature information of the current frame image into the decoder for decoding, and output the final segmentation mask of the current frame image as the moving target detection result.

2. The method for detecting moving targets in complex indoor and outdoor scene surveillance videos according to claim 1, characterized in that: The prompt information in step S1 includes original image information and corresponding annotated true value image information.

3. The method for detecting moving targets in complex indoor and outdoor scene surveillance videos according to claim 1, characterized in that: In step S2, the visual model segments all N frames of original images in the surveillance video according to the prompt information of the selected two frames of original images to obtain the preliminary segmentation mask as follows: S21: Input all N frames of original images of the surveillance video into the visual big model in sequence; S22: The large visual model is trained using random masking and reconstruction based on the prompt information provided in step S1; S23: After training is completed, the large visual model outputs a preliminary segmentation mask to segment each frame image into the moving target area and the background area.

4. The method for detecting moving targets in complex indoor and outdoor scene surveillance videos according to claim 1, characterized in that: In step S3, the visual model performs local contrast enhancement on the original image of the current frame to be processed using the following method: S31: Divide the current frame original image to be processed into multiple local blocks of equal size, and calculate the grayscale frequency of each local block according to formula (1); （1）； in: Indicates the gray level is The gray level frequency, Indicates the gray level is The number of gray levels, Represents a local patch The total number of gray levels; S32: Calculate the cumulative probability of the gray level based on the gray level frequency of each local block according to formula (2): （2）； in: represents the cumulative probability of gray level, Indicates the total number of gray levels; S33: normalize the cumulative probability of each gray level to obtain the normalized cumulative probability of the gray level, calculate the processed gray value according to formula (3) based on the normalized cumulative probability of the gray level, and perform local contrast enhancement on each local block according to the calculated processed gray value: （3）； in: Represents the grayscale value after processing, represents the minimum value of the cumulative probability of the gray level, Indicates the maximum threshold of the set gray value; S34: Splicing all the local small blocks after local contrast enhancement into an image after local contrast enhancement of the current frame.

5. The method for detecting moving targets in complex indoor and outdoor scene surveillance videos according to claim 4, characterized in that: In step S3, the image with enhanced local contrast of the current frame is superimposed on the original image of the current frame according to the preliminary segmentation mask to obtain the target information of the current frame image: If the area of the original image of the current frame belongs to the moving target area, the corresponding area of the image after local contrast enhancement of the current frame is used to replace the moving target area of the original image of the current frame. If the area of the original image of the current frame belongs to the background area, the background area of the original image of the current frame is retained. The final image information is the target information of the current frame image.

6. The method for detecting moving targets in complex indoor and outdoor scene surveillance videos according to claim 5, characterized in that: In step S4, the background compensation is performed on the previous frame image of the current frame original image using the following method to obtain the previous frame image after background compensation: S41: The motion information module performs Gaussian blurring of different scales on all original images in the surveillance video and uses a Gaussian difference kernel to build a Gaussian difference pyramid; S42: Take 3×3 adjacent grayscale points in the central layer of the Gaussian difference pyramid and the upper and lower layers adjacent to the central layer, then compare the central grayscale point of the central layer with all adjacent grayscale points to determine whether the central grayscale point of the central layer is an extreme point in the scale space. If the central grayscale point of the central layer is an extreme point in the scale space, the central grayscale point of the central layer is used as a candidate key point to proceed to the next step. If the central grayscale point of the central layer is not an extreme point in the scale space, the central grayscale point of the central layer is screened out. S43: Obtain accurate scales and positions of candidate key points by fitting a three-dimensional quadratic function, and remove edge candidate key points and low-contrast candidate key points to obtain key points; S44: Calculate the gradient magnitude and direction of the grayscale of the adjacent area of each key point according to formula (4), and select the direction with the largest gradient magnitude of the grayscale of the adjacent area of the key point as the main direction of the key point: （4）； in: Represents key points The gradient component in the axial direction, represents the coordinates of the key points, Represents the grayscale value of the key point, Represents key points The gradient component in the axial direction, Represents the gradient amplitude of the grayscale of the adjacent area of the key point, Indicates the gradient direction of the grayscale of the adjacent area of the key point; S45: Construct a 16×16 grayscale area with the key point as the center, and rotate the constructed grayscale area to make the grayscale area The axis direction is aligned with the main direction of the key point, and then the constructed grayscale area is divided into 16 sub-areas, each sub-area is 4×4 grayscale, the gradient amplitude and direction of each grayscale are calculated in each sub-area, and an 8-direction gradient histogram is constructed. The gradient histograms of all sub-areas are merged to obtain a 128-dimensional key point feature vector; S46: Calculate the Euclidean distance based on the key point feature vector of the current frame and the key point feature vector of the previous frame according to formula (5), and select the key point with the smallest Euclidean distance as the initial feature point matching pair: （5）； in: represents the Euclidean distance, The first key point feature vector of the current frame dimensional component, The first key point feature vector of the previous frame Dimensional component; S47: Divide the current frame image and the previous frame image into set grids, retain the feature points with the highest contrast in each grid, filter out other feature points, and obtain filtered feature point matching pairs; S48: Randomly select four pairs of feature points from the filtered feature point matching pairs and calculate the projection error under the current estimated transmission transformation matrix according to formula (6). The feature point pairs whose projection error under the current estimated transmission transformation matrix is less than the set threshold are regarded as inliers. Multiple rounds of iterations are performed until all feature point pairs are selected. The transmission transformation matrix with the most inliers is selected as the final estimation result: （6）； in: Represents the projection error under the currently estimated transmission transformation matrix, represents the two-norm, represents the currently estimated transmission transformation matrix, Indicates the Keypoints in the current frame of the matching pair The coordinates of Indicates the Keypoints from the previous frame in the matching pair coordinates of S49: Based on the final estimation result, the key point coordinates of the previous frame image after background compensation are calculated according to formula (7), thereby obtaining the previous frame image after background compensation: （7）； in: Represents the coordinate matrix of the previous frame image after background compensation, Represents the original image coordinate matrix before transformation.

7. The method for detecting moving targets in complex indoor and outdoor scene surveillance videos according to claim 6, characterized in that: In step S5, the following method is used to calculate the motion information of the current frame image by using the frame difference method between the current frame original image and the previous frame image after background compensation: S51: The motion information module performs a differential calculation on the grayscale of the current frame original image and the grayscale of the previous frame image after background compensation according to formula (8), and obtains the grayscale difference between the current frame original image and the previous frame image after background compensation: （8）； in: Indicates the coordinates of the current frame original image and the previous frame image after background compensation The grayscale difference at Indicates the original image of the current frame at the coordinate point The grayscale at Indicates the coordinate point of the previous frame image after background compensation The grayscale at Indicates time; S52: The original image of the current frame and the image of the previous frame after background compensation are combined at the coordinate point The grayscale difference at the coordinate point is compared with the set threshold. If the original image of the current frame and the previous frame image after background compensation are at the coordinate point The grayscale difference at is less than the set threshold, then the original image of the current frame is placed at the coordinate point The image at is set as the background. If the original image of the current frame and the image of the previous frame after background compensation are at the coordinate point The grayscale difference at the point is greater than or equal to the set threshold, then the original image of the current frame is placed at the coordinate point The image at is set as the moving target: S53: Output the mask image of the moving target to obtain the motion information of the current frame image.

8. The method for detecting moving targets in surveillance videos of complex indoor and outdoor scenes according to claim 7, characterized in that: The feature fusion module includes a target information feature fusion module and a motion information feature fusion module. In step S7, the following method is used to obtain the fused feature information of the current frame image: S71: The dilated convolution layers of the target information feature fusion module and the motion information feature fusion module respectively receive the current frame image depth target information and the current frame image depth motion information input by the corresponding encoders. The first dilated convolution layer performs 2×2 maximum pooling downsampling on the input current frame image depth target information and the current frame image depth motion information, and then performs 1×1 convolution to obtain feature information after convolution by the first dilated convolution layer. S72: The second atrous convolution layer performs a 3×3 convolution on the input current frame image depth target information and the current frame image depth motion information to obtain feature information after the convolution of the second atrous convolution layer; S73: The third dilated convolution layer concatenates the input current frame image depth target information and the current frame image depth motion information with the feature information after convolution by the second dilated convolution layer, and then performs a 3×3 convolution with a dilation rate of 4 to obtain the feature information after convolution by the third dilated convolution layer; S74: The fourth atrous convolution layer concatenates the input current frame image depth target information and the current frame image depth motion information with the feature information after convolution by the third atrous convolution layer, and then performs a 3×3 convolution with a dilation rate of 8 to obtain the feature information after convolution by the fourth atrous convolution layer; S75: The fifth atrous convolution layer concatenates the input current frame image depth target information and the current frame image depth motion information with the feature information after convolution by the fourth atrous convolution layer, and then performs a 3×3 convolution with a dilation rate of 16 to obtain the feature information after convolution by the fifth atrous convolution layer; S76: Connecting the feature information after convolution of the first dilated convolution layer, the feature information after convolution of the second dilated convolution layer, the feature information after convolution of the third dilated convolution layer, the feature information after convolution of the fourth dilated convolution layer, and the feature information after convolution of the fifth dilated convolution layer along the depth dimension to obtain multi-scale feature information; S77: Process the multi-scale feature information through an instance normalization layer and a spatial dropout layer to obtain fused feature information of the current frame image.

9. The method for detecting moving targets in complex indoor and outdoor scene surveillance videos according to claim 7, characterized in that: In step S8, the following method is used to input the fused feature information of the current frame image into the decoder for decoding, and the final segmentation mask of the current frame image is output as the result of moving target detection: S81: The fused feature information of the current frame image is input to the decoder; S82: The decoder is provided with a first 3×3 convolutional layer, a first instance normalization layer, a first ReLU activation function, a second 3×3 convolutional layer, a second instance normalization layer, and a second ReLU activation function in sequence, and a first feature attention layer is provided after the first instance normalization layer, and a second feature attention layer is provided after the second instance normalization layer, the first feature attention layer receives the output of the second convolutional layer of the target information encoder and the output of the second convolutional layer of the motion information encoder, and the second feature attention layer receives the output from the first convolutional layer of the target information encoder and the output from the first convolutional layer of the motion information encoder, and fuses the low-level features of the encoder with the high-level features of the decoder; S83: Setting a third 3×3 convolutional layer after the second ReLU activation function to perform feature processing on the fused encoder low-level features and decoder high-level features; S84: A 1×1 convolution layer is set after the third 3×3 convolution layer to perform mapping from the feature space to the image space, thereby outputting a final segmentation mask of the current frame image as a result of moving target detection.

10. A moving target detection system for monitoring videos of complex indoor and outdoor scenes, characterized by: Used to execute a motion target detection method for monitoring videos of complex indoor and outdoor scenes as described in any one of claims 1 to 9, which comprises a monitoring video, a visual large model, a motion information module, an encoder, a feature fusion module and a decoder in sequence, the encoder comprising a target information encoder and a motion information encoder, the feature fusion module comprising a target information feature fusion module and a motion information feature fusion module, the monitoring video is used to provide video frames, the visual large model is used to obtain the target information of the current frame image, the motion information module is used to obtain the motion information of the current frame image, the encoder is used to obtain the depth target information of the current frame image and the depth motion information of the current frame image, the feature fusion module is used to obtain the fused feature information of the current frame image, the decoder is used to decode the fused feature information of the current frame image, and output the final segmentation mask of the current frame image as the result of motion target detection.

Images