Video moving object detection, foreground and background separation method, device, equipment and medium

Abstract

The embodiment of the specification discloses a video moving object detection method, which comprises the following steps: cutting a sample video containing a moving object into sample image frames, and training an image classification model through the sample image frames; cutting a target video into image frames, determining the complexity of any image frame by using the image classification model, and separating the foreground and the background of the image frame by using a foreground and background separation algorithm corresponding to the complexity of the image frame to obtain a foreground image corresponding to the image frame; forming a foreground video from the foreground images corresponding to the image frames, and taking the foreground video as a detection result of the moving object in the target video.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to methods, apparatus, devices and media for video moving object detection and foreground / background separation. Background Technology

[0002] The field of computer vision has seen rapid development in recent years, with the AI ​​wave sweeping through academia and industry. Moving object detection is a crucial task in computer vision, referring to the extraction of targets of interest from video sequences, typically moving objects such as people, vehicles, and animals. It involves two main tasks: target localization and target tracking. Target localization involves accurately determining the target's position and bounding box in each video frame, while target tracking involves tracing the target's trajectory and motion across consecutive video frames.

[0003] Commonly used methods include background-based modeling, optical flow-based, feature-based, and deep learning-based methods. Background-based modeling methods assume the background in the video sequence is static and separate foreground objects from the background by modeling and updating the background model. Common background modeling methods include Gaussian Mixture Models (GMM), adaptive background modeling, and pixel-difference-based methods. Optical flow-based methods utilize motion information between pixels to detect and track objects. Common optical flow algorithms include Lucas-Kanade optical flow, Horn-Schunck optical flow, and dense optical flow-based methods. Feature-based methods extract features from the image and use these features to detect and track objects. Commonly used features include color features, texture features, shape features, and motion features. Common feature extraction algorithms include Histogram of Oriented Gradients (HOG), Scale-Invariant Feature Transform (SIFT), and Speeded Up Robust Features (SURF). Deep learning-based methods learn the feature representations and spatial relationships of targets by training on large-scale labeled data, such as Faster R-CNN, YOLO (You Only LookOnce), and SSD (Single Shot MultiBox Detector).

[0004] However, although the above methods can detect moving objects to a certain extent, the detection effect still needs to be improved. Summary of the Invention

[0005] This specification provides a method, apparatus, device, and medium for detecting moving objects in videos and separating foreground and background, addressing both the technical problem of how to detect moving objects in videos more effectively and efficiently, and the technical problem of how to separate foreground and background in videos more effectively and efficiently.

[0006] To address the aforementioned technical problems, the embodiments in this specification provide the following technical solutions:

[0007] This specification provides an embodiment of a video moving object detection method, the method comprising:

[0008] The sample video containing moving objects is segmented into sample image frames, and an image classification model is trained using the sample image frames.

[0009] The target video is divided into image frames. For any image frame, the complexity of the image frame is determined using the image classification model. The foreground and background separation algorithm corresponding to the complexity of the image frame is used to separate the foreground and background of the image frame to obtain the foreground image corresponding to the image frame.

[0010] A foreground video is formed from the foreground images corresponding to each image frame, and the foreground video serves as the detection result of moving objects in the target video.

[0011] Optionally, training the image classification model using the sample image frames includes:

[0012] One or more rounds of model training are performed using the sample image frames, and the model obtained after one or more rounds of model training is used as the image classification model.

[0013] The first round of model training includes: inputting one or more sample image frames into the original model, having the original model calculate the complexity of the input sample image frames, comparing the complexity calculated by the original model with the actual complexity of the input sample image frames, updating the parameters of the original model based on the comparison result, and obtaining a new model.

[0014] Any non-first-round model training operation includes: inputting one or more sample image frames into the latest obtained model, calculating the complexity of the input sample image frames by the latest obtained model, comparing the complexity calculated by the latest obtained model with the actual complexity of the input sample image frames, updating the parameters of the latest obtained model based on the comparison result, and obtaining a new model.

[0015] Optionally, the method further includes:

[0016] If the complexity of an image frame indicates that the image frame does not contain moving objects from the target video, then the foreground / background separation algorithm corresponding to the complexity of the image frame is a Gaussian mixture model (MOG).

[0017] And / or,

[0018] If the complexity of an image frame indicates that the image frame contains moving objects from the target video, and the number of moving objects in the image frame is within a first preset range, then the foreground and background separation algorithm corresponding to the complexity of the image frame is the K-nearest neighbor algorithm.

[0019] And / or,

[0020] If the complexity of an image frame indicates that the image frame contains moving objects from the target video, and the number of moving objects in the image frame is within a second preset range, then the foreground / background separation algorithm corresponding to the complexity of the image frame is the Vibe algorithm.

[0021] Optionally, forming the foreground video from the foreground images corresponding to each image frame includes:

[0022] For any image frame, perform a binarization operation on the foreground image corresponding to that image frame to obtain a binary foreground image;

[0023] If the foreground binary image is determined to be usable, then a noise filtering operation is performed on the foreground binary image, and the foreground binary image after the noise filtering operation is superimposed on the image frame to obtain the foreground superimposed image corresponding to the image frame.

[0024] A foreground video is formed by overlaying various foreground images.

[0025] Optionally, the method further includes:

[0026] Based on the binarization result of the foreground image, determine whether the foreground binary image is usable.

[0027] This specification provides a video moving object detection device, the device comprising:

[0028] The training module is used to segment sample videos containing moving objects into sample image frames, and to train an image classification model using the sample image frames.

[0029] The separation module is used to segment the target video into image frames. For any image frame, the image classification model is used to determine the complexity of the image frame. The foreground and background separation algorithm corresponding to the complexity of the image frame is used to separate the foreground and background of the image frame to obtain the foreground image corresponding to the image frame.

[0030] An imaging module is used to form a foreground video from the foreground images corresponding to each image frame, and the foreground video serves as the detection result of moving objects in the target video.

[0031] This specification provides an embodiment of a video moving object detection and foreground / background separation device, the device comprising:

[0032] At least one processor;

[0033] as well as,

[0034] A memory that is communicatively connected to the at least one processor;

[0035] in,

[0036] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the video moving object detection method described above.

[0037] This specification provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the video moving object detection method described above.

[0038] The above-mentioned technical solution adopted in this application can achieve the following beneficial effects:

[0039] Image classification models can accurately classify image frames obtained from segmenting a target video. Image frames of different complexities can be separated into foreground and background using different algorithms, thereby obtaining foreground images containing moving objects more accurately, and thus obtaining foreground videos containing moving objects, improving the detection effect of moving objects in videos.

[0040] By treating moving objects as the foreground, the moving object detection task is transformed into a foreground-background separation task, thereby improving the effectiveness and efficiency of moving object detection.

[0041] This specification provides an embodiment of a video foreground and background separation method, the method comprising:

[0042] The sample video containing moving objects is segmented into sample image frames, and an image classification model is trained using the sample image frames.

[0043] The target video is segmented into image frames. For any image frame, the complexity of the image frame is determined using the image classification model. The foreground and background separation algorithm corresponding to the complexity of the image frame is used to separate the foreground and background of the image frame to obtain the foreground and background images corresponding to the image frame.

[0044] The foreground video is formed from the foreground images corresponding to each image frame, and the background video is formed from the background images corresponding to each image frame.

[0045] This specification provides a video moving object detection device, the device comprising:

[0046] The training module is used to segment sample videos containing moving objects into sample image frames, and to train an image classification model using the sample image frames.

[0047] The separation module is used to segment the target video into image frames. For any image frame, the image classification model is used to determine the complexity of the image frame. The foreground and background separation algorithm corresponding to the complexity of the image frame is used to separate the foreground and background of the image frame to obtain the foreground image corresponding to the image frame.

[0048] An imaging module is used to form a foreground video from the foreground images corresponding to each image frame, and the foreground video serves as the detection result of moving objects in the target video.

[0049] This specification provides an embodiment of a video moving object detection and foreground / background separation device, the device comprising:

[0050] At least one processor;

[0051] as well as,

[0052] A memory that is communicatively connected to the at least one processor;

[0053] in,

[0054] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the video foreground and background separation method described above.

[0055] This specification provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the aforementioned video foreground and background separation method.

[0056] The above-mentioned technical solution adopted in this application can achieve the following beneficial effects:

[0057] Image classification models can accurately classify image frames obtained from segmenting a target video. Image frames of different complexities can be separated into foreground and background using different foreground / background separation algorithms, thereby obtaining foreground and background images more accurately, and ultimately foreground and background videos, thus improving the foreground / background separation effect of the video.

[0058] The foreground and background separation method described above can be used for moving object detection in videos. The obtained foreground video is used as the detection result of moving objects in the video, thereby improving the detection effect and efficiency of moving objects in videos. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the description of the embodiments of this specification or the prior art will be briefly introduced below. Obviously, the drawings described below are only some of the drawings that may be involved in the embodiments described in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram of the execution entity of the video moving object detection method in the first embodiment of this specification.

[0061] Figure 2 This is a flowchart illustrating the video moving object detection method in the first embodiment of this specification.

[0062] Figure 3 This is a flowchart illustrating the video foreground and background separation method in the second embodiment of this specification.

[0063] Figure 4 This is a schematic diagram of the video moving object detection device in the third embodiment of this specification.

[0064] Figure 5 This is a schematic diagram of the video foreground and background separation device in the fourth embodiment of this specification. Detailed Implementation

[0065] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions of the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described in this specification are merely some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application. The terms "comprising" and "having," and any variations thereof, in the embodiments of this specification are intended to cover non-exclusive inclusion.

[0066] The first embodiment of this specification (hereinafter referred to as "Embodiment 1") provides a video moving object detection method. The executing entity of Embodiment 1 can be a terminal (including but not limited to mobile phones, computers, tablets, and televisions), a server, an operating system, an application, a video moving object detection platform, or a video moving object detection system, etc. That is, the executing entity can be diverse, and can be set, used, or changed as needed. Alternatively, a third-party application can assist the executing entity in performing Embodiment 1. For example... Figure 1 As shown, the video moving object detection method in Embodiment 1 can be executed by a server, and a corresponding application can be installed on the terminal (held by the user). Data can be transmitted between the terminal or the application and the server. Data can be collected, input, or output, or pages or information can be processed (to the user) through the terminal or the application, thereby assisting the server in executing the video moving object detection method in Embodiment 1.

[0067] like Figure 2 As shown, the video moving object detection method provided in Embodiment 1 includes:

[0068] S101: (Executing entity) Divides the sample video containing moving objects into sample image frames, and trains an image classification model using the sample image frames;

[0069] In Example 1, several videos containing moving objects can be collected or acquired as sample videos. These moving objects can be people or other various objects; Example 1 does not impose any specific limitations.

[0070] A sample video can be divided into image frames (also called "video frames"), and the image frames obtained from the sample video can be called sample image frames. For example, each sample video can be divided into sample image frames at a frequency of one frame per 0.1 seconds.

[0071] For any sample image frame, a label can be assigned based on whether the sample image frame contains moving objects from the sample video to which it belongs and / or the number of moving objects it contains. This label characterizes the complexity (i.e., complexity level) of the sample image frame. For example, a sample image frame might be segmented from a sample video containing moving people and vehicles. If the sample image frame does not contain moving objects from the sample video (i.e., it does not contain the people or vehicles), its label can be low complexity. If the sample image frame contains moving objects from the sample video, and the number of moving objects in the sample image frame is within a first preset range (the first preset range can be a single value, such as 1), its label can be medium complexity. If the sample image frame contains moving objects from the sample video, and the number of moving objects in the sample image frame is within a second preset range (generally, the value in the second preset range exceeds the value in the first preset range, and the value in the second preset range is greater than or equal to 2), its label can be high complexity.

[0072] In addition, different complexity labels can be represented by different numbers. For example, low complexity labels can be represented by 0, medium complexity labels can be represented by 1, and high complexity labels can be represented by 2.

[0073] In Example 1, an image classification model can be trained using sample image frames. These sample image frames can be divided into a training set and a test set, with the training set used to train the model. For example, the sample image frames can be divided into the training and test sets in a 7:3 ratio, preferably with both sets containing sample image frames labeled with various complexity levels.

[0074] In Example 1, training an image classification model using sample image frames may include: performing one or more rounds of model training operations using sample image frames (specifically, sample image frames in the test set), and using the model obtained after the one or more rounds of model training operations as the image classification model.

[0075] The first round of model training includes: inputting one or more sample image frames into the original model, having the original model calculate the complexity of each input sample image frame, comparing the complexity calculated by the original model with the actual complexity of the input sample image frames, updating the parameters of the original model based on the comparison results, and obtaining a new model.

[0076] For example, the original model can be a ResNet model (network architecture model). The ResNet model is a milestone model in deep convolutional neural networks, significant because it introduces residual modules to address the model degradation problem as network depth increases, greatly improving model depth and enhancing image content recognition capabilities. Residual modules refer to convolutional modules that introduce skip connections. The convolutional layers learn additional residual information based on the input data and add it to the input data. Compared to ordinary convolutional modules, residual modules retain more information from the input data. Therefore, when multiple residual modules are stacked to form a deep network, the top-level modules can still acquire sufficient information from the input data. Simultaneously, during training, the gradients from backpropagation can smoothly propagate to the input layer without gradient vanishing or exploding. Specifically, the ResNet50 model can be used as the original model to extract depth image features. ResNet50 is one of the ResNet series models, containing 50 convolutional layers, achieving a good trade-off between model capacity and inference speed.

[0077] Of course, other types of models can also be used as the original model; Example 1 does not impose any specific limitations.

[0078] For any non-first round of model training operation, the model training operation includes: inputting one or more sample image frames into the latest model obtained (after the previous round of model training operation), calculating the complexity of the input sample image frame by the latest model obtained (after the previous round of model training operation), comparing the complexity calculated by the latest model obtained (after the previous round of model training operation) with the true complexity of the input sample image frame (i.e., the complexity label assigned above, the same below), updating the parameters of the latest model obtained (after the previous round of model training operation) according to the comparison result, and obtaining a new model.

[0079] In any round of model training: a sample image frame is input into the model. The model passes through convolutional layers to obtain image features, and then through fully connected layers to obtain the complexity of the sample image frame. A loss function, Cross Entropy Loss, can be defined to calculate the loss between the complexity calculated or predicted by the model and the true complexity label of the sample image frame in that round of model training. An optimizer (e.g., the Adam optimizer) can be selected to update or optimize the model's parameters (i.e., using the optimizer to optimize the parameters of each layer during Loss backpropagation, thereby updating the training parameters). In this way, the model is trained by comparing the complexity calculated by the model with the true complexity to obtain the loss, calculating the gradient through the backpropagation algorithm, and updating the model's parameters using the optimizer.

[0080] After performing one or more rounds of model training as described above, save the model structure and its parameters as an image classification model.

[0081] After obtaining the image classification model, a test set can be used to test the image classification model, and the test results can be used to determine whether the image classification model is qualified.

[0082] S103: (Executing entity) The target video is divided into image frames. For any image frame, the image classification model is used to determine the complexity of the image frame. The foreground and background separation algorithm corresponding to the complexity of the image frame is used to separate the foreground and background of the image frame to obtain the foreground image corresponding to the image frame.

[0083] The video for which moving object detection is required is used as the target video, which is then segmented into image frames. For any given image frame, its complexity can be determined using an image classification model.

[0084] Specifically, for any given image frame, a foreground / background separation algorithm corresponding to the complexity of that image frame can be used to separate the foreground and background, obtaining the foreground image corresponding to that image frame. That is, different foreground / background separation algorithms can be used for image frames with different complexities.

[0085] In Example 1, if the complexity of an image frame indicates that it does not contain moving objects from the target video (i.e., low complexity), then the foreground / background separation algorithm corresponding to the complexity of the image frame can be a Gaussian Mixture Model (MOG). That is, the MOG is used to separate the foreground and background of the image frame. The MOG models the scene by modeling the color intensity of each pixel as a linear combination of multiple Gaussian distributions. At each time step, for a new input pixel, it calculates the difference between its color intensity and the background model's Gaussian distribution, and determines whether it belongs to the foreground or background by comparing the differences. When a pixel is classified as background, its color value is used to update the background model. This algorithm uses a learning rate parameter to balance the contributions of new and old frames to the background model to adapt to scene changes. Although the complexity of the image frame indicates that it does not contain moving objects from the target video, foreground / background separation is still performed on this image frame. This is to select a suitable foreground / background separation algorithm based on the complexity of the image frame, and to avoid potential complexity calculation errors in image classification models.

[0086] And / or,

[0087] If the complexity of an image frame indicates that it contains moving objects from the target video, and the number of moving objects in the image frame is within a first preset range (i.e., medium complexity), then the foreground / background separation algorithm corresponding to the complexity of the image frame can be the K-nearest neighbor algorithm, i.e., the K-nearest neighbor algorithm is used to separate the foreground and background of the image frame. The K-nearest neighbor algorithm (background subtractor) is a pixel-based classification method. It treats each pixel as a finite state machine and determines whether it belongs to the foreground or background by comparing each pixel with its surrounding pixels. It uses a buffer to store the pixel history information for modeling the scene. At each time step, for a new input pixel, it calculates the distance between its features and historical samples and determines whether it belongs to the foreground or background. When a pixel is classified as background, its color distribution information is used to update the background model. The background estimate is updated by selecting the background models of neighboring pixels to adapt to changes in the scene.

[0088] And / or,

[0089] If the complexity of an image frame indicates that the image frame contains moving objects from the target video, and the number of moving objects in the image frame is within a second preset range (i.e., high complexity), then the foreground and background separation algorithm corresponding to the complexity of the image frame can be the Vibe algorithm, that is, the Vibe algorithm is used to separate the foreground and background of the image frame.

[0090] The Vibe algorithm consists of four steps: initialization, foreground segmentation, background update, and model maintenance. In the initialization phase, for each pixel, a background model is maintained, including the current pixel value, the number of background pixels (a counter), and pixel samples. Pixel samples are several randomly selected pixels from previous frames. In the foreground segmentation phase, for each pixel, the difference (Euclidean distance) between the current pixel and the pixel samples in its background model is calculated. If the difference is less than a predefined threshold, the pixel is classified as background; otherwise, it is classified as foreground. In the background update phase, for pixels classified as background, the background model is updated with a certain probability (update rate). A neighboring pixel is randomly selected as a new sample and overlaid on an old sample in the sample set. In the model maintenance phase, when a pixel is classified as background for several consecutive frames, it is considered a reliable background pixel, and the counter is incremented. If a pixel is classified as foreground for several consecutive frames, the model for that pixel is reinitialized.

[0091] Specifically, the Vibe algorithm may include the following steps:

[0092] For each pixel, a background model is maintained. This background model includes the current pixel value, the number of background pixels, and the pixel sample set maintained by the current sample itself. The number of pixel sample sets maintained by the background model is N, and v(x) is the pixel value at x, M(x) = {v1, v2, v3, ..., v N Let} be the sample set maintained by the background model at location x. During the initialization phase, to build the background model for pixel samples, several random pixels are selected as initial samples from the first few frames or previous frames of the target video. These randomly selected initial samples converge quickly using the ViBe algorithm to accurately represent background features. By selecting random pixels as initial samples, initial information about the background can be obtained. These initial samples can represent background pixels in previous frames and provide a starting point for subsequent pixel change detection. Over time, in subsequent frames, these samples are updated and adjusted based on pixel changes to reflect the latest background features. This allows the background model to adapt to scene changes and accurately detect and segment foreground objects.

[0093] For pixel x, calculate the Euclidean distance d between the current pixel value and the sample value. i =||v(x)-v i ||, where i∈{1,2,3,...,N}. The obtained Euclidean distance is compared with a predefined distance R. If the number of pixels with an Euclidean distance less than R is less than a predefined threshold θ, then the pixel x is classified as background; if the number of pixels with an Euclidean distance less than R is greater than or equal to the predefined threshold θ, then the pixel x is classified as foreground.

[0094] When a pixel x is classified as a background pixel, we have The probability of updating the pixel x background model is given, where These are pre-defined hyperparameters. Specifically, a sample point v is randomly selected from the sample set of pixel x. i Simultaneously, a pixel within the neighborhood of pixel x is randomly selected to overwrite and update the sample point; that is, this pixel is used for filling and updating. Furthermore, pixel x also has... The probability is used to update the background model of the eight neighboring points, using the same method as updating pixel x. The Euclidean distance between the pixel and the sample value is calculated and compared with a predefined distance R to determine whether the pixel is background. Furthermore, if the current point y is detected as background K times consecutively, pixel y is changed to a background point. Similarly, pixel y has... It updates its background model based on the probability of the changes.

[0095] S105: (Execution subject) forms a foreground video from the foreground images corresponding to each image frame, and the foreground video serves as the detection result of moving objects in the target video.

[0096] Through the foreground and background separation operations described above, each image frame can be separated into a foreground image and a background image (the background can also be called "background"). The foreground and background images separated from a single image frame are the foreground and background images corresponding to that image frame. The foreground images corresponding to each image frame can be used to form a foreground video, which can then be used as the detection result of moving objects in the target video. This will be explained in detail below.

[0097] In Example 1, forming a foreground video from the foreground images corresponding to each image frame may include:

[0098] For any image frame, the corresponding foreground image is binarized to obtain a binary foreground image (i.e., the foreground image after binarization). Furthermore, the usability of the binary foreground image can be determined based on the binarization result. Specifically, if the binarization result of the foreground image is not completely black (indicating that the foreground image contains moving objects), then the binary foreground image is usable; if the binarization result of the foreground image is completely black (indicating that the foreground image does not contain moving objects), then the binary foreground image is unusable. Similarly, the corresponding background image can also be binarized to obtain a binary background image (i.e., the background image after binarization).

[0099] If the foreground binary image is determined to be usable, then a noise filtering operation is performed on the foreground binary image. Specifically, morphological erosion operations can be used to set small cross-shaped structuring elements (kernels) to remove relatively small, isolated noise points. Correspondingly, a similar noise filtering operation can be performed on the background binary image.

[0100] Additionally, hole repair can be performed on the foreground binary image as needed. Specifically, morphological dilation operations can be used for hole repair. The first operation is combined with the denoising operation as an opening operation (i.e., dilation after erosion), this time based on a small kernel. Subsequent operations use a larger kernel (circular kernel) for three dilation operations. Correspondingly, hole repair can also be performed on the background binary image.

[0101] The foreground binary image, after noise filtering (or both noise filtering and hole repair if applicable), is overlaid on the image frame to obtain the corresponding foreground overlay image. In this foreground overlay image, only the foreground region is retained, and the background region in the foreground overlay image becomes a solid color (usually black, because pixels with a value of 0 in the foreground binary image are black). Similarly, the background binary image, after noise filtering (or both noise filtering and hole repair if applicable), is overlaid on the image frame to obtain the corresponding background overlay image. In this background overlay image, only the background region is retained, and the foreground region in the background overlay image becomes a solid color (usually black, because pixels with a value of 0 in the background binary image are black).

[0102] Based on the above, foreground overlay images can be obtained from each image frame (the number of foreground overlay images may be less than the number of image frames because some foreground binary images may be unavailable). These foreground overlay images can then be combined, assembled, or synthesized into a video, serving as the foreground video. Specifically, the foreground overlay images can be arranged according to the sequential order of the image frames to form, combine, or synthesize the foreground video. Similarly, background videos can be formed from the background images corresponding to each image frame. In other words, background overlay images can be obtained from each image frame, and these background overlay images can be combined, assembled, or synthesized into a video, serving as the background video. Specifically, the background overlay images can be arranged according to the sequential order of the image frames to form, combine, or synthesize the background video.

[0103] Since the foreground overlay image is obtained from each available foreground binary image, and the available foreground binary images contain images of each moving object in the target video, each foreground overlay image contains (generally only contains) images of each moving object in the target video. Therefore, the foreground video formed, combined, or synthesized from each foreground overlay image contains (generally only contains) the motion of each moving object in the target video (including but not limited to motion content or motion trajectory). In other words, the foreground video highlights the motion of each moving object in the target video, and the foreground video can serve as the detection result of each moving object in the target video, thereby achieving moving object detection.

[0104] In practice, instead of determining whether the foreground binary images are usable, noise filtering (or hole repair) can be performed on each foreground binary image. Then, each foreground binary image can be superimposed with its corresponding image frame (the image frame used to obtain a single foreground binary image is the image frame corresponding to that single foreground binary image) to obtain various foreground superimposed images. These foreground superimposed images can then be used to form, combine, or synthesize a foreground video. In the foreground video obtained in this way, when displaying the motion of various moving objects in the target video, some frames may be completely black (because unusable foreground binary images are completely black and are not discarded, the completely black frames in the foreground video are the frames containing the foreground superimposed images formed from the unusable foreground binary images).

[0105] Correspondingly, the background binary image is an image that includes the background (non-moving objects can be used as the background) in the target video. Therefore, each background superimposed image is an image that includes (generally only includes) the background in the target video. The background video formed, combined, or synthesized from these background superimposed images then includes (generally only includes) the background content of the target video. In other words, the background video highlights the background content of the target video and can serve as the background content video within the target video.

[0106] As can be seen, the above steps have also achieved the separation of the foreground and background of the target video, resulting in foreground and background videos respectively.

[0107] Example 1 can achieve the following beneficial effects:

[0108] Image classification models can accurately determine the complexity of image frames obtained from segmenting a target video. Image frames with different complexities can be separated into foreground and background using different algorithms, thereby improving the efficiency of foreground and background separation, obtaining foreground images containing moving objects more accurately, and thus obtaining foreground videos containing moving objects, thereby improving the detection effect of moving objects in videos.

[0109] By treating moving objects as the foreground, the moving object detection task is transformed into a foreground-background separation task, thereby improving the effectiveness and efficiency of moving object detection.

[0110] A model training approach is employed, using a progressive learning strategy to gradually learn the features of moving objects, resulting in an image classification model. This model is then used to determine the complexity of image frames. For image frames of varying complexity, different pixel-level foreground and background recognition methods are adaptively applied. Specifically, a simple foreground / background separation algorithm is used for low-complexity backgrounds, while a more complex and efficient algorithm is employed for medium- and high-complexity foreground images containing moving objects. This approach saves computational resources while enabling rapid detection of moving objects.

[0111] Using morphological algorithms to perform noise filtering and hole repair on foreground images can improve the effect of foreground overlay images, thereby obtaining more accurate foreground videos, thus optimizing the moving object detection results, avoiding missing areas of moving object activity or the inclusion of a large number of non-moving object areas, and improving the moving object detection effect.

[0112] The second embodiment of this specification (hereinafter referred to as "Embodiment Two") provides a method for video foreground and background separation. The executing entity in Embodiment Two can be a terminal (including but not limited to mobile phones, computers, tablets, and televisions), a server, an operating system, an application, a video foreground and background separation platform, or a video foreground and background separation system, etc. That is, the executing entity can be diverse and can be set, used, or changed as needed. Alternatively, a third-party application can assist the executing entity in performing Embodiment Two. For example... Figure 1 As shown, the video foreground and background separation method in Embodiment 2 can be executed by the server, and a corresponding application can be installed on the terminal (held by the user). Data can be transmitted between the terminal or the application and the server. Data can be collected, input, or output, or pages or information can be processed (to the user) through the terminal or the application, thereby assisting the server in executing the video foreground and background separation method in Embodiment 2.

[0113] like Figure 3 As shown, the video foreground and background separation method provided in Embodiment 2 includes:

[0114] S202: (Executing entity) Divides the sample video containing moving objects into sample image frames, and trains an image classification model using the sample image frames;

[0115] S204: (Executing entity) Divides the target video into image frames. For any image frame, the image classification model is used to determine the complexity of the image frame. The foreground and background separation algorithm corresponding to the complexity of the image frame is used to separate the foreground and background of the image frame to obtain the foreground image and background image corresponding to the image frame.

[0116] S206: (Executing entity) The foreground video is formed from the foreground images corresponding to each image frame, and the background video is formed from the background images corresponding to each image frame.

[0117] The details of the above content in Example 2 have been described in Example 1 and will not be repeated here.

[0118] Example 2 can achieve the following beneficial effects:

[0119] Image classification models can accurately determine the complexity of image frames obtained from segmenting a target video. Image frames with different complexities can be separated into foreground and background using different foreground / background separation algorithms, thereby obtaining foreground and background images more accurately, and ultimately foreground and background videos, thus improving the foreground / background separation effect of the video.

[0120] The foreground and background separation method described above can be used for moving object detection in videos. The obtained foreground video is used as the detection result of moving objects in the video, which improves the detection effect and efficiency of moving objects in the video. Based on this, the beneficial effects in Example 1 can be achieved.

[0121] Example 1 and / or Example 2 can be applied to various fields, such as moving object detection and foreground / background separation in surveillance videos. In Example 1 and / or Example 2, the target video may be continuously generated; for example, the target video is surveillance video, which can be continuously generated. Therefore, moving object detection and / or foreground / background separation of the target video can be performed continuously, meaning that the foreground or background video can also be continuously generated.

[0122] like Figure 4 As shown, the third embodiment of this specification provides a video moving object detection device corresponding to the video moving object detection method described in Embodiment 1, comprising:

[0123] The training module (first training module) 301 is used to segment the sample video containing moving objects into sample image frames, and train the image classification model through the sample image frames.

[0124] The separation module (first classification module) 303 is used to segment the target video into image frames. For any image frame, the image classification model is used to determine the complexity of the image frame. The foreground and background separation algorithm corresponding to the complexity of the image frame is used to separate the foreground and background of the image frame to obtain the foreground image corresponding to the image frame.

[0125] The imaging module (first imaging module) 305 is used to form a foreground video from the foreground images corresponding to each image frame, and the foreground video serves as the detection result of moving objects in the target video.

[0126] Optionally, training the image classification model using the sample image frames includes:

[0127] One or more rounds of model training are performed using the sample image frames, and the model obtained after one or more rounds of model training is used as the image classification model.

[0128] The first round of model training includes: inputting one or more sample image frames into the original model, having the original model calculate the complexity of the input sample image frames, comparing the complexity calculated by the original model with the actual complexity of the input sample image frames, updating the parameters of the original model based on the comparison result, and obtaining a new model.

[0129] Any non-first-round model training operation includes: inputting one or more sample image frames into the latest obtained model, calculating the complexity of the input sample image frames by the latest obtained model, comparing the complexity calculated by the latest obtained model with the actual complexity of the input sample image frames, updating the parameters of the latest obtained model based on the comparison result, and obtaining a new model.

[0130] Optionally, if the complexity of the image frame indicates that the image frame does not contain moving objects in the target video, then the foreground / background separation algorithm corresponding to the complexity of the image frame is a Gaussian mixture model (MOG).

[0131] And / or,

[0132] If the complexity of an image frame indicates that the image frame contains moving objects from the target video, and the number of moving objects in the image frame is within a first preset range, then the foreground and background separation algorithm corresponding to the complexity of the image frame is the K-nearest neighbor algorithm.

[0133] And / or,

[0134] If the complexity of an image frame indicates that the image frame contains moving objects from the target video, and the number of moving objects in the image frame is within a second preset range, then the foreground / background separation algorithm corresponding to the complexity of the image frame is the Vibe algorithm.

[0135] Optionally, forming the foreground video from the foreground images corresponding to each image frame includes:

[0136] For any image frame, perform a binarization operation on the foreground image corresponding to that image frame to obtain a binary foreground image;

[0137] If the foreground binary image is determined to be usable, then a noise filtering operation is performed on the foreground binary image, and the foreground binary image after the noise filtering operation is superimposed on the image frame to obtain the foreground superimposed image corresponding to the image frame.

[0138] A foreground video is formed by overlaying various foreground images.

[0139] Optionally, the imaging module 305 is further configured to determine whether the foreground binary image is usable based on the binarization operation result of the foreground image.

[0140] like Figure 5As shown, the fourth embodiment of this specification provides a video foreground / background separation device corresponding to the video foreground / background separation method described in Embodiment 1, comprising:

[0141] The training module (second training module) 402 is used to segment the sample video containing moving objects into sample image frames, and train the image classification model through the sample image frames;

[0142] The separation module (second separation module) 404 is used to segment the target video into image frames. For any image frame, the image classification model is used to determine the complexity of the image frame. The foreground and background separation algorithm corresponding to the complexity of the image frame is used to separate the foreground and background of the image frame to obtain the foreground image and background image corresponding to the image frame.

[0143] The imaging module (second imaging module) 406 is used to form a foreground video from the foreground images corresponding to each image frame and to form a background video from the background images corresponding to each image frame.

[0144] The fifth embodiment of this specification provides a video moving object detection and foreground / background separation device, the device comprising:

[0145] At least one processor;

[0146] as well as,

[0147] A memory that is communicatively connected to the at least one processor;

[0148] in,

[0149] The memory stores instructions that can be executed by the at least one processor. When the at least one processor executes the instructions, it enables the at least one processor to perform the video moving object detection method described in Embodiment 1 or the video foreground / background separation method described in Embodiment 2.

[0150] The sixth embodiment of this specification provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the video moving object detection method described in Embodiment 1 or the video foreground / background separation method described in Embodiment 2.

[0151] The above embodiments can be used in combination, and modules with the same name in different embodiments or within the same embodiment can be the same or different modules.

[0152] The foregoing has described specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily have to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0153] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer-readable storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0154] The apparatus, device, non-volatile computer-readable storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, device and non-volatile computer storage medium will not be repeated here.

[0155] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0156] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0157] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0158] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0159] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0163] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0164] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0165] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0166] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0167] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0168] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0169] The above description is merely an embodiment of this specification and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A video moving object detection method, characterized in that, The method includes: The sample video containing moving objects is segmented into sample image frames, and an image classification model is trained using the sample image frames. The target video is divided into image frames. For any image frame, the complexity of the image frame is determined using the image classification model. The foreground and background separation algorithm corresponding to the complexity of the image frame is used to separate the foreground and background of the image frame to obtain the foreground image corresponding to the image frame. A foreground video is formed from the foreground images corresponding to each image frame, and the foreground video serves as the detection result of moving objects in the target video; Wherein, if the complexity of an image frame indicates that the image frame does not contain moving objects in the target video, then the foreground and background separation algorithm corresponding to the complexity of the image frame is Gaussian Mixture Model (MOG). And / or, If the complexity of an image frame indicates that the image frame contains moving objects from the target video, and the number of moving objects in the image frame is within a first preset range, then the foreground and background separation algorithm corresponding to the complexity of the image frame is the K-nearest neighbor algorithm. And / or, If the complexity of an image frame indicates that the image frame contains moving objects from the target video, and the number of moving objects in the image frame is within a second preset range, then the foreground / background separation algorithm corresponding to the complexity of the image frame is the Vibe algorithm.

2. The method as described in claim 1, characterized in that, Training an image classification model using the sample image frames includes: One or more rounds of model training are performed using the sample image frames, and the model obtained after one or more rounds of model training is used as the image classification model. The first round of model training includes: inputting one or more sample image frames into the original model, having the original model calculate the complexity of the input sample image frames, comparing the complexity calculated by the original model with the actual complexity of the input sample image frames, updating the parameters of the original model based on the comparison result, and obtaining a new model. Any non-first-round model training operation includes: inputting one or more sample image frames into the latest obtained model, calculating the complexity of the input sample image frames by the latest obtained model, comparing the complexity calculated by the latest obtained model with the actual complexity of the input sample image frames, updating the parameters of the latest obtained model based on the comparison result, and obtaining a new model.

3. The method as described in claim 1, characterized in that, The foreground video, formed from the foreground images corresponding to each image frame, includes: For any image frame, perform a binarization operation on the foreground image corresponding to that image frame to obtain a binary foreground image; If the foreground binary image is determined to be usable, then a noise filtering operation is performed on the foreground binary image, and the foreground binary image after the noise filtering operation is superimposed on the image frame to obtain the foreground superimposed image corresponding to the image frame. A foreground video is formed by overlaying various foreground images.

4. The method as described in claim 3, characterized in that, The method further includes: Based on the binarization result of the foreground image, determine whether the foreground binary image is usable.

5. A video foreground and background separation method, characterized in that, The method includes: The sample video containing moving objects is segmented into sample image frames, and an image classification model is trained using the sample image frames. The target video is segmented into image frames. For any image frame, the complexity of the image frame is determined using the image classification model. The foreground and background separation algorithm corresponding to the complexity of the image frame is used to separate the foreground and background of the image frame to obtain the foreground and background images corresponding to the image frame. Foreground video is formed from the foreground images corresponding to each image frame, and background video is formed from the background images corresponding to each image frame. Wherein, if the complexity of an image frame indicates that the image frame does not contain moving objects in the target video, then the foreground and background separation algorithm corresponding to the complexity of the image frame is Gaussian Mixture Model (MOG). And / or, If the complexity of an image frame indicates that the image frame contains moving objects from the target video, and the number of moving objects in the image frame is within a first preset range, then the foreground and background separation algorithm corresponding to the complexity of the image frame is the K-nearest neighbor algorithm. And / or, If the complexity of an image frame indicates that the image frame contains moving objects from the target video, and the number of moving objects in the image frame is within a second preset range, then the foreground / background separation algorithm corresponding to the complexity of the image frame is the Vibe algorithm.

6. A video moving object detection device, characterized in that, The device includes: The training module is used to segment sample videos containing moving objects into sample image frames, and to train an image classification model using the sample image frames. The separation module is used to segment the target video into image frames. For any image frame, the image classification model is used to determine the complexity of the image frame. The foreground and background separation algorithm corresponding to the complexity of the image frame is used to separate the foreground and background of the image frame to obtain the foreground image corresponding to the image frame. An imaging module is used to form a foreground video from the foreground images corresponding to each image frame, and the foreground video serves as the detection result of moving objects in the target video; Wherein, if the complexity of an image frame indicates that the image frame does not contain moving objects in the target video, then the foreground and background separation algorithm corresponding to the complexity of the image frame is Gaussian Mixture Model (MOG). And / or, If the complexity of an image frame indicates that the image frame contains moving objects from the target video, and the number of moving objects in the image frame is within a first preset range, then the foreground and background separation algorithm corresponding to the complexity of the image frame is the K-nearest neighbor algorithm. And / or, If the complexity of an image frame indicates that the image frame contains moving objects from the target video, and the number of moving objects in the image frame is within a second preset range, then the foreground / background separation algorithm corresponding to the complexity of the image frame is the Vibe algorithm.

7. A video foreground and background separation device, characterized in that, The device includes: The training module is used to segment sample videos containing moving objects into sample image frames, and to train an image classification model using the sample image frames. The separation module is used to segment the target video into image frames. For any image frame, the image classification model is used to determine the complexity of the image frame. The foreground and background separation algorithm corresponding to the complexity of the image frame is used to separate the foreground and background of the image frame to obtain the foreground image and background image corresponding to the image frame. The imaging module is used to form a foreground video from the foreground images corresponding to each image frame and a background video from the background images corresponding to each image frame. Wherein, if the complexity of an image frame indicates that the image frame does not contain moving objects in the target video, then the foreground and background separation algorithm corresponding to the complexity of the image frame is Gaussian Mixture Model (MOG). And / or, If the complexity of an image frame indicates that the image frame contains moving objects from the target video, and the number of moving objects in the image frame is within a first preset range, then the foreground and background separation algorithm corresponding to the complexity of the image frame is the K-nearest neighbor algorithm. And / or, If the complexity of an image frame indicates that the image frame contains moving objects from the target video, and the number of moving objects in the image frame is within a second preset range, then the foreground / background separation algorithm corresponding to the complexity of the image frame is the Vibe algorithm.

8. A video moving object detection and foreground / background separation device, characterized in that, The device includes: At least one processor; as well as, A memory that is communicatively connected to the at least one processor; in, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the video moving object detection method of any one of claims 1 to 4 or the video foreground / background separation method of claim 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the video moving object detection method of any one of claims 1 to 4 or the video foreground / background separation method of claim 5.

Images