Systems and methods for stabilizing the bounding boxes of objects in a video stream

Abstract

This disclosure relates to a system and method for stabilizing bounding boxes of objects in a video stream. The method includes: receiving a video stream comprising a sequence of image frames; detecting objects in the image frames and generating bounding boxes around the objects; measuring a noise level in the video stream; and performing temporal filtering on the bounding boxes across multiple image frames based on the measured noise level, thereby stabilizing the bounding boxes in the video stream. This disclosure also relates to an image processing system.

Description

Technical Field

[0001] This disclosure relates to a method for stabilizing bounding boxes of objects in a video stream. By using this method, the user experience regarding the stability of the bounding boxes is improved. This disclosure also relates to an image processing system for implementing this method. Background Technology

[0002] A bounding box is a type of annotation that refers to a box drawn around an object in an image or video. A bounding box is typically (but not always) a rectangle defined by x (vertical) and y (horizontal) coordinates, with its edges surrounding the object. Bounding boxes are not strictly limited to a rectangular shape and can take any suitable shape. The size of the bounding box usually depends on the height and width of the object. Bounding boxes are often labeled with the name of the type of object they surround; for example, "car" indicates a bounding box around a car. It is also common to use different colors for different types of objects.

[0003] Machine learning models can be trained to perform object detection and identify certain object types. In a video stream comprising a sequence of image frames, bounding boxes can be updated for each image frame, or, for example, for every Nth image frame. For instance, in a surveillance system including cameras, bounding boxes in a video stream can be updated virtually in real time, allowing users to track multiple objects. If a new object enters the camera view or an object leaves the view, the bounding box updates accordingly.

[0004] The problem with existing systems and methods for generating bounding boxes in video streams is that they cannot generate stable bounding boxes for objects within the video stream. Although the human eye doesn't always see a video stream, there are often pixel variations between image frames. These variations cause the system to calculate bounding boxes slightly differently frame by frame, resulting in variations in the position and / or size of the bounding boxes between frames. To the user, this can be perceived as a flickering effect on the bounding boxes. This undesirable effect can be unpleasant for the user or lead to an experience where the application malfunctions.

[0005] The 2016 paper "Video Object Extraction and Tracking in Complex Environments Using Background Subtraction" by Kumar et al. (https: / / doi.org / 10.1016 / j.pisc.2016.04.064) discloses a method for studying motion blobs in the foreground and updating the background to improve tracking accuracy.

[0006] Another paper by Shah et al., “Improving the performance of CNN-based vehicle detection and tracking by median algorithm” (2021, DOI:10.1109 / ICCE-Asia53811.2021.9641942), proposes a median-based label estimation scheme that uses the median of historical labels stored in previous frames to predict the detection labels of objects in the frame.

[0007] US Patent 10511846 discloses a method and apparatus for adaptive denoising of source video in video conferencing applications. Temporal denoising is adaptively applied to blocks of source frames based on noise estimation and moving object detection. Summary of the Invention

[0008] This invention relates to a method for defining the bounding box of an object in a stable video stream, the method comprising:

[0009] Receive a video stream including a sequence of image frames;

[0010] Detect objects in an image frame and generate bounding boxes around those objects;

[0011] Measure the noise level of the video stream; and

[0012] The bounding boxes are temporally filtered across multiple image frames based on the measured noise level, thereby stabilizing the bounding boxes in the video stream.

[0013] Among them, the bounding boxes are temporally filtered on multiple previous image frames;

[0014] Its features are:

[0015] The number of previous image frames is adapted such that a higher noise level means temporal filtering on a larger number of previous image frames, and a lower noise level means temporal filtering on a smaller number of previous image frames.

[0016] The inventors have realized that measured noise levels can be used to adapt temporal filtering to achieve a positive stabilization effect on bounding boxes in a video stream. This can be done by averaging the positions of bounding boxes in a given image over multiple previous image frames. Previous frames can be represented as multiple frames or time windows. The number of previous frames or the length of the time window can be adapted to the noise level. According to one embodiment, the number of previous image frames is adapted such that a higher noise level implies temporal filtering over a larger number of previous image frames or a longer time window, and a lower noise level implies temporal filtering over a smaller number of previous image frames or a shorter time window.

[0017] Noise in video is typically not constant but varies over time and depends on the environment. When suppressing instability by averaging the positions of bounding boxes, the inventors found it beneficial to update the length of the time window accordingly. Videos with high levels of noise tend to generate very unstable bounding boxes, while low levels of noise tend to generate less stable ones. By dynamically and continuously updating the number of previous frames used in the time window or temporal filtering, it is possible to make the bounding boxes more stable while maintaining the best possible responsiveness to, for example, movement.

[0018] Averaging is typically a stabilization technique known to those skilled in the art. However, averaging over a long time window is not always useful because it introduces latency. In other words, a user or application may gain bounding box stability in noisy video at the same time as losing latency. For the user, bounding boxes with excessive latency can be as unpleasant as the flickering effect caused by noise on the bounding boxes. The currently disclosed methods describe approaches and systems that can adapt stability based on noise levels, wherein latency is reduced for video sequences with lower noise levels, and a slightly increased latency can be tolerated for video sequences with higher noise levels due to the increased stability of the bounding boxes.

[0019] The present invention also relates to an image processing system, comprising:

[0020] At least one camera; and

[0021] The processing unit is configured as follows:

[0022] Receive a video stream comprising a sequence of image frames from the camera;

[0023] Detect objects in an image frame and generate bounding boxes around those objects; and

[0024] Measure the noise level of the video stream;

[0025] Temporal filtering of bounding boxes is performed on multiple image frames based on the measured noise level to stabilize bounding boxes in the video stream.

[0026] The processing unit is configured to perform temporal filtering on the bounding boxes across multiple previous image frames;

[0027] Its features are:

[0028] The number of previous image frames is adapted such that a higher noise level means temporal filtering on a larger number of previous image frames, and a lower noise level means temporal filtering on a smaller number of previous image frames.

[0029] The system may also include a display for showing the video stream and a stable bounding box. This image processing system can be used, for example, in camera-based surveillance systems.

[0030] Those skilled in the art will recognize that any embodiment of the currently disclosed image processing system can be used to perform the currently disclosed method of bounding boxes of objects in a stable video stream, and vice versa. Attached Figure Description

[0031] Various embodiments are described below with reference to the accompanying drawings. The drawings are examples of embodiments and are intended to illustrate some features of the currently disclosed methods and systems for stabilizing bounding boxes of objects in a video stream, but are not limited to the currently disclosed methods and systems.

[0032] Figure 1 A method flowchart illustrating an embodiment of a method for defining bounding boxes of objects in a currently disclosed stable video stream is shown.

[0033] Figures 2A to 2C An example of unstable bounding boxes between image frames of a stationary object is shown.

[0034] Figure 3 A conceptual diagram illustrates an embodiment of stabilizing bounding boxes by temporally filtering them across multiple image frames based on measured noise levels.

[0035] Figure 4 A schematic diagram of an embodiment of the currently disclosed image processing system is shown. Detailed Implementation

[0036] This invention relates to a method for defining the bounding boxes of objects in a stable video stream. Figure 1 A method flowchart of an embodiment of method 100 based on the bounding box of an object in a currently disclosed stable video stream is shown. Method 100 includes the following steps:

[0037] Receive a video stream (101) including a sequence of image frames;

[0038] Detect objects in the image frame and generate bounding boxes around the objects (102);

[0039] Measure the noise level of the video stream (103); and

[0040] The bounding boxes are temporally filtered across multiple image frames based on the measured noise level, thereby stabilizing the bounding boxes in the video stream (104).

[0041] The step of temporally filtering bounding boxes across multiple image frames based on the measured noise level is such that the position of the temporally filtered bounding box in a given image frame is a combination of the bounding box positions of multiple previous image frames, such as an average value.

[0042] Figure 3 A conceptual diagram illustrates an embodiment of temporal filtering of bounding boxes across multiple image frames to stabilize the bounding boxes based on measured noise levels. Figure 3 In the example, video 300 includes a sequence of image frames 301. For a given image frame 301', the temporal filtering of the bounding boxes in image frame 301' is based on the number of previous image frames. In this example, the number of previous image frames 302 is 5. Figure 3 The number N in the equation represents the number of previous image frames 302, which is not a fixed number but can be adjusted based on the noise level. For example, if the noise level increases, the number of previous image frames 302 used for temporal filtering of the bounding boxes can increase from 5 to 6, 7, 10, or any other larger number. If the noise level decreases, the number of previous image frames 302 used for temporal filtering of the bounding boxes can decrease from 5 to 1, 2, 3, or 4 previous image frames. The concept is the same for another given image frame 301". The temporal filtering of the bounding boxes in image frame 301 is based on multiple previous image frames 302. Previous frames can be represented as multiple frames or time windows. The noise level at any given time point or for any given image frame can be measured backward in time for multiple frames and / or time windows. For example, for a given image frame, the noise level can be given as the average pixel brightness of multiple previous image frames. The average can then be subtracted from the pixel brightness of the given image frame. As those skilled in the art will recognize, pixel brightness can alternatively or in combination be, for example, color intensity. The number of previous frames or the length of the time window can be adapted to the noise level.

[0043] The step of temporally filtering bounding boxes across multiple image frames can be based on the measured noise level and can be performed on every Nth image frame, where N is an integer greater than 1. Alternatively, the step of temporally filtering bounding boxes across multiple image frames based on the measured noise level can be performed for each image frame. Figure 3 In the example, there are two image frames, 301' and 301", for which bounding boxes are generated.

[0044] The step of temporally filtering bounding boxes across multiple image frames may include temporally smoothing the bounding boxes. Temporal smoothing is a term commonly known to those skilled in the art. It can refer to averaging the bounding boxes across multiple image frames to create a more stable position and / or size. Temporal filtering can employ simple techniques such as ordinary averaging, but it can also apply techniques such as exponential smoothing and / or Kalman filtering.

[0045] Figures 2A to 2C An example of unstable bounding boxes between image frames of a stationary object is shown. Figure 2AIn image frame 200, three stationary objects 201 are detected, all of which are cars in this example. A bounding box 202 has been generated for each object 201. Figure 2B and Figure 2C Other image frames of the same scene at later time points in the image frame sequence are shown. Although there is no visible noise in this image sequence, it can be noticed that the bounding box 202 of the leftmost object 201 is... Figure 2A , Figure 2B and Figure 2C They do not have the same position in the same frame. Therefore, if this sequence of image frames is presented to the user as a video, the user will see the bounding box 202 moving between image frames. As a simplified example, if based on Figure 2A and Figure 2B Position Figure 2C If the bounding boxes in the data are averaged over time, the overall result will be a more stable bounding box.

[0046] More flexible temporal filtering can be achieved if the length of the time window or the number of previous image frames on which temporal filtering is based is adapted to the noise level.

[0047] Noise level, measuring noise level

[0048] Noise, often referred to as electrostatic noise, white noise, or snowflakes, can be a dotted pixel pattern appearing in a video. Noise may refer to random variations in brightness or color within the video. It may appear as a grainy or speckled texture in the video. In one embodiment, noise is visual noise in a video stream. Noise can be a dotted pattern, such as a pixel pattern, that varies (e.g., randomly) between image frames, superimposed on image frames. Optionally or in combination, noise may include fluctuations in color and / or brightness and / or contrast. Video noise may (but is not necessarily) be visible to the user.

[0049] Noise can be caused by a variety of factors, such as the camera's sensitivity, ISO setting, or digital magnification settings. Noise can include internal noise, such as noise caused by electrical, thermal, or lighting levels, and / or compression artifacts, and / or interference noise, such as Gaussian noise and / or fixed-pattern noise and / or salt-and-pepper noise and / or shot noise and / or quantization and / or anisotropic noise.

[0050] Those skilled in the art will generally understand what noise in a video refers to. In the context of this invention, the term should be interpreted broadly to encompass any noise that may cause instability in the bounding box of the video. Those skilled in the art will appreciate that object detection and the calculation of the final bounding box may differ slightly when noise is present in the video.

[0051] Measuring the noise level of a video stream can be done in several ways. Typically, noise can be represented as the magnitude of variation in the point pattern between image frames. For example, if the similarity between image frames is high, disregarding other events like moving objects, the video can be said to have lower noise than a video with lower similarity. The noise level can be considered a measure of the random variations or distortions present in the video.

[0052] One metric that can be used to represent noise levels is the signal-to-noise ratio (SNR), which is a measure of the ratio of useful information to noise in a video. A higher SNR indicates a lower noise level. This can be an average over a period of time or across multiple image frames. Peak signal-to-noise ratio (PSNR) is another example of how those skilled in the art express noise levels. The above and other techniques and standards for determining video noise levels are readily available to those skilled in the art.

[0053] Noise levels can be measured over the entire area of ​​an image frame, the area covering the bounding box, the area covering the bounding box, and the additional area surrounding the bounding box. Figures 2A-2C In the example, there is region 203, which is slightly larger than the bounding box. Noise level measurements can be performed on local regions such as region 203. For example, it may be useful to focus on the region inside and near each specific bounding box. One reason for doing this is that noise levels can vary locally within an image frame. For example, if the lower left region of an image frame has more noise, the bounding box in that region may be less stable than the bounding box in another region of the image frame. Currently disclosed methods for stabilizing bounding boxes of objects in a video stream can apply different corrections to the individual bounding boxes of different regions in the video accordingly.

[0054] Similarly, in the step of measuring noise levels, sub-regions including moving objects can be ignored because moving objects can be considered pixel variations that are treated as noise, even though moving objects are not noise in the video.

[0055] Object detection

[0056] Object detection is a computer vision technique that involves locating and identifying objects within an image frame. Those skilled in the art are generally familiar with these techniques and know how to implement them.

[0057] Convolutional neural networks (CNNs) or other machine learning-based methods have become popular because they are generally very accurate and fast, but there are a number of object detection techniques that do not rely on CNNs or machine learning.

[0058] An example of an object detection algorithm is the Viola-Jones detection framework. In this method, a sliding window is used to scan image frames, where each region is classified as either containing or not containing an object. The method uses Haar features and a cascaded classifier to detect objects.

[0059] Various types of object detection methods are known in the prior art, among which object detection uses cascaded identifiers, such as those described by Viola, Paul, and Michael Jones, “Fast Object Detection Using Enhanced Cascades of Simple Features,” Computer Vision and Pattern Recognition 2001, CVPR 2001, Proceedings of the IEEE Computer Society Conference 2001, Vol. 1, IEEE, 2001. Because visual features are important for these algorithms, groups of objects with similar visual features can be detected; examples of such groups include faces, vehicles, and people. Any of these methods can be used individually or in combination to detect objects in image data. It is also possible to detect several objects in the same set of image data.

[0060] When an object is detected, a set of recognition features can be created to describe the visual appearance of the detected object. Image data from a single image frame or video sequence can be used to create recognition features of the detected object. Various image and / or video analysis algorithms can be used to extract and create recognition features from image data. Examples of such image or video analysis algorithms are various algorithms, such as those for extracting facial features, such as Turk, Matthew A., and Alex P. Pentland, “Face Recognition Using Eigenfaces,” Computer Vision and Pattern Recognition, 1991. CVPR Proceedings 1991, IEEE Computer Society Conference, IEEE, 1991; Lee, Lily, and W. Eric L. Grimson, “Gait Features, Recognition and Classification of Gait Analysis,” Automatic Face and Gesture Recognition, 2002, Proceedings, Fifth IEEE International Conference, IEEE, 2002; or color in Brogren et al., US8472714.

[0061] The database may include multiple objects and multiple recognition features. In currently disclosed methods for bounding boxes of objects in a stable video stream, the step of detecting objects in an image frame may include matching recognition features against recognition features in the database to classify the object into a specific type of object, such as a car, person, or any other item. In one embodiment, the step of detecting objects in an image frame includes comparing the image frame with reference images in the database to match features corresponding to objects in the image frame and the reference images.

[0062] Scale-Invariant Feature Transform (SIFT) is another feature-based object detection technique that uses keypoint extraction and matching to detect and track objects in video. It works by identifying unique features in an image, such as edges, corners, and freckles, and matching them across multiple frames.

[0063] The steps of detecting objects in an image frame include processing the image frame to identify predefined features of the object's characteristics, such as shape.

[0064] As described above, optionally or in combination, the step of detecting objects in an image frame may include applying a machine learning model, such as a neural network, trained to recognize objects. The neural network may include, for example, a deep learning model.

[0065] Neural networks can be used to classify objects and / or events. Classification neural networks are commonly used in applications such as character recognition, surveillance, monitoring, image analysis, and natural language processing. Various neural network algorithms / techniques exist for object classification, such as convolutional neural networks and recurrent neural networks.

[0066] Based on a non-limiting example of a setup for object detection using a neural network, the neural network is fed labeled data. For example, labeled data is an image of the object to be classified, where the image is labeled with the correct category; that is, the labeled data includes both the image data and the ground truth of the image data itself. The image data is fed into a classifier, and the ground truth is sent to a loss function calculator. The classifier processes the data representing the object to be classified and generates a classification identifier. The processing in the classifier may include applying weights to the values ​​as the data passes through the classifier. The classification identifier can be a feature vector, a classification vector, or a single value identifying the category. In the loss function, the classification identifier is compared to the ground truth using, for example, a loss function. The result of the loss function is then passed to a weight adjustment function, which is configured to adjust the weights used in the classifier. When the classifier is fully trained, it can be used to perform classification by loading the data to be classified into the classifier. The data to be classified may have the same form as the labeled data used during training, but without labels. The classifier can then output data that is identified as belonging to the category determined by the input data.

[0067] The present invention also relates to an image processing system, comprising:

[0068] At least one camera; and

[0069] The processing unit is configured as follows:

[0070] Receive a video stream, including a sequence of image frames, from the camera;

[0071] Detect objects in an image frame and generate bounding boxes around those objects; and

[0072] Measure the noise level of the video stream;

[0073] Temporal filtering of bounding boxes is performed on multiple image frames based on the measured noise level to stabilize the bounding boxes in the video stream.

[0074] Figure 4 A schematic diagram of an embodiment of a currently disclosed image processing system 400 is shown. The disclosed image processing system 400 includes a camera 410, a display 403, and a processing unit 402.

[0075] The system may, but does not necessarily, include a display for showing the video stream and the stable bounding boxes. For a user viewing the display, the stability of the bounding boxes of objects in the video stream can be a useful visual feature. However, in further applications, the bounding boxes are not necessarily displayed, but rather used in additional applications. Additional applications may include, for example, extracting statistical data or other information from the video, such as analyzing the size or orientation of objects. As those skilled in the art will recognize, the system and method are not necessarily limited to displaying the bounding boxes on a display.

[0076] The system may also include peripheral components, such as one or more storage units for storing instructions executable by the processing unit. The system may also include any of the following: internal and external network interfaces, input and / or output ports, a keyboard or mouse, etc.

[0077] As those skilled in the art will understand, the processing unit can also be a single processor in a multi-core / multi-processor system. Both the computing hardware accelerator and the central processing unit can be connected to the data communication infrastructure.

[0078] The system may include storage units, such as random access memory (RAM) and / or read-only memory (ROM) or any suitable type of memory. The system may also include a communication interface that allows software and / or data to be transferred between the system and external devices. The software and / or data transferred via the communication interface can be any suitable form of electrical signal, optical signal, or radio frequency signal. The communication interface may include, for example, a cable or wireless interface.

[0079] The present invention also relates to a computer program having instructions that, when executed by a computing device or computing system, cause the computing device or computing system to perform any embodiment of the currently disclosed methods for defining bounding boxes of objects in a stable video stream. The computer program can be stored on any suitable type of storage medium, such as a non-transitory storage medium.

Claims

1. A computer-implemented method of stabilizing a bounding box of an object in a video stream, the method comprising: receiving a video stream comprising a sequence of image frames; detecting an object in an image frame and generating a bounding box around the object; measuring a noise level of the video stream; and temporally filtering the bounding box over a plurality of image frames based on the measured noise level, thereby stabilizing the bounding box in the video stream, wherein the bounding box is temporally filtered over a plurality of previous image frames, characterized in that: a number of previous image frames is adapted such that a higher noise level means temporally filtering over a larger number of previous image frames and a lower noise level means temporally filtering over a smaller number of previous image frames. a position of the temporally filtered bounding box in a given image frame is a combination of the bounding box positions of a plurality of previous image frames.

2. The method of stabilizing a bounding box of an object in a video stream of claim 1, wherein, the noise is visual noise in the video stream, and / or wherein the noise is a pattern of dots superimposed on the image frames that varies between the image frames, and / or wherein the noise level is a magnitude of variation of the pattern of dots between the image frames.

3. The method for stabilizing a bounding box of an object in a video stream of claim 1, wherein, the noise comprises fluctuations in color and / or brightness and / or contrast, and / or wherein the noise comprises internal noise and / or compression artifacts and / or interference noise.

4. The method for stabilizing a bounding box of an object in a video stream of claim 1, wherein, the noise level is measured for an entire area of the image frames, or for an area covering the bounding box, or for an area covering the bounding box and an additional area surrounding the bounding box.

5. The method for stabilizing a bounding box of an object in a video stream of claim 1, wherein, a noise level of a sub-area comprising a moving object is ignored in the step of measuring the noise level.

6. The method for stabilizing a bounding box of an object in a video stream of claim 1, wherein, the step of temporally filtering the bounding box over a plurality of image frames based on the measured noise level is performed for every Nth image frame, wherein N is an integer greater than 1, or wherein the step of temporally filtering the bounding box over a plurality of image frames based on the measured noise level is performed for every image frame.

7. The method of stabilizing a bounding box of an object in a video stream according to any of the preceding claims, wherein, the noise level at any given point in time is measured temporally backwards for a plurality of image frames and / or for a time window.

8. The method for stabilizing a bounding box of an object in a video stream of claim 1, wherein, the step of temporally filtering the bounding box over a plurality of image frames comprises temporally smoothing the bounding box.

9. The method for stabilizing a bounding box of an object in a video stream of claim 1, wherein, the step of detecting an object in an image frame comprises applying a machine learning model trained to recognize the object.

10. The method for stabilizing a bounding box of an object in a video stream of claim 1, wherein, the step of detecting an object in an image frame comprises processing the image frame to recognize a predefined feature as a characteristic of the object.

11. The method for stabilizing a bounding box of an object in a video stream of claim 1, wherein, 12. A computer program having instructions which, when executed by a computing device or computing system, cause the computing device or computing system to perform the method of stabilizing a bounding box of an object in a video stream according to claim 1.

13. An image processing system comprising: at least one camera; and a processing unit configured to: receive a video stream comprising a sequence of image frames from the camera; detect an object in an image frame and generate a bounding box around the object; measure a noise level of the video stream; temporally filter the bounding box over a plurality of image frames based on the measured noise level to stabilize the bounding box in the video stream, ​ ​ ​ wherein the processing unit is configured to temporally filter the bounding box over a number of previous image frames, characterized in that the number of previous image frames is adapted such that a higher noise level means temporally filtering over a larger number of previous image frames and a lower noise level means temporally filtering over a smaller number of previous image frames.

Images