Method and system for producing streams of image frames

JP2022090619A5Active Publication Date: 2024-12-03AXIS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2021190667
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-12-07
Filing Date
2021-11-25
Publication Date
2024-12-03
Estimated Expiration
2041-11-25

AI Technical Summary

Technical Problem

Conventional video encoders encode entire image frames uniformly, leading to increased bandwidth and storage requirements due to less interesting parts of the image contributing significantly, especially with moving objects in the background.

Method used

Segment image frames into background and instance segments, classify objects of interest and non-interest, encode foreground objects at a high frame rate and background at a lower rate, using panoramic segmentation and motion detection to update background frames incrementally.

Benefits of technology

Reduces streaming bandwidth by encoding only interesting information at a high rate and less interesting information at a lower rate, allowing operators to focus on relevant content and detect threats accurately while optimizing visual quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

To provide methods, systems, and computer program products, for producing streams of image frames.SOLUTION: A method comprises: segmenting image frames in streaming video into background segments and instance segments composed of movable objects of interest and movable objects of non-interest; creating a background image frame that contains the background segments; and, when a movable object of non-interest has moved to reveal a background area, updating the background image frame to include the revealed background area. The method further comprises: creating a foreground image that contains the movable objects of interest; and producing a stream of encoded foreground image frames having at a first frame rate and a stream of encoded background image frames having a second frame rate that is lower than the first frame rate.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to video encoding, and more particularly to generating a video stream in a bandwidth-efficient manner.

Background Art

[0002] Cameras such as surveillance cameras are used in many different applications both indoors and outdoors to monitor various environments. Images depicting the captured scenes can be monitored, for example, by an operator or a security guard. In many situations, certain objects in the captured image may be of more interest to the operator than others. For example, an operator of a surveillance camera may be very interested in human activities, but may not be very interested in other objects in the image that move or change but are not important, such as animals, passing vehicles, or trees swaying in the wind.

[0003] However, conventional encoders generally encode the entire image frame of the video stream in the same way regardless of the operator's interest. As a result, the "less interesting" parts of the image frame are often a significant factor in bandwidth, especially when small changes due to moving objects are in the background. This can also lead to higher storage usage than may be considered necessary when only the most "interesting" information in the image or video stream is preserved. Therefore, it would be interesting to find a solution for video encoding that further reduces the bandwidth used by surveillance cameras and the storage requirements for long-term storage.

Summary of the Invention

[0004] According to a first aspect, the present invention relates to a method for generating a stream of image frames in an encoding system. The method ● Segmenting image frames in a stream of image frames into one or more background segments and one or more instance segments, ●Create a background image frame that contains one or more background segments, ● Classify at least one or more instance segments into objects of interest and objects of non-interest, ●When an indifferent movable object moves during the background update period to reveal an additional background area, the background image frame is updated to include the revealed / additional background area. ● Create a foreground image frame that encompasses the movable object of interest, ● Encoding blocks of pixels in the updated background image frame, ● Encoding blocks of pixels in the foreground image frame, ●Generating a stream of encoded foreground image frames having a first frame rate, ● To generate a stream of encoded, updated background image frames having a second frame rate lower than the first frame rate. It is equipped with.

[0005] This method reduces the streaming bandwidth required for streaming video compared to when conventional encoding is used, because only information of interest to the operator is streamed at a high image frame rate, e.g., 30 image frames per second, while information of little or no interest to the operator is streamed at a much lower rate, such as 1 image frame per minute. Because the background is streamed at a lower bitrate, it is also possible to capture slow, overall changes in the scene, such as changes in light due to a slow transition from night to day, or from sunny to cloudy, or when streetlights are turned on in the evening. This helps the operator better understand the overall scene compared to having a completely fixed background and ensures that the lighting scene is updated to be roughly the same between the two streams. Furthermore, by reducing the amount of information (i.e., image data) that the operator needs to intelligently process, the operator can focus their attention on the most important aspects of the surveillance situation and more accurately detect any potential hazards or threats. Moreover, reducing the bitrate also allows for the provision of optimal visual quality for the specific use case under consideration, as the saved bandwidth can be used to enhance instance segments. A background segment can be a background area, and an instance segment can be an object detected by object discovery. A moving object of interest can be a moving object of interest to the user, and a moving object of indifference can be a moving object of no interest to the user.

[0006] According to one embodiment, the segmentation of an image frame is performed using panoramic segmentation, in which pixels in the image frame are assigned either to a background segment representing an area containing a group of objects of a particular type, or to an instance segment representing an individual object. Panoramic segmentation is a technique well known to those skilled in the art and can be described as a combination of instance segmentation (i.e., identification and segmentation of individual instances in an image) and semantic segmentation (i.e., segmenting pixels in an image based on the class to which the pixel belongs (rather than a specific instance)). Panoramic segmentation is therefore particularly useful in this type of application where a portion of an image (e.g., the background) should be treated differently from individual objects (e.g., objects of interest and / or objects of no interest) for encoding and transmission purposes. This facilitates the integration of the present invention with existing systems that may already use panoramic segmentation for various purposes.

[0007] According to one embodiment, the method receives a user selection from a list of object types, the user selection further comprises receiving a user selection indicating which types of objects should be considered interest-moving objects and which types of objects should be considered non-interest-moving objects. Having the ability to select which objects are interest-moving and which are non-interest-moving objects provides the operator with great versatility, as this decision may change based on the time of day or the time of week. For example, a person stopping to look at a storefront during normal business hours may not be of much interest to the operator to track, whereas a person exhibiting the same behavior at 3 a.m. may be a basis for the operator's more careful attention. Thus, in such a case, the person may be selected as an interest-moving object (although the operator's interest may fluctuate depending on the time of day). However, a dog sitting outside the same storefront may be more likely to be considered a non-interest-moving object, regardless of the time of day. Furthermore, by providing a list of possible object types, the user may be presented with a limited selection of easily viewable objects. For example, even if it is possible to identify a boat, generally speaking, there are no situations in which one would find a boat outside a storefront, and therefore, even if the system may have the capability, the boat does not need to be included on the list from which the operator can select a movable object of interest.

[0008] According to one embodiment, a movable object of interest includes one or more of the following: a person, a vehicle, a weapon, a bag, and a face mask. While every surveillance situation is unique, this list represents some of the more common movable objects of interest in a typical surveillance situation. Some of these objects may not be movable on their own, but may become movable when acted upon by a person. For example, a bicycle may not be a movable object of interest on its own, but a bicycle ridden by a person into an area under surveillance is very likely to be considered a movable object of interest.

[0009] According to one embodiment, the motion of an indifferent moving object is tracked by a motion and object detector during a background update period, and the background image frame is updated several times before the expiration of the background update period. Motion and object detection can be achieved using various deep learning algorithms known to those skilled in the art. A non-exhaustive list of these techniques includes region-based convolutional networks (R-CNN), fast region-based convolutional networks (Fast R-CNN), faster region-based convolutional networks (Faster R-CNN), region-based fully convolutional networks (R-FCN), You Only Look Once (YOLO), single-shot detector (SSD), neural architecture search net (NASNet), and masked region-based convolutional networks (Mask R-CNN). Each of these explanations, along with further references to sources, can be found at https: / / medium.com / zylapp / review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852.

[0010] For example, suppose an indifferent moving object is a dog sitting in front of a wall. When the dog moves, parts of the wall previously hidden by the dog are revealed, and when the background frame is sent to the operator, it needs to be filled in so that it shows the wall where the dog was, rather than a "gap." If the background update period is 1 minute, for example, the dog may move several times and be in a completely different part of the image frame compared to where the dog was at the end of the most recent background update period. This can look awkward to the operator, and (particularly) for this reason, it is advantageous to update the background frame several times during the background update period. Furthermore, if the dog moves enough during the update time to leave the scene and reveal the entire background, the dog does not need to be rendered at all, and the entire background can be updated. This can be done in a single update (or several updates). However, if the dog moves gradually or changes direction (for example, partially to the right, then partially to the left), the update cannot be done as a single update, and several incremental updates are required during the background update period.

[0011] According to one embodiment, encoding a foreground image frame involves encoding pixel data only for pixels corresponding to objects of interest and encoding the rest of the foreground image frame as black pixels. By encoding only the pixel data for objects of interest in the conventional way and encoding the rest of the image frame as black pixels, a substantially reduced bitrate and, consequently, reduced / less bandwidth requirements are achieved when the encoded foreground image frame is transmitted. While black pixels (generally encoded as zero) are mentioned here, it should be noted that the same or very similar savings can be achieved with any consistent pixel value. Since repeating the same pixel value (black or otherwise) does not actually add any new information, the same pixel value can be compressed very efficiently into a compact representation.

[0012] Conceptually, the bitrate savings achieved through using this technique can be more easily understood by considering how black squares are encoded. Encoding all pixels in a black square by sending a byte for each pixel requires a significant amount of memory, even though the bytes always have the same value (i.e., zero). However, an equivalent way of representing a black square is to send only four values: the coordinates of the top-left corner of the black square, and the width and height of the black square. For large black squares, the data required to send this representation is almost zero compared to sending zero values ​​for all pixels in the black square. Thus, the bandwidth required when using this technique is essentially the same as the bandwidth required to send only the object of interest. Of course, different real-world encoders have many encoder-specific ways of saving bandwidth, using different encoding schemes and achieving similar results to this technique. However, the general principle remains the same: blacking out areas of interest, or encoding them using some other bandwidth-saving method, leads to substantially the same bandwidth usage as if those parts of the image were not sent at all.

[0013] According to one embodiment, the first frame rate is approximately 30 image frames per second, and the second frame rate is approximately 1 image frame per minute. Having a background image frame update rate substantially lower than a typical image frame update rate significantly reduces the amount of data transmitted and the bandwidth required.

[0014] According to one embodiment, the method further comprises classifying instance segments as indifferent static objects and updating the background image to include the indifferent static objects. For example, trees, flags, and flashing neon signs may be identified as instance segments, and parts of the objects may be moving, but generally, it is not necessary to update such objects using a high frame rate. Instead, the objects are classified as indifferent static objects and included in the background image frame, which can then be updated at a lower frame rate, again contributing to substantial savings in transmitted data.

[0015] According to one embodiment, the method further comprises verifying the completeness of the update to the background image frame at the end of the background update period, determining which indifferent moving object caused the incompleteness in response to the determination that the background image frame update was incomplete, and including the indifferent moving object that caused the incompleteness in the foreground image frame. Referring again to the above example with the dog, if the dog has not moved enough by the end of the background update period to reveal all of the background behind the dog, the method reverts to the old encoding for that region by including the dog in a faster foreground image stream. Otherwise, there would be pixel regions that do not contain the update information. This allows the background update to complete on time.

[0016] According to one embodiment, verifying integrity involves determining whether the entire background image frame has been updated. This is a standard and simple method for determining integrity, as it uses information that is already available in most systems. In one implementation, this can be done by checking, for every pixel coordinate, whether the background pixel was seen at that coordinate at any given time during the background update interval (in other words, in any image). Of course, the exact mechanism of how this is implemented in the encoder depends on the specific encoder under consideration, but the explanation of the concept above remains the same for different types of decoders.

[0017] According to one embodiment, updating the background image frame when an indifferent movable object moves to expose a background area includes comparing the movement of the indifferent movable object to one or more of area-dependent thresholds, distance-dependent thresholds, and time-dependent thresholds, and updating the background image frame when the movement of the indifferent movable object exceeds at least one threshold. For example, a minimum area of ​​movement, a minimum distance of movement, and / or a minimum duration of movement may be set for the indifferent movable object during the background update period, before the background update is triggered. Again, using the dog example, if the dog moves only a few inches or wags its tail, that may not be enough movement to trigger a background image frame update. However, if the dog moves two feet to the left or moves from lying down to sitting up, that may be enough movement to justify a background update. Strict thresholds may be configured by the operator based on a number of factors, including the type of object and the specific circumstances in the scene.

[0018] According to one embodiment, the threshold is set based on available computing resources, for example, so that the frequency of background image frame updates is limited to a frequency acceptable by the available computing resources. For example, if the camera system has limited computing resources, it may be advantageous to try to postpone any background image frame updates for as long as possible, whereas if the camera system has abundant computing resources, more frequent updates may be possible.

[0019] According to a second aspect, the present invention relates to an encoding system for generating a stream of image frames. The system includes a motion and object detector and an encoder. The motion and object detector is ● Segmenting image frames in a stream of image frames into one or more background segments and one or more instance segments, ● Classify at least one or more instance segments into objects of interest and objects of non-interest. It was configured to perform the following actions.

[0020] The encoder is ●Create a background image frame that contains one or more background segments, ●When an indifferent movable object moves during the background update period to reveal an additional background area, the background image frame is updated to include the revealed / additional background area. ● Create a foreground image frame that encompasses the movable object of interest, ● Encoding blocks of pixels in the updated background image frame, ● Encoding blocks of pixels in the foreground image frame, ● Generating a stream of encoded foreground image frames having a first frame rate; ● Generating a stream of encoded updated background image frames having a second frame rate lower than the first frame rate; and is configured to perform the following.

[0021] The advantages of the system correspond to those of the method and can be similarly modified.

[0022] According to a third aspect, the present invention relates to a computer program product for generating a stream of image frames. The computer program:[[]] ● Segmenting image frames in a stream of image frames into one or more background segments and one or more instance segments; ● Creating a background image frame including one or more background segments; ● Classifying at least some of the one or more instance segments into moving objects of interest and moving objects of no interest; ● During a background update period, when a moving object of no interest moves to reveal a (further) background area, updating the background image frame to include the revealed / further background area in the background image frame; ● Creating a foreground image frame including a moving object of interest; ● Encoding blocks of pixels of the updated background image frame; [[ID=?]]● Encoding blocks of pixels of the foreground image frame; ● Generating a stream of encoded foreground image frames having a first frame rate; ● To generate a stream of encoded, updated background image frames having a second frame rate lower than the first frame rate. It includes instructions corresponding to the steps.

[0023] Computer programs can be modified in a similar manner, each with advantages corresponding to the advantages of the method.

[0024] Details of one or more embodiments of the present invention are described in the accompanying drawings and the following description. Other features and advantages of the present invention will become apparent from the description and drawings, as well as from the claims. [Brief explanation of the drawing]

[0025] [Figure 1] This is a schematic diagram showing an exemplary environment 100 according to one embodiment, to which various methods and systems of the present invention may be applied. [Figure 2] This is a block diagram showing a detailed view of the camera system 108 illustrated in Figure 1, according to one embodiment. [Figure 3] This is a flowchart showing a method for processing a stream of image frames according to one embodiment. [Figure 4] This is a schematic diagram of a foreground image video stream and a background image video stream sent from a camera system to a receiver, according to one embodiment. [Modes for carrying out the invention]

[0026] Similar reference numerals in various drawings indicate the same elements.

[0027] As described above, the objective of various embodiments of the present invention is to reduce the bandwidth required for streaming video compared to when conventional encoding is used. This is achieved by streaming only the information of interest to the operator and streaming that information at a high rate. Information of little or no interest to the operator is streamed at a significantly lower rate.

[0028] In short, various embodiments of the present invention may be described as relating to a camera system comprising a camera that takes images of a scene, for example, a fixed camera, where the operator is interested in, for example, human activity. A fixed camera is a camera whose field of view does not change during the operation of the installation. However, the camera may be a pan-tilt-zoom (PTZ) camera, which is capable of changing the field of view of the camera in the pan and tilt directions in order to zoom in and zoom out the camera's field of view. If the camera is a PTZ camera, it should be understood that the PTZ camera should be in a static stage or static mode, in other words, the PTZ camera should be configured to have the same pan, tilt and zoom settings when capturing images of a scene to which the present invention applies. Due to the operator's interest in human activity, it is desirable to identify any information relating to such activity and send it frequently from the camera to a receiver to which the operator can view the images and monitor the human activity. This image information is called image information about the object of interest. In contrast, the background in the scene mainly works to place the actions of the foreground object in an understandable context and therefore may be updated / sent less frequently. The background is either static or may contain motion that, in a sense, is not of interest and should not be rendered. One example of such motion is tree branches swaying in the wind.

[0029] A further measure to conserve bandwidth is to ideally not transmit any image information about mobile objects other than the object of interest (i.e., objects that can change their geographical location). This not only saves bandwidth but also allows the operator to focus solely on the object of interest. Such objects are referred to herein as “non-interest mobile objects.” An example of a non-interest mobile object is an animal. Another example is some kind of vehicle. For example, if a security camera is monitoring the entrance to a building, it is generally of greater interest to track the behavior of a person walking just outside the door rather than a person on a bicycle or in a car quickly speeding through the entrance. Yet another example is that what is considered a non-interest object may be determined based on other rules, such as location. For example, one could choose to treat a person outside the surveillance area as a non-interest mobile object, while a person inside the surveillance area is treated as an object of interest, whereas a person outside the surveillance area is treated as an object of interest, as described above for animals.

[0030] During the background update period, the movement of an uninterested movable object is tracked, and the background image is updated using the portion of the background that was obscured by the uninterested movable object and revealed when the uninterested movable object moved. The background image may be updated incrementally as the uninterested object moves during the background update period. The background image is transmitted at a lower frame rate compared to the frame rate of the information containing the object of interest. As described above, the uninterested object is not shown to the operator. For this purpose, one embodiment includes refraining from including the uninterested movable object in either the background frame or the foreground frame.

[0031] In one embodiment, if an object of no interest does not move sufficiently during the background update period to reveal any background portion, the image information of the object of no interest is sent in the stream of the object of interest so as not to create a "gap" in the image. The receiver can combine the image of the image information containing the object of interest (and possibly the object of no interest) with the most recent background image. Various embodiments of the present invention are then described by reference to the drawings.

[0032] Figure 1 shows a schematic diagram of an exemplary environment 100 in which various embodiments of the present invention may be carried out. As seen in Figure 1, a scene 102 in which a person 104 is walking toward a building 106 is captured by a camera system 108. It should be noted that the depiction of scene 102 is a simplified diagram for illustrative purposes only. In a more general sense, scene 102 can be described as any three-dimensional physical space whose size and shape are defined by the field of view of the camera recording the scene.

[0033] A camera system 108, for example, a fixed camera system or a PTZ camera system in static mode, in other words, a PTZ camera system having the same fixed PTZ setting, when capturing image frames to which the present invention should be applied, is illustrated in more detail in Figure 2. The camera system 108 has a lens 110 that captures a scene 102 and projects the scene 102 onto an image sensor 112. Together, the lens 110 and the image sensor 112 form an image capturing system 109. The image sensor 112 captures a series of image frames, together forming a video stream. The image capturing system 109 is coupled to an image processing and encoding system 205, which includes an image processing pipeline 202 and an encoder 204, both of which are described in more detail below. The image processing and encoding system 205 is preferably located within the camera system 108, but may be located outside the camera system 108. For example, in a modular camera system, the image capturing system 109 and the image processing and encoding system 205 may be arranged separately from each other, or they may communicate with each other and be arranged accordingly. Furthermore, the image capturing system 109 may be movable, while the image processing and encoding system 205 may be static.

[0034] In some embodiments, such as those shown in Figure 2, the image processing and encoding system 205 further includes a motion and object detector 206. The encoder 204 and the motion and object detector 206 are provided in the encoding system 210, which is sometimes referred to in this disclosure as the extended encoding system 210 because it is an extension of a conventional encoding system. As will be described in more detail below, the image processing pipeline 202 receives a signal from the image sensor 112 and performs various types of image processing operations, and then the extended encoding system 210 encodes the video stream into a format suitable for transmission over the network to an operator via the input / output interface 208. In Figure 1, the encoded video is transmitted wirelessly via a radio link 116 to a wired network 118 and finally to a client 120 connected to the network 118, although of course there are many combinations of wireless and wired transmission models that can be used.

[0035] Client 120 has a display from which the operator can view the image video stream from the camera. Generally, client 120 is also connected to a server from which the video may be stored and / or further processed. Often, client 120 is also used to control camera 108, for example, by the operator issuing control commands on client 120. For example, the operator may instruct the camera to zoom in on a particular detail in scene 102, or to track person 104 if person 104 begins to move away from tree 106. However, there are also situations in which the operator does not control the camera, and the camera is static, merely providing an image stream for the operator to view on client 120.

[0036] As shown in Figure 2, the camera system 108 includes a lens 110 that images the scene 102 onto an image sensor 112, an image processing pipeline (IPP) 202, an encoder 204, a motion and object detector 206, and input and output interfaces 208 for communication with other devices. The IPP performs various operations on the image data received from the image sensor 112. Such operations may include filtering, demosaicing, color correction, noise filtering (to remove spatial and / or temporal noise), distortion correction (e.g., to remove the effects of barrel distortion), global and / or local tone mapping (e.g., to enable imaging of scenes encompassing a wide range of saturation), transformations (e.g., rotation), flat-field correction (e.g., to remove the effects of vignetting), and the application of overlays (e.g., privacy masks, descriptive text, etc.). The IPP 202 may be associated with the motion and object detector 206, which is used to perform object detection and classification, as well as various other functions described in more detail below. It should be noted that in some embodiments, some of these operations (e.g., barrel distortion correction, rotation, and transformation operations) may be performed by one or more subsystems outside the IPP202 in a unit between the IPP202 and the encoder204.

[0037] Following the image IPP 202, the image is transferred to the encoder 204, where the information is encoded according to the encoding protocol and transferred to the receiving client 120 via the network 118 using the input / output interface 208. The motion and object detector 206 is used to perform object detection and classification, as well as various other functions described in more detail below, in order to provide the encoder 204 with the essential information required to perform the encoding operation. It should be noted that the camera system 108 illustrated in Figure 2 also includes many other components, such as processors and memory, which are common in conventional camera systems and whose purposes and operations are well known to those skilled in the art. Such components are omitted from the examples and descriptions in Figure 2 for clarity. There are several conventional video encoding formats. Some common video encoding formats that work with various embodiments of the present invention include, to name just a few, High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2; Advanced Video Coding (AVC), also known as H.264 and MPEG-4 Part 10; Versatile Video Coding (VVC), also known as H.266, MPEG-I Part 3, and Future Video Coding (FVC); VP9, ​​VP10, and AOMedia Video1 (AV1).

[0038] Figure 3 shows a method for processing a stream of image frames captured by a camera according to one embodiment. As seen in Figure 3, the method begins in step 302 by segmenting the image frames into background segments and instance segments. This step may be performed by a motion and object detector 206. For example, the motion and object detector 206 may perform segmentation in response to a request from the encoder 204. As described above, the encoder 204 and the motion and object detector 206 are provided in the extended coding system 210. As discussed above, in one embodiment, segmentation is performed using panoramic segmentation. Panoramic segmentation creates instances of objects of interest (e.g., people) and instances of objects of no interest (e.g., animals), i.e., each individual object is identifiable. Panoramic segmentation further creates one or more background segments, i.e., regions that do not contain any instance segmentation (e.g., trees, where individual trees are not distinguishable from one another). Having this segmentation allows us to treat different objects of interest separately from each other, and also treat the background separately from the objects of interest. It should be noted that the encoding may vary depending on the particular embodiment and the scene under consideration. For example, a forest may be better encoded as a background segment, while a potted plant in an indoor scene may be encoded as a static but movable object of disinterest (because the potted plant may be moved by someone). As a result, many variations can be envisioned by those skilled in the art, depending on the specific situation under consideration.

[0039] Next, in step 304, a background image frame is created. This step may be performed by the encoder 204. The background image frame encompasses the background segments identified in step 302. In some embodiments, the background image frame also encompasses indifferent static objects, as will be described in more detail below. In other embodiments, the background encompasses only background segments. It should be understood that the creation of a background image frame is not necessarily performed for every frame. Furthermore, it should be understood that the created background image frame may be updated during the background update period using information from subsequent image frames, as will be described below with reference to step 314.

[0040] Next, in step 306, the instance segments are classified into objects of interest and objects of non-interest. This step may be performed by the motion and object detector 206. What is considered an object of interest and what is considered a object of non-interest may be determined based on the specific use case under consideration. For example, in some embodiments, the operator may choose that on a given day, cows are objects of interest and people are objects of non-interest. On a different day, the situation may be reversed, and the operator may also include cars as objects of interest, and so on. In general, the operator can select from a list of object categories that the system has been pre-trained to recognize which objects are considered objects of interest and which are considered objects of non-interest. By making this selection, only information about objects of interest is sent to the operator, and the operator is not distracted by "irrelevant" information in the video stream.

[0041] In some embodiments, there is yet another classification, namely, indifferent static objects. These objects are instance segments that, despite being static, contain some movement. An example of an indifferent static object is a tree. A tree is an instance of an object that can be identified using panoramic segmentation. A tree is static in the sense that the tree does not change location. The branches of the tree may move in the wind, but this movement is generally of little or no interest to most monitoring situations. Thus, a tree is an indifferent static object and, to save bandwidth, can be included in a background image frame that is only updated infrequently. In most embodiments, the operator is provided with an option to define which movement is "acceptable" for including an indifferent static object in a background image frame, or there may be predefined criteria for the camera system to make such a decision automatically.

[0042] In general, non-interesting moving objects, as described above, are of little or no interest and are therefore neither encoded nor sent to the operator. However, static but moving non-interesting objects (e.g., potted plants) may sometimes be included in the background, in contrast to animals that are moving and not expected to be static. In many situations, the decision of whether to include static but moving non-interesting objects in the background section depends on what the operator deems acceptable. Background image frames may be sent to the receiver and operator at a rate of approximately one image frame per minute, as will be described in more detail below. After classification in step 306, the process splits into a fast branch related to processing objects of interest, i.e., foreground image frames, and a slow branch related to processing background images. Each of these branches is then described.

[0043] In step 308, a foreground image frame containing the object of interest is created. This step may be performed by encoder 204. As described above, including only the object of interest in the foreground image frame and excluding non-interesting objects from both the foreground and background image frames makes it possible to provide the operator monitoring the scene with the most relevant information. Using scene 102 in Figure 1 as an example, if the operator is only interested in human activity, only the person 104 will be included in the foreground image frame. As noted above, a person is just one example of an object of interest. Other common examples, depending on the specific scene or surveillance situation under consideration, may include a vehicle, weapon, bag, or face mask.

[0044] After creating the foreground image frame, in step 310, blocks of pixels in each frame are encoded by the encoder 204. For the foreground image frame, the encoder 204 encodes blocks of pixels belonging to the object of interest 104 using conventional techniques, and encodes the rest of the foreground image frame as black pixels. Encoding pixels as black pixels (or any other color) allows blocks of pixels to be encoded as having location coordinates, width, and height, as discussed above, which saves a significant amount of data compared to conventional encoding. In step 312, a stream of encoded foreground image frames having a first frame rate is generated. This may be done by the encoder 204. The stream of encoded foreground image frames may be sent to a receiver using the first frame rate, or the stream of encoded foreground image frames may be sent to storage.

[0045] Next, in step 314, a timer is set that defines the background update period. During this background update period, the background image frame is updated when background areas are exposed as a result of indifferent moving objects changing their positions. This step can be performed by the encoder 204 updating the background image frame and the motion and object detector 206 determining the motion of the indifferent moving objects. These updates are performed to avoid the appearance of "gaps" in the background at the end of the background update period. The background update period is generally related to the frame rate for the background image frame, which is typically about one minute. Depending on the number of indifferent moving objects and the amount of movement, the background image frame may be updated several times during the background update period to fill any "empty areas" created as a result of the movement of the indifferent moving objects.

[0046] The movement of any indifferent moving object is tracked using the motion and object detector 206, as described above. In some embodiments, the motion and object detector 206 acts as a trigger for determining when a background image frame update is required. For example, a threshold may be set so that a background image frame update is triggered when an indifferent moving object moves beyond a certain number of pixels in the background image frame. The threshold may be set based, for example, on available computing resources. For example, a camera system with limited computing resources may update the background image frame less frequently than a camera with abundant computing resources.

[0047] In some embodiments, at the end of the background update period, the completeness of the update to the background image frame is verified to ensure a complete background image frame. This may be done by the encoder 204. In this context, “completeness” simply means ensuring that there are no “gaps” in the background image resulting from the movement of indifferent moving objects that were not filled with background pixel information at the end of the background update period. If it is determined that the update to the background image frame was incomplete, the motion and object detector 206 may be used to determine which indifferent moving object caused the incompleteness, and that object may instead be treated as part of the foreground image frame together with the moving objects of interest, as described above.

[0048] Next, similar to the high-speed branching, in the low-speed branching, in step 316, the updated background image is encoded by the encoder. Note that even if the background image frame may be updated several times during the background update period, the encoding of the background image frame is performed only once per update period, for example, at the end of each background update period. Conventional encoding techniques can be used for encoding the background image.

[0049] Finally, in step 318, a stream of encoded updated background image frames having a second frame rate is generated. As mentioned above, the second frame rate is lower than the first frame rate. The stream of encoded background image frames may be sent to the receiver at a slower frame rate compared to the frame rate of the foreground image frames. Figure 4 schematically illustrates how the stream of encoded foreground image frames 404 is sent from the camera system 108 to the receiver 402 at the first frame rate and the stream of encoded background image frames 406 is sent at the second, slower frame rate. Note that for ease of illustration in Figure 4, the background image frames are illustrated as being sent every three foreground image frames. However, in a typical scenario, the frame rate for the foreground image frame stream is generally 30 frames per second, and the frame rate for the background image frame stream is generally about 1 frame per minute. Thus, in a real-world scenario, the difference between the two streams is considerably larger than that illustrated in Figure 4. When the two image streams leave the camera system, they are in a format that can be decoded and possibly processed by a receiver 402, for example, a conventional decoder. The receiver 402 may be provided in or connected to a client 120, as illustrated in Figure 1.

[0050] At receiver 402, the two image streams are merged together to create a composite image stream for the operator to view. This can be done using a wide range of standard techniques familiar to those skilled in the art. For example, there may be gradual merging along the edges of objects to make the viewing experience more pleasing to the operator. There are many ways to achieve this gradual merging, familiar to those skilled in the art. For example, object and background pixels may be added together and averaged, weights may be applied so that higher weights are given to the background, and a blend curve specifying the weights may be used (also known as alpha blending). footnote

[0051] Although the above example was explained in the context of visible light, the same general principle of encoding and transmitting background and foreground frames at different frame rates can also be applied in the context of thermal cameras, provided that appropriate modifications are made due to the nature of the image sensors used in cameras that primarily operate in the infrared range compared to the visible light range.

[0052] The systems, image processing pipelines, encoders, motion and object detectors, and other components of the systems and methods disclosed herein may be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, the division of tasks among the functional units or components mentioned above does not necessarily correspond to the division into physical units; rather, one physical component may perform multiple functionalities, and one task may be performed jointly by several physical components.

[0053] Some or all components may be implemented as software executed by a digital signal processor or microprocessor, or as hardware or as application-specific integrated circuits. Such software may be distributed on computer-readable media, which may include computer storage media (or non-temporary media) and communication media (or temporary media). As is well known to those skilled in the art, the term computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any way or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other media that may be used to store desired information and may be accessed by a computer.

[0054] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of the system, method, and computer program product according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of an instruction comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in a block may occur outside the order shown in the figure. For example, two blocks shown consecutively may be executed virtually simultaneously, or blocks may sometimes be executed in reverse order depending on the functionality involved. It should also be noted that each block shown in the block diagram and / or flowchart, as well as combinations of blocks shown in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs a specified function or action, or a combination of dedicated hardware and computer instructions.

[0055] Those skilled in the art will understand that the embodiments described above can be modified in many ways and the advantages of the invention shown in the above embodiments can still be used. Thus, the invention should not be limited to the embodiments shown, but should be defined solely by the appended claims. Additionally, as those skilled in the art will understand, the embodiments shown can be combined.

Claims

1. 1. A method for generating a stream of image frames in an encoding system, comprising: Segmenting image frames in the stream of image frames into one or more background areas and one or more objects; creating a background image frame encompassing said one or more background areas; classifying at least some of the one or more objects into movable objects of interest and movable objects of non-interest; updating the background image frame to include the additional background area in the background image frame when, during a background update period, a non-interest movable object moves to reveal the additional background area; verifying the completeness of the updates to the background image frame at the end of the background update period; in response to determining that an update of the background image frame is incomplete, determining which non-interest movable object caused the incompleteness; including in the foreground image frame the non-interest movable object that caused the imperfection; in response to determining that the entire background image frame has been updated, refraining from including the non-interest movable object in any of the background image frame and the foreground image frame; creating a foreground image frame containing said movable object of interest; encoding blocks of pixels of the updated background image frame; encoding a block of pixels of the foreground image frame; generating a stream of encoded foreground image frames having a first frame rate; generating a stream of encoded updated background image frames having a second frame rate lower than the first frame rate; A method comprising:

2. 2. The method of claim 1, wherein the segmentation of an image frame is performed using panoramic segmentation, where pixels in the image frame are assigned either to background areas representing regions containing groups of objects of a particular type, or to instance segments representing individual objects.

3. The method of claim 1 or 2, further comprising receiving a user selection from a list of object types, the user selection indicating which types of objects should be considered as interesting movable objects and which types of objects should be considered as non-interesting movable objects.

4. The method of claim 1 , wherein the movable object of interest comprises one or more of a human, a vehicle, a weapon, a bag, and a face mask.

5. 5. The method of claim 1, wherein the movement of the non-interested movable object is tracked by a motion and object detector during the background update period, and the background image frame is updated several times before expiration of the background update period.

6. 6. The method of claim 1, wherein encoding the foreground image frame comprises encoding pixel data only for pixels corresponding to a movable object of interest and encoding the remainder of the foreground image frame as black pixels.

7. 7. The method of claim 1, wherein the first frame rate is 30 image frames per second and the second frame rate is 1 image frame per minute. Classifying an object as a non-interesting static object; updating the background image frame to include the non-interest static object; The method of claim 1 , further comprising:

9. updating the background image frame when a non-interest movable object moves to reveal a background area; comparing the movement of the non-interest movable object to one or more of an area-dependent threshold, a distance-dependent threshold and a time-dependent threshold; updating the background image frame when the movement of the non-interest movable object exceeds at least one threshold; The method of any one of claims 1 to 8, comprising:

10. The method of claim 9 , further comprising setting the threshold based on available computing resources.

11. The method of claim 10 , wherein setting the threshold comprises setting the threshold such that a frequency of updates of the background image frame is limited to a frequency of updates that can be accommodated by available computing resources.

12. 1. An encoding system for generating a stream of image frames, comprising an encoder and a motion and object detector, the system comprising: The motion and object detector, Segmenting image frames in the stream of image frames into one or more background areas and one or more objects; classifying at least some of the one or more objects into movable objects of interest and movable objects of non-interest; configured to: The encoder comprises: creating a background image frame encompassing said one or more background areas; updating the background image frame to include the additional background area in the background image frame when, during a background update period, a non-interest movable object moves to reveal the additional background area; verifying the completeness of the updates to the background image frame at the end of the background update period; in response to determining that an update of the background image frame is incomplete, determining which non-interest movable object caused the incompleteness; including in the foreground image frame the non-interest movable object that caused the imperfection; in response to determining that the entire background image frame has been updated, refraining from including the non-interest movable object in any of the background image frame and the foreground image frame; creating a foreground image frame containing said movable object of interest; encoding blocks of pixels of the updated background image frame; encoding a block of pixels of the foreground image frame; generating a stream of encoded foreground image frames having a first frame rate; generating a stream of encoded updated background image frames having a second frame rate lower than the first frame rate; configured to: Coding system.

13. 1. A computer program product for generating a stream of image frames, the computer program product comprising a computer readable storage medium having program instructions embodied therein, the computer readable storage medium being not a transitory signal itself, the program instructions being: Segmenting image frames in the stream of image frames into one or more background areas and one or more objects; creating a background image frame encompassing said one or more background areas; classifying at least some of the one or more objects into movable objects of interest and movable objects of non-interest; updating the background image frame to include the additional background area in the background image frame when, during a background update period, a non-interest movable object moves to reveal the additional background area; verifying the completeness of the updates to the background image frame at the end of the background update period; in response to determining that an update of the background image frame is incomplete, determining which non-interest movable object caused the incompleteness; including in the foreground image frame the non-interest movable object that caused the imperfection; in response to determining that the entire background image frame has been updated, refraining from including the non-interest movable object in any of the background image frame and the foreground image frame; creating a foreground image frame containing said movable object of interest; encoding blocks of pixels of the updated background image frame; encoding a block of pixels of the foreground image frame; generating a stream of encoded foreground image frames having a first frame rate; generating a stream of encoded updated background image frames having a second frame rate lower than the first frame rate; To carry out the method comprising: A computer program product executable by a processor.