Method and system for generating a stream of image frames

Abstract

Methods and systems are provided for generating a stream of image frames. Image frames in a streaming video are segmented into background segments and instance segments. A background image frame containing the background segments is created. At least some of the instance segments are classified as a movable object of interest and a movable object of non-interest. During a background update time period, when the movable object of non-interest has moved to reveal further background area, the background image frame is updated to include the further background area in the background image frame. A foreground image containing the movable object of interest is created. Pixel blocks of the updated background image frame and pixel blocks of the foreground image frame are encoded. A stream of encoded foreground image frames is generated having a first frame rate. A stream of encoded, updated background image frames is generated having a second, lower frame rate.

Description

Technical Field

[0001] This invention relates to video coding, and more specifically, to generating video streams in a bandwidth-efficient manner. Background Technology

[0002] Cameras, such as surveillance cameras, are used in many different applications, both indoors and outdoors, to monitor a variety of environments. Images depicting the captured scene can be monitored by, for example, operators or security personnel. In many cases, operators may be more interested in certain objects in the captured image than others. For example, a surveillance camera operator may be very interested in human activity but less interested in other moving or changing (but not important) objects in the image, such as animals, passing vehicles, or trees moving in the wind.

[0003] However, traditional encoders typically encode the entire image frame of a video stream in the same way, regardless of the operator's interest. Therefore, the "less interesting" portions of an image frame often significantly increase bandwidth usage, especially when the background changes slightly due to moving objects. This can also lead to higher storage usage than if only the most "interesting" information in the image or video stream were retained. Therefore, it is of interest to find video encoding solutions that can further reduce the bandwidth used by surveillance cameras and reduce long-term storage requirements. Summary of the Invention

[0004] According to a first aspect, the present invention relates to a method for generating an image frame stream in an encoding system. The method includes:

[0005] • Divide the image frames in the image frame stream into one or more background segments and one or more instance segments;

[0006] • Create a background image frame containing one or more background segments;

[0007] • Classify at least some of the instance segments in one or more instance segments into movable objects of interest and non-movable objects of interest;

[0008] • During the background update period, when a non-interesting movable object has moved to reveal a (further) background area, update the background image frame to include the revealed / further background area in the background image frame.

[0009] • Create a foreground image frame containing the movable object of interest;

[0010] • Encode the pixel blocks of the updated background image frame;

[0011] • Encode the pixel blocks of the foreground image frame;

[0012] • Generate an encoded foreground image frame stream with a first frame rate; and

[0013] • Generate an encoded, updated background image frame stream with a second frame rate lower than the first frame rate.

[0014] Compared to traditional encoding, this method reduces the streaming bandwidth required for streaming video because only information of interest to the operator is streamed at a high image frame rate (e.g., 30 image frames per second), while information of little or no interest to the operator is streamed at a much lower rate (e.g., one image frame per minute). Because the background is streamed at a low bit rate, slow overall changes in the scene, such as changes in light due to a slow transition from night to day, or from sunny to cloudy, or when streetlights come on at night, can also be captured. This helps the operator better understand the overall scene compared to having a completely fixed background and ensures that the lighting settings are updated to be roughly the same between the two streams. Furthermore, by reducing the amount of information (i.e., image data) that the operator needs to process in their mind, she can focus her attention on the most important aspects of the monitored scene and more accurately detect any potential dangers or threats. Even further, the reduced bit rate can also provide optimal visual quality for the specific use case at hand, as the saved bandwidth can be used to enhance instance segments. Background segments can be background areas, and instance segments can be objects detected through object detection. The movable objects that the user is interested in can be movable objects that the user is interested in, and the movable objects that the user is not interested in can be movable objects that the user is not interested in.

[0015] According to one embodiment, image frame segmentation is performed using panoramic segmentation, where pixels in the image frame are assigned to either background segments representing regions comprising a set of objects of a specific type, or to instance segments representing a single object. Panoramic segmentation is a technique well known to those skilled in the art and can be described as a combination of instance segmentation (i.e., identifying and segmenting individual instances in an image) and semantic segmentation (i.e., segmenting pixels in an image based on the class to which they belong (rather than a specific instance)). Therefore, panoramic segmentation is particularly suitable for this type of application, where a portion of an image (e.g., the background) should be treated differently from individual objects (e.g., objects of interest and / or objects of non-interest) for encoding and transmission. This facilitates the integration of the present invention with existing systems that may have already used panoramic segmentation for various purposes.

[0016] According to one embodiment, the method further includes receiving a user selection from a list of object types, the user selection indicating which types of objects should be considered as movable objects of interest and non-movable objects of interest. Having the ability to select which objects are movable objects of interest and non-movable objects of interest provides operators with great versatility, as this determination can vary based on the time of day or the time of week. For example, an individual stopping to look at the storefront during normal business hours may not be of great interest to the operator, while someone exhibiting the same behavior at 3 a.m. might be worthy of closer attention. Therefore, in this case, a human could be selected as a movable object of interest (even though the operator's interest may vary depending on the time of day). However, a dog sitting outside the same storefront, regardless of the time of day, might be considered a movable object of non-interest. Furthermore, by providing a list of possible object types, a limited and easily understood selection of objects can be presented to the user. For example, even if a boat can be identified, it is not typically found in a storefront, and therefore, even if the system might have the capability to do so, it is not necessary to include boats in the list from which the operator can select movable objects of interest.

[0017] According to one embodiment, movable objects of interest include one or more of the following: humans, vehicles, weapons, bags, and masks. Each surveillance scenario is unique, but this list represents some of the more common movable objects in common surveillance scenarios. While some of these objects may not be movable by themselves, they may move when manipulated by humans. For example, a bicycle itself may not be a movable object of interest, but a bicycle ridden into the monitored area by a person is likely to be considered a movable object of interest, and so on.

[0018] According to one embodiment, during the background update period, the movement of non-interesting movable objects is tracked by a motion and object detector, and the background image frame is updated multiple times before the background update period expires. Motion and object detection can be performed using a range of deep learning algorithms familiar to those skilled in the art. A non-exhaustive list of these techniques includes: Region-based Convolutional Networks (R-CNN), Region-based Fast Convolutional Networks (Fast R-CNN), Region-based Faster Convolutional Networks (Faster R-CNN), Region-based Fully Convolutional Networks (R-FCN), YouOnly Look Once (YOLO), Single Shot Monitor (SSD), Neural Architecture Search Network (NASNet), and Mask Region-based Convolutional Networks (Mask R-CNN). Descriptions of each of these can be found at https: / / medium.com / zylapp / review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852, and further details can be found in other resources.

[0019] For example, suppose the moving object that isn't of interest is a dog sitting in front of a wall. When the dog moves, the portion of the wall that was previously hidden by the dog will become visible, and the wall needs to be filled in to show it, rather than leaving an "open hole" where the dog was when the background frame is sent to the operator. For example, if the background update cycle is one minute, the dog might move multiple times, and it will end up in a completely different part of the image frame compared to where it was at the end of the last background update period. This can look awkward to the operator, and for this reason (among others), it is advantageous to update the background frame multiple times during the background update period. Furthermore, if the dog moves enough during the update period to leave the scene and reveal the entire background, the dog doesn't need to be rendered at all; the entire background can be updated. This can be done in a single update (or multiple updates). However, if the dog moves bit by bit or changes direction (e.g., partially to the right and then partially to the left), the update cannot be done as a single update and requires multiple incremental updates during the background update period.

[0020] According to one embodiment, encoding a foreground image frame includes encoding only the pixel data corresponding to the movable object of interest, and encoding the remainder of the foreground image frame as black pixels. Encoding only the pixel data for the movable object of interest in a conventional manner and encoding the remainder of the image frame as black pixels results in a significant reduction in bit rate, and thus reduces / decreases the bandwidth requirements when transmitting the encoded foreground image frame. It should be noted that although black pixels (typically encoded as zero) are mentioned here, the same or very similar savings can be achieved for any consistent pixel value. Since repeating the same pixel value (black or otherwise) does not really add any new information, it can be compressed into a compact representation very efficiently.

[0021] Conceptually, the bit rate savings achieved by using this technique are easier to understand by considering how black squares are encoded. Encoding each pixel in a black square by sending a byte for each pixel, even if that byte always has the same value (i.e., zero), requires a significant amount of memory. However, an equivalent way to represent a black square is to send the coordinates of its top-left corner, along with the width and height of the black square—only four values. For large black squares, the amount of data required to send this representation is negligible compared to sending a zero value for each pixel in the black square. Therefore, the bandwidth required using this technique is essentially the same as the bandwidth required to send only the object of interest. Of course, different real-world encoders use different encoding schemes, and many encoder-specific bandwidth-saving methods exist to achieve similar results. However, the general principle remains the same: blacking out the non-interested regions, or using other bandwidth-saving methods to encode the non-interested regions, makes the bandwidth usage essentially the same as if these parts of the image were not sent at all.

[0022] According to one embodiment, the first frame rate is approximately thirty image frames per second, and the second frame rate is approximately one image frame per minute. A background image frame update rate that is significantly lower than the typical image frame update rate significantly reduces the amount of data transmitted and the required bandwidth.

[0023] According to one embodiment, the method further includes classifying the instance segment as a stationary object of non-interest, and updating the background image to include the non-interested stationary object. For example, trees, signs, flashing neon lights, etc., can be identified as instance segments, and although a portion of the object may be movable, it is generally not necessary to update such objects at a fast frame rate. Instead, the object can be classified as a stationary object of non-interest and included in the background image frame, and thus updated at a slower frame rate, which again helps to significantly save transmitted data.

[0024] According to one embodiment, the method further includes: at the end of the background update period, verifying the completeness of the update to the background image frame; in response to determining that the background image frame update is incomplete, determining which non-interested movable object caused the incompleteness; and including the non-interested movable object causing the incompleteness in the foreground image frame. Referring again to the dog example above, if the dog has not yet moved sufficiently to reveal all the background behind it at the end of the background update period, a fallback to conventional encoding for that region can be made by including the dog in the faster foreground image stream. Otherwise, there will be pixel regions that do not contain update information. This allows the background update to complete on time.

[0025] According to one embodiment, verifying integrity involves determining whether the entire background image frame has been updated. This is a standard, straightforward method for determining integrity because it uses information already available in most systems. In one implementation, this can be done by checking whether, for each pixel coordinate, a background pixel has been seen at any time during the background update interval (i.e., in any image). Of course, the exact mechanism used to implement this in the encoder will depend on the specific encoder at hand, but the above conceptual description will remain unchanged for different types of decoders.

[0026] According to one embodiment, updating a background image frame when a non-interesting movable object has moved into an exposed background area includes: comparing the movement of the non-interesting movable object with one or more of the following: a region-related threshold, a distance-related threshold, and a time-related threshold; and updating the background image frame when the movement of the non-interesting movable object exceeds at least one threshold. For example, during a background update period, a minimum movement area, a minimum movement distance, and / or a minimum time period can be set for the non-interesting movable object before triggering a background update. Similarly, taking a dog as an example, if the dog only moves a few inches or only wags its tail, it may not be enough movement to trigger a background image frame update. However, if the dog moves two feet to the left or moves from lying down to sitting up, it may be enough movement to make a background update necessary. The operator can configure the precise thresholds based on various factors, such as the type of object and the specific circumstances of the scene.

[0027] According to one embodiment, a threshold is set based on available computing resources, for example, such that the frequency of background image frame updates is limited to an update frequency that the available computing resources can accommodate. For example, if the camera system has limited computing resources, it may be advantageous to try to postpone any background image frame update for as long as possible, while if the camera system has sufficient computing resources, more frequent updates can be performed.

[0028] According to a second aspect, the present invention relates to an encoding system for generating image frame streams. The system includes a motion and object detector and an encoder. The motion and object detector is configured to:

[0029] • Divide the image frames in the image frame stream into one or more background segments and one or more instance segments; and

[0030] • Classify at least some of the instance segments in one or more instance segments into movable objects of interest and movable objects of non-interest.

[0031] The encoder is configured as follows:

[0032] • Create a background image frame containing one or more background segments;

[0033] • During the background update period, when a non-interesting movable object has moved to reveal a (further) background area, update the background image frame to include the revealed / further background area in the background image frame.

[0034] • Create a foreground image frame containing the movable object of interest;

[0035] • Encode the pixel blocks of the updated background image frame;

[0036] • Encode the pixel blocks of the foreground image frame;

[0037] • Generate an encoded foreground image frame stream with a first frame rate; and

[0038] • Generate an encoded, updated background image frame stream with a second frame rate lower than the first frame rate.

[0039] The advantages of the system correspond to those of the method and can vary similarly.

[0040] According to a third aspect, the present invention relates to a computer program product for generating image frame streams. The computer program includes instructions corresponding to the following steps:

[0041] • Divide the image frames in the image frame stream into one or more background segments and one or more instance segments;

[0042] • Create a background image frame containing one or more background segments;

[0043] • Classify at least some of the instance segments in one or more instance segments into movable objects of interest and non-movable objects of interest;

[0044] • During the background update period, when a non-interesting movable object has moved to reveal a (further) background area, update the background image frame to include the revealed / further background area in the background image frame.

[0045] • Create a foreground image frame containing the movable object of interest;

[0046] • Encode the pixel blocks of the updated background image frame;

[0047] • Encode the pixel blocks of the foreground image frame;

[0048] • Generate an encoded foreground image frame stream with a first frame rate; and

[0049] • Generate an encoded, updated background image frame stream with a second frame rate lower than the first frame rate.

[0050] Computer programs have advantages corresponding to the advantages of methods and can be similarly varied.

[0051] Details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features and advantages of the invention will be apparent from the description, the drawings, and the claims. Attached Figure Description

[0052] Figure 1 This is a schematic diagram illustrating an exemplary environment 100 in which various methods and systems of the present invention can be applied according to one embodiment.

[0053] Figure 2 This illustrates an embodiment. Figure 1 A block diagram showing a detailed view of the camera system 108 shown in the figure.

[0054] Figure 3 This is a flowchart illustrating a method for processing an image frame stream according to one embodiment.

[0055] Figure 4 This is a schematic diagram of a foreground image video stream and a background image video stream transmitted from a camera system to a receiver according to one embodiment.

[0056] The same reference numerals in the various figures denote the same elements. Detailed Implementation

[0057] As described above, the goal of various embodiments of the present invention is to reduce the bandwidth required for streaming video compared to using conventional encoding. This is achieved by streaming only the information of interest to the operator and streaming that information at a high rate. Information of little or no interest to the operator is streamed at a significantly lower rate.

[0058] In short, various embodiments of the invention can be described as relating to camera systems including cameras (e.g., fixed cameras) that, for example, capture images of scenes in which an operator is of interest to human activity. A fixed camera is one whose field of view does not change during operation once installed. However, the camera can be a translation-tilt-zoom (PTZ) camera capable of changing its field of view in both translation and tilt directions and magnifying and reducing its field of view. In the case of a PTZ camera, it should be understood that when capturing images of the scene to which the invention is applied, the PTZ camera will be in a stationary phase or static mode, i.e., the PTZ camera should be set to have one or more translation, tilt, and zoom settings. Because the operator is of interest to human activity, it is necessary to identify any information related to such activity and frequently transmit any information related to such activity from the camera to a receiver, where the operator can view the images and monitor human activity. This image information is referred to as image information of the object of interest. Conversely, the background in the scene is primarily used to place the actions of foreground objects within an understandable context and therefore can be updated / transmitted less frequently. The background is either static or may contain objects whose motion is, in some sense, not of interest and should not be rendered. An example of this movement is tree branches swaying in the wind.

[0059] As a further means of reducing bandwidth, image information relating to movable objects other than the object of interest (i.e., objects whose geographical location can change) is ideally not sent at all. This not only saves bandwidth but also allows the operator to focus only on the object of interest. Such objects will be referred to as "non-interesting movable objects" in this document. Examples of non-interesting movable objects are animals. Another example is certain types of vehicles. For instance, if a security camera is monitoring the entrance to a building, it is generally more interested in tracking the behavior of people walking outside the door than people on bicycles or in cars quickly passing by the building entrance. In yet another example, what is considered a non-interesting object can be determined based on other rules such as location. For example, one could choose to consider people outside the monitored area as non-interesting movable objects and treat them as described above for animals, while considering people within the monitored area as objects of interest.

[0060] During the background update period, the movement of non-interested movable objects is tracked, and the background image is updated with portions of the background (parts obscured by the non-interested movable objects and revealed as they move). The background image can be gradually updated as the non-interested objects move during the background update period. The background image is sent at a lower frame rate compared to the frame rate containing information about the objects of interest. As described above, the non-interested objects will not be shown to the operator. To this end, one embodiment includes avoiding including non-interested movable objects in either the background frame or the foreground frame.

[0061] In one embodiment, if a non-interested object has not moved sufficiently to reveal any background portion during the background update period, image information relating to the non-interested object is sent in the stream of the object of interest to avoid creating "holes" in the image. The receiver can then combine the image information with the image of the object of interest (and potentially the non-interested object) and the latest background image. Various embodiments of the invention will now be described by way of example and with reference to the accompanying drawings.

[0062] Figure 1 A schematic diagram of an exemplary environment 100 in which various embodiments of the present invention can be implemented is shown. Figure 1 As can be seen, scene 102, in which person 104 walks towards building 106, is captured by camera system 108. It should be noted that the scene 102 depicted is only a simplified view for illustrative purposes. Scene 102 can be described in a more general sense as any three-dimensional physical space, the size and shape of which are defined by the field of view of the camera recording the scene.

[0063] exist Figure 2A camera system, such as a fixed camera system or a PTZ camera system in still mode (i.e., a PTZ camera system having the same fixed PTZ setting when capturing image frames to which the present invention is applied), is illustrated in more detail below. Camera system 108 has a lens 110 that captures scene 102 and projects it onto image sensor 112. Lens 110 and image sensor 112 together form image capture system 109. Image sensor 112 captures a series of image frames that collectively form a video stream. Image capture system 109 is coupled to image processing and encoding system 205, which includes image processing pipeline 202 and encoder 204 (both of which will be described in more detail below). Image processing and encoding system 205 is preferably located inside camera system 108, but may also be located outside of camera system 108. For example, in a modular camera system, image capture system 109 and image processing and encoding system 205 may be arranged separately from each other and arranged to communicate with each other. Furthermore, image capture system 109 may be movable, while image processing and encoding system 205 may be fixed.

[0064] In some embodiments (such as) Figure 2 In the embodiment shown, the image processing and encoding system 205 further includes a motion and object detector 206. The encoder 204 and the motion and object detector 206 are included in the encoding system 210, which is sometimes referred to herein as the enhanced encoding system 210 because it is enhanced compared to a conventional encoding system. The image processing pipeline 202 receives signals from the image sensor 112 and performs various types of image processing operations, and then the enhanced encoding system 210 encodes the video stream into a format suitable for transmission to an operator via an input / output interface 208 over a network, as will be described in more detail below. Figure 1 In this process, the encoded video is wirelessly transmitted to the wired network 118 via radio link 116 and eventually to the client 120 connected to the network 118. However, there are, of course, various combinations of wireless and wired transmission modes that can be used.

[0065] Client 120 has a display on which an operator can view the video stream from the camera. Typically, client 120 is also connected to a server where the video can be stored and / or further processed. Client 120 is also typically used to control camera 108, for example, by issuing control commands from the operator at client 120. For example, the operator could instruct the camera to zoom in on specific details of scene 102, or instruct it to track person 104 as they begin to move away from tree 106. However, there are also cases where the operator does not control the camera, and the camera is stationary and only provides a stream of images for the operator to view on client 120.

[0066] like Figure 2 As shown, the camera system 108 includes: a lens 110 that images the scene 102 onto an image sensor 112, an image processing pipeline (IPP) 202, an encoder 204, a motion and object detector 206, and an input and output interface 208 for communicating with other devices. The IPP performs a range of operations on the image data received from the image sensor 112. Such operations may include: filtering, desacrifice, color correction, noise filtering (for eliminating spatial and / or temporal noise), distortion correction (for eliminating effects such as barrel distortion), global and / or local tone mapping (e.g., enabling imaging of scenes with a wide intensity range), transformation (e.g., rotation), flattening correction (e.g., for eliminating vignetting effects), and applying overlays (e.g., privacy masking, explanatory text, etc.). The IPP 202 may be associated with the motion and object detector 206, which performs object detection and classification, as well as a range of other functions described in further detail below. It should be noted that in some embodiments, some of these operations (e.g., transformation operations such as barrel distortion correction, rotation, etc.) may be performed by one or more subsystems outside of IPP 202 (e.g., in a unit between IPP 202 and encoder 204).

[0067] After image IPP 202, the image is forwarded to encoder 204, where the information is encoded according to the encoding protocol and forwarded to receiving client 120 via network 118 using input / output interface 208. Motion and object detector 206 performs object detection and classification, as well as a series of other functions, which will be described in further detail below, to provide encoder 204 with the necessary information required to perform the encoding operation. It should be noted that... Figure 2 The camera system 108 illustrated also includes many other components such as a processor and memory, which are common in conventional camera systems, and whose purpose and operation are well known to those skilled in the art. For clarity, in Figure 2 These components are omitted from the illustrations and descriptions. Many conventional video coding formats exist. Some common video coding formats that work with the various embodiments of the present invention include (to name only): High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2; Advanced Video Coding (AVC), also known as H.264 and MPEG-4 Part 10; Universal Video Coding (VVC), also known as H.266, MPEG-I Part 3, and Future Video Coding (FVC); VP9, ​​VP10, and AOMedia Video 1 (AV1).

[0068] Figure 3 A method for processing a stream of image frames captured by a camera, according to one embodiment, is shown. Figure 3 As can be seen, the method first segments the image frame into background segments and instance segments (step 302). This step can be performed by the motion and object detector 206. For example, the motion and object detector 206 can perform segmentation in response to a request from the encoder 204. As described above, the encoder 204 and the motion and object detector 206 are included in the enhanced coding system 210. As described above, in one embodiment, this segmentation is accomplished using panoramic segmentation. Panoramic segmentation creates instances of objects of interest (e.g., people) and instances of non-objects of interest (e.g., animals), i.e., each individual object is identifiable. Panoramic segmentation further creates one or more background segments, i.e., regions that do not contain any instance segments (e.g., trees, and the individual trees are indistinguishable from each other). With this segmentation, different objects of interest can be processed differently from each other, and backgrounds that are different from the objects of interest can also be processed. It should be noted that the encoding can vary depending on the specific embodiment and scenario at hand. For example, a forest may be better encoded as a background segment, while potted plants in an indoor environment may be encoded as non-objects of interest that are stationary but movable (because potted plants may also be moved by someone). Therefore, depending on the specific circumstances at hand, those skilled in the art can conceive of many variations.

[0069] Next, a background image frame is created (step 304). This step can be performed by encoder 204. The background image frame contains the background segments identified in step 302. In some embodiments, the background image frame also contains stationary objects of non-interest, as will be described in further detail below. In other embodiments, the background contains only background segments. It should be understood that the creation of the background image frame is not performed for each frame. It should also be understood that the created background image frame can be updated during the background update time period using information from subsequent image frames, as will be described below with reference to step 314.

[0070] Next, the instance segments are categorized into moving objects of interest and non-moving objects of interest (step 306). This step can be performed by the motion and object detector 206. What is considered a moving object of interest and what is not can be determined based on the specific use case at hand. For example, in some embodiments, an operator might select cows as moving objects of interest and people as non-moving objects of interest on a given day. On different days, the situation might be reversed, and the operator might also select cars as moving objects of interest, etc. Typically, the operator can select which objects are considered moving objects of interest and what are not from a pre-trained list of object categories already identified by the system. By making this selection, only information about moving objects of interest is sent to the operator, and she is not distracted by "irrelevant" information in the video stream.

[0071] In some embodiments, another category exists: non-interesting stationary objects. These objects are instance segments that, although stationary, still contain some movement. An example of a non-interesting stationary object is a tree. A tree is an instance of an object that can be identified using panorama segmentation. A tree is stationary in a sense because it does not change position. Branches may move in the wind, but for most monitoring scenarios, this movement is usually of little or no interest. Therefore, trees are non-interesting stationary objects, and to save bandwidth, trees can be added to background image frames that are not frequently updated. In most embodiments, the operator is given the option to define what movement is "acceptable" for including non-interesting stationary objects in the background image frame, or predefined criteria may exist for the camera system to make such a decision automatically.

[0072] Typically, non-interesting movable objects are neither encoded nor sent to the operator, as they are rarely of interest or are not of interest, as stated above. However, unlike movable but not expected to be stationary animals, non-interesting stationary but movable objects (e.g., potted plants) can sometimes be included in the background. In many cases, the decision of whether to include non-interesting stationary but movable objects in the background depends on what the operator considers acceptable items. As will be described in further detail below, background image frames can be sent to the receiver and operator at a rate of approximately one image frame per minute. Following the classification in step 306, the process is divided into a fast branch related to the processing of moving objects of interest (i.e., foreground image frames) and a slow branch related to the processing of background images. Each of these branches will now be described.

[0073] In step 308, a foreground image frame containing the movable object of interest is created. This step can be performed by encoder 204. As described above, including the movable object of interest only in the foreground image frame and excluding non-movable objects of interest from the foreground and background image frames allows the most relevant information to be provided to the operator monitoring the scene. Figure 1 Using scenario 102 as an example, if the operator is only interested in human activity, the foreground image frame will only include people 104. As mentioned above, people are just one example of movable objects of interest. Other common examples include vehicles, weapons, bags, or masks, depending on the specific scene or surveillance situation at hand.

[0074] After the foreground image frames are created, the pixel blocks in each frame are encoded by encoder 204 (step 310). For the foreground image frames, encoder 204 encodes the pixel blocks belonging to the moving object 104 of interest using conventional techniques, and encodes the remainder of the foreground image frame as black pixels. Encoding pixels as black pixels (or any other color) allows pixel blocks to be encoded with position coordinates, width, and height, as described above, which saves a significant amount of data compared to conventional encoding. In step 312, an encoded foreground image frame stream with a first frame rate is generated. This can be performed by encoder 204. The encoded foreground image frame stream can be sent to a receiver at the first frame rate, or the encoded foreground image frame stream can be sent to storage.

[0075] Now we move to the slow branch of process 300, where in step 314, a timer is set to define a background update period. During this background update period, background image frames are updated when background areas are exposed due to changes in the position of non-interested movable objects. This step can be performed by encoder 204 updating the background image frames and motion and object detector 206 determining the movement of non-interested movable objects. These updates are performed to avoid “holes” in the background at the end of the background update period. The background update period is typically related to the frame rate of the background image frames, generally around one minute. Depending on the number and amount of movement of non-interested movable objects, the background image frames may be updated multiple times during the background update period to fill any “blank areas” created by the movement of non-interested movable objects.

[0076] As described above, a motion and object detector 206 is used to track the movement of any non-interesting movable objects. In some embodiments, the motion and object detector 206 serves as a trigger for determining when a background image frame needs to be updated. For example, a threshold can be set such that an update of the background image frame is triggered if a non-interesting movable object moves within the background image frame by more than a certain number of pixels. For example, the threshold can be set based on available computing resources. For instance, a camera system with limited computing resources may update the background image frame less frequently than a camera with abundant computing resources.

[0077] In some embodiments, at the end of the background update period, the integrity of the update to the background image frame is verified to ensure a complete background image frame. This can be performed by encoder 204. In this context, "integrity" means only ensuring that there are no "holes" in the background image caused by the movement of a non-interesting movable object that have not been filled with background pixel information at the end of the background update period. If it is determined that the update to the background image frame is incomplete, motion and object detector 206 can be used to determine which non-interesting movable object caused the incompleteness, and that object can instead be processed as part of the foreground image frame along with the movable object of interest, as described above.

[0078] Next, similar to the fast branch, in the slow branch, the updated background image is encoded by the encoder (step 316). It should be noted that even though the background image frame can be updated multiple times during the background update time period, the background image frame is encoded only once per update time period (e.g., once at the end of each background update time period). The background image can be encoded using conventional encoding techniques.

[0079] Finally, in step 318, an encoded, updated background image frame stream with a second frame rate is generated. As mentioned above, the second frame rate is lower than the first frame rate. Compared to the frame rate of the foreground image frames, the encoded background image frame stream can be sent to the receiver at a slower frame rate. Figure 4 The diagram schematically illustrates how an encoded foreground image frame stream 404 is transmitted from camera system 108 to receiver 402 at a first frame rate, and how an encoded background image frame stream 406 is transmitted from camera system 108 to receiver 402 at a slower second frame rate. It should be noted that in Figure 4 In the illustration, for ease of explanation, background image frames are shown as being sent every three foreground image frames. However, in typical scenarios, the frame rate of the foreground image frame stream is typically 30 frames per second, and the frame rate of the background image frame stream is typically about one frame per minute. Therefore, in real-world scenarios, the difference between the two streams is significant. Figure 4The differences illustrated in the diagram are much greater. When the two image streams leave the camera system, they are in a format that can be decoded by receiver 402 (e.g., a conventional decoder) and otherwise processed. Receiver 402 can be included in... Figure 1 The client 120 shown in the diagram can connect to... Figure 1 The client shown in the diagram is 120.

[0080] At receiver 402, the two image streams are merged together to create a composite image stream for the operator to view. This can be accomplished using a variety of standard techniques familiar to those skilled in the art. For example, the blending can be gradual along the edges of the object to enhance the operator's viewing experience. There are many ways to achieve this gradual blending, which are familiar to those skilled in the art. For example, object pixels and background pixels can be added and averaged, weights can be applied such that higher weights are assigned to the background, and a blending curve with specified weights (also known as alpha blending) can be used.

[0081] Conclusion

[0082] Although the example above is described against a visible light background, the same general principles for encoding and transmitting background and foreground frames at different frame rates can also be applied to the background of thermal imagers with appropriate modifications, mainly due to the nature of the image sensors used in cameras that operate in both the visible and infrared light ranges.

[0083] The systems, system components (such as image processing pipelines, encoders, and motion and object detectors) and methods disclosed herein can be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, the task division between functional units or components described above does not necessarily correspond to the division of physical units; rather, a physical component can perform multiple functions, and a task can be completed collaboratively by multiple physical components.

[0084] Specific components or all components may be implemented as software executed by a digital signal processor or microprocessor, or as hardware or application-specific integrated circuits. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (e.g., computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disc storage, magnetic recording tape, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer.

[0085] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a portion of a module, segment, or instruction comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions marked in the blocks may occur in a non-consecutive order. For example, depending on the function involved, two consecutively shown blocks may actually be executed substantially simultaneously, or sometimes these blocks may be executed in reverse order. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by systems based on dedicated hardware that perform the specified functions or actions or execute combinations of dedicated hardware and computer instructions.

[0086] It should be understood that those skilled in the art can modify the above embodiments in various ways and still utilize the advantages of the invention as shown in the above embodiments. Therefore, the invention should not be limited to the illustrated embodiments but should be defined only by the appended claims. Furthermore, as those skilled in the art will understand, the illustrated embodiments can be combined.

Claims

1. A method for generating an image frame stream in an encoding system, comprising: The image frames in the image frame stream are divided into one or more background segments and one or more instance segments; Create a background image frame that contains the one or more background segments; Classify at least some of the instance segments into movable objects of interest and movable objects of non-interest; During the background update period, when a non-interesting movable object has moved to reveal a further background area, the background image frame is updated to include the further background area in the background image frame. Create a foreground image frame containing the movable object of interest, wherein the background image frame and the foreground image frame exclude the movable objects of non-interest. The pixel blocks of the updated background image frame are encoded; The pixel blocks of the foreground image frame are encoded; Generate an encoded foreground image frame stream with a first frame rate; and An encoded, updated background image frame stream with a second frame rate lower than the first frame rate is generated.

2. The method according to claim 1, wherein, The segmentation of the image frame is accomplished using panoramic segmentation, wherein pixels in the image frame are assigned to either background segments representing regions comprising a set of objects of a particular type, or to instance segments representing a single object.

3. The method of claim 1, further comprising receiving a user selection from a list of object types, the user selection indicating which types of objects should be considered as movable objects of interest and non-movable objects of interest.

4. The method according to claim 1, wherein, The movable objects of interest include one or more of the following: humans, vehicles, weapons, bags, and masks.

5. The method according to claim 1, wherein, During the background update period, the movement of the non-interested movable object is tracked by a motion and object detector, and the background image frame is updated multiple times before the background update period expires.

6. The method according to claim 1, wherein, Encoding the foreground image frame includes: encoding only pixel data for pixels corresponding to movable objects of interest, and encoding the remainder of the foreground image frame as black pixels.

7. The method according to claim 1, wherein, The first frame rate is thirty image frames per second, and the second frame rate is one image frame per minute.

8. The method of claim 1, further comprising: Classify instance segments as static objects that are not of interest; as well as Update the background image frame to include the static object that is not of interest.

9. The method of claim 1, further comprising: At the end of the background update period, verify the integrity of the update to the background image frame; In response to determining that the update to the background image frame is incomplete, determine which non-interesting movable object caused the incompleteness; as well as The non-interesting movable object that would cause the incompleteness will be included in the foreground image frame.

10. The method according to claim 9, wherein, Verifying integrity includes: Determine whether the entire background image frame has been updated.

11. The method according to claim 1, wherein, When a non-interesting movable object has moved to the exposed background area, updating the background image frame includes: The movement of the non-interesting movable object is compared with one or more of the following: a region-related threshold, a distance-related threshold, and a time-related threshold; and The background image frame is updated when the movement of the non-interesting movable object exceeds at least one threshold.

12. The method of claim 11, further comprising: The threshold is set based on available computing resources.

13. The method according to claim 12, wherein, Setting the threshold includes: Setting the threshold limits the frequency of updates to the background image frame to a frequency that available computing resources can accommodate.

14. An encoding system for generating an image frame stream, comprising an encoder and motion and object detectors, wherein The motion and object detector is configured to: The image frames in the image frame stream are divided into one or more background segments and one or more instance segments; as well as Classify at least some of the instance segments in the one or more instance segments into movable objects of interest and movable objects of non-interest; and wherein The encoder is configured as follows: Create a background image frame that contains the one or more background segments; During the background update period, when a non-interesting movable object has moved to reveal a further background area, the background image frame is updated to include the further background area in the background image frame. as well as Create a foreground image frame containing the movable object of interest, wherein the background image frame and the foreground image frame exclude the movable objects of non-interest. The pixel blocks of the updated background image frame are encoded; The pixel blocks of the foreground image frame are encoded; Generate an encoded foreground image frame stream with a first frame rate; and An encoded, updated background image frame stream with a second frame rate lower than the first frame rate is generated.

15. A computer program product for generating an image frame stream, the computer program product comprising a computer-readable storage medium having program instructions, wherein, The computer-readable storage medium itself is not a transient signal, and the program instructions can be executed by a processor to perform a method including the following steps: The image frames in the image frame stream are divided into one or more background segments and one or more instance segments; Create a background image frame that contains the one or more background segments; Classify at least some of the instance segments into movable objects of interest and movable objects of non-interest; During the background update period, when a non-interesting movable object has moved to reveal a further background area, the background image frame is updated to include the further background area in the background image frame. Create a foreground image frame containing the movable object of interest, wherein the background image frame and the foreground image frame exclude the movable objects of non-interest. The pixel blocks of the updated background image frame are encoded; The pixel blocks of the foreground image frame are encoded; Generate an encoded foreground image frame stream with a first frame rate; and An encoded, updated background image frame stream with a second frame rate lower than the first frame rate is generated.

Images