Method and apparatus for managing encoded image frames in a data buffer

Abstract

A method and apparatus for managing encoded image frames in a data buffer are disclosed. The method includes obtaining encoded image frames arranged in groups of frames (GOPs), each GOP including a single intra-frame and two or more frame subgroups. Each frame subgroup includes a single directly coded inter-frame and zero or more indirectly coded inter-frames. The method includes determining whether a buffer fullness requirement is met. When the buffer fullness requirement is met, the method includes selecting one or more frames to be removed from the data buffer and removing the selected one or more frames. The selected one or more frames are at least one of the following: a non-reference encoded image frame and a frame subgroup in the two or more frame subgroups. The method also includes storing the one or more obtained encoded image frames.

Description

Technical Field

[0001] The present invention relates to an apparatus and method for managing encoded image frames in a data buffer. Background Technology

[0002] The use of video surveillance is steadily increasing across various applications. Surveillance or monitoring cameras can be used, for example, for crime prevention and traffic monitoring, and can be installed in shopping malls, retail stores, private residences, vehicles, and roads. So-called body-worn cameras (BWCs) can be worn by, for example, police officers and guards.

[0003] Typically, surveillance cameras capture image frames of a scene over time, thus obtaining a time series of image frames. This time series of image frames can be called a video sequence. Before the video sequence is stored in memory or transmitted to a receiver via a communication network, it is compressed to reduce its size. Compression can be performed using various video coding methods. In many digital video coding systems, two main modes are used to compress the image frames of a video sequence: intra-frame / modal coding and inter-frame / modal coding.

[0004] In intra-frame mode coding, image frames are encoded using prediction, transform, and entropy coding, taking advantage of the spatial redundancy of pixels within the frame itself. Image frames encoded in intra-frame mode are called intra-frames, I-frames, or keyframes.

[0005] Inter-frame mode coding utilizes temporal redundancy between individual image frames and relies on motion-compensated prediction techniques to predict portions of an image frame from one or more other image frames by encoding motion in selected pixel blocks from one image frame to another. The image frame encoded in inter-frame mode is called an inter-frame frame and can be either a P-frame or a B-frame. A P-frame is a forward-predicted frame encoded based on forward predictions from a previous I-frame or P-frame. However, a B-frame is a bidirectional prediction frame encoded based on both forward predictions from a previous I-frame or P-frame and backward predictions from a subsequent I-frame or P-frame. Therefore, a B-frame is based on both the preceding and subsequent image frames, while a P-frame is based only on the preceding frame. Some coding systems also use another type of inter-frame frame, sometimes called an inter-refresh frame, or simply a refresh frame or R-frame. This type of inter-frame frame can also be called a fast-forward frame or F-frame. Unlike P-frames, F-frames do not use the most recently decoded P-frame as a reference image frame; instead, they reference back to the most recently preceding intra-frame.

[0006] Encoded image frames are arranged in picture groups, or simply GOPs. A GOP can contain a fixed number of pictures, i.e., a fixed number of image frames, or, when using the concept of dynamic GOPs, a GOP can contain a variable number of image frames. Each picture group begins with an intra-frame, which does not reference any other image frames. Intra-frames are followed by multiple inter-frames, which do reference other image frames. Image frames do not necessarily have to be encoded and decoded in the same order they were captured or displayed. The only inherent limitation is that an image frame used as a reference frame must be decoded before the image frame it references can be encoded. In surveillance or monitoring applications, encoding is generally done in real time, meaning the most practical approach is to encode and decode image frames in the same order they were captured and displayed, as this would otherwise introduce undesirable latency.

[0007] Specifically, in surveillance or monitoring applications, video sequence recording can be based on one or more event triggers, such as a motion detection event trigger that is triggered when motion is detected within a monitored area. Thus, recording can be initiated when an event occurs, such as when movement occurs in a previously static scene or when a fire alarm or burglary alarm is triggered. When recording is based on an event trigger, it is often useful to also record video sequences captured before the event is triggered. Such video sequences are referred to herein as pre-event video sequences. For example, if recording is triggered by a person moving within an area of ​​interest that represents a part of the monitored scene, it might be of interest to also record a video sequence showing how that person moved within the area of ​​interest (e.g., entered the area of ​​interest) before the event was triggered, as such video sequences can have evidentiary value. To enable the recording of pre-event video sequences as an event occurs, encoded image frames can be continuously buffered in a buffer such as a first-in-first-out (FIFO) buffer. When the FIFO buffer is full, the oldest buffered encoded image frame is overwritten by the latest encoded image frame. Upon event triggering, a buffered encoded image frame is retrieved from the FIFO buffer and transmitted or transferred to the recorder, allowing the retrieved encoded image frame to be recorded prior to the video sequence that began at the event. Recording can then continue during the event and for a predetermined period after the event has passed. Alternatively, recording can continue until an event trigger indicating a stop recording event is received. The duration of the video sequence prior to the event can be set by the user.

[0008] Figure 1 schematically illustrates a sequence of I-frames and P-frames captured by a surveillance camera. In the figure, the letter I represents an I-frame, and the letter P represents a P-frame. Figure 1 also illustrates when an event begins and when it ends. The start of an event can be when a motion alarm is triggered, and the end of an event can be when the motion alarm has been acknowledged by an operator. However, the end of an event does not necessarily have to be associated with an action taken by the operator. Rather, the end of an event can be considered to occur within a predefined time period after the event is triggered or within a predefined time period after the detected motion has ceased. The pre-event video sequence of image frames captured by the surveillance camera before the start of the event can be temporarily stored in a FIFO buffer. Because the buffer stores the pre-event video sequence of image frames, it is sometimes called the pre-event buffer. When the event begins, the pre-event video sequence of image frames is retrieved from the buffer, and the retrieved pre-event image sequence is combined with the video sequence of image frames captured from the start of the event onwards. Therefore, in addition to the pre-event video sequence, the combined video sequence of image frames can also include an event video sequence containing the video sequence of image frames captured during the event and a post-event video sequence containing the video sequence of image frames captured after the end of the event. The combined image frames of the video sequence can, for example, be displayed to an operator and / or permanently recorded in memory for later playback and investigation.

[0009] Some surveillance cameras can capture image frames continuously but record them only when an event is triggered, while other surveillance cameras capture and record image frames only when an event is triggered. The first type of surveillance camera mentioned typically stores captured image frames in a FIFO buffer so that they can be recorded later in memory. The storage in the FIFO buffer is considered temporary storage compared to the recording in memory, which is more permanent storage. Because the buffer is a FIFO buffer, when the buffer is full, the oldest image frame in the FIFO buffer will be overwritten by the most recently captured image frame. Typically, the user decides the length of the pre-event video sequence of encoded image frames to be buffered in the buffer. For example, it might be of interest to be able to record and / or display a video sequence that begins 3 seconds before the event is triggered. This time period is sometimes referred to as the pre-event recording time, and in the given example, the pre-event recording time is 3 seconds. In order to be able to decode and view these 3 seconds of encoded image frames, the buffered pre-event video sequence must begin with an I-frame. However, the first image frame in these 3 seconds is likely to be a P-frame, as illustrated in Figure 1. Because a P-frame does not contain all the necessary data when there is no I-frame to reference for encoding, decoding of the pre-event video sequence cannot begin at the P-frame. To ensure that at least 3 seconds of decodable pre-event video is present, as described in US2016 / 277759 A1, the capacity of an additional sequence of encoded image frames corresponding to a GOP is added to the buffer. Therefore, the actual length of the encoded image frames stored in the pre-event buffer will depend on the user-selected pre-event recording time, the current frame rate, and the GOP length. For example, if the user sets a 3-second pre-event recording time, a frame rate of 30fps, and a GOP length of 128 image frames, the total time of the pre-event video sequence buffered in the pre-event buffer will be 7.3 seconds (3 seconds + 128 / 30 seconds). Therefore, in this example, the total time of the pre-event video sequence that needs to be buffered in the pre-event buffer is more than twice the expected pre-event recording time. Therefore, because an additional GOP is always required, the total time of the pre-event video sequence that needs to be buffered in the pre-event buffer is always longer than the expected pre-event recording time. This is a drawback when attempting to optimize buffer storage usage, and especially when trying to optimize the buffering of encoded image frames to provide only the buffer needed for the pre-event video sequence with the desired pre-event recording time. Furthermore, because the GOP length can vary, and because the length of the pre-event video sequence varies based on the GOP length, the pre-event buffer must be able to hold pre-event video sequences of varying lengths to ensure that a decodeable pre-event image sequence with the pre-event recording time can be retrieved from the buffer.As a safety measure, the pre-event buffer can be configured to buffer pre-event video sequences that include an additional GOP with the longest GOP length, in addition to the pre-event video sequence of the desired length. Therefore, improved buffering of encoded image frames is needed. In particular, improved management of the buffered encoded image frames within the data buffer is required.

[0010] ISOVIC D et al., “Quality-Aware MPEG-2 Stream Adaptation in Resource-Constrained Systems” (Proceedings of the European Conference on Real-Time Microprocessors, January 1, 2004, pp. 23-32, XP002447140), describes a method for MPEG decoding under limited resources, in which frame skipping is applied to the MPEG stream to achieve a customized MPEG stream that includes only frames that can be guaranteed to be decoded and displayed in a timely manner.

[0011] CA2 786 200A1 discloses a capacity- and spectrum-constrained multiple access communication system that optimizes performance by selectively discarding packets. Packets are prioritized, and discard decisions are made based on these priorities. Summary of the Invention

[0012] In view of the foregoing, the object of the present invention is to alleviate the disadvantages of the prior art and to provide an improved buffering of encoded image frames. In particular, the object is to manage the encoded image frames in a data buffer (e.g., a pre-event buffer) in such a way that all stored (e.g., buffered) encoded image frames can be retrieved from the data buffer as a decodable video sequence, and that only those encoded image frames needed to obtain a decodable video sequence with a predefined recording time can be stored in the data buffer. This can be described as pruning the data buffer in such a way that all stored encoded image frames can be retrieved from the data buffer as a decodable video sequence with a predefined recording time. Thus, only the encoded image frames needed to generate the video sequence can be stored in the data buffer. Therefore, no additional space is needed in the data buffer for storing additional GOPs.

[0013] The objective is to select which one or more stored encoded image frames to delete when the data buffer is full, so that new encoded image frames can be stored. If an "erroneous" encoded image frame, i.e., a reference image frame used as a reference to one or more other encoded image frames, is deleted from the data buffer, those one or more other encoded image frames will be undecodeable, and therefore it will be impossible to retrieve a decodeable video sequence including all buffered encoded image frames from the data buffer.

[0014] This invention is defined by the appended claims.

[0015] According to a first aspect of the invention, the above objective is achieved by a method for managing encoded image frames in a pre-event buffer, wherein the encoded image frames depict a scene, and the pre-event buffer is configured to temporarily store the encoded image frames, the stored encoded image frames constituting a video sequence captured prior to the event when an event in the scene or a scene-related event is detected. The method includes obtaining an input stream of encoded image frames depicting a scene and arranged in groups of pictures (GOPs), each GOP comprising a single intra-frame and two or more subgroups of frames. Each subgroup of frames includes a single directly encoded inter-frame that is encoded directly with reference to the intra-frame, and zero or more indirectly encoded inter-frames that are encoded indirectly with reference to the intra-frame via the single directly encoded inter-frame. Further, the method includes determining whether a buffer fullness requirement is met, wherein the buffer fullness requirement is a buffer time requirement met when the total recording time of the stored encoded image frames is equal to a predefined recording time, or when the total recording time of the stored encoded image frames would become greater than a predefined recording time if one or more encoded image frames in the obtained input stream are stored. When the buffer fullness requirement is not met, the method includes storing one or more encoded image frames to be included in the acquired input stream in a pre-event buffer until the buffer fullness requirement is met. When the buffer fullness requirement is met, the method includes selecting one or more encoded image frames to be removed from the data buffer from the encoded image frames stored in the data buffer. The selected one or more encoded image frames are non-reference encoded image frames and at least one of two or more frame subgroups, wherein the non-reference encoded image frames are encoded image frames that are reference image frames to any encoded image frames not used as stored encoded image frames. Furthermore, the method includes removing the selected one or more encoded image frames from the pre-event buffer and storing one or more encoded image frames included in the acquired input stream in the pre-event buffer. Thus, the encoded image frames temporarily stored in the pre-event buffer are managed as only those encoded image frames constituting a decodable video sequence with a predefined recording time.

[0016] When used in this disclosure, the term "intraframe" should be understood as an intra-coded image frame, that is, an image frame encoded using an intra-frame mode without referencing another image frame. An intraframe may be referred to as an I-frame or a keyframe.

[0017] When used in this disclosure, the term "inter-frame" should be understood as an inter-frame coded image frame, that is, an image frame encoded using inter-frame mode coding with reference to at least one other image frame. An image frame used as a reference frame when encoding another image frame is referred to herein as a "reference / reference image frame", and an encoded image frame not used as a reference is referred to herein as a "non-reference coded image frame".

[0018] The term "directly coded inter-frame" should be understood as an inter-frame that is encoded by directly referencing intra-frames.

[0019] When used in this disclosure, the term "indirectly coded inter-frame" should be understood as an inter-frame encoded by indirectly referencing an intra-frame via a directly coded inter-frame. Therefore, an indirectly coded inter-frame can be encoded by directly referencing a directly coded inter-frame, which in turn directly refers to an intra-frame. In other words, an indirectly coded inter-frame can be encoded by indirectly referencing an intra-frame via a directly coded inter-frame. To decode an indirectly coded inter-frame, both the directly coded inter-frame referenced by the indirectly coded inter-frame and the intra-frame referenced by the directly coded inter-frame are required. Alternatively, an indirectly coded inter-frame can be encoded by directly referencing a prior indirectly coded inter-frame.

[0020] When used herein, the term “non-reference encoded image frame” should be understood as an encoded image frame that is not used as a reference image frame for any of the encoded image frames stored in the data buffer.

[0021] By managing the encoded image frames in the pre-event buffer (such as by pruning the pre-event buffer when it is full, as suggested by this invention), new encoded image frames can be continuously stored in the pre-event buffer, and all stored encoded image frames can be retrieved from the pre-event buffer as a decodable video sequence. Preferably, the size of the pre-event buffer, for example, measured in frames or seconds of a video sequence, is suitable for storing only the encoded image frames needed to obtain a decodable video sequence with a predefined recording time. As will be described herein, when an event occurs, all stored encoded image frames can be retrieved to replay what happened before the event occurred. In this case, a decodable video sequence with a predefined recording time can be referred to as a decodable pre-event video sequence with a predefined pre-event recording time.

[0022] Furthermore, by storing in the pre-event buffer one or more encoded image frames from the obtained input stream included in the GOP, wherein each GOP comprises a single intra-frame and two or more frame subgroups, and wherein each frame subgroup comprises a single directly encoded inter-frame encoded with direct reference to the intra-frame and zero or more indirectly encoded inter-frames encoded with indirect reference to the intra-frame via the single directly encoded inter-frame, it is possible to select and remove one or more encoded image frames from the pre-event buffer without affecting the ability to decode the remaining encoded image frames stored in the pre-event buffer.

[0023] Therefore, the advantage of the present invention is that it is possible to store only those encoded image frames needed to obtain a decodeable video sequence with a predefined recording time in the pre-event buffer.

[0024] According to a second aspect of the invention, the above objective is achieved by an apparatus for managing encoded image frames in a pre-event buffer, wherein the encoded image frames depict a scene, and the pre-event buffer is configured to temporarily store the encoded image frames, wherein the stored encoded image frames constitute a video sequence captured before the event is detected, or when an event in the scene or a scene-related event is detected. The apparatus includes an acquisition module configured to acquire an input stream of encoded image frames depicting a scene and arranged in a group of pictures (GOP), each GOP comprising a single intra-frame and two or more subgroups of frames. Each subgroup of frames includes a single directly encoded inter-frame that is encoded directly with reference to the intra-frame, and zero or more indirectly encoded inter-frames that are encoded indirectly with reference to the intra-frame via the single directly encoded inter-frame. The apparatus further includes a determination module configured to determine whether a buffer fullness requirement is met, wherein the buffer fullness requirement is a buffer time requirement met when the total recording time of the stored encoded image frames is equal to a predefined recording time, or when the total recording time of the stored encoded image frames would become greater than the predefined recording time if one or more encoded image frames in the acquired input stream are stored. The apparatus includes a storage module configured to store one or more encoded image frames included in the acquired input stream in an event-before buffer until the buffer fullness requirement is met, provided that a buffer fullness requirement is not met. Furthermore, the apparatus includes a selection module configured to select one or more encoded image frames to be removed from the event-before buffer from the stored encoded image frames when the buffer fullness requirement is met. The selected one or more encoded image frames are non-reference encoded image frames and at least one of two or more frame subgroups, wherein the non-reference encoded image frames are encoded image frames that are not used as reference image frames for any of the stored encoded image frames. Further, the apparatus includes a removal module configured to remove the selected one or more encoded image frames from the event-before buffer, and the storage module is configured to store one or more encoded image frames included in the acquired input stream in the event-before buffer.

[0025] According to a third aspect of the invention, the above objective is achieved by a non-transitory computer-readable medium having computer code instructions stored thereon, which, when executed by a processing-capable device, are adapted to perform the method of the first aspect.

[0026] The second and third aspects can generally have the same features and advantages as the first aspect. Embodiments of the invention are defined in the appended dependent claims. It should be further noted that, unless otherwise expressly stated, the invention relates to all possible combinations of the features disclosed herein. Attached Figure Description

[0027] The above and other objects, features, and advantages of the invention will be better understood by referring to the following illustrative and non-limiting detailed description of embodiments of the invention, wherein like reference numerals will be used for similar elements, wherein:

[0028] Figure 1 schematically illustrates an event video sequence according to the prior art.

[0029] Figure 2A Exemplary environments in which various embodiments of the invention may be implemented are schematically illustrated.

[0030] Figure 2B An embodiment of a camera including means for managing encoded image frames in a data buffer is illustrated schematically.

[0031] Figure 2C An embodiment in which a means for managing encoded image frames in a data buffer is connected to a camera is schematically illustrated.

[0032] Figure 2D An embodiment of a camera including or connected to an encoder and a recorder is schematically illustrated.

[0033] Figure 3 A flowchart illustrating a method for managing encoded image frames in a data buffer according to an embodiment is shown schematically.

[0034] Figure 4A A sequence of encoded image frames according to an embodiment is illustrated schematically.

[0035] Figure 4B A group of pictures (GOP) according to an embodiment is illustrated schematically.

[0036] Figure 4C An event video sequence according to an embodiment is illustrated schematically.

[0037] Figure 5 A block diagram of an apparatus for managing encoded image frames in a data buffer, according to an embodiment, is schematically illustrated. Specific Implementation

[0038] The invention will now be described more fully with reference to the accompanying drawings, in which embodiments of the invention are illustrated.

[0039] Figure 2A A schematic diagram of an exemplary environment 200 in which various embodiments of the invention may be implemented is shown. (As in...) Figure 2AAs can be seen, camera 208 captures a scene 202 with object 204, such as a person walking towards building 206. It should be noted that the depiction of scene 202 is merely a simplified view for illustrative purposes. In a more general sense, scene 202 can be described as any three-dimensional physical space whose size and shape are defined by the field of view of the camera monitoring the scene. Camera 208 can be a surveillance camera, sometimes also called a monitoring camera. Further, the camera can be a fixed camera (e.g., a still camera), a movable camera (e.g., a pan, tilt, and zoom (PTZ) camera), or a body-worn camera (BWC). Camera 208 can be a visible light camera, a thermal imaging camera, or a camera combining visible light and thermal imaging methods. Further, to give only a few examples, camera 208 can be arranged to monitor areas such as the area in front of a building's door, an area inside a building, or an area outside a building in a scene in front of a person wearing camera 208.

[0040] like Figure 2A As illustrated, camera 208 is configured to transmit video streams directly to wired network 212 via a network (such as directly via wired network 212 or wirelessly via radio link 210), and ultimately to a client 214 and / or server 216 that can connect to network 212. It should be understood that multiple combinations of wireless and wired transmission models can be used, and Figure 2A Only one example is illustrated. Furthermore, in this disclosure, the terms "stream" and "sequence" are used interchangeably, for example, in "video stream," "video sequence," "stream of image frames," and "sequence of image frames."

[0041] Client 214 may be a console equipped with input devices and a display, allowing an operator to input control commands to camera 208 and view images and / or video sequences from camera 208. Server 216 may be a video management system capable of storing and / or further processing images and / or video sequences. Typically, client 214 is also connected to server 216.

[0042] Typically, camera 208 continuously captures and encodes video sequences of image frames of a scene, and buffers (i.e., temporarily stores) the encoded video sequence of image frames in a data buffer. However, recording of the buffered video sequence to a more permanent memory than the data buffer is only performed when an event in or related to the scene is detected, and only when such a detected event triggers recording. Detected events can trigger alarms associated with the monitored scene (e.g., burglary alarms or fire alarms) by detecting motion or sound in the scene. For example, events can be detected by a video analysis module (not shown) included in camera 208 and configured to detect motion and / or sound in the video sequence captured by camera 208. The video analysis module may include an image analysis module and an audio analysis module configured to detect motion and sound in the video sequence, respectively. As another example, events can be detected by external sensors such as door contacts, smoke detectors, or passive infrared (PIR) sensors communicatively connected to camera 208. Events can also be user input events such as command input. For example, an event can be detected by activating a registration recording button on a body-worn camera.

[0043] After an event is detected, an event notification can be transmitted to the receiving module. As described below, the receiving module can be included in a means for managing encoded image frames in a data buffer.

[0044] In the event that an event is detected in scene 202 monitored by camera 208, or an event related to scene 202 is detected, it is desirable to record a video sequence capturing the event so that the video sequence can be replayed at a later time point. As previously mentioned, it is often also desirable to include a video sequence prior to the event in the recording. Therefore, camera 208 may include or be operatively connected to a data buffer configured to buffer (i.e., temporarily store) encoded image frames that will constitute the video sequence prior to the event in the event. Reference will be made below. Figure 5 In more detail, the data buffer 520 may be included in or operatively connected to a means 500 for managing encoded image frames in the data buffer 520. The data buffer 520 may be a memory or a region of memory for temporarily storing encoded image frames. The memory may be implemented in physical memory in hardware, or it may be implemented by using virtual memory in software that points to locations in physical memory. Because the data buffer 520 may be included in or connected to the means 500, therefore... Figure 5 The data buffer 520 is illustrated as a dashed box arranged inside and outside the device 500. The device 500 may be included in the camera 208, such as... Figure 2B The diagram is shown schematically.

[0045] exist Figure 2C The diagram schematically illustrates that device 500 can be operatively connected to camera 208, and data buffer 520 can be included in device 500, or external to device 500 and operatively connected to device 500. It is also conceivable that device 500 can be included in camera 208, and data buffer 520 can be arranged externally to camera 208 but operatively connected to device 500 (not shown).

[0046] Figure 2D An embodiment of camera 208 is schematically illustrated, including an encoder 220 configured to encode a sequence of captured image frames into a sequence of encoded image frames, and a recorder 230 configured to record a sequence of encoded image frames retrieved from a data buffer 520. However, the recorder 230 may be arranged externally to the camera 208 but operationally connected to it. It should also be understood that both the encoder 220 and the recorder 230 may be arranged externally to the camera 208. Figure 2D The diagram also schematically illustrates a controller 240, such as a bitrate controller, which may be included in or operatively connected to camera 208. Controller 240 controls the output bitrate of encoder 220 and is therefore operatively connected to encoder 220. Controller 240 can control encoder 220 to have the same output bitrate for a sequence of encoded image frames, regardless of whether the sequence of encoded image frames is a pre-event sequence, an event sequence, or a post-event sequence. However, the output bitrate of the sequence may differ for one or more of the pre-event, event, and post-event sequences. Thus, the pre-event sequence may have a first bitrate, the event sequence may have a second bitrate, and the post-event sequence may have a third bitrate; these bitrates may be the same, or one or more of them may differ from the other bitrates.

[0047] Encoder 220 may be a video encoder configured to encode a sequence of image frames into a sequence of encoded image frames conforming to a standard video coding format. Some examples of standard video coding formats are H.264, H.265, AV1, and VVC.

[0048] Recorder 230 may be a data recorder, such as a video recorder, configured to record / store data / video. Recorder 230 may further be a network recorder 230 configured to connect to other devices, such as camera 208 and / or encoder 220, via a communication network (e.g., network 212).

[0049] Compared to the more temporary storage provided by data buffer 520, the recording of a video sequence provided by recorder 230 can be considered a more permanent storage of the video sequence. However, it should be understood that permanent storage of a video sequence does not mean that the video sequence is permanently stored, but rather that it is stored over a longer period of time compared to the shorter time period provided by data buffer 520. For example, to give just a few examples, the storage time in data buffer 520 can range from one or more seconds to several minutes or even several hours, while the storage time in recorder 230 can range from one or more days to several months or even several years.

[0050] Now refer to Figure 3 This describes a method for managing encoded image frames in a data buffer configured to store encoded image frames. References will also be made to... Figure 4A , Figure 4B and Figure 4C as well as Figure 5 . Figure 4A The diagram schematically illustrates a sequence of encoded image frames according to an embodiment. Figure 4B A GOP according to an embodiment is schematically illustrated, and Figure 4C An event video sequence according to an embodiment is illustrated schematically. Figure 5 A block diagram schematically illustrates an apparatus 500 for managing encoded image frames in a data buffer 520 according to an embodiment. The data buffer 520 may be an event-pre-event buffer configured to buffer (i.e., temporarily store) an event-pre-event video sequence having a predefined event-pre-recording time. In this disclosure, the invention will be described with reference to a data buffer as an event-pre-event buffer storing an event-pre-event video sequence of encoded image frames. However, it should be understood that the data buffer need not be an event-pre-event buffer storing an event-pre-event video sequence, but may be another type of data buffer storing another type of video sequence of encoded image frames having a predefined recording time. Furthermore, it should be understood that some of the actions may be optional and may be taken in another suitable order.

[0051] Action 302

[0052] In action 302, an input stream 400 of encoded image frames arranged in a group of pictures (GOP) is obtained, wherein each GOP includes a single intra-frame 401 and two or more frame subgroups SG1, SG2. Further, each frame subgroup SG1, SG2 includes a single directly encoded inter-frame 402 that is encoded directly with reference to the intra-frame 401, and zero or more indirectly encoded inter-frames 403 that are encoded indirectly with reference to the intra-frame 401 via the single directly encoded inter-frame 402.

[0053] exist Figure 4AIn the diagram, the input stream 400 of the encoded image frames is arranged in multiple Groups of Pictures (GOPs), with the first GOP (GOP1) and the second GOP (GOP2) illustrated. As indicated by the timeline, the first GOP (GOP1) precedes the second GOP (GOP2) in time, meaning that the first GOP (GOP1) is older than the second GOP (GOP2) and was captured before GOP2. Furthermore, each GOP includes an intra-frame 401 and two frame subgroups SG1 and SG2. Figure 4A In this context, reference numeral 4011 is used to refer to intraframes of GOP1, where the subscript number indicates the GOP number, and reference numeral SG is used. 11 and SG 12 The two frame subgroups of GOP1 are referred to by the first subscript, which indicates the GOP number, and the second subscript indicates whether the subgroup is the first or second subgroup of the GOP. GOP2 uses the same notation. However, it should be understood that in this disclosure, the GOP subscripts are used only when it is desired to distinguish between two GOPs such as GOP1 and GOP2.

[0054] exist Figure 4B The diagram schematically illustrates GOP1, which includes a single intraframe 4011 and two subgroups SG. 11 and SG 12 Furthermore, it illustrates each frame subgroup SG. 11 and SG 12 This includes corresponding directly encoded inter-frames 4021 and 4022, where the subscript of each directly encoded inter-frame indicates which of the two frame subgroups it belongs to. As indicated by the arrows from each of the directly encoded inter-frames 4021 and 4022 to the single intra-frame 4011, both directly encoded inter-frames 4021 and 4022 are encoded directly with reference to the single intra-frame 4011 of GOP1. In some embodiments, the frame subgroup consists of only one frame, namely, only a single directly encoded inter-frame 402. The single directly encoded inter-frame 402 can be a P-frame, F-frame, or R-frame encoded directly with reference to the intra-frame 401. In this disclosure, the single directly encoded inter-frame 402 will be referred to as an F-frame.

[0055] As also Figure 4B The diagram shows the subgroup SG of GOP1. 11 It may include one or more indirectly encoded inter-frames 403 11 Up to 403 1n And GOP1's subgroup SG 12 It may include one or more indirectly encoded inter-frames 403 21 Up to 403 2mTo indicate that one or more indirectly coded inter-frames are optional, they are indicated by dashed boxes in the figure. Each of the indirectly coded inter-frames 403 can be a P-frame or a B-frame.

[0056] Therefore, each frame subgroup SG1, SG2 includes a single directly encoded inter-frame 402 that is directly referenced to the intra-frame 401 for encoding. As mentioned above, sometimes the single directly encoded inter-frame 402 is the only frame included in the subgroup, and thus no indirectly encoded inter-frame 403 is included in the subgroup. However, in addition to the single directly encoded inter-frame 402, each subgroup may also include one or more indirectly encoded inter-frames 403.

[0057] Each subgroup comprising a single directly coded inter-frame 402 and zero or more indirectly coded inter-frames 403 can be expressed as each subgroup comprising a single directly coded inter-frame 402 and none, or one or more indirectly coded inter-frames 403. Alternatively, it can be expressed as each subgroup comprising a single directly coded inter-frame 402 and may include one or more indirectly coded inter-frames 403.

[0058] In addition to a single directly encoded inter-frame 402, whether a subgroup includes zero or more indirectly encoded inter-frames 403 depends on how the input stream 400 of the encoded image frame is encoded.

[0059] The input stream 400 is encoded by the encoder 220 under the control of the controller 240. Therefore, the controller 240 controls the encoder 220 to encode captured image frames into the input stream 400 of encoded image frames arranged in GOPs, where each GOP includes a single intraframe 401 and two or more subgroups of frames SG1, SG2. The controller 240 can employ a constant bit rate (CBR), maximum bit rate (MBR), or variable bit rate (VBR). CBR means that the encoder 220 will strive to always output the same bit rate regardless of what happens in the captured scene. If bandwidth is limited, this may result in low-quality images when there is motion in the scene, but high-quality images when the scene is static. This is generally not useful in surveillance or monitoring situations, as scenes with motion are usually more interesting than static scenes. With MBR, the bit rate is allowed to vary as long as it does not exceed the set bit rate limit. The problems involving this method are similar to those associated with CBR. If the MBR limit is set too low, images of scenes with motion may be of low quality. However, if the limits are set too high to accommodate motion, the output bit rate may be unnecessarily high when encoding images of static scenes. VBR, also known as constant bit rate, means that the quality of the encoded image should remain constant, but the output bit rate is allowed to vary depending on what is happening in the scene. This approach can lead to high output bit rates when there is motion in the scene. This is particularly problematic if bandwidth is limited (such as when transmitting encoded image frames over a mobile network). Similarly, it is problematic if storage is limited (such as when buffering encoded image frames in a data buffer or storing them on-board in the camera (e.g., on an SD card). High output bit rates can also be problematic in large camera systems if several cameras are simultaneously transmitting images of scenes with motion.

[0060] Regardless of the bitrate control scheme used by controller 240, one parameter that can be adjusted to conform to the bitrate set by controller 240 is the GOP length. In some applications, the GOP length is dynamically determined by controller 240. For example, to reduce the bitrate, the GOP length, i.e., the number of encoded image frames included in the GOP, can be increased. Conversely, to increase the bitrate, the GOP length can be decreased.

[0061] Based on knowledge of the total storage capacity of data buffer 520, the predefined pre-event recording time, and the desired frame rate, controller 240 determines how to encode each image frame. For example, if the total storage capacity of data buffer 520 is 270 MB, the predefined pre-event recording time is 90 seconds, and the frame rate is 30 image frames per second, then 2700 (30 fps × 90 s) encoded image frames will be stored in data buffer 520. If these 2700 encoded image frames have the same size, then each of them must be 0.1 MB (0.8 Mbit). By encoding image frames in a group of frames (GOPs), where each GOP includes a single intraframe as described herein and one or more subgroups of frames, the utilization of available storage capacity can be optimized to store encoded image frames with a predefined pre-event recording time while optimizing the quality of the video sequence.

[0062] For example, it can be advantageous to encode image frames such that each GOP consists of only a single intra-frame 401 and multiple directly encoded inter-frames 402, all encoded by directly referencing the single intra-frame 401 of the GOP. Such a GOP would not include any indirectly encoded inter-frames 403. This can be the case when a GOP is encoded to consist of an I-frame and multiple F-frames, all encoded by referencing the corresponding I-frame. In this scenario, each of the F-frames is considered to constitute the corresponding frame subgroup. Encoding the image frames of a GOP as an I-frame and multiple F-frames can be advantageous in a scenario where the encoded image frames are to be stored in an event-pre-event buffer, such as a data buffer 520, with sufficient storage space for recording a predetermined length (e.g., a predefined event-pre-event recording time) of video sequence prior to the start of the event. If the event-pre-event video sequence consists only of multiple F-frames and one or more I-frames required to decode the F-frames, the event-pre-event video sequence will always be decodable regardless of whether one or more of the F-frames of the GOP previously stored in the data buffer 520 have been removed. Therefore, if the data buffer 520 has sufficient storage space, it is advantageous to store the encoded image frames in GOPs, where each GOP consists of one I-frame and multiple (e.g., two or more) F-frames, as this facilitates the management of the data buffer 520 to enable the retrieval of decodable pre-event video sequences with predefined pre-event recording times.

[0063] However, F-frames are typically larger than P-frames, especially in non-static scenes, and when the temporal distance between an F-frame and its directly encoded I-frame increases, and when the increased content difference between F-frames and I-frames results in less redundancy being available in inter-frame coding compared to encoding P-frames which encode frames with smaller reference temporal distances and content differences. Therefore, F-frames generally require more storage space than P-frames. Thus, to reduce the required storage space while still being able to buffer pre-event video sequences of encoded image frames with predefined pre-event recording times, it may be advantageous to also include one or more P-frames in one or more frame subgroups of the input stream 400 of the encoded image frames. The controller 240 can control the encoder 220 to encode one or more image frames into one or more indirectly encoded inter-frame frames 403 (e.g., one or more P-frames), and thus, when there is a lot of motion in the scene and the difference between a single intra-frame 401 of the GOP and each directly encoded inter-frame frame 402 is too large to result in a too high bitrate, it increases the number of frame subgroups to include one or more indirectly encoded inter-frame frames 403 in addition to a single directly encoded inter-frame frame 402 (i.e., a single F-frame in the frame subgroup). By increasing the number of frames in the frame subgroup, the number of frame subgroups in the GOP can be reduced, and thus the number of directly encoded inter-frame frames 402 (i.e., F-frames) in the GOP can also be reduced. This often results in a lower bitrate because the indirectly encoded inter-frame frames 403 (e.g., P-frames) are typically smaller than the directly encoded intra-frame frames 402 (i.e., F-frames).

[0064] exist Figure 4B The diagram schematically illustrates that one or more indirectly coded inter-frames 403 (i.e., one or more P-frames) can be included in each frame subgroup. Each such P-frame is directly referenced in the coding order of the image frame that precedes it, i.e., directly references either the indirectly coded inter-frame 403 or the directly coded inter-frame 402 that precedes it in the coding order. Therefore, each such P-frame is indirectly coded only via its directly coded inter-frame 402 with direct or indirect reference, and references the intra-frame 401 of the GOP.

[0065] The input stream 400 of encoded image frames can be obtained by the acquisition module 511. The acquisition module 511 can be included in a device 500 for managing the encoded image frames in the data buffer 520. The device 500 can be included in a camera, such as camera 208, to capture image frames and encode them into the input stream 400 of encoded image frames. Alternatively, when camera 208 does not include an encoder, the device 500 can be included in an encoder 220 connected to camera 208. In the latter scenario, encoder 220 receives captured image frames from camera 208 and encodes them into the input stream 400 of encoded image frames. Thus, the device 500 obtains the input stream 400 of encoded image frames. As another alternative, and only a few examples are given, the device 500 can be included in a recorder 230 configured to receive or retrieve the input stream 400 of encoded image frames from camera 208 or encoder 220.

[0066] As mentioned above, controller 240 can determine how to encode the captured image frames and transmit control signals to encoder 220, which is configured to encode the captured image frames according to the control signals. Therefore, controller 240 can determine whether to include one or more indirectly encoded inter-frame frames 403, i.e., one or more P-frames, in each frame subgroup. Whether to include one or more indirectly encoded inter-frame frames 403 can be based on the size of a preceding single directly encoded inter-frame frame 402, i.e., based on the size of a preceding F-frame. If the size of the preceding single directly encoded inter-frame frame 402 is large, it indicates a large difference between the preceding single directly encoded inter-frame frame 402 and the single intra-frame frame 401 on which it is encoded. This large difference, in turn, is an indication of multiple movements in the scene between the capture of the single intra-frame frame 401 and the capture of the single directly encoded inter-frame frame 402. If subsequent image frames are to be encoded as subsequent single directly encoded inter-frame frames 402 in subsequent frame subgroups, the subsequent single directly encoded inter-frame frames 402 will also have a large size due to movement in the scene. In this scenario, referencing a preceding single directly encoded inter-frame 402, encoding subsequent image frames as indirectly encoded inter-frames 403 (e.g., P-frames) can be advantageous because the size of the indirectly encoded inter-frame 403 can be smaller than the size of the directly encoded inter-frame 402. Therefore, the controller 240 can decide to reduce the number of frame subgroups in a GOP, instead of increasing the number of indirectly encoded inter-frames 403 in one or more frame subgroups, in order to reduce the bit rate.

[0067] Action 304

[0068] Before one or more encoded image frames from the acquired input stream 400 are stored in the data buffer 520, it is determined whether the data buffer 520 has the capacity to store one or more encoded image frames. This is performed in action 304, where it is determined whether the buffer fullness requirement is met.

[0069] If the buffer full requirement is not met, the data buffer 520 has the capacity to store one or more encoded image frames from the acquired input stream 400. If the buffer full requirement is not met, the method continues with operation 310.

[0070] On the other hand, if the buffer fullness requirement is met, one or more stored encoded image frames must be removed from the data buffer 520 before one or more encoded image frames from the acquired input stream 400 can be stored. In such a case, actions 306 and 308 must be executed before the method proceeds to action 310.

[0071] The buffer fullness requirement can be a buffer time requirement that is met when the total recording time of the stored encoded image frames equals a predefined recording time, or when the total recording time of the stored encoded image frames becomes greater than the predefined recording time if one or more encoded image frames from the acquired input stream are stored. When the buffer time requirement is met, one or more encoded image frames stored in the data buffer 520 should be removed to free up sufficient storage space in the data buffer for storing one or more encoded image frames captured at a time point after the last stored encoded image frame was captured. In particular, the first encoded image frame among the one or more encoded image frames to be stored is captured directly consecutively with the last encoded image frame stored in the data buffer 520. However, it is not possible to arbitrarily remove one or more stored encoded image frames from the data buffer 520, because the remaining encoded image frames in the data buffer 520 must be decodable and must have a predefined recording time. If the remaining encoded image frames are undecodeable, it will be impossible to replay and display the pre-event video sequence based on all the remaining encoded image frames, especially pre-event video sequences with a predefined pre-event recording time. In order to display the retrieved pre-event video sequence, all encoded image frames of the retrieved pre-event video sequence need to be decodeable. Therefore, if multiple inter-frames are retrieved from data buffer 520, intra-frames used when encoding the inter-frames are also needed to enable decoding of the inter-frames. If the reference intra-frame has been removed from data buffer 520, the retrieved inter-frames encoded with reference to that reference intra-frame will be undecodeable, and therefore it will be impossible to replay and view them. Because those retrieved inter-frames are part of the pre-event video sequence but cannot be displayed due to being undecodeable, the displayable length of the pre-event video sequence will be shorter than expected, i.e., shorter than the predefined pre-event recording time. Consequently, important information preceding the start of the event may not be replayed to the operator. Therefore, how to select one or more encoded image frames to be removed from data buffer 520 is important.

[0072] The selection of one or more encoded image frames to be removed will be described in more detail below with reference to action 308, but non-reference encoded image frames or subgroups of frames can also be selected. Furthermore, the selected one or more encoded image frames to be removed can be so-called old non-reference encoded image frames. Old non-reference encoded image frames are non-reference encoded image frames captured at a time point prior to the current time point that is greater than a predefined recording time (i.e., greater than a predefined recording time before the event). The current time point can be the event time point, i.e., the time point when the event occurs or is detected. Therefore, the selected one or more encoded image frames can be captured at a time point prior to the current time point minus the predefined recording time. For example, the predefined recording time can be in the range of several seconds to tens of minutes, such as in the range of 5 seconds to 10 minutes.

[0073] Alternatively or additionally, the buffer fullness requirement can be a buffer size requirement. For example, the buffer fullness requirement can be a buffer size requirement that is met when the total size of the stored encoded image frames exceeds a predetermined buffer size threshold. The predetermined buffer size threshold can be a value in the range of 50 megabytes (MB) to 500 MB. Sometimes, the predetermined buffer size threshold can be 100 MB.

[0074] As another example, the buffer fullness requirement can be a buffer size requirement that is satisfied when the total number of stored encoded image frames exceeds a predetermined number of stored encoded image frames. The predetermined number of stored encoded image frames to be buffered in data buffer 520 can be set based on usage. In some embodiments, it may be desirable to buffer a certain number of encoded image frames regardless of their respective sizes. When setting the predetermined number of stored encoded image frames, the desired number of GOPs and / or the desired number of encoded image frames in each GOP can be considered. The predetermined number of stored encoded image frames can be set by the user. In some embodiments, the predetermined number of stored encoded image frames does not include older stored encoded image frames that serve as reference image frames during counting, as they are held in the data buffer only to enable decoding of one or more stored inter-frame frames that reference them. Sometimes, such older encoded image frames are only used to enable decoding of one or more stored inter-frame frames that reference them, and they will not be displayed when playing back the video sequence before the event that includes them. When the frame rate is 30 frames per second and the pre-event video sequence has a desired pre-event recording time in the range of 1 second to 90 seconds, the predetermined number of stored encoded image frames can be a value in the range of 30 to 2700. However, the predetermined number of stored image frames can be another number, as it will vary with the frame rate and the desired pre-event recording time of the pre-event video sequence.

[0075] The determination of whether the buffer fullness requirement is met can be performed by the determination module 513. The determination module 513 can be included in the device 500.

[0076] Action 306

[0077] When the buffer fullness requirement is met, one or more encoded image frames to be removed from the data buffer 520 are selected from the stored encoded image frames. The selected one or more encoded image frames are non-reference encoded image frames and at least one of two or more frame subgroups.

[0078] A non-reference encoded image frame is an encoded image frame that is not used as a reference image frame for any of the stored encoded image frames. Therefore, it should be understood that a non-reference encoded image frame can be an I-frame that is not used as a reference frame for another encoded image frame (e.g., for an F-frame, P-frame, or B-frame). Further, a non-reference encoded image frame can be an F-frame that is not used as a reference frame for another encoded image frame (e.g., for a P-frame or B-frame). Further, a non-reference encoded image frame can be a P-frame or B-frame that is not used as a reference frame for another encoded image frame (e.g., for another P-frame or another B-frame). In other words, if an encoded image frame is not used as a reference for encoding other image frames, then that encoded image frame is referred to in this disclosure as a non-reference encoded image frame.

[0079] When selecting one or more encoded image frames, an alternative approach is to select one or more encoded image frames from a set of non-reference encoded image frames. Another alternative approach is to select one or more encoded image frames as a subgroup of two or more frame subgroups.

[0080] When a non-reference coded image frame is selected as one or more coded image frames to be removed, the non-reference coded image frame can be selected based on size (e.g., the largest frame size), based on the size of a group of frames (GOP) or a subgroup of frames including the non-reference coded image frame, or according to a predefined pattern (e.g., every two or every three frames to be selected). Therefore, when one or more coded image frames are selected as non-reference coded image frames, the non-reference coded image frame is at least one of the following:

[0081] - A non-reference encoded image frame with a size greater than the size of any other stored non-reference encoded image frame. By selecting the largest non-reference encoded image frame, and by removing only one encoded image frame, the maximum possible storage space in data buffer 520 will be made available.

[0082] - Non-reference encoded image frames included in a GOP with a number of encoded image frames greater than any other stored GOP. A large GOP can indicate that the scene captured by the encoded image frames of the GOP has not changed much and indicates that the inter-frames of the GOP are small. This can be the case when the GOP length is dynamically controlled based on motion detected in the scene, other detected events such as detected sound, or pre-arranged changes in the GOP length. Selecting non-reference encoded image frames from the largest GOP for removal will not affect the display of the pre-event video sequence as much as selecting and deleting larger encoded image frames from a smaller GOP.

[0083] - Non-reference encoded image frames included in a frame subgroup that has a larger number of encoded image frames than any other stored frame subgroup. A large frame subgroup indicates that the scene captured by the encoded image frames of that subgroup has not changed much, and also indicates that the inter-frame frames within that subgroup are smaller. Therefore, selecting non-reference encoded image frames from the largest frame subgroup for removal will not affect the display of the pre-event video sequence as much as selecting and removing potentially larger encoded image frames from a smaller frame subgroup.

[0084] - Non-reference encoded image frames selected according to a predefined pattern. By selecting non-reference encoded image frames according to a predefined pattern (e.g., a first predefined pattern), such that every two or three image frames are selected and then removed, the potential negative visual impact of "missing" image frames is reduced when displaying the pre-event video sequence because the "missing" image frames are distributed according to the predefined pattern in the pre-event video sequence. Furthermore, the uniform distribution removal of image frames can also lead to a reduction in the uniform distribution of the frame rate.

[0085] When a subgroup of frames is selected as one or more encoded image frames to be removed, the subgroup can be selected based on size; for example, the subgroup with the largest size can be selected. Alternatively, the subgroup can be selected based on the number of its frames; for example, the subgroup with the largest number of frames can be selected. As another alternative, the subgroup can be selected according to a predefined pattern, for example, every two or three subgroups to be selected, or three out of four subgroups, or nine out of ten subgroups to be selected. Therefore, when one or more encoded image frames are selected as a subgroup, the subgroup can be at least one of the following:

[0086] - A subgroup of frames with a size greater than the size of any other subgroup of frames stored in bytes. By selecting the subgroup of frames with the largest byte size, and by removing only one subgroup of frames, the maximum possible storage space in the data buffer will be made available.

[0087] - A frame subgroup that includes a larger number of encoded image frames than any other stored frame subgroup. A frame subgroup with a large number of encoded image frames indicates that the scene captured by the encoded image frames of that subgroup has not changed much, and indicates that the inter-frame ratio of the frame subgroup is small. Therefore, by selecting the largest frame subgroup (i.e., the one containing the largest number of encoded image frames) for removal, the display of the video sequence before the event will not be affected as much as if smaller frame subgroups were selected and removed.

[0088] - Frame subgroups selected according to a predefined pattern. By selecting frame subgroups according to a predefined pattern (e.g., a second predefined pattern that may be different from the first predefined pattern mentioned above), such that every two or every three frame subgroups are selected and then removed, the potential negative visual impact of "missing" image frames is reduced when displaying the pre-event video sequence because the "missing" image frames are distributed in the pre-event video sequence according to the predefined pattern.

[0089] In order to keep the latest encoded image frame in the data buffer 520, it is sometimes desirable to select a non-reference encoded image frame or frame subgroup from the oldest GOP stored in the data buffer 520 (i.e., from the oldest GOP in time). Therefore, one or more encoded image frames selected may be included in a stored GOP that was captured at a time point earlier than any other stored GOP.

[0090] For example, the oldest subgroup of frames included in the oldest GOP can be selected. Thus, one or more selected encoded image frames are included in the subgroup of frames captured at a time point earlier than any other subgroup of frames included in the stored GOP.

[0091] The selection of one or more encoded image frames can be performed by the selection module 514. The selection module 514 can be included in the device 500.

[0092] Action 308

[0093] Remove one or more selected encoded image frames from data buffer 520. This means deleting one or more stored encoded image frames from data buffer 520 to free up data space for new encoded image frames to be stored in data buffer 520. It should be understood that the one or more stored encoded image frames can be removed from data buffer 520 and thus deleted by rewriting them with one or more encoded image frames to be stored in data buffer 520.

[0094] The removal of one or more encoded image frames can be performed by the removal module 515. The removal module 515 can be included in the device 500.

[0095] Action 310

[0096] In action 310, one or more encoded image frames from the input stream 400, which are arranged in the GOP, are stored in the data buffer 520.

[0097] Encoded image frames can be stored one after another in the encoding order in the data buffer 520. This can be performed by the storage module 512, which is configured to store encoded image frames in the data buffer 520. As mentioned in action 308, one or more selected stored encoded image frames can be removed by rewriting them with one or more encoded image frames to be stored. This can be performed by the storage module 512.

[0098] When the data buffer 520 is started, all the storage space of the data buffer 520 is available, and therefore the encoded image frames corresponding to the pre-event video sequence with a predefined pre-event recording time can be stored in the data buffer 520.

[0099] However, during the operation of data buffer 520, one or more encoded image frames have been stored in the data buffer, and action 306 must be performed before one or more other encoded image frames in the obtained input stream 400 can actually be stored in data buffer 520, and actions 308 and 310 may also have been performed.

[0100] Therefore, as the input stream 400 is continuously acquired, one or more encoded image frames are stored in the data buffer 520 either immediately or after removing one or more stored encoded image frames, as described above in actions 306, 308, and 310. The one or more encoded image frames can be stored one after another in the encoding order. This can be performed by the storage module 512. The storage module 512 can be included in the device 500.

[0101] In some embodiments, the above method may further include additional actions 312, 314, and 316 to be taken when an event has occurred, such as when an event has been detected. These actions will be described in more detail below. However, it should be understood that they may be performed interleaved with prior actions in any suitable manner.

[0102] Action 312

[0103] In action 312, an event indication is received. The event indication indicates that an event has been detected. This event can be an alarm or an indication that movement has been detected in the captured scene. As another example, the event indication could be the activation of the record button on the BWC. The event indication can be received by a receiving module 516 configured to receive event indications. The receiving module 516 can be included in the device 500.

[0104] Action 314

[0105] Upon receiving an event indication, in action 314, the stored encoded image frame is retrieved from the data buffer 520. The stored encoded image frame is also referred to herein as the pre-event video sequence. This can be performed by the retrieval module 517. The retrieval module 517 can be included in the device 500.

[0106] Action 316

[0107] In action 316, the retrieved encoded image frame is transmitted in the output stream of the encoded image frames. Transmission module 518 can perform the transmission of the retrieved encoded image frame from the device 500 to the receiver in the output stream of the encoded image frames. In addition to the retrieved encoded image frames (i.e., the pre-event video sequence), the output stream of encoded image frames received at the receiver may also include encoded image frames captured during the event (i.e., the event sequence), and may also include encoded image frames captured after the event (i.e., the post-event sequence of encoded image frames). Transmission module 518 may be included in device 500. The receiver may be recorder 230, client 214, or server 216.

[0108] One or more reference encoded image frames, such as one or more reference I-frames, transmitted in the output stream of encoded image frames may be transmitted with a no-display indication. This may be, for example, when the reference I-frame is old, too old to be included in the predefined recording time before the event, and is only stored in the data buffer 520 for use when decoding one or more encoded image frames encoded with reference to it. Because the old reference I-frame may have been captured at a time point corresponding to a time point several frames earlier than the time point when the encoded image frame using it as a reference was captured, the scene may change too much between the two image frames, so it would not look good if they were displayed sequentially in the video sequence. Therefore, it is more pleasing to the viewer to watch the encoded image frames using only the old reference I-frame as a reference, and this is the reason for transmitting reference I-frames with a no-display indication. For example, a no-display indication can be added to the old reference I-frame when it is stored in the data buffer, or when or after retrieving the old reference I-frame from the data buffer. Therefore, the transmitted retrieved encoded image frames may include one or more reference encoded image frames transmitted in the output stream of encoded image frames with a no-display indication. The term "reference encoded image frame" is used in this disclosure for a encoded image frame that is used as a reference image frame for any of the retrieved encoded image frames.

[0109] The embodiments also relate to a means 500 for managing encoded image frames in a data buffer 520, which will now be referred to... Figure 5 The data buffer 520 is configured to store encoded image frames.

[0110] A data buffer is an area of ​​memory used to temporarily store data as it moves from one place to another. Data buffer 520 can be implemented in a fixed memory location in hardware or by using a virtual data buffer in software that points to a location in physical memory. The memory can be random access memory (RAM) or flash memory, but it can also be network-attached storage.

[0111] The apparatus 500 includes an acquisition module 511 configured to acquire an input stream of encoded image frames arranged in a group of frames (GOPs), each GOP comprising: a single intra-frame and two or more subgroups. Each subgroup comprises a single directly encoded inter-frame that is encoded with direct reference to the intra-frame, and zero or more indirectly encoded inter-frames that are encoded with indirect reference to the intra-frame via the single directly encoded inter-frame.

[0112] The device 500 includes a determination module 513 configured to determine whether the buffer fill requirement is met.

[0113] The apparatus 500 further includes a selection module 514 configured to select one or more encoded image frames to be removed from the data buffer 520 from the encoded image frames stored in the data buffer 520 when a buffer fullness requirement is met. The selected one or more encoded image frames are non-reference encoded image frames and at least one of two or more subgroups, wherein the non-reference encoded image frames are encoded image frames that are not used as reference image frames for any of the stored encoded image frames.

[0114] Furthermore, the apparatus 500 includes a removal module 515 configured to remove one or more selected encoded image frames from the data buffer.

[0115] In addition, the device 500 includes a storage module 512 configured to store a stream of encoded image frames arranged in a GOP in a data buffer 520.

[0116] In some embodiments, the apparatus 500 is further configured to perform the method according to one or more actions described herein.

[0117] The device 500 may be included in a camera 208, an encoder 220, or a recorder 230.

[0118] The embodiments also relate to a non-transitory computer-readable medium storing computer code instructions that, when executed by a processing-capable device, are adapted to perform embodiments of the methods described herein.

[0119] As described above, apparatus 500 can be configured to implement a method for managing encoded image frames in a data buffer. For this purpose, apparatus 500 may include circuitry configured to implement the various method steps described herein.

[0120] In hardware implementation, the circuit may be dedicated and specifically designed to implement one or more steps in the method steps. The circuit may take the form of one or more application-specific integrated circuits (ASICs) or one or more field-programmable gate arrays (FPGAs). For example, apparatus 500 may therefore include circuitry for obtaining an input stream 400 of encoded image frames arranged in a group of frames (GOPs), wherein each GOP includes: a single intra-frame 401 and two or more frame subgroups SG1, SG2, and wherein each frame subgroup SG1, SG2 includes a single directly encoded inter-frame 402 encoded directly with reference to the intra-frame 401, and zero or more indirectly encoded inter-frames 403 encoded indirectly with reference to the single intra-frame 401 via the single directly encoded inter-frame 402. Further, the circuitry determines whether a buffer fullness requirement is met. When the buffer fullness requirement is met, the circuitry selects one or more encoded image frames to be removed from the data buffer from the encoded image frames stored in the data buffer. The selected one or more encoded image frames are non-reference encoded image frames and at least one of two or more frame subgroups, wherein the non-reference encoded image frame is an encoded image frame that is a reference image frame not used as a reference for any of the encoded image frames stored. Furthermore, the circuit removes the selected one or more encoded image frames from the data buffer 520 and stores the obtained stream 400 of the encoded image frames arranged in the GOP in the data buffer 520.

[0121] In a software implementation, the circuitry may be replaced by a processor, such as a microprocessor, associated with computer code instructions stored on a (non-transitory) computer-readable medium, such as non-volatile memory, causing device 500 to perform any of the methods disclosed herein. Examples of non-volatile memory include read-only memory, flash memory, ferroelectric RAM, magnetic computer storage devices, and optical discs. In the software case, each of the above method steps may therefore correspond to a portion of computer code instructions stored on a computer-readable medium, which, when executed by a processor, causes device 500 to perform any of the methods disclosed herein.

[0122] It should be understood that a combination of hardware and software implementations is also possible, meaning that some method steps are implemented in hardware while others are implemented in software.

[0123] It will be understood that those skilled in the art can modify the above embodiments in various ways while still utilizing the advantages of the invention as shown in the above embodiments. Therefore, the invention should not be limited to the illustrated embodiments, but should be defined only by the appended claims. Furthermore, as those skilled in the art will understand, the illustrated embodiments can be combined.

Claims

1. A method for managing encoded image frames in an event-before buffer (520), wherein the encoded image frames depict a scene, wherein the event-before buffer (520) is configured to temporarily store the encoded image frames, wherein when an event in the scene or an event related to the scene is detected, the stored encoded image frames constitute a video sequence captured before the event, and wherein the method is characterized in that: (302) An input stream (400) is obtained of encoded image frames depicting the scene and arranged in a group of pictures (GOP), each GOP comprising: A single intraframe (401), and Two or more frame subgroups (SG1, SG2), wherein each frame subgroup (SG1, SG2) comprises: A single directly encoded inter-frame (402) is encoded by directly referencing the intra-frame (401). Wherein, the single directly encoded inter-frame (402) is a P-frame, a fast-forward frame, or an inter-frame refresh frame, wherein the fast-forward frame or inter-frame refresh frame is a forward-predicted inter-frame that references the nearest preceding intra-frame, and One or more indirectly encoded inter-frames (403) are encoded by indirectly referencing the single intra-frame (401) via the single directly encoded inter-frame (402), wherein each of the one or more indirectly encoded inter-frames (403) is a P-frame. Before one or more encoded image frames in the obtained input stream (400) are stored in the pre-event buffer (520), it is determined (304) whether the total recording time of all encoded image frames stored in the pre-event buffer is equal to a predefined pre-event recording time, or whether the total recording time of all encoded image frames stored in the pre-event buffer will become greater than the predefined pre-event recording time if one or more encoded image frames in the obtained input stream are stored in the pre-event buffer. One or more encoded image frames to be included in the obtained input stream (400) will be stored (310) in the pre-event buffer (520) until the total recording time of all stored encoded image frames in the pre-event buffer is equal to the predefined pre-event recording time, or if one or more encoded image frames in the obtained input stream are stored in the pre-event buffer and the total recording time of all stored encoded image frames will not become greater than the predefined pre-event recording time; and The total recording time of all stored encoded image frames is equal to the predefined pre-event recording time, or the total recording time of all stored encoded image frames becomes greater than the predefined pre-event recording time if one or more encoded image frames in the obtained input stream are stored in the pre-event buffer. Select (306) one of the two or more frame subgroups to be removed from the event pre-buffer (520) from the encoded image frames stored in the event pre-buffer (520); Remove (308) the selected frame subgroup from the pre-event buffer (520); and One or more encoded image frames included in the obtained input stream (400) are stored (310) in the pre-event buffer (520) until the total recording time of all encoded image frames stored in the pre-event buffer is equal to the predefined pre-event recording time, or if one or more encoded image frames in the obtained input stream are stored in the pre-event buffer, the total recording time of all encoded image frames stored in the pre-event buffer will become greater than the predefined pre-event recording time. Thus, the encoded image frames temporarily stored in the pre-event buffer (520) are managed as only those encoded image frames that constitute a decodable video sequence having the predefined pre-event recording time.

2. The method of claim 1, wherein the selected frame subgroup is captured at a time point prior to the current time point minus the predefined recording time.

3. The method according to claim 1, wherein, The predefined recording time ranges from a few seconds to tens of minutes, such as from 5 seconds to 10 minutes.

4. The method of claim 1, wherein the selected frame subgroup is at least one of the following: A subgroup of frames that is larger than the size of any other stored subgroup of frames; A frame subgroup comprising the number of encoded image frames greater than any other stored frame subgroup; and The frame subgroup is selected based on a predefined pattern.

5. The method of claim 1, wherein the selected frame subgroup is included in a stored GOP captured at a time point prior to the capture time point of any other stored GOP.

6. The method of claim 5, wherein the selected frame subgroup is captured at a time point prior to the capture time point of any other frame subgroup included in the stored GOP.

7. The method of claim 1, further comprising: Receive (312) an event indication that the event has been detected; Upon receiving the event indication, the stored encoded image frame is retrieved (314) from the event pre-buffer (520), and The retrieved encoded image frame is transmitted (316) in the output stream of the encoded image frame.

8. An apparatus (500) for managing encoded image frames in an event-before buffer (520), wherein the encoded image frames depict a scene, wherein the event-before buffer (520) is configured to temporarily store the encoded image frames, wherein when an event in the scene or an event related to the scene is detected, the stored encoded image frames constitute a video sequence captured before the event, and wherein the apparatus (500) is characterized in that: The acquisition module (511) is configured to acquire an input stream (400) of encoded image frames depicting the scene and arranged in a group of pictures (GOP), each GOP comprising: A single intraframe (401), and Two or more frame subgroups (SG1, SG2), each of which includes: A single directly encoded inter-frame (402) that is directly encoded with reference to the intra-frame, wherein, The single directly encoded inter-frame (402) is a P-frame, a fast-forward frame, or an inter-frame refresh frame, wherein the fast-forward frame or inter-frame refresh frame is a forward-predicted inter-frame that references the nearest preceding intra-frame, and One or more indirectly encoded inter-frames (403) are encoded by indirectly referencing the single intra-frame (401) via the single directly encoded inter-frame (402), wherein each of the one or more indirectly encoded inter-frames (403) is a P-frame. The determining module (513) is configured to determine, before one or more encoded image frames in the obtained input stream (400) are stored in the pre-event buffer (520), whether the total recording time of all encoded image frames stored in the pre-event buffer is equal to a predefined pre-event recording time, or whether the total recording time of all encoded image frames stored in the pre-event buffer will become greater than the predefined pre-event recording time if one or more encoded image frames in the obtained input stream are stored in the pre-event buffer. The storage module (512) is configured to store one or more encoded image frames included in the obtained input stream (400) in the pre-event buffer (520) until the total recording time of all stored encoded image frames in the pre-event buffer is equal to the predefined pre-event recording time, or if one or more encoded image frames in the obtained input stream are stored in the pre-event buffer, the total recording time of all stored encoded image frames will not become greater than the predefined pre-event recording time; and The total recording time of all stored encoded image frames is equal to the predefined pre-event recording time, or the total recording time of all stored encoded image frames becomes greater than the predefined pre-event recording time if one or more encoded image frames in the obtained input stream are stored in the pre-event buffer. The selection module (514) is configured as follows: Select one of the two or more frame subgroups to be removed from the event pre-buffer (520) from the encoded image frames stored in the event pre-buffer (520); The removal module (515) is configured to remove the selected frame subgroup from the pre-event buffer (520); and The storage module (512) is configured to store one or more encoded image frames included in the obtained stream (400) in the pre-event buffer (520) until the total recording time of all encoded image frames stored in the pre-event buffer is equal to the predefined pre-event recording time, or if one or more encoded image frames in the obtained input stream are stored in the pre-event buffer, the total recording time of all encoded image frames stored in the pre-event buffer will become greater than the predefined pre-event recording time; Thus, the device (500) is configured to manage the encoded image frames temporarily stored in the pre-event buffer (520) as only those encoded image frames that constitute a decodable video sequence having the predefined pre-event recording time.

9. The apparatus (500) according to claim 8 is further configured to perform the method according to claim 2.

10. The device (500) according to claim 8, wherein the device is included in a camera (208), an encoder (220), or a recorder (230).

11. A non-transitory computer-readable medium having computer code instructions stored thereon, the computer code instructions being adapted to perform the method according to claim 1 when executed by a processing-capable device (500).

Images