Image processing apparatus and method of encoding an image captured by a camera

Abstract

The invention provides an image processing apparatus and a method of encoding an image captured by a camera. The invention relates to the field of video encoding. In particular, the invention relates to a method 300 of encoding an image captured by a camera and an image processing apparatus. An image sequence captured with an image sensor of the camera is acquired S310 and an oscillation frequency of a periodic movement of the camera during capturing the image sequence is determined S320. A base subset of images of the image sequence corresponding to the oscillation frequency is identified S330 and the base subset of images is encoded S340 into an encoded video stream comprising intra frames and inter frames.

Description

TECHNICAL FIELD

[0001] The present invention relates to video encoding, and in particular to encoding video captured by a camera. BACKGROUND

[0002] For example, police officers use cameras to capture video streams during patrols and incidents. Such cameras can also be referred to as body worn cameras, BWC. The cameras are typically powered by a battery. Therefore, there is a limit to the available power of the camera. Furthermore, the camera can be configured to transmit the captured video stream via a wireless connection. Therefore, the bit rate available for such transmission is also a limiting factor in relation to the video stream encoding. SUMMARY

[0003] It is an object of the present invention to achieve a bit rate saving for a video stream generated by a camera.

[0004] According to a first aspect, there is provided a method for encoding a video captured by a camera. The method comprises obtaining a sequence of images captured with an image sensor of the camera, and determining an oscillation frequency of a periodic movement of the camera during capturing the sequence of images. The method further comprises identifying a base subset of images of the sequence of images corresponding to the oscillation frequency, and encoding the base subset of images into an encoded video stream comprising intra frames and inter frames.

[0005] The inventors have realized that a camera having an oscillation frequency is subject to a periodic movement when in use, and that this periodic movement can be advantageously taken into account in an encoding scheme that enables a reduction of the bit rate while maintaining the possibility to watch and understand the video content. For this periodic movement, a subset of images (denoted in the present disclosure as a base subset of images) corresponding to the oscillation frequency can be identified (or selected) and encoded into an encoded video stream.

[0006] More movement between consecutive images in a sequence of images to be encoded will typically result in a higher bit rate of the resulting encoded video stream than for a sequence of images with less movement between consecutive images, i.e. where the consecutive images are more similar. The increase in bit rate can for example be due to a failure of motion estimation between consecutive images of the sequence of images, where intra coding of image blocks needs to be performed instead of a less costly inter coding.

[0007] Furthermore, in a sequence of images that has been captured by an image sensor of a camera, when the camera is periodically moved at an oscillation frequency, the periodic movement, i.e. the component of the camera movement that repeats, results in a periodic movement in the sequence of images that is of less interest for the video stream to be encoded. Typically, what is of interest, e.g. of forensic interest, is the scene and objects in the video stream and their relationship to each other, and not the effect on the sequence of images due to the camera movement.

[0008] By identifying the base subset of images comprising only a subset of the sequence of images and encoding them into the video stream, the bitrate of the video stream can be reduced (even substantially reduced) compared to encoding all the sequence of images using the same encoding principles.

[0009] Reducing the bitrate is beneficial, for example, it can reduce the bandwidth required for wireless transmission of the resulting video stream and reduce the storage space required for storing the resulting video stream. This becomes more severe when the camera moves between different locations, for example due to movement of the wearer of the camera, as the wireless capacity can be different between different locations.

[0010] Identifying the base subset of images corresponding to the oscillation frequency enables the base subset of images to consist of images captured by the camera at the same or similar locations related to the periodic movement. This enables the base subset of images to consist of similar consecutive images, for example by having similar directions, and / or where the movement between consecutive images due to the periodic movement is reduced.

[0011] The method can further comprise identifying additional subsets of images of the sequence of images, wherein each additional subset comprises images captured between two consecutive images of the base subset of images, and encoding the additional subsets of images. Each additional subset of images can consist of all images or a subset of images captured between two consecutive images of the base subset of images.

[0012] Encoding, in addition to the base subset of images into the video stream, the remaining images of the sequence of images or at least a subset of the remaining images, denoted in this disclosure as additional subsets of images, into the encoded video stream, such that the encoded video stream comprises the encoded base subset of images and the encoded additional subsets of images, introduces flexibility to enable lower bitrates, for example by transmitting only the encoded base subset of images, or to include more images of the sequence of images in the encoded video stream, for example by transmitting the encoded base subset of images and the encoded additional subsets of images.

[0013] The encoded base subset of images of the video stream comprises intra frames and inter frames that are independent of the additional subsets of images, such that the encoded base subset of images of the video stream has been encoded and can be decoded independently of the additional subsets of images and any frames encoded from the additional subsets of images.

[0014] The base subset of images can be encoded as intra frames and inter frames using predictions that only rely on images within the base subset.

[0015] The additional subsets of images can be encoded as inter frames (P frames or B frames) using predictions that rely on the respective preceding one of the two consecutive images of the base subset of images.

[0016] One or more images of the additional subset of images can be encoded as a null frame. As used herein, a "null frame" generally denotes a frame that has been encoded using skip blocks (P-skip blocks, null blocks, etc.) for at least some macroblocks (MBs) of the encoded frame. In some embodiments, a majority of the MBs of the null frame are encoded as skip blocks. In other embodiments, all of the MBs are set / encoded as skip blocks. Furthermore, the null frame includes a reference to another frame in the video stream. In other words, the null frame is an inter-coded frame in which at least one MB is encoded as a skip block.

[0017] The method can further include estimating an available bandwidth for transmitting the video stream to the remote unit. On condition that the available bandwidth is below a threshold, only the base subset of encoded images of the encoded video stream is transmitted to the remote unit; and on condition that the available bandwidth is equal to or above the threshold, the base subset of encoded images and the additional subset of encoded images of the encoded video stream are transmitted to the remote unit.

[0018] The inclusion of only the base subset of encoded images of the encoded video stream in the transmission to the remote unit, and the alternative inclusion of the additional subset of encoded images in the transmission to the remote unit, enables the bit rate of the video stream to be adapted to the varying bandwidth of the communication channel, such as a wireless communication channel, to which the video stream is to be transmitted. The threshold of the estimated available bandwidth via which both the base subset of encoded images and the additional subset of encoded images of the encoded video stream are transmitted can be selected based on the bit rate of the encoded video stream including the base subset of encoded images and the additional subset of encoded images.

[0019] In the act of identifying the base subset of images, the base subset of images can consist of images captured at an oscillating frequency. For example, one image can be captured every cycle at the oscillating frequency.

[0020] By identifying the base subset of images such that it consists of images captured at an oscillating frequency, the base subset of images will consist of images captured at the same position in relation to the periodic movement of the camera. Thus, in the base subset of images, the movement between consecutive images due to the periodic movement is omitted or at least significantly reduced.

[0021] In the act of identifying the base subset of images, the base subset of images can be further identified based on respective contrast or frequency content of the images of the image sequence. The periodic movement of the camera can cause a variation in the contrast or frequency content of the images of the image sequence, which variation has the same or a similar frequency as the oscillation frequency of the camera. Thus, by identifying, for example, peaks in the contrast or frequency content of the images of the image sequence, the base subset of images can be identified in correspondence with the oscillation frequency of the camera. Furthermore, the base subset of images can also be identified such that it consists of images having desired properties related to the contrast or frequency content.

[0022] The oscillation frequency can be determined using data from one or more motion sensors in the camera.

[0023] The oscillation frequency can be determined using data from the image sequence. The periodic movement of the camera can cause a variation in a property of the images of the image sequence, which variation has the same or a similar frequency as the oscillation frequency of the camera. Such a variation in the property of the images of the image sequence can cause a corresponding variation in the data of the image sequence, which can be used to identify the oscillation frequency of the camera.

[0024] The camera having captured the image sequence can be a camera adapted to be worn on a body of a person. When the camera is worn on the body of the person and the person is moving, the camera will move along with the person. The movement will depend on the type of movement of the person and the position on the body of the person where the camera is worn. For example, when the person wearing the camera is walking or running, the periodic movement of the camera can cause a variation in the property of the images of the image sequence, which variation has the same or a similar frequency as the oscillation frequency of the camera. The camera can also be adapted to be worn on a body of an animal, such as a dog or a horse.

[0025] According to a second aspect, there is provided a non-transitory computer- readable storage medium. The non-transitory computer-readable storage medium of the second aspect has stored thereon instructions which, when executed on a device having processing capability, are used to implement the method according to the first aspect.

[0026] The above-mentioned features of the method according to the first and second aspects also apply to the camera of the fourth aspect, where applicable.

[0027] According to a third aspect, there is provided an image processing device. The image processing device comprises circuitry configured to execute an image sequence acquisition function configured to acquire an image sequence of images captured with an image sensor of a camera. The circuitry is further configured to execute an oscillation frequency determination function configured to determine an oscillation frequency of a periodic movement of the camera during the capturing of the image sequence. The circuitry is further configured to execute a base subset identification function configured to identify a base subset of images of the image sequence corresponding to the oscillation frequency. The image processing device further comprises an encoder configured to encode the base subset of images into an encoded video stream comprising intra frames and inter frames.

[0028] The circuitry can further be configured to perform an additional subset identification function configured to identify an additional subset of images of the sequence of images, wherein each additional subset comprises images captured between two consecutive images of the base subset of images. The encoder can further be configured to encode the additional subset of images into the encoded video stream.

[0029] The encoder can further be configured to encode the base subset of images as intra frames and inter frames using predictions that only rely on images within the base subset.

[0030] The encoder can further be configured to encode the additional subset of images as empty frames or as inter frames using predictions that rely on a respective preceding one of the two consecutive images of the base subset of images.

[0031] In the base subset identification function, the base subset of images can consist of images captured at an oscillation frequency.

[0032] In the base subset identification function, the base subset of images can be further identified based on respective contrast or frequency content of the images of the sequence of images.

[0033] The oscillation frequency determination function can be configured to determine the oscillation frequency using data from one or more motion sensors in the camera.

[0034] The oscillation frequency determination function can be configured to determine the oscillation frequency using data from the sequence of images.

[0035] According to a fourth aspect, there is provided a camera comprising the image processing apparatus of the third aspect. The camera is preferably a surveillance camera or a security camera.

[0036] The camera of the fourth aspect can comprise further circuitry configured to perform a bandwidth estimation function configured to estimate an available bandwidth for transmission to a remote unit. The further circuitry can further be configured to perform a conditional transmission function configured to transmit only the base subset of encoded images of the encoded video stream to the remote unit on condition that the available bandwidth is below a threshold value, and to transmit the base subset of encoded images and the additional subset of encoded images of the encoded video stream to the remote unit on condition that the available bandwidth is equal to or above the threshold value. The further circuitry of the camera of the fourth aspect can be separate from or the same as the circuitry of the image processing apparatus of the third aspect.

[0037] The above-mentioned features of the methods according to the first and second aspects apply also to the camera of the fourth aspect, where applicable.

[0038] Further scope of the applicability of the present application will become apparent from the detailed description given hereinafter. However, it should be understood that the detailed description and specific examples, while indicating preferred embodiments of the application, are given by way of illustration only, since various changes and modifications within the scope of the application will become apparent to those skilled in the art from this detailed description.

[0039] It should be understood, therefore, that the present application is not limited to the particular combinations of components described or the particular acts described, but is amenable to variations and modifications within the scope of the application. It should be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It must be noted that, as used in the specification and the appended claims, the articles "a," "an," and "the" are intended to mean one or more of the elements to which the articles refer and the articles are not intended to mean the only one of the elements to which the articles refer. Thus, for example, reference to "a unit" or "the unit" can include more than one such unit. In addition, the words "comprising," "containing," "including," and similar words are to be construed as meaning "including, but not limited to." BRIEF DESCRIPTION OF DRAWINGS

[0040] The above and other aspects of the present application will now be described in more detail with reference to the accompanying drawings. The drawings should not be considered limiting; the purpose is to explain and understand. Like reference numbers refer to like elements throughout.

[0041] Figure 1 is a schematic block diagram of an image processing device included in a camera.

[0042] Figure 2 is a schematic block diagram of a layered structure of an encoded video stream.

[0043] Figure 3 is a flowchart of a method for encoding a video captured by a camera. DETAILED DESCRIPTION

[0044] The present application will now be described more fully hereinafter with reference to the accompanying drawings, in which current preferred embodiments of the application are shown. The application may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0045] Figure 1A schematic block diagram of an image processing apparatus 100 comprised in a camera 105 is shown. The camera 105 can be any type of camera that can experience periodic movements, such as a security camera or a surveillance camera. For example, the camera 105 can be a body worn camera, BWC, for which the camera can experience periodic movements, e.g. when the person or animal wearing the body worn camera is walking or running. The camera 105 can be used as a body worn camera by, e.g., a police officer, a security guard or a police dog, for capturing video and other possible data during patrols and incidents. The camera 105 can also be used as a body worn camera on a person's body when the person is riding a horse, a bicycle, a motorcycle, a car, etc., and in this case the camera 105 can also be worn by the horse or mounted on the bicycle, the motorcycle, the car, etc. The data (forensic information) that can be captured later can be needed as evidence, e.g. when investigating a crime and prosecuting a suspect. The camera 105 can also be a surveillance camera that is positioned such that it experiences periodic movements, e.g. on a pole that can start to oscillate when exposed to wind or for some other reason. Another example of a camera that can experience periodic movements is a camera on a boat that is exposed to waves. For saving the captured data, a data management system external to the camera 105 can be used, such as a video management system or an evidence management system. Such a data management system typically provides storage of the captured data, as well as viewing of the captured data in real time or as recorded data in a playback. The camera 105 and the image processing apparatus 100 can be powered by a battery and are typically subject to a bit rate limitation. The latter can be due to limited local data storage and / or a bandwidth limitation to a remote unit, such as a data management system or a central for watching live. The connection to the remote unit can be wireless. Furthermore, the bandwidth limitation for the connection can vary over time and between different locations, such that the bit rate that can be transmitted occasionally becomes even more limited. The camera 105 comprises an image sensor 110 configured to capture image data. The image data can be, e.g., data of image frames. The image sensor and the capturing of image data are well known to the person skilled in the art and will not be discussed in more detail in this disclosure.

[0046] The image processing apparatus 100 comprises an encoder 120 and a circuit 130.

[0047] The encoder 120 is configured to encode image data captured by the image sensor 110 into a video stream. Sometimes, the video stream output by the encoder 120 is referred to as an encoded video stream. Typically, the video encoder 120 is configured to encode some image frames of the video stream as intra frames and to encode some image frames of the video stream as inter frames. An intra frame is an encoded video frame that does not require information from other video frames to be decoded. Thus, an intra frame is encoded based on information from the image frame of the video data for which the intra frame is set to correspond. Typically, similarities within an image frame are used to encode the image frame as an intra frame. In video encoding, an intra frame is often referred to as an I-frame. Image frames of the video stream between two intra frames are encoded as inter frames. An inter frame is encoded based on information from at least one other image frame of the video data to be encoded, for which the inter frame is set to correspond. Inter frames typically contain less data than intra frames. In video encoding, an inter frame is often referred to as a P-frame or a B-frame. A P-frame refers to a previous frame for data reference. Thus, to decode a P-frame, the content of the previous frame must be known. A B-frame can refer to a previous frame and a forward frame for data reference. Thus, to decode a B-frame, the content of the previous frame and the forward frame must be known. When encoding an inter frame, the image frame is divided into groups of pixels. The groups of pixels can for example be referred to as blocks, macroblocks or coding tree units. The image frame is compared to a reference frame. For example, the reference frame for encoding a P-frame is the previous image frame. A matching algorithm is used to identify matching groups of pixels between the image frame to be encoded and the reference frame, and if a match is found for a group of pixels, the group of pixels can be encoded based on a motion vector specifying how the group of pixels has moved in the image frame since the reference frame. Determining the motion vector is sometimes referred to as motion estimation. If the movement is large due to fast movement of the camera 105, the motion estimation can fail to identify a motion vector. The fewer motion vectors identified for the image frame to be encoded, the larger the bit size of the resulting encoded inter frame will become, and thus, the larger bandwidth required to transmit the encoded inter frame.

[0048] The circuitry 130 is configured to perform the functions of the image processing apparatus 100. The circuitry 130 can comprise a processor 132, such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 132 is configured to execute program code. The program code can for example be configured to perform the functions of the image processing apparatus 100.

[0049] The image processing device 100 can further comprise a memory 140. The memory 140 can be one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, a random access memory (RAM), and other suitable devices. In a typical arrangement, the memory 140 can comprise a non-volatile memory for long term data storage, and a volatile memory for use as system memory for the circuit 130. The memory 140 can exchange data with the circuit 130 over a data bus. There can also be accompanying control lines and address buses between the memory 140 and the circuit 130.

[0050] The camera can further comprise a circuit 135 configured to perform the functions of the camera 105. The circuit 135 can comprise a processor 137, such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 137 is configured to execute program code. The program code can for example be configured to perform the functions of the camera 105.

[0051] As an alternative to having separate circuits 130, 135 comprising separate processors 132, 137 in the image processing device 100 and the camera 105 as shown in Fig. 1, a common circuit comprising a common processor for the image processing device 100 and the camera 105 can be used. Figure 1

[0052] The camera 105 can further comprise a motion sensor 150 for determining motion data of the camera. The motion sensor 150 is configured to measure movement data of the camera. The motion sensor 150 can comprise a gyroscope, an accelerometer, a pedometer, and / or any other sensor that senses motion. The gyroscope is configured to measure movement data in the form of orientation and / or angular velocity of the camera 105. The accelerometer is configured to measure movement data in the form of acceleration (or rate of change of velocity) of the camera 105 in its own instantaneous frame of rest. The motion sensor 150 is configured to sample the movement data as a function of time.

[0053] The camera 105 can further comprise a local data storage 160. The local data storage 160 can be configured to store a video stream. The local data storage typically has a limited data storage capacity. The local data storage 160 can be any type of local data storage suitable for storing a video stream. For example, the local data storage 160 can be in the form of an SD card reader and an SD card. Another example of the local data storage 160 can be in the form of a flash memory, such as a NAND flash memory.

[0054] ​Camera 105 may further include transmitter 170. Transmitter 170 may be configured to transmit video streams to a data management system, for example, via a wireless connection. Transmitter 170 may be configured to continuously transmit video streams to the video management system. Transmission is typically limited due to the bandwidth available for wireless transmission.

[0055] Camera 105 may further include memory 145. Memory 145 may be one or more of a buffer, flash memory, hard disk drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), and other suitable devices. In a typical arrangement, memory 145 may include non-volatile memory for long-term data storage and volatile memory serving as system memory for circuit 135. Memory 145 may exchange data with circuit 135 via a data bus. Accompanying control lines and address buses may also exist between memory 145 and circuit 135.

[0056] As such Figure 1 The alternative shown, which has separate memories 140 and 145 in the image processing device and camera 105, can use a common memory for the image processing device 100 and camera 105.

[0057] The functionality of the image processing apparatus 100 can be embodied in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored on a non-transitory computer-readable medium (e.g., memory 140) of the image processing apparatus 100 and executed by circuitry 130 (e.g., using processor 132). Furthermore, the functionality of the image processing apparatus 100 can be a standalone software application or part of a software application that performs additional tasks related to the image processing apparatus 100. The described functionality can be considered as a method configured to execute by a processing unit (e.g., processor 132 of circuitry 130). Moreover, while the described functionality can be implemented in software, such functionality can also be executed by dedicated hardware or firmware, or some combination of hardware, firmware, and / or software.

[0058] The circuit 130 of the image processing device 100 is configured to perform an image sequence acquisition function 181. The image sequence acquisition function 181 is configured to acquire an image sequence captured using the image sensor of a camera (such as the image sensor 110 of camera 105).

[0059] The acquired image sequence typically comprises images captured during a plurality of cycles of a periodic movement of the camera 105 containing the image sensor 110 that captured the image sequence. By periodic movement is meant a movement that is regularly repeated with a certain period. The periodic movement can be due to a number of reasons, for example the camera 105 being worn by a person or animal walking or running. Thus, the movement in the image sequence will comprise a component related to the periodic movement, and for some cases the movement in the image sequence will comprise one or more other components related to non-periodic movement.

[0060] For example, for the case where the camera 105 is worn by a person or animal walking or running, the component related to the periodic movement will be due to the wearer taking steps that cause the camera to move up and down and tilt to one side and then to the other side in the periodic movement. The component related to the non-periodic movement will be due to movement of the wearer in the direction in which the wearer is walking or running.

[0061] For the case where the camera 105 is a surveillance camera positioned such that a periodic movement occurs, for example on a pole, when exposed to wind from the side, oscillations can start such that the camera moves periodically to the left and right.

[0062] Another example of a periodic movement occurring is a camera on a boat that is exposed to waves.

[0063] The periodic movement or the total movement in the image sequence that is related to the periodic movement is typically not of interest for the video stream to be encoded. Typically, what is of interest is the content of the video stream, i.e. the scene and objects and their relation to each other, and not the effect of the movement of the camera on the image sequence. Furthermore, the periodic movement will typically contribute to increase the bit rate of the resulting encoded video stream, for example due to failed motion estimation between consecutive images of the image sequence.

[0064] What is common for different types of periodic movement is that the camera will periodically return to the same or a similar position and / or orientation, such that images of the image sequence captured at the same relative position in the periodic movement will be similar.

[0065] The circuit 130 is further configured to perform an oscillation frequency determination function 182. The oscillation frequency determination function 182 is configured to determine an oscillation frequency of the periodic movement of the camera 105 during capturing of the image sequence.

[0066] The oscillation frequency determining function 182 can be configured to determine the oscillation frequency using data from a motion sensor 150 in the camera 105. For example, if the camera 105 is worn by a user walking or running, the motion sensor 150 can comprise a pedometer that records each time the user takes a step. Thus, the oscillation frequency can be determined by the recorded steps. Other ways of using data from a motion sensor 150, e.g. comprising a gyroscope and / or an accelerometer, to identify the oscillation frequency of a periodic movement are well known to the person skilled in the art and will not be discussed in more detail in this disclosure.

[0067] Alternatively or additionally, the oscillation frequency determining function 182 can be configured to determine the oscillation frequency using data from the image sequence or data from processing of the image sequence. For example, due to the periodic movement of the camera 105, the bit rate of the encoded video stream can vary with the periodic movement such that the bit rate has the same oscillation frequency as the oscillation frequency of the camera 105. This is due to bit rate spikes that occur when the camera 105 is moved fast, where the encoding has difficulties to perform motion estimation on the respective end position images. Fast movements can occur when the camera 105 is moved between end positions or if the camera 105 is rotated. The movement of the camera is mainly caused by the movement of the wearer, e.g. when he / she is running, walking, etc. Thus, images captured during fast camera movements require a relatively large amount of intra coding.

[0068] Thus, encoding an image sequence, a part of an image sequence or even a preceding image sequence and analyzing the bit rate of the encoded image sequence, the encoded part of the image sequence or the encoded preceding image sequence can determine or at least estimate the oscillation frequency of the periodic movement of the camera 105.

[0069] In general, the oscillation frequency of a periodic movement can be relatively, but not completely, constant over time. For example, if the camera 105 is worn on a human or animal and the human or animal is walking or running, the oscillation frequency in the form of the step frequency of the human or animal can vary slightly over time, but can be relatively constant over a number of consecutive cycles, where a cycle is two steps: one left foot and one right foot.

[0070] The oscillation frequency determining function 182 can be configured to determine the oscillation frequency as an average over the image sequence, i.e. the number of identified cycles of the periodic movement divided by the capturing time of the image sequence.

[0071] The circuit 130 is further configured to perform a basis subset identifying function 183. The basis subset identifying function 183 is configured to identify a basis subset of images of the image sequence corresponding to the oscillation frequency.

[0072] The purpose of the base subset identification function 183 can be to identify a base subset of images such that the base subset consists of images of an image sequence with the same or similar relative position in relation to the periodic movement, such that the images of the base subset of images are similar, e.g. by having similar direction and / or reducing the movement between consecutive images caused by the periodic movement. This is achieved by identifying a base subset of images corresponding to the oscillation frequency. In particular, the base subset of images can be identified in the base subset identification function 183 such that the base subset consists of images captured at the oscillation frequency. This means that one image is identified in each cycle of the periodic movement, with a cycle distance between consecutive images. An alternative to identifying a base subset of images in the base subset identification function 183 is possible such that the base subset consists of images captured at half the oscillation frequency. This means that one image is identified in every two cycles of the periodic movement, with a two-cycle distance between consecutive images. For some types of periodic movement, an alternative is to identify a base subset of images in the base subset identification function 183 such that the base subset consists of images captured at twice the oscillation frequency. This means that two images are identified in each cycle of the periodic movement, with a cycle distance of half between consecutive images. The latter is for example applicable in case the camera 105 passes the same position and / or direction twice per cycle, e.g. a camera on a pole oscillating sideways due to being exposed to wind. The camera 105 will then pass the central position halfway from one extreme side position to the other extreme side position twice per cycle in the periodic movement from one side to the other side. There is no universal best base subset or universal best method to identify a base subset. The suitable base subset will depend on the application and can also vary over time. For example, the suitable base subset can be identified differently depending on whether the camera is a wearable camera or a camera positioned on a pole. The suitable base subset can also be identified differently depending on the position of the wearable camera on the body of the person or animal wearing the camera and can be identified differently depending on whether the person or animal wearing the camera is walking or running. The skilled person will understand how to identify a suitable base subset, e.g. how to identify a base subset of images such that the base subset consists of images of an image sequence with the same or similar relative position in relation to the periodic movement, such that the images of the base subset of images are similar, e.g. by having similar direction and / or reducing the movement between consecutive images caused by the periodic movement.

[0073] In addition to identifying the base subset corresponding to the oscillation frequency (e.g. one image per cycle), the base subset identification function 183 can be configured to identify a desired position (timing) of the images of the base subset of images in each cycle. For example, after being encoded into the encoded video stream, the images of the image sequence can vary in contrast, frequency content, direction or other quality metric, desired attribute, and resulting bitrate during each cycle. For example, the image with the highest contrast can be identified in the periodically moving cycle, and the base subset of images can be selected to consist of the images of the image sequence identified from the image with the highest contrast at the oscillation frequency. The same can be done in terms of frequency content, direction or other quality metric, desired attribute, and resulting bitrate after being encoded into the encoded video stream as well.

[0074] Alternatively or in addition to identifying the desired position (timing) of the images of the base subset of images in each cycle, the base subset identification function 183 can be configured to identify an undesired position (timing) of the images of the base subset of images in each cycle. For example, the identification can be based on an evaluation of an image attribute that typically varies during each cycle. For example, the images of the image sequence can vary during each cycle such that at a certain position or area of each cycle, one or more images of the image sequence are undesired for the base subset of images. For example, if the camera is worn by a person or animal walking or running, one or more images captured by the user each time she / he steps (puts down her / his foot) can be the least stable / blurred, and it is advantageous to avoid in the base subset of images. Such images can be identified by image analysis, or indirectly by means of a motion sensor 150 when the user steps. The identified undesired position or area can then be used as a constraint when identifying the desired position.

[0075] Other criteria can be used to identify the desired position (timing) of the images of the base subset of images in each cycle, such as an end position of the periodic movement. If the camera moves up and down, for example when the camera is worn by a user walking or running, this will be at the uppermost position and the lowermost position. Alternatively, a central position can be identified between the uppermost position and the lowermost position.

[0076] The encoder 120 is configured to encode the base subset of images into the encoded video stream comprising intra frames and inter frames. The encoder can further be configured to encode the base subset of images into the intra frames and inter frames using predictions that only rely on images within the base subset. The encoder 120 can for example be adapted to encode according to the H.264 or H.265 video compression standard.

[0077] The circuit 130 can further be configured to perform an additional subset identification function 184. The additional subset identification function 184 can be configured to identify additional subsets of images of the sequence of images, wherein each additional subset comprises images captured between two consecutive images of the base subset of images. The encoder 120 can then further be configured to encode the additional subsets of images into the encoded video stream, the encoded video stream comprising the encoded base subset of images and the encoded additional subsets of images.

[0078] The additional subsets of images comprise images that do not belong to the base subset of images.

[0079] It is noted that the combination of the base subset of images and the additional subsets of images can together constitute all images of the sequence of images, or they can constitute a subset of all images of the sequence of images. Thus, some images of the sequence of images can not be encoded into the encoded video stream.

[0080] The encoder 120 can be configured to encode the additional subsets of images into inter frames using prediction, wherein a first frame depends on a respective preceding one of the two consecutive images of the base subset of images.

[0081] Alternatively, the encoder 120 can encode the additional subsets of images into empty frames.

[0082] Alternatively, the encoder 120 can encode some of the additional subsets into inter frames using prediction that depends on a respective preceding one of the two consecutive images of the base subset of images, and encode some of the additional subsets of images into empty frames.

[0083] The general concept of a base layer and one or more additional temporal layers is known from video compression standards. The novel aspect of the method disclosed herein is the identification of the base subset of images such that it consists of images of the sequence of images that have the same or similar relative positions related to the periodic movement, so that the images of the base subset of images are similar (e.g. by having similar directions and / or reducing the movement between consecutive images caused by the periodic movement).

[0084] Turning to Figure 2 which shows an encoding structure with layers in the form of a base layer comprising the encoded base subset of images BS and additional layers comprising the encoded additional subsets of images AS1 to AS4. This encoding structure can be referred to as a temporal encoding structure and is a kind of hierarchical encoding structure. With respect to Figure 2, a base subset of images has been identified that corresponds to the oscillation frequency of the periodic movement of the camera, so that it consists of one image per cycle of the periodic movement during five cycles, wherein the distance between successive images of the base subset of images is equal to the length. The base subset of encoded images BS comprises one intra frame I and four successive inter frames P1 to P4. An additional subset of images has been identified as all or some of the images of the sequence of images between the successive images of the base subset of images. The additional subsets of encoded images AS1 to AS4 can for example have been encoded as inter frames P1' to P4' using a prediction of the initial frame of each additional subset AS1 to AS4 from a respective preceding one of the two successive images of the base subset of images. The inter frame P1' of the first additional subset of encoded images AS1 has been encoded using a prediction that depends on the first image of the base subset of images, i.e. the image encoded into the intra frame I, and the inter frame P2' of the second additional subset of encoded images AS2 has been encoded using a prediction that depends on the second image of the base subset of images, i.e. the image encoded into the first inter frame P1, and so on. Preferably, the time between intra frames in the encoded video stream should not be too long. However, since the base layer only comprises the base subset of encoded images identified for example at the oscillation frequency, which can correspond to the pace at which a user walks or runs while wearing the camera, each image of the base subset of images can be spaced apart by up to one second. Therefore, if the time between intra frames is too short, the number of inter frames between successive intra frames in the base layer will become too small. As a non-limiting example, a suitable number of frames between intra frames can be set to correspond to 5 seconds of encoded video. Depending on the frame rate, i.e. the number of frames per second (FPS), of the video to be encoded, the number of frames can vary. For a frame rate of 30 FPS, the number of frames between intra frames is 150 frames for a video period of 5 seconds. With such a group of pictures (GOP) structure, several frames of the base layer can originate from a single GOP.

[0085] The images related to the base subset of encoded images BS in the base layer are encoded independently from the images related to the additional subsets of encoded images AS1 to AS4 in the additional layers. On the other hand, the images related to the images of the additional subsets of images AS1 to AS4 of the additional layers are encoded depending on the images related to the base subset of encoded images BS in the base layer. Therefore, even if two different layers are encoded into one video stream, two transmission modes can be chosen when the encoded video stream is to be transmitted. In a first transmission mode, only the base layer of the encoded video stream is transmitted, while in a second transmission mode, the base layer and the additional layers of the encoded video stream are transmitted. The first transmission mode will typically result in a much lower bit rate than the second transmission mode and can therefore be used in case the available bit rate for transmission is reduced.

[0086] As an alternative to encoding the images of the additional subsets AS1 to AS4 of images as inter frames P1' to P4' as shown in Figure 2 , the images of the additional subsets AS1 to AS4 of images can be encoded as empty frames.

[0087] Furthermore, as an alternative to encoding the images of the additional subsets AS1 to AS4 of images as P frames using a prediction dependent on a respective preceding one of the two consecutive images of the base subset of images as shown in Figure 2 , the images of the additional subsets AS1 to AS4 of images can be encoded as B frames using a prediction dependent on a respective following one of the two consecutive images of the base subset of images.

[0088] Although two layers are shown in Figure 2 in the form of a base layer and an additional layer, more layers can be added, such as for example another additional layer, so that there is a first additional layer and a second additional layer. In this case, at least three alternative transmission modes for the encoded video stream associated with different bit rates can be used; a first transmission mode in which only the base layer of the video stream, i.e. the encoded images of the base subset of images, is transmitted, a second transmission mode in which the base layer and the first additional layer of the video stream, i.e. the encoded images of the base subset of images and the encoded images of the first additional layer, are transmitted, and a third transmission mode in which the base layer, the first additional layer and the second additional layer of the video stream, i.e. the encoded images of the base subset of images, the encoded images of the first additional layer and the encoded images of the second additional layer, are transmitted.

[0089] Now referring again to Figure 1 , the camera 105 can further comprise a circuit 135 configured to execute a bandwidth estimation function 185 configured to estimate the available bandwidth for transmission to a remote unit, and a conditional transmission function 186 configured to use different transmission modes for the transmission of the encoded video stream via the transmitter 170 depending on a condition regarding the estimated available bandwidth.

[0090] On condition that the available bandwidth is below a threshold value, the conditional transmission function 186 is configured to use a first transmission mode in which only the base subset of encoded images of the video stream is transmitted to a remote unit (not shown), for example via wireless transmission. On condition that the available bandwidth is equal to or above the threshold value, the conditional transmission function 186 is configured to use a second transmission mode in which the base subset of encoded images and the additional subsets of encoded images are transmitted to the remote unit.

[0091] The available bandwidth can be estimated in a conventional manner, e.g. based on feedback from the network, such as feedback indicating packet loss. The conditional transmission function 186 can be configured to transmit the encoded video stream using the second transmission mode until feedback is received indicating packet loss. This indicates that the bandwidth is below the threshold. The conditional transmission function 186 can be configured to subsequently use the first transmission mode, in which only the base subset of encoded images of the encoded video stream is transmitted. The conditional transmission function 186 can be configured to attempt (e.g. at regular intervals) to use the second transmission mode, in which both the base subset of encoded images and the additional subset of encoded images are transmitted, and to monitor whether any feedback is received indicating packet loss. If such feedback is received, the first transmission mode is used, if not, the second transmission mode is used. Typically, at least the base subset of encoded images of the encoded video stream is always transmitted.

[0092] One situation in which wireless transmission of the encoded video stream is used is live streaming of the encoded video stream for live viewing by another user. In this case, it is particularly important that the video content is perceivable by the other user, regardless of how the bandwidth capacity varies. This can be the case, for example, of a police officer getting assistance from another police officer or a police operator. The police officer wears a body camera, and the video stream from the body camera is wirelessly transmitted as a live stream to a remote viewing police officer or police operator who views the live stream.

[0093] For situations in which the camera does not move periodically or when the oscillation frequency cannot be determined, the base subset of images can be identified on the basis other than the oscillation frequency. The basis can be, for example, the latest identified oscillation frequency or a predetermined default frequency or other. The inclusion of such a basis is to ensure that the lower bit rate encoded video stream comprising only the base subset of encoded images can always be transmitted when required, i.e. even when the camera does not move periodically or when the oscillation frequency cannot be determined.

[0094] In combination Figure 3A method 300 for encoding a video captured by a camera will be discussed. The method 300 exploits an implementation by the inventors, namely for a sequence of images captured by an image sensor of a camera that undergoes a periodic movement, the encoded video stream has a much lower bit rate than what would be possible when determining the oscillation frequency of the periodic movement, and a base subset of images corresponding to the oscillation frequency is identified and encoded into the video stream, instead of encoding all images of the image stream. This enables the base subset of images to consist of similar consecutive images, e.g. by having similar directions and / or reducing the movement between consecutive images due to the periodic movement. Thus, the reduction in bit rate is not only due to the reduction in number of image frames, but also due to the reduction in movement between consecutive images in the base sequence of images compared to the movement between consecutive images in the image sequence. Furthermore, in a sequence of images that have been captured by an image sensor of a camera, when the camera is periodically moved at an oscillation frequency, the periodic movement, i.e. the component of the camera movement that repeats, causes a periodic movement of the sequence of images that is of less interest in relation to the video stream to be encoded. Typically, the scene and objects in the video stream and the relationship between them are of interest, and not the effect of the camera movement on the sequence of images. By identifying a base subset of images corresponding to the oscillation frequency, the movement of the base subset of images between consecutive images will be less than the movement between consecutive images of the image sequence. This is due to the reduced effect of the periodic movement on the base subset of images, since the base subset of images is identified corresponding to the oscillation frequency of the periodic movement. In particular, if the base subset of images consists of images of the image sequence that are captured at the same relative position in the periodic movement, i.e. at the oscillation frequency, every period (or every two periods, etc.), the effect of the camera components related to the periodic movement on the movement between consecutive images of the base subset of images will be omitted or greatly reduced.

[0095] Some or all of the steps of the method 300 can be performed by the functionality of the image processing apparatus 100 and / or the camera 105 described above. Unless one step specifically depends on the result of another step, the steps can be performed in any suitable order.

[0096] The method 300 comprises capturing S310 a sequence of images with an image sensor of a camera, and determining S320 an oscillation frequency of a periodic movement of the camera during capturing of the sequence of images. The method further comprises identifying S330 a base subset of images of the sequence of images corresponding to the oscillation frequency, and encoding S340 the base subset of images as an encoded video stream comprising intra frames and inter frames.

[0097] The base subset of images can be encoded as intra frames and inter frames using predictions that only rely on images within the base subset.

[0098] The method 300 can further comprise identifying S350 additional subsets of images of the sequence of images, and encoding S360 the additional subsets of images, wherein each additional subset comprises images captured between two consecutive images of the base subset of images.

[0099] The additional subsets of images can be encoded as one of an inter-frame and a null frame, the inter-frame using a prediction dependent on a respective preceding one of the two consecutive images of the base subset of images.

[0100] The method 300 can further comprise estimating S370 an available bandwidth for transmitting the encoded video stream to a remote unit. On condition C375 that the available bandwidth is below a threshold, the first transmission mode is used to transmit S380 only the base subset of encoded images of the encoded video stream to the remote unit. On condition C375 that the available bandwidth is equal to or above the threshold, the second transmission mode is used to transmit S390 the base subset of encoded images and the additional subsets of encoded images to the remote unit.

[0101] The step of estimating S370 the available bandwidth can be performed in a conventional manner, e.g. based on feedback from the network such as feedback indicating packet loss. The condition C375 can be configured to select to use the second transmission mode, to transmit S390 using the encoded video stream, until feedback is received indicating packet loss. This indicates that the bandwidth is below the threshold. The condition C375 can be configured to subsequently select to use the first transmission mode, to transmit S380, wherein only the base subset of encoded images of the encoded video stream is transmitted. The condition C375 can be configured to subsequently (e.g. at regular intervals) select to use the second transmission mode, to transmit S390, and to monitor whether any feedback is received indicating packet loss. If such feedback is received, the condition C375 is configured to select to use the first transmission mode, to transmit S380, and if not, to select to use the second transmission mode, to transmit S390.

[0102] In the step S330 of identifying the base subset of images, the base subset of images can be identified such that it comprises images captured at an oscillation frequency.

[0103] In the step S330 of identifying the base subset of images, the base subset of images can be further identified based on respective contrast or frequency content of the images of the sequence of images.

[0104] In the step of determining S320 the oscillation frequency, data from one or more motion sensors in the camera can be used to determine the oscillation frequency.

[0105] In the step of determining S320 the oscillation frequency, data from the sequence of images can be used to determine the oscillation frequency.

[0106] Since the oscillation frequency will typically vary over time, the determining step S320 has to be performed repeatedly over time. For example, the determining step S320 can be performed at predetermined time intervals, and the determined oscillation frequency is used after each time. Alternatively, the oscillation frequency is continuously monitored. The oscillation frequency is determined 320 at a first time. Then the basis subset of images is determined S330 based on the determined oscillation frequency until the monitoring of the oscillation frequency indicates that the oscillation frequency varies with respect to the determined oscillation frequency more than a certain percentage or absolute value. Then a new oscillation frequency is determined 320, and the basis subset of images is subsequently determined based on the newly determined oscillation frequency until the monitoring of the oscillation frequency indicates that the oscillation frequency varies with respect to the newly determined oscillation frequency more than a certain percentage or absolute value, and so on.

[0107] The camera can be a camera suitable for being worn on the body of a human or animal.

[0108] The method can further comprise determining the basis subset of images with respect to Figure 1 Features corresponding to the features disclosed for the image processing apparatus 100 and / or the camera 105 are described.

[0109] Those skilled in the art will understand that the application is not limited to the embodiments described above. Rather, many modifications and variations are possible within the scope of the appended claims. Such modifications and variations are understood to be within the scope of the application as disclosed and claimed, and are appreciated by those skilled in the art upon study of the drawings, the disclosure and the appended claims.

Claims

1. A method of encoding images captured by a camera, the method comprising: acquiring a sequence of images captured using an image sensor of the camera; determining an oscillation frequency of a periodic movement of the camera during capture of the sequence of images; identifying an undesirable location or region in each cycle of the periodic movement in which one or more images of the sequence of images are not desired for a base subset of images; selecting a location in each cycle of the periodic movement to avoid the undesirable location or region in each cycle of the periodic movement; identifying a base subset of the images of the sequence of images corresponding to the oscillation frequency such that the base subset of images consists of images captured when the camera is in the same or similar location relative to the periodic movement, wherein in each cycle of the periodic movement, each image of the base subset of images is captured at the selected location; and encoding the base subset of images as an encoded video stream comprising intra frames and inter frames.

2. The method of claim 1, further comprising: identifying additional subsets of images of the sequence of images, wherein each additional subset comprises images captured between two consecutive images of the base subset of images; and encoding the additional subsets of images into the encoded video stream.

3. The method of claim 2, wherein, The act of encoding each additional subset of images comprises: encoding the additional subset of images as one of: an inter frame using a prediction dependent on a respective preceding one of the two consecutive images of the base subset of images, and a null frame, wherein a null frame is a frame that has been encoded using a skip block for at least some macroblocks of the frame.

4. The method of claim 2, further comprising: estimating an available bandwidth for transmission of the encoded video stream to a remote unit; transmitting only the encoded base subset of images of the encoded video stream to a remote unit on condition that the available bandwidth is below a threshold value; and transmitting both the encoded base subset of images and the encoded additional subsets of images of the encoded video stream to the remote unit on condition that the available bandwidth is equal to or above a threshold value. In the act of identifying the base subset of images, the base subset of images consists of images captured at the oscillation frequency.

5. The method of claim 1, wherein, The act of determining the oscillation frequency comprises:

6. The method of claim 1, wherein, using data from one or more motion sensors in the camera to determine the oscillation frequency. The act of determining the oscillation frequency comprises:

7. The method of claim 1, wherein, using a variation in data from the sequence of images or a variation in data from processing of the sequence of images corresponding to a variation in an attribute of images of the sequence of images having a same or similar frequency to the oscillation frequency of the periodic movement of the camera during capture of the sequence of images. The camera is a camera adapted to be worn on a person's body.

8. The method of claim 1, wherein, ​ 9. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed on a device having processing capability, are to implement the method of any one of claims 1 to 8.

10. An image processing device comprising: circuitry configured to perform: an image sequence acquisition function configured to acquire an image sequence captured with an image sensor of a camera; an oscillation frequency determination function configured to determine an oscillation frequency of a periodic movement of the camera during capturing of the image sequence; and an unwanted position identification function configured to identify an unwanted position or area in each cycle of the periodic movement in which one or more images of the image sequence are not wanted for a base subset of images; a position selection function configured to select a position in each cycle of the periodic movement to avoid the unwanted position or area in each cycle of the periodic movement; a base subset identification function configured to identify the base subset of images of the image sequence corresponding to the oscillation frequency such that the base subset of images consists of images captured when the camera is in a same or similar position related to the periodic movement, wherein each image of the base subset of images is captured at the selected position in each cycle of the periodic movement; and an encoder configured to encode the base subset of images into an encoded video stream comprising intra frames and inter frames. the circuitry is further configured to perform: an additional subset identification function configured to identify additional subsets of images of the image sequence, wherein each additional subset comprises images captured between two consecutive images of the base subset of images, and 11. The image processing apparatus according to claim 10, wherein wherein the encoder is further configured to encode the additional subsets of images into the encoded video stream. the encoder is configured to encode the additional subsets of images as one of: inter frames using predictions dependent on a respective preceding one of the two consecutive images of the base subset of images, and 12. The image processing apparatus according to claim 11, wherein null frames, wherein a null frame is a frame that has been encoded using a skip block for at least some macroblocks of the frame.

13. A camera comprising the image processing device of claim 10 and the image sensor for capturing the image sequence. circuitry configured to perform: a bandwidth estimation function configured to estimate an available bandwidth for transmission to a remote unit; and 14. The camera of claim 13, further comprising: a conditional transmission function configured to: on condition that the available bandwidth is below a threshold, transmit only the encoded base subset of images of the encoded video stream to a remote unit; and on condition that the available bandwidth is equal to or above a threshold, transmit both the encoded base subset of images and the encoded additional subsets of images to the remote unit. ​ ​ ​ ​

Images