Apparatus and method for encoding an image including a privacy filter

Abstract

The invention provides an apparatus and method of encoding images comprising a privacy filter. For each image in a sequence of images captured by a camera, the image is pre-processed by filtering the image by applying a privacy filter (S110), the privacy filter being configured to distort the image such that privacy is achieved in the filtered image, and for at least a subset of the filtered images, the filtered images are color-corrected (S120) by changing the color of the pixels of a plurality of dispersed regions of the filtered images such that the respective color of one or more pixels of each region in the plurality of dispersed regions represents one or more original colors of the one or more pixels prior to filtering in the filtered image at the location of the region. The pre-processed images are then encoded (S130) into an encoded video stream.

Description

Technical Field

[0001] This invention relates to encoding images, and more particularly to encoding images that have been preprocessed by applying a privacy filter. Background Technology

[0002] When using surveillance cameras to capture images that are encoded into video streams, in some cases it may be necessary, or at least desirable, to apply privacy filters to the captured images before encoding to achieve privacy. Privacy is typically achieved by distorting the captured images to prevent image-related facial recognition, vehicle registration number identification, etc. For example, one scenario might be that a surveillance camera might not be installed unless privacy is achieved. Another scenario might be that permission to install the surveillance camera is not required, or that permission can be granted more quickly if privacy can be achieved. Summary of the Invention

[0003] The purpose of this invention is to facilitate the extraction of enhanced information from encoded images that have been preprocessed with privacy filters before being encoded into video streams, without compromising the privacy of the encoded images themselves.

[0004] According to a first aspect, a method for encoding images captured by a camera is provided. In this method, for each image in a sequence of images captured by the camera, the image is preprocessed by applying a privacy filter configured to distort the image to achieve privacy in the filtered image. Furthermore, for at least a subset of the filtered images, the filtered images are color-corrected by changing the colors of pixels in multiple dispersed regions, such that the color of one or more pixels in each of the multiple dispersed regions represents one or more original colors of the one or more pixels at the location of that region in the filtered image before filtering. The preprocessed images are then encoded into an encoded video stream.

[0005] Privacy is achieved by applying a privacy filter to distort an image and then encoding it into an encoded video stream. This encoded video stream should not be used to identify people, vehicles, etc. However, if police are conducting an investigation and can identify people, vehicles, etc., relevant to the investigation within the encoded video stream, such identification might be desirable.

[0006] The inventors have realized that by introducing information about the original colors of pixels in scattered regions of each image that has been filtered by applying a privacy filter, the seemingly contradictory need to maintain privacy in an encoded video stream while simultaneously allowing subsequent identification can be eliminated or at least mitigated. By changing the colors of pixels in multiple scattered regions of each of the filtered images to represent the original colors of pixels at the corresponding locations, information about the original colors of scattered regions in each image can later be extracted. Thus, in multiple sequentially encoded images of an encoded video stream, moving objects such as people or vehicles may be positioned such that a hat, jacket, bicycle, car, etc., overlaps with one of the scattered regions in which the pixels have been color-corrected. Therefore, at least an approximate color of the hat, jacket, bicycle, car, etc., can be determined by extracting the colors of one or more pixels in that region from multiple scattered regions. Such color information alone cannot identify people, vehicles, or others. However, it can be used, for example, by the police in investigations, where people, vehicles, etc., relevant to the investigation can be identified in the encoded video stream. For example, the extracted colors of the hat, jacket, bicycle, car, etc., can be compared with information from other sources. Based on such comparisons, people, vehicles, or others can be excluded from or included in the investigation.

[0007] Privacy filters are configured to distort an image to achieve privacy in the filtered image. This means that after applying a privacy filter, the image is distorted to prevent identification of people, such as by recognizing facial features, vehicle registration numbers, or other features that may be associated with a particular individual. The term privacy filter is intended to encompass image processing that achieves such prevention. An example of such processing is applying a filter that removes image details that are irrelevant to high gradients in the image (such as a Sobel filter or other filters that use gradient operators). Other examples of such processing are pixelation of all or selected portions of an image, for example, by significantly reducing resolution, blurring all or selected portions of an image, and overlaying or removing selected objects from an image.

[0008] While all images in the image sequence need to be filtered to achieve privacy, only a subset of the filtered images requires color correction, such as every two filtered images or other subsets. Of course, the entire filtered image sequence can also be color corrected.

[0009] In this paper, a dispersed region refers to a region in an image that is separated such that there is a distance between adjacent regions, and that there are regions between adjacent regions that do not belong to any dispersed region. By using dispersed regions, the original color information of pixels representing dispersed regions can be extracted in the form of the color of pixels with color changes, thereby removing or at least reducing the risk of privacy violations.

[0010] Multiple scattered regions can be arranged in a predetermined pattern in at least one subset of the filtered image. This simplifies the decoder's identification of scattered regions, including pixels whose colors have been altered.

[0011] For all filtered images of at least a subset of the filtered images, i.e., all color-corrected filtered images, the predetermined pattern can be fixed (i.e., the same). In this way, the location of the scattered regions can be known on the decoder side without needing to provide metadata indicating the current pattern between the encoder and decoder sides.

[0012] The predetermined pattern can also vary (i.e., be different) among at least a subset of the filtered images. In this case, the pattern should preferably vary in a predetermined manner so that the location of the dispersed region can be known at the decoder side without requiring metadata indicating the current pattern to be provided between the encoder and decoder sides.

[0013] Multiple dispersed regions can be arranged in a sparse grid within at least one subset of the filtered image. This is advantageous because, for a moving object captured in an image sequence, a portion of the moving object that may be of color interest lies within at least one color-corrected filtered image, such that at least one of the dispersed regions lies on that portion.

[0014] A privacy filter can be a privacy filter that includes the application of gradient operators. Such gradient operators can distort the image to enhance edges associated with high gradients and reduce details associated with low gradients. Such a privacy filter will be able to identify objects in the filtered image, but prevent identification of people, such as by recognizing facial features. A privacy filter can be configured to change the original color of the image, and specifically, to generate a monochrome image. Specifically, a privacy filter can be an edge filter such as a Sobel filter.

[0015] Each of the corresponding colors of one or more pixels in each of the multiple dispersed regions can represent the original color of the selected pixel at its location in the filtered image before filtering. Therefore, the corresponding color of one or more pixels can be changed to represent the original color of a pixel before filtering. Preferably, the colors of two or more pixels in each of the multiple dispersed regions can be changed to represent the original color of a pixel before filtering. This is advantageous because changing the colors of two or more pixels in each region to the original color of a single pixel reduces the risk of that color being altered by the encoding of the color-corrected filtered image. This is especially true if the two or more pixels are adjacent pixels, such as, for example, k × k pixels, where k is an integer.

[0016] At least one of the corresponding colors of one or more pixels in each of the multiple regions can represent the average or median of one or more original colors of the one or more pixels at the location of that region in the filtered image before filtering. Preferably, the corresponding color of one or more pixels in each of the multiple regions can represent the average of the colors of two or more pixels at the location of that region in the filtered image before filtering. This is advantageous because the average of the original colors of two or more pixels before filtering can provide a better estimate of the color of an object or part of an object containing a scattered region in the image compared to the color of a single pixel. For example, an image may be affected by noise, and the effect of noise will be less if the average of the colors of two or more pixels is used. The average or median of the colors of two or more pixels before filtering can be determined by a median filter.

[0017] Further preprocessing of images in an image sequence can be performed by applying a low-pass filter to the color-corrected filtered image. This is beneficial because the encoding cost is reduced when encoding the color-corrected filtered image.

[0018] The method may further include: for each image in a sequence of images captured by a camera, receiving one or more sub-regions of a filtered image corresponding to an object identified in the image, wherein, for at least one subset of the filtered image, multiple dispersed regions are arranged within the sub-regions of the received filtered image. Objects identified in the image may typically correspond to information in the form of color that may be of interest for extraction at a later stage. Therefore, arranging all or at least most of the multiple dispersed regions within the sub-regions of the filtered image corresponding to the objects identified in the image increases the likelihood that such information can be extracted.

[0019] Images in a sequence of images captured by a camera can be further preprocessed by reducing the intensity of color-changing pixels in multiple scattered regions of the color-corrected filtered image. By reducing the intensity of color-changing pixels, these pixels become less noticeable and therefore less distracting when watching the decoded version of the encoded video. This is beneficial because it reduces distraction when watching the video for reasons other than extracting color information.

[0020] According to a second aspect, a non-transitory computer-readable storage medium is provided, having stored thereon instructions for implementing the method according to the first aspect when executed on a processing-capable device.

[0021] The aforementioned features of the method described in the first aspect can also be applied to the second aspect when applicable. To avoid unnecessary repetition, please refer to the above.

[0022] According to a third aspect, an image processing apparatus is provided. The image processing apparatus includes circuitry configured to preprocess each image in a sequence of images captured by a camera by performing a filtering function and a color correction function. The filtering function is configured to filter the image by applying a privacy filter, the privacy filter being configured to distort the image to prevent identification of a person in the filtered image. The color correction function is configured to, for at least a subset of the filtered image, color correct the filtered image by changing the color of pixels in a plurality of dispersed regions of the filtered image, such that the color of one or more pixels in each of the plurality of dispersed regions represents the original color of one or more pixels at the location of that region in the filtered image before filtering. The image processing apparatus further includes an encoder configured to encode the preprocessed image into an encoded video stream.

[0023] The aforementioned features of the method described in the first aspect can also be applied to the third aspect when applicable. To avoid unnecessary repetition, please refer to the above.

[0024] According to the fourth aspect, a camera is provided that includes the image processing apparatus according to the third aspect.

[0025] The further scope of the invention will become apparent from the detailed description given below. However, it should be understood that while the detailed description and specific examples illustrate preferred embodiments of the invention, they are given by way of illustration only, as various variations and modifications within the scope of the invention will become apparent to those skilled in the art based on this detailed description.

[0026] Therefore, it should be understood that the present invention is not limited to the specific components of the described apparatus or the operation of the described method, as such apparatus and method can vary. It should also be understood that the terminology used herein is merely for describing particular embodiments and not for limitation. It must be noted that, as used in the specification and appended claims, the articles “a,” “the,” and “described” are intended to indicate the presence of one or more elements unless the context clearly specifies otherwise. Thus, for example, references to “a unit” or “the unit” can include several devices, etc. Furthermore, the words “comprising,” “including,” and similar wording do not exclude other elements or steps. Attached Figure Description

[0027] The above and other aspects of the invention will now be described in more detail with reference to the accompanying drawings. The drawings should not be considered limiting, but rather for explanation and understanding.

[0028] Figure 1 This is a flowchart related to an embodiment of the method disclosed herein.

[0029] Figure 2 These are schematic diagrams relating to embodiments of the image processing apparatus and the camera disclosed herein.

[0030] Figure 3a An example of a filtered image after applying the Sobel filter is shown.

[0031] Figure 3b Show Figure 3a The image shown is a color-corrected version of a portion of the filtered image.

[0032] Figure 4 This shows a color-corrected version of the image after further filtering, showing individuals that have been identified and removed. Detailed Implementation

[0033] The invention will now be described below with reference to the accompanying drawings, which illustrate presently preferred embodiments of the invention. However, the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0034] This invention can be applied to scenarios where privacy is desired or necessary in relation to encoded video streams, i.e., people, vehicles, or other entities should not be identified from the encoded video stream itself. Such privacy can be achieved through image distortion, for example, by applying privacy filters.

[0035] Now refer to Figure 1 And further reference Figure 3a and Figure 3b as well as Figure 4An embodiment of a method 100 for encoding images captured by a camera is described below. In step S102, an image sequence consisting of n images captured by the camera is obtained. This image sequence is used for preprocessing and encoding into an encoded video stream. For example, the camera may be a surveillance camera or a monitoring camera used to monitor an area. Then, each image i (i = 1 → n) should be preprocessed and encoded into a video stream. The variable i is set to 1 in step S106, and the first image in the image sequence is preprocessed by applying a privacy filter to the first image in step S110. The privacy filter is configured to distort the image to achieve privacy in the filtered image.

[0036] Privacy is achieved when an image is distorted to prevent identification of people, such as by recognizing facial features, vehicle registration numbers, or other features that might be associated with a specific individual. Such distortion can be achieved by applying filters that remove image details irrelevant to high gradients in the image (such as edge filters, e.g., Sobel filters, or any other filters using gradient operators). Such edge filters are kernel-based filters, which give a strong response to both ascending and descending gradients in all directions. The absolute value of such gradient operators, along with a gain factor, can be used to generate a filtered image that includes all edges but omits or reduces all other details of the image. Edge filtering can be applied to the image's color, but often brightness-based edge filtering is sufficient to provide a filtered image in which objects can be identified at a general level, but specific individuals cannot be identified. Preprocessing an image by applying a brightness-based edge filter will produce a monochrome image. For an example of a filtered image after applying a Sobel filter, see [reference needed]. Figure 3a and Figure 3b .about Figure 3a and Figure 3b The image has been filtered using a negative gain factor to generate a darker image with brighter areas where there are no edges.

[0037] Other examples of privacy-enhancing processing include pixelating all or selected portions of an image, such as by significantly reducing resolution, blurring all or selected portions of an image, and overlaying or removing selected objects from an image. For an example of processing an image to remove selected objects, see [reference needed]. Figure 4 .exist Figure 4 In the image, the two objects 410 and 420 corresponding to the person have been removed and replaced by a blurred representation of the background. For clarity, the outlines of the two objects 410 and 420 are... Figure 4 The middle section is marked with a solid line.

[0038] What these identified examples of privacy-enhancing processing share is that the color of pixels in an image is altered through processing, making the color of the processed pixel significantly different from its original color. Furthermore, it is impossible to identify the original color of a pixel before processing based on the processed pixel.

[0039] Then, determine whether the first image after C114 filtering should undergo color correction.

[0040] It can be predetermined whether images in an image sequence should undergo color correction after the first filtering, such that, for example, all images or only a predetermined subset, such as every two, every three, or some other predetermined subset, are color corrected. Alternatively, an identifier can be obtained for the image sequence in step S104 indicating which images should undergo color correction after filtering, such that this can change over time, for example, causing a larger proportion or even all images in the image sequence to be color corrected at some times, and a smaller proportion of images in the image sequence to be color corrected at other times. This can be determined, for example, based on the amount of movement identified.

[0041] When only a subset of the filtered images should be color corrected, the decoder needs to determine in some way which filtered images have been color corrected to avoid attempting to identify color-corrected pixels in images that have not been color corrected within the image sequence. The subset is preferably predetermined, such as every two, every three, or any other given pattern starting from a given time point. In this case, the pattern can be synchronized with the Group of Pictures (GOP) structure of the encoded video stream so that the decoder knows which filtered images have been color corrected. If the pattern is not predetermined, metadata indicating whether each image has been color corrected should preferably be provided to the decoder, for example, in a side channel.

[0042] If the first image after filtering should not undergo color correction, then the first image is encoded into an encoded video stream.

[0043] If the filtered first image should be color corrected, then the filtered first image is color corrected by changing the color of the pixels in multiple scattered regions of the filtered image S120, and the first image is further preprocessed so that the corresponding color of one or more pixels in each of the multiple scattered regions represents one or more original colors of one or more pixels at the position of that region in the filtered image before filtering.

[0044] The corresponding color of one or more pixels in each of a plurality of dispersed regions of one or more original colors before filtering, representing the location of one or more pixels in a region of the encoded video stream, can be extracted later from the decoded image of the encoded video stream. Thus, in multiple sequentially encoded images of the encoded video stream, a moving object such as a person or vehicle can be positioned such that at least one of the dispersed regions where the pixels have been color-corrected is located on a hat, jacket, bicycle, car, etc. Therefore, at least an approximate color of the hat, jacket, bicycle, car, etc., can be determined by extracting the color of one or more pixels in that region from the plurality of dispersed regions. Such color information itself cannot identify a person, vehicle, or other entity. However, it can be used, for example, by the police in investigations, where persons, vehicles, etc., relevant to the investigation can be identified in the encoded video stream. For example, the extracted colors of the hat, jacket, bicycle, car, etc., can be compared with information from other sources. Based on such comparisons, persons, vehicles, or other entities can be excluded from or included in the investigation.

[0045] Dispersed regions are separated in an image such that there are distances between adjacent regions within the dispersed regions, that is, there are regions between adjacent regions that do not belong to any dispersed region.

[0046] Multiple scattered regions can be arranged in a predetermined pattern in the color-corrected filtered image. This simplifies the identification of scattered regions on the decoder side, as the predetermined pattern is also known on the decoder side. Specifically, the scattered regions can be sparse and associated with only one or a few pixels. If the decoder side, which decodes and analyzes the encoded video stream to extract the original colors of the pixels, does not know the location of each region in the scattered regions, it is possible to identify pixels whose colors have changed, but this can be difficult and time-consuming.

[0047] The predefined pattern can be fixed, meaning it is identical for all color-corrected filtered images. In this way, the location of scattered regions can be determined on the decoder side without requiring metadata indicating the current pattern to be provided between the encoder and decoder sides.

[0048] Alternatively, the predetermined pattern can vary among at least a subset of filtered images. For the varying pattern, the pattern should preferably vary in a predetermined manner such that the location of the dispersed region can be known at the decoder side, without requiring metadata indicating the current pattern to be provided between the encoder and decoder sides. If the pattern varies in a non-predetermined manner, metadata indicating the current pattern needs to be provided to the decoder side.

[0049] As an alternative to arranging the dispersed regions in a predetermined pattern, metadata indicating the pattern of each color-corrected filtered image can be provided to the decoder side. Such metadata can be provided, for example, in a side channel.

[0050] Multiple dispersed regions are preferably arranged in a uniform distribution in each image of the image sequence, such that for an object located at any position in the images of the image sequence, at least one of the dispersed regions may be located on some portion of that object. Furthermore, if the object is moving, it is possible to locate each portion of the moving object that may be of color interest in at least one image of the image sequence, such that at least one of the dispersed regions is located on that portion.

[0051] For example, multiple dispersed regions can be arranged in a sparse grid such as Figure 3b In at least one subset of the filtered images illustrated in the figure, Figure 3b yes Figure 3a The magnified portion of the filtered image, including scattered regions with color-corrected pixels. Figure 3b In the image, the dispersed areas of color-corrected pixels are arranged in a circular pattern, comprising eight columns, each with five rows. The color correction is indicated by different shades within these circular areas. Figure 3b As can be seen, the dispersed regions are close enough that multiple dispersed regions are located on vehicle 310. Furthermore, even if the dispersed regions are not close enough that at least one of the dispersed regions is located on each part of the person in the image (such as hat, helmet, jacket, pants, etc.), when the person 320, 330 moves between images in the image sequence, at least one of the dispersed regions may be located on one of these parts in one of the images.

[0052] Alternatively, multiple dispersed regions can be arranged in a pseudo-random pattern. The pseudo-random pattern is preferably predetermined and can be fixed or varied in a predetermined manner between images in the image sequence.

[0053] Multiple dispersed regions can also be arranged only in selected portions of each image in the image sequence. For example, the selected portions could refer to a part of the scene of interest captured in the image sequence, such as a road, a building entrance, etc. It is also possible that the selected portions refer to areas of the scene captured in the image sequence, for which additional information, such as information about the original color of the pixels, is allowed to be provided in the encoded video stream. The selected portions could, for example, refer to areas inside the houses of people or organizations where cameras have been set up to capture the image sequence, and other portions could refer to prohibited areas of the scene outside those houses, for which information about the original color of the pixels is not allowed to be provided in the encoded video stream.

[0054] Each region within the dispersed area is preferably small relative to the image, such as a square of 1 pixel, 3×3 pixels, or 5×5 pixels. It can also have a rectangular, circular, or other shape. Even if the size can be larger, it is advantageous if each region is small relative to the image so that it does not distract viewers of the video stream for reasons other than extracting original color information. Furthermore, since the color of the pixels in the dispersed area is altered to represent the original color of the pixels at the location of the dispersed area, the size of each region should not be large enough to compromise privacy.

[0055] Each of the corresponding colors of one or more pixels in each of the multiple dispersed regions can represent the original color of one of the pixels at that location in the filtered image before filtering. For example, if each of the multiple dispersed regions comprises multiple pixels, the corresponding colors of the multiple pixels in each region can be changed to represent or be equivalent to the original color of one pixel in that region before filtering. Preferably, a pixel is predetermined such that it is known at the decoder side, without needing to provide metadata indicating which pixel in the image has the original color to the decoder side, for example, in a side channel. Changing the colors of multiple pixels in each region to the original color of the selected pixel is beneficial because it reduces color distortion caused by encoding the color-corrected filtered image.

[0056] As an example, each region within the dispersed areas can comprise a 3x3 pixel square. Then, the color of each of the corresponding 3x3 pixels in each of the multiple dispersed areas can be changed to the original color of one of the 3x3 pixel pixels. For example, one of the pixels could be the center pixel. Other examples are possible, such as a 5x5 pixel square, circular pixels, rectangular pixels, etc.

[0057] Furthermore, at least one of the corresponding colors of one or more pixels in each of the multiple regions can represent the average or median of one or more original colors of the one or more pixels at the location of that region in the filtered image before filtering. For example, if each of the scattered regions comprises multiple pixels, the corresponding color of one or more pixels in each of the multiple regions can represent the average of the colors of the multiple pixels at the location of that region in the filtered image before filtering. This is advantageous because the average of the original colors of multiple pixels can provide a better estimate of the color of the object or part of an object in the image where the scattered region is located, compared to the color of a single pixel. For example, an image may be affected by noise, and the effect of noise will be less if the average of the colors of two or more pixels is used. The average or median of the colors of two or more pixels before filtering can be determined by applying a median filter to the multiple pixels.

[0058] As an example, each region in the dispersed regions can include a 3×3 pixel square. Then, one or more of the corresponding colors of the 3×3 pixels in each region of the multiple dispersed regions can be changed to the average or median of the original colors of the 3×3 pixels. Other examples are possible, such as 5×5 pixel squares, circular shapes of pixels, rectangular shapes of pixels, etc.

[0059] The preprocessed first image is then encoded into the encoded video stream via S130.

[0060] Then, the variable i is incremented by i = i + 1, and it is checked whether i > n in C134. If not, the next image is preprocessed by filtering the next image S110, and the method is repeated for the next image i. In this case, i = 2 and n > 1, so the method is repeated by applying a privacy filter to preprocess the second image i = 2 in the image sequence.

[0061] Once the C134 indicator i>n is checked, it means that all n images in the image sequence have been preprocessed and encoded into a video stream, and therefore the method ends.

[0062] Images in a sequence of images captured by a camera can be further preprocessed by reducing the intensity of color-changed pixels in multiple scattered regions of the filtered image after color correction (S122). By reducing the intensity of color-changed pixels, they become less noticeable and therefore less distracting when a person watches the decoded version of the encoded video for reasons other than extracting the original color information of the pixels. The color gain can be reduced, for example, from the normal 100% to 30%. The amount of reduction can be predetermined so that it is known on the decoder side, and the reduction is compensated for if the image is intended to be used to extract information about the original color of the pixels. Alternatively, metadata indicating the amount of reduction can be provided to the decoder side, for example, in a side channel.

[0063] Further preprocessing of images in an image sequence can be performed by applying a low-pass filter to the filtered image after color correction in S124. By applying the low-pass filter to the filtered image after color correction, color information is "smeared" onto neighboring pixels, thus reducing the color gradient. Therefore, some additional coding costs caused by high-frequency changes due to the addition of pixels of the original color can be avoided.

[0064] In some scenarios, sub-regions of a filtered image may have been identified as representing objects in each image of a sequence. This could be done, for example, by using object tracking algorithms to track objects within an image sequence. Information associated with the identified sub-regions can be used to filter the image by adding opaque blocks that cover the sub-regions in each image of the sequence, or it can be used to remove objects from each image in the sequence. An example of the latter is... Figure 4 As shown in, Figure 4 In the filtered image, sub-regions 410 and 420 corresponding to the two individuals have been identified and removed, and replaced by blurred indications of the background behind each of the two individuals. Then, method 100 may further include: for each image in the image sequence captured by the camera, receiving one or more sub-regions of the filtered image corresponding to the objects identified in the image. For each of the filtered images, multiple dispersed regions can then be arranged within the received sub-regions of the filtered image. An example of this is... Figure 4 As shown in the figure, in Figure 4As can be seen, all the scattered regions are illustrated as small circles 412 and 414 within the left sub-region 410 and similar circles within the right sub-region 420. The sub-regions identified in the filtered image can typically correspond to color information that may be of interest for extraction in later stages. Therefore, arranging all or at least most of the scattered regions within the sub-regions corresponding to the identified objects in the filtered image increases the likelihood of extracting such information.

[0065] Since the sub-regions corresponding to the identified objects typically vary between images in an unpredictable manner within an image sequence, the patterns of the dispersed regions will also vary unpredictably. Therefore, metadata indicating the current pattern will need to be provided to the decoder side, for example via a side channel, so that the decoder side can easily identify the locations of pixels whose colors have been changed to represent the original colors before filtering.

[0066] from Figure 4 As can be seen, the dispersed regions 412 and 414 of the left sub-region 410 and the corresponding dispersed regions of the right sub-region 420 are based on... Figure 3b The diagram illustrates a subset of multiple dispersed regions in a general pattern similar to a sparse grid. These subsets are dispersed regions within sub-regions 410 and 420, respectively. Therefore, the metadata provided to the decoder can identify sub-regions 410 and 420. If the general pattern in the form of a sparse grid is predetermined, the dispersed regions within sub-regions 410 and 420 can be determined at the decoder based on the predetermined pattern and the received metadata identifying sub-regions 410 and 420.

[0067] It should be noted that even though method 100 has been described as involving the preprocessing and encoding of n images in an image sequence from the first image (i.e., image 1) to the last image (i.e., image n), the images do not necessarily have to be preprocessed in the order of the image sequence (i.e., the order in which they were captured by the camera). They can be preprocessed in any order and then encoded into a video stream such that the video stream includes preprocessed images encoded in an order corresponding to the order in which the images in the image sequence were captured by the camera.

[0068] Further attention should be paid to, Figure 3b and Figure 4 The dimensions of the dispersed regions shown are for illustrative purposes only. The actual dimensions of the dispersed regions are likely much smaller.

[0069] Figure 2This is a schematic diagram relating to embodiments of image processing apparatus 200 and camera 202, which includes an image sensor 208 configured to capture images in an image sequence. Image sensors and image capture are well known to those skilled in the art and will not be discussed in detail herein. Camera 202 may be a surveillance camera or a monitoring camera. Camera 202 may be a standalone unit or it may be integrated into another unit such as a helmet, glasses, etc. Camera 202 may be used to capture video in relation to a monitored area without specifically needing to identify people, vehicles, or others in the video. However, the captured data may subsequently be used as evidence, for example, in investigating crimes and prosecuting criminal suspects. To store the captured data, an external data management system, such as a video management system or an evidence management system, may be used. Such a data management system typically provides storage for the captured data and also allows viewing of the captured data in real time or as playback of recorded data.

[0070] The image processing device 200 includes an encoder 220 and a circuit 210.

[0071] Encoder 220 is configured to encode images (e.g., images in an image sequence) captured by image sensor 208 of camera 202 into a video stream. The video stream provided by encoder 220 may be referred to as an encoded video stream.

[0072] Circuit 210 is configured to perform the functions of image processing device 200. Circuit 210 may include processor 212, such as a central processing unit (CPU), microcontroller, or microprocessor. Processor 212 is configured to execute program code. The program code may, for example, be configured to perform the functions of image processing device 200.

[0073] The image processing apparatus 200 may further include a memory 230. The memory 230 may be one or more of a buffer, flash memory, hard disk drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or other suitable devices. In a typical arrangement, the memory 230 may include non-volatile memory for long-term data storage and volatile memory serving as system memory for circuit 210. The memory 230 may exchange data with the camera circuit 210 via a data bus. Accompanying control lines and address buses may also exist between the memory 230 and the circuit 210.

[0074] The functionality of the image processing apparatus 200 can be implemented in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored on a non-transitory computer-readable medium (e.g., memory 230) of the image processing apparatus 200 and executed by circuitry 210 (e.g., using processor 212). Furthermore, the functionality of the image processing apparatus 200 can be a standalone software application or part of a software application that performs additional tasks related to the image processing apparatus 200. The described functionality can be considered as a method configured to be executed by a processing unit (e.g., processor 212 of circuitry 210). Moreover, while the described functionality can be implemented in software, such functionality can also be implemented using dedicated hardware or firmware, or a combination of hardware, firmware, and / or software.

[0075] Circuit 210 is configured to preprocess images in an image sequence by performing filtering function 231 and color correction function 232. Filtering function 231 is configured to filter the image by applying a privacy filter, which is configured to distort the image to prevent the identification of people in the filtered image. Color correction function 232 is configured to color correct the filtered image for at least one subset by changing the color of pixels in multiple dispersed regions of the filtered image, such that the color of one or more pixels in each of the multiple dispersed regions represents the original color of one or more pixels at that location in the filtered image before filtering.

[0076] Camera 202 may further include a local data storage device (not shown) configured to store video streams and / or a transmitter (not shown) configured to, for example, wirelessly transmit video streams (e.g., continuously transmit captured video streams to a remote site).

[0077] Circuit 210 can be further configured to perform image sequence acquisition function 233. Image sequence acquisition function 233 is configured to acquire images from an image sequence captured by camera 202.

[0078] Circuit 210 can be further configured to perform image identification acquisition function 234. Image identification acquisition function 234 is configured to acquire identifiers of which images in the image sequence should undergo color correction after filtering.

[0079] Circuit 210 may be further configured to perform sub-region receiving function 235. Sub-region receiving function 235 is configured to receive one or more sub-regions of a filtered image corresponding to an object identified in the image for each image in a sequence of images captured by the camera, wherein, for at least one subset of the filtered image, multiple dispersed regions are arranged within the received sub-regions of the filtered image.

[0080] Circuit 210 can be further configured to perform low-pass filter application function 236. Low-pass filter application function 236 is configured to further preprocess each image in the image sequence by applying a low-pass filter to the color-corrected filtered image.

[0081] Circuit 210 can be further configured to perform intensity reduction function 237. Intensity reduction function 237 is configured to further preprocess each image in the image sequence by reducing the intensity of color-changed pixels in multiple scattered regions of the color-corrected filtered image.

[0082] The functions performed by encoder 220 and circuit 210 can be further adapted to be related to Figure 1 The steps corresponding to the described method 100.

[0083] Those skilled in the art will recognize that the present invention is not limited to the embodiments described above. Rather, many modifications and variations are possible within the scope of the appended claims. Such modifications and variations can be understood and implemented by those skilled in the art in practicing the claimed invention through a study of the drawings, the disclosure, and the appended claims.

Claims

1. A method for encoding an image captured by a camera, the method comprising: For each image in the image sequence captured by the camera, the image is preprocessed using the following steps: The image is filtered by applying a privacy filter configured to distort the image to prevent the identification of people in the filtered image. The privacy filter changes the color of pixels in the image from their original color. Identify one or more sub-regions of the filtered image that correspond to objects identified in the image, and Color correction is performed on the filtered image by changing the colors of pixels in multiple dispersed regions, such that the color of one or more pixels in each of the multiple dispersed regions represents the original color of the one or more pixels at that location in the filtered image before filtering. The multiple dispersed regions are arranged within identified sub-regions of the filtered image, and each dispersed region is smaller than a size that compromises privacy. The preprocessed image is encoded into an encoded video stream.

2. The method according to claim 1, wherein, The multiple dispersed regions are arranged in a predetermined pattern in the filtered image.

3. The method according to claim 2, wherein, The predetermined pattern is fixed for all filtered images in the filtered images.

4. The method according to claim 2, wherein, The predetermined pattern varies between filtered images in the filtered image.

5. The method according to claim 1, wherein, The multiple dispersed regions are arranged in a sparse grid in the filtered image.

6. The method according to claim 1, wherein, Applying privacy filters includes applying gradient operators.

7. The method according to claim 1, wherein, The privacy filter is configured to generate a monochrome image.

8. The method according to claim 1, wherein, The privacy filter is an edge filter.

9. The method according to claim 1, wherein, Each of the corresponding colors of the one or more pixels in each of the plurality of dispersed regions represents the original color of the selected pixel in the one or more pixels at the location of that region in the filtered image before filtering.

10. The method according to claim 1, wherein, At least one of the corresponding colors of the one or more pixels in each of the plurality of dispersed regions represents the average value of the one or more original colors of the one or more pixels at the location of that region in the filtered image before filtering.

11. The method according to claim 1, wherein, The images in the image sequence captured by the camera are further preprocessed through the following steps: Reduce the intensity of the color-changed pixels in the plurality of dispersed regions of the color-corrected filtered image.

12. A non-transitory computer-readable storage medium having stored thereon instructions for implementing the method according to any one of claims 1-11 when executed on a processing-capable device.

13. An image processing apparatus, comprising: The circuit is configured to preprocess each image in a sequence of images captured by the camera by performing the following functions: The filtering function is configured to filter the image by applying a privacy filter, the privacy filter being configured to distort the image to prevent the identification of people in the filtered image, wherein, by applying the privacy filter, the color of the pixels in the image is changed from the original color of the pixels; The recognition function is configured to recognize one or more sub-regions of the filtered image corresponding to objects identified in the image; and A color correction function is configured to color correct the filtered image by changing the colors of pixels in a plurality of dispersed regions, such that the color of one or more pixels in each of the plurality of dispersed regions represents the original color of one or more pixels at the location of that region in the filtered image before filtering. The plurality of dispersed regions are arranged within identified sub-regions of the filtered image, and each dispersed region is smaller than a size that would compromise privacy. An encoder is configured to encode a preprocessed image into an encoded video stream.

14. A camera comprising the image processing apparatus according to claim 13.

Images