DEVICE AND METHOD FOR ENCODING IMAGES THAT INCLUDE PRIVACY FILTER - Patent application

Abstract

To provide an image processing device, a camera, and a method of encoding images captured by the camera.SOLUTION: A method comprises the steps of: filtering (S110) the image by applying a privacy filter to each image of an image sequence captured by a camera, the privacy filter being configured to distort the image in such a way that privacy is achieved in the filtered image; for at least a subset of the filtered images, color revising (S120) the filtered image by changing colors of pixels of a plurality of scattered areas of the filtered image such that a respective color of one or more pixels of each area of the plurality of scattered areas represents one or more original colors of one or more pixels before filtering at a location of that area in the filtered image; and encoding (S130) the images into a video stream.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to encoding images, and in particular to encoding images that have been pre-processed by application of a privacy filter. [Background technology]

[0002] When using surveillance cameras that capture images that are encoded into a video stream, in some scenarios it may be necessary, or at least desirable, to apply a privacy filter to the captured images before they are encoded in order to achieve privacy. Privacy is typically achieved by distortion of the captured images such that facial recognition on the images, identification of vehicle registration numbers, etc. are prevented. For example, such a scenario may be that surveillance cameras cannot be installed unless privacy is achieved. Another scenario may be that if privacy can be achieved, permission to install surveillance cameras is not required or is granted sooner. Summary of the Invention

[0003] It is an object of the present invention to facilitate enhanced information extraction from coded images that have been pre-processed with a privacy filter before encoding into a video stream, without compromising privacy in the coded images.

[0004] According to a first aspect, a method for encoding images captured by a camera is provided. In the method, for each image of an image sequence captured by the camera, the image is preprocessed by filtering the image by applying a privacy filter configured to distort the image in a manner such that privacy is achieved in the filtered image, and for at least a subset of the filtered images, color-correcting the filtered image by changing the colors of pixels of a plurality of scattering areas of the filtered image so that the respective colors of one or more pixels in each of the plurality of scattering areas represent one or more original colors of one or more pixels at the location of that area in the filtered image before filtering. The preprocessed images are then encoded into an encoded video stream.

[0005] By applying a privacy filter to distort an image in a manner that achieves privacy, and encoding such an image into an encoded video stream, the encoded video stream should not be able to be used by itself to identify people, vehicles, etc. However, such identification may be desired if the police are conducting an investigation and people, vehicles, etc. relevant to the investigation can be found in the encoded video stream.

[0006] The inventors have discovered that the seemingly contradictory desire to maintain privacy in an encoded video stream while simultaneously enabling later identification can be eliminated or at least alleviated by introducing information about the original color of pixels in each scattered area of ​​an image filtered by application of a privacy filter. By modifying the color of pixels in each of a plurality of scattered areas of a filtered image to represent the original color of the pixel at the location of the respective area, information about the original color can be later extracted for each scattered area. Thus, over several consecutive encoded images of an encoded video stream, a moving object, such as a person or vehicle, may be in a position such that a hat, jacket, bicycle, car, etc. matches an area of ​​one of the scattered areas whose pixels have been color-modified. Thus, at least an approximate color of the hat, jacket, bicycle, car, etc. can be determined by extracting the color of one or more pixels in that area of ​​the plurality of scattered areas. Such color information does not, by itself, enable identification of a person, vehicle, or the like. However, such color information can be used, for example, by police in an investigation where a person, vehicle, etc. relevant to the investigation may be found in the encoded video stream. For example, extracted colors of hats, jackets, bicycles, cars, etc. may be compared with information from other sources. From such comparisons, people, vehicles, or others may be excluded or included in an investigation.

[0007] A privacy filter is configured to distort an image in such a way that privacy is achieved in the filtered image. By this, it is meant that after application of the privacy filter, the image is distorted in such a way that identification of a person is hindered, such as by recognizing their facial features, vehicle registration numbers, or other features that can be associated with a particular person. The term privacy filter is intended to encompass processing of an image that achieves such hindering. One example of such processing is applying a filter that removes image details that are not related to high gradients in the image, such as a Sobel filter or other filter that uses a gradient operator. Other examples of such processing are, for example, a substantial reduction in resolution, blurring all or selected parts of the image, and pixelization of all or selected parts of the image by covering or removing selected objects in the image.

[0008] All images in an image sequence need to be filtered to achieve privacy, but only a subset of the filtered images need be color corrected, for example, every other image or some other subset of the filtered images. Of course, all of the filtered images may be color corrected in the same way.

[0009] By scattered areas it is meant here that areas are separated in the image such that there is a distance between adjacent areas and such that there are areas between adjacent areas that are not part of any of the scattered areas. By using information that can be extracted in the form of scattered areas, color-changed pixel colors that represent the original colors of the pixels in the scattered areas, the risk of compromising privacy can be eliminated or at least reduced.

[0010] The scattering areas may be arranged in a predetermined pattern in at least one subset of the filtered image, which simplifies identification at the decoder side of scattering areas comprising pixels whose pixel color has been changed.

[0011] The predetermined pattern may be fixed, i.e., the same, for all filtered images of at least one subset of filtered images, in other words, in all color-corrected filtered images. In this way, the location of the scattering areas may be known at the decoder side without the need to provide metadata indicating the current pattern between the encoder and decoder side.

[0012] The predetermined pattern may also vary, in other words be different, between the filtered images of at least a subset of filtered images, in which case the pattern should preferably vary in a predetermined manner such that the locations of the scattering areas can be known at the decoder side without the need to provide metadata indicating the current pattern between the encoder and decoder side.

[0013] The plurality of scattering areas may be arranged in a sparse grid in at least one subset of the filtered images, which is beneficial because, for a moving object captured in the image sequence, a portion of the moving object for which color may be noted is likely to be located in at least one of the color-corrected filtered images such that at least one of the scattering areas is located on the portion.

[0014] The privacy filter may be a privacy filter that includes applying a gradient operator. Such a gradient operator allows image distortion to enhance edges associated with high gradients and reduce details associated with low gradients. Such a privacy filter allows identification of objects in the filtered image but prevents identification of people, such as by recognizing their facial features. The privacy filter may be configured to alter the original color of the image, and in particular to generate a monochrome image. In particular, the privacy filter may be an edge filter, such as a Sobel filter.

[0015] Each of the colors of one or more pixels in each of the plurality of scattering areas may represent the original color of a selected pixel of the one or more pixels at that area's location in the filtered image before filtering. Thus, the color of each of the one or more pixels may be changed to represent the original color of the pixel before filtering. Preferably, the colors of two or more pixels in each of the plurality of scattering areas may be changed to represent the original color of the pixel before filtering. This is beneficial because changing the colors of two or more pixels in each of the areas to the original color of the pixel reduces the risk that the color will be changed due to encoding of the color-corrected filtered image. This is particularly true when the two or more pixels are adjacent pixels, e.g., k×k pixels, where k is an integer.

[0016] At least one of the respective colors of one or more pixels in each of the multiple areas may represent the average or median of one or more original colors of one or more pixels at the location of that area in the filtered image before filtering. Preferably, the respective colors of one or more pixels in each of the multiple areas may represent the color average of two or more pixels at the location of that area in the filtered image before filtering. This is beneficial because the average of the original colors of two or more pixels before filtering may provide a better estimate of the color of an object or portion of an object in the image on which the scattering area is located than the color of a single pixel. For example, the image may be noisy, and if the average of the colors of two or more pixels is used, the noise will have less of an impact. The average or median of the colors of two or more pixels before filtering may be determined by a median filter.

[0017] The images of the image sequence may be further pre-processed by applying a low-pass filter to the color-corrected filtered images, which is beneficial because the coding cost is then reduced when encoding the color-corrected filtered images.

[0018] The method may further include receiving, for each image of the image sequence captured by the camera, one or more sub-areas of the filtered image corresponding to an object identified in the image, wherein for at least a subset of the filtered images, the plurality of scattering areas are arranged within the received sub-areas of the filtered image. The identified object in the image may generally correspond to an object about which information in the form of color may be noted for subsequent extraction. Thus, arranging all or at least a majority of the plurality of scattering areas within the sub-areas of the filtered image corresponding to the object identified in the image increases the likelihood that such information can be extracted.

[0019] The images of the image sequence captured by the camera may be further preprocessed by reducing the intensities of the color-modified pixels in multiple scattered areas of the color-corrected filtered image. By reducing the intensities of the color-modified pixels, the color-modified pixels become less noticeable and therefore less distracting when viewing a decoded version of the encoded video. This is beneficial because reducing the intensities of the color-modified pixels reduces distraction when the video is viewed for reasons other than extracting color information.

[0020] According to a second aspect, there is provided a non-transitory computer-readable storage medium storing instructions for implementing a method according to the first aspect when executed on a device having processing capability.

[0021] The above-mentioned features of the method according to the first aspect, where applicable, apply analogously to this second aspect, and in order to avoid excessive repetition, reference is made to the above.

[0022] According to a third aspect, an image processing device is provided. The image processing device comprises a circuit configured to preprocess each image of an image sequence captured by a camera by performing a filtering function and a color correction function. The filtering function is configured to filter the image by applying a privacy filter, the privacy filter being configured to distort the image in a manner that prevents people from being identified in the filtered image. The color correction function is configured to color-correct the filtered image by, for at least a subset of the filtered image, changing the color of pixels of a plurality of scattering areas of the filtered image such that the color of each of one or more pixels in each of the plurality of scattering areas represents the original color of the one or more pixels at that area's location in the filtered image before filtering. The image processing device further comprises an encoder configured to encode the preprocessed images into an encoded video stream.

[0023] The above-mentioned features of the method according to the first aspect, where applicable, apply equally to this third aspect, and in order to avoid excessive repetition, reference is made to the above.

[0024] According to a fourth aspect, there is provided a camera comprising an image processing device according to the third aspect.

[0025] Further scope of the applicability of the present invention will become apparent from the following detailed description. It should be understood, however, that the detailed description and specific examples, while indicating preferred embodiments of the invention, are given by way of illustration only, since various modifications and alterations within the scope of the invention will become apparent to those skilled in the art from this detailed description.

[0026] Therefore, it is to be understood that the present invention is not limited to the particular component parts of the devices described or the method acts described as such devices, and that methods may vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not limiting. It should be noted that, as used in this specification and the appended claims, the articles "a," "an," "the," and "said" are intended to mean that there are one or more elements, unless the context clearly dictates otherwise. Thus, for example, reference to "a unit" or "the unit" may include several devices and the like. Furthermore, the words "comprising," "including," "containing," and similar phrases do not exclude other elements or steps.

[0027] These and other aspects of the present invention will now be described in more detail with reference to the accompanying figures, which should not be considered limiting, but instead are used for purposes of illustration and understanding. [Brief explanation of the drawings]

[0028] [Figure 1] 1 is a flowchart of a method embodiment of the present disclosure. [Figure 2] 1 is a schematic diagram of an image processing device embodiment and a camera embodiment of the present disclosure. [Figure 3a] FIG. 1 shows an example of a filtered image to which a Sobel filter has been applied. [Figure 3b] FIG. 3b shows a color-corrected version of a portion of the filtered image shown in FIG. 3a. [Figure 4] FIG. 10 shows a color-corrected version of the further filtered image in which people have been identified and removed. DETAILED DESCRIPTION OF THE INVENTION

[0029] The present invention will now be described with reference to the accompanying drawings, in which presently preferred embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0030] The present invention is applicable in scenarios where privacy is desired or necessary to be achieved with respect to the encoded video stream, in other words, when identification of people, vehicles or other entities in the image should not be possible from the encoded video stream per se. Such privacy can be achieved by distortion of the image, for example by application of a privacy filter.

[0031] An embodiment of a method 100 for encoding images captured by a camera will now be described with reference to FIG. 1 and with further reference to FIGS. 3a, 3b, and 4. An image sequence consisting of n images captured by a camera is obtained S102 for preprocessing and encoding into an encoded video stream. The camera may, for example, be a surveillance camera or a monitoring camera used to monitor an area. Each image i, i = 1 → n, is then to be preprocessed and encoded into a video stream. A variable i is set S106 to 1, and a first image in the image sequence is preprocessed by filtering the first image by applying a privacy filter S110. The privacy filter is configured to distort the image in a manner such that privacy is achieved in the filtered image.

[0032] Privacy is achieved when an image is distorted in a manner that prevents identification of a person, such as by recognizing their facial features, vehicle registration numbers, or other features that can be associated with a particular person. Such distortion can be achieved by applying a filter that removes image details that are not related to high gradients in the image, such as an edge filter (e.g., a Sobel filter) or any other filter that uses a gradient operator. Such edge filters are kernel-based filters that provide a strong response to rising and falling gradients in all directions. The absolute value of such a gradient operator along with a gain factor can be used to generate a filtered image that includes all edges but omits or reduces all other image details. While it is possible to edge filter the color of an image, luminance-based edge filtering is generally sufficient to provide a filtered image in which objects can be identified at a general level, but identification of specific people is prevented. Preprocessing an image by applying a luminance-based edge filter produces a monochrome image. Reference is made to Figures 3a and 3b for an example of a filtered image after application of a Sobel filter. For the images of Figures 3a and 3b, a negative gain factor was used to produce a filtered image in which edges are darker and areas without edges are lighter.

[0033] Other examples of processing to achieve privacy are, for example, substantial reduction in resolution, blurring of all or selected portions of the image, and pixelation of all or selected portions of the image by covering or removing selected objects in the image. Reference is made to FIG. 4 for an example of an image after processing to remove selected objects in the image. In FIG. 4, two objects 410, 420 corresponding to people have been removed from the image and replaced with blurred representations of the background. For clarity of illustration, the outlines of the two objects 410, 420 have been identified by solid lines in FIG. 4.

[0034] What the identified examples of processing that achieve privacy have in common is that the color of pixels in an image is altered by the processing in a way that the color of the pixel after the processing can be significantly different from the original color of the pixel before the processing, and it is not possible to distinguish the original color of the pixel before the processing from the pixel after the processing.

[0035] It is then determined C114 whether the filtered first image should be color corrected.

[0036] Whether the images in the image sequence should be color corrected after being initially filtered may be predetermined, for example, so that all images are color corrected, or so that only a predetermined subset, such as every other image, every third image, or some other predetermined subset, is color corrected. In an alternative identification, identification of which images in the image sequence should be color corrected after being filtered may be obtained for the image sequence S104 such that this may be varied over time, for example, so that a larger portion or even all images in the image sequence are color corrected at some times and a smaller portion of the images in the image sequence are color corrected at other times. This may be determined, for example, based on an identified amount of movement.

[0037] In the case where only a subset of the filtered images should be color corrected, the decoder side needs to decide which of the filtered images have been color corrected in order to avoid trying to identify color-corrected pixels in images of the image sequence that have not been color corrected in some way. The subset is preferably predetermined, such as every other image, every third image, or any other given pattern starting from a given time point. In this case, the pattern can be synchronized with the group of pictures (GOP) structure of the coded video stream so that the decoder side knows which of the filtered images have been color corrected. If the pattern is not predetermined, metadata indicating whether the image has been color corrected or not should preferably be provided to the decoder side, for example in a side channel.

[0038] If the filtered first image is not to be color corrected, the first image is encoded into the encoded video stream.

[0039] If the filtered first image is to be color corrected, the first image is further preprocessed by color correcting S120 the filtered first image by changing the colors of the pixels of the plurality of scattering areas of the filtered image so that the respective colors of the one or more pixels in each of the plurality of scattering areas represent one or more original colors of the one or more pixels at the location of that area in the filtered image before filtering.

[0040] The respective colors of one or more pixels in each of the plurality of scattering areas, representing one or more original colors of the one or more pixels at the location of the area in the filtered image before filtering, can be subsequently extracted from a decoded image of the encoded video stream. Thus, over several consecutive encoded images of the encoded video stream, a moving object, such as a person or vehicle, may be in a position such that at least one of the scattering areas whose pixels are color-modified is located above a hat, jacket, bicycle, car, etc. Thus, at least an approximate color of the hat, jacket, bicycle, car, etc. can be determined by extracting the color of one or more pixels in that area of ​​the plurality of scattering areas. Such color information does not, by itself, enable identification of the person, vehicle, or the like. However, such color information can be used, for example, by police in an investigation where a person, vehicle, etc. relevant to the investigation may be found in the encoded video stream. For example, the extracted colors of the hat, jacket, bicycle, car, etc. can be compared to information from other sources. From such a comparison, the person, vehicle, or the like can be excluded or included in the investigation.

[0041] The scattered areas are separated in the image such that there is a distance between adjacent areas of the scattered area, in other words there is an area between adjacent areas that is not part of either of the scattered areas.

[0042] The scattering areas may be arranged in a predetermined pattern in the color-corrected filtered image. This simplifies the identification of the scattering areas at the decoder side, since the predetermined pattern may also be known at the decoder side. In particular, the scattering areas may be sparse and only relate to one or a few pixels. If the location of each scattering area is not known at the decoder side, where the encoded video stream is decoded and analyzed to extract the original color of the pixel, identifying the pixel for which the color has been changed, although possible, may be difficult and time-consuming.

[0043] The predetermined pattern may be fixed, in other words such that the predetermined pattern is the same for all color-corrected filtered images. In this way, the location of the scattering areas may be known at the decoder side without the need to provide metadata indicating the current pattern between the encoder and decoder side.

[0044] Alternatively, the predetermined pattern may vary between filtered images of at least one subset of filtered images. In the case of a varying pattern, the pattern should preferably vary in a predetermined manner such that the locations of the scattering areas can be known at the decoder side without the need to provide metadata indicating the current pattern between the encoder and decoder side. If the pattern varies in a non-predetermined manner, metadata indicating the current pattern needs to be provided at the decoder side.

[0045] Instead of arranging the scattering areas in a predetermined pattern, metadata indicating the pattern for each color-corrected filtered image can be provided to the decoder side, such metadata can be provided in a side channel, for example.

[0046] The plurality of scattering areas are preferably evenly spread and arranged in each image of the image sequence such that for an object located anywhere in an image of the image sequence, at least one of the scattering areas is likely to be located on some portion of the object. Moreover, if the object is moving, each portion of the moving object for which color may be noted in at least one image of the image sequence is likely to be positioned such that at least one of the scattering areas is located on that portion.

[0047] The scattering areas may be arranged in a sparse grid in at least one subset of the filtered image, for example, as shown in FIG. 3b, which is an enlarged portion of the filtered image in FIG. 3a with the included scattering areas having color-modified pixels. In FIG. 3b, the scattering areas with color-modified pixels are in the form of circular areas arranged in eight columns, each including five rows. The color modification is indicated by the different shading of the circular areas. As can be seen in FIG. 3b, the scattering areas are close enough together so that several scattering areas are located on the vehicle 310. Furthermore, even if the scattering areas are not close enough together so that at least one of the scattering areas is located on each part of the person in the image, such as the hat, helmet, jacket, or pants, as the person 320, 330 moves between images in the image sequence, at least one of the scattering areas may be located on one of these parts in one of the images.

[0048] Alternatively, the plurality of scattering areas may be arranged in a pseudo-random pattern, which is preferably predetermined and may be fixed or may vary in a predetermined manner between images of the image sequence.

[0049] The multiple scattering areas may also be arranged only in selected portions of each image of the image sequence. For example, the selected portions may relate to portions of the scene captured in the image sequence that are of interest, such as roads, building entrances, etc. The selected portions may also relate to areas of the scene captured in the image sequence for which additional information, such as information about the original color of pixels, is enabled to be provided in the encoded video stream. The selected portions may relate to areas inside the premises of a person or entity that set up a camera that captured the image sequence, for example, while other portions may relate to prohibited areas of the scene, relating to areas outside those premises for which information about the original color of pixels is not enabled to be provided in the encoded video stream.

[0050] Each scattering area is preferably small relative to the image, for example, a 1 pixel, 3x3 pixel, or 5x5 pixel square. It can also have a rectangular shape, a circular shape, or other shapes. Although the size can be larger, it is beneficial for the size of each area to be small relative to the image so that the area is not distracting to anyone viewing the video stream for reasons other than extracting the original color information. Moreover, because the color of the pixels in the scattering area is changed to represent the original color of the pixel at the location of the scattering area, the size of each scattering area should not be large enough to impair privacy.

[0051] Each of the respective colors of one or more pixels in each of the plurality of scattering areas may represent the original color of one pixel of the one or more pixels at that area's location in the filtered image before filtering. For example, if each of the plurality of scattering areas comprises a plurality of pixels, the color of each of the plurality of pixels in each area may be changed to represent or equal the original color of one pixel in that area before filtering. The one pixel is preferably predetermined so that the one pixel can be known at the decoder side without having to provide metadata to the decoder side, e.g., in a side channel, indicating which pixels in the image had their original color. Changing the color of the plurality of pixels in each area to the original color of the selected pixel is beneficial because changing the color of the plurality of pixels in each area reduces color distortion due to encoding of the color-corrected filtered image.

[0052] As an example, each scattering area may comprise a 3x3 pixel square. The color of each of the 3x3 pixels in each of the multiple scattering areas may then be changed to the original color of one of the pixels in the 3x3 pixels. One of the pixels may be, for example, a central pixel. Other examples are possible, such as a 5x5 pixel square, a circular shape of pixels, a rectangular shape of pixels, etc.

[0053] Furthermore, at least one of the respective colors of one or more pixels in each of the multiple areas may represent the average or median of one or more original colors of one or more pixels at the location of that area in the filtered image before filtering. For example, if each scattering area comprises multiple pixels, the respective colors of one or more of the pixels in each of the multiple areas may represent the color average of the multiple pixels at the location of that area in the filtered image before filtering. This is beneficial because the average of the original colors of the multiple pixels may provide a better estimate of the color of an object or portion of an object in the image on which the scattering area is located than the color of a single pixel. For example, the image may be subject to noise, and the noise will have less of an impact if the average of the colors of two or more pixels is used. The average or median of the colors of two or more pixels before filtering may be determined by a median filter applied to the multiple pixels.

[0054] As an example, each scattering area may comprise a 3x3 pixel square. One or more of the colors of each of the 3x3 pixels in each of the plurality of scattering areas may then be changed to the average or median of the original colors of the 3x3 pixels. Other examples are possible, such as a 5x5 pixel square, a circular shape of pixels, a rectangular shape of pixels, etc.

[0055] The pre-processed first image is then encoded S130 into an encoded video stream.

[0056] The variable i is then incremented i=i+1 and it is checked C134 whether i>n. If not, the method repeats for the next image i and then preprocessing the next image by filtering the next image S110. In this case, i=2 and n>1, so the method repeats by preprocessing the second image i=2 in the image sequence by applying a privacy filter.

[0057] If the check C134 indicates that i>n, this means that all n images of the image sequence have already been preprocessed and coded into a video stream, and therefore the method ends.

[0058] Images of the image sequence captured by the camera may be further preprocessed by reducing S122 the intensities of color-altered pixels in multiple scattered areas of the color-corrected filtered image. By reducing the intensities of the color-altered pixels, the color-altered pixels become less noticeable and therefore less distracting when a decoded version of the encoded video is viewed by a person for reasons other than extracting the original color information of the pixels. The color gain may be reduced, for example, from the usual 100% to 30%. The amount of reduction may be predetermined so that the amount of reduction is known at the decoder side so that the reduction can be corrected if the images are to be used to extract information about the original color of the pixels. Alternatively, metadata indicating the amount of reduction may be provided to the decoder side, for example, in a side channel.

[0059] The images of the image sequence may be further preprocessed by applying a low-pass filter to the color-corrected filtered image S124. By applying a low-pass filter to the color-corrected filtered image, the color information is "smeared" relative to neighboring pixels such that color gradients are reduced. Thus, some of the additional coding cost due to the high frequency changes in the added original color pixels may be avoided.

[0060] In some scenarios, a subarea of ​​the filtered image representing an object in each image of the image sequence may have been identified. This may have been done, for example, based on an object tracking algorithm for tracking the object in the image sequence. Information related to the identified subarea may be used to filter the image by adding opaque blocks covering the subarea in each image of the image sequence, or information related to the identified subarea may be used to remove the object from each image of the image sequence. An example of the latter is shown in FIG. 4, in which subareas 410 and 420 corresponding to two people have been identified and removed from the filtered image and replaced with a blurred representation of the background behind each of the two people. Method 100 may then further include receiving S112, for each image of the image sequence captured by the camera, one or more subareas of the filtered image corresponding to the object identified in the image. For each filtered image, multiple scattering areas may then be arranged within the received subarea of ​​the filtered image. An example of this is shown in Figure 4, where it can be seen that all of the scattering areas are illustrated as small circles 412, 414 in the left sub-area 410 and similar circles in the right sub-area 420. Identified sub-areas in the filtered image may generally correspond to objects about which information in the form of color may be noted for subsequent extraction. Thus, arranging all or at least the majority of the multiple scattering areas in sub-areas of the filtered image corresponding to objects identified in the image increases the likelihood that such information can be extracted.

[0061] Since the sub-areas corresponding to the identified objects generally vary in an undetermined manner between images of the image sequence, the pattern of the scattering areas also varies in an undetermined manner, and therefore metadata indicating the current pattern needs to be provided to the decoder side, e.g., via a side channel, in order for the decoder side to easily identify the locations of pixels whose color has been changed to represent their original color before filtering.

[0062] As can be seen in Figure 4, the scattering areas 412, 414 of the left sub-area 410 and the corresponding scattering areas of the right sub-area 420 are subsets of a plurality of scattering areas of a general pattern in the form of a low-density grid similar to the low-density grid illustrated in Figure 3b. The subsets of the plurality of scattering areas are the scattering areas within sub-area 410 and sub-area 420, respectively. Thus, the metadata provided to the decoder side may be metadata identifying the sub-areas 410, 420. If the general pattern in the form of a low-density grid is predetermined, the scattering areas within the sub-areas 410, 420 may be determined at the decoder side based on the predetermined pattern and the received metadata identifying the sub-areas 410, 420.

[0063] Although method 100 has been described with respect to preprocessing and encoding n images of an image sequence, from the first image, i.e., image 1, to the last image, i.e., image n, it should be noted that the images need not be preprocessed in the order of the image sequence, i.e., the order in which the images were captured by the camera. The images may be preprocessed in any order and then encoded into a video stream such that the video stream includes the encoded preprocessed images in an order corresponding to the order in which the images of the image sequence were captured by the camera.

[0064] Furthermore, it should be noted that the sizes of the scattering areas in Figure 3b and in Figure 4 are for illustration purposes only: the actual sizes of the scattering areas may be much smaller.

[0065] FIG. 2 is a schematic diagram of an embodiment of an image processing device 200 and an embodiment of a camera 202 including an image sensor 208 configured to capture images of an image sequence. Image sensors and image capturing are well known to those skilled in the art and will not be discussed in further detail in this disclosure. The camera 202 may be a surveillance camera. The camera 202 may be a separate unit, or the camera 202 may be incorporated into another unit, such as a helmet, glasses, etc. The camera 202 may be used to capture video in connection with monitoring an area without the specific need to identify people, vehicles, or the like in the video. However, the captured data may subsequently be used as evidence, for example, when investigating a crime and prosecuting criminal suspects. A data management system external to the camera 202, such as a video management system or evidence management system, may be used to store the captured data. Such data management systems typically provide for storage of the captured data as well as viewing of the captured data either in real time or as playback of recorded data.

[0066] The image processing device 200 includes an encoder 220 and a circuit 210 .

[0067] The encoder 220 is configured to encode images captured by the image sensor 208 of the camera 202, e.g., images of an image sequence, into a video stream. The video stream provided by the encoder 220 may be referred to as an encoded video stream.

[0068] The circuit 210 is configured to perform the functions of the image processing device 200. The circuit 210 may include a processor 212, such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 212 is configured to execute program code. The program code may, for example, be configured to perform the functions of the image processing device 200.

[0069] The image processing device 200 may further include memory 230. The memory 230 may be one or more of a buffer, flash memory, a hard drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or another suitable device. In a typical arrangement, the memory 230 may include non-volatile memory for long-term data storage and volatile memory that serves as system memory for the circuit 210. The memory 230 may exchange data with the camera circuit 210 via a data bus. Associated control lines and an address bus between the memory 230 and the circuit 210 may also be present.

[0070] The functionality of image processing device 200 may be embodied in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored in a non-transitory computer-readable medium (e.g., memory 230) of image processing device 200 and executed by circuit 210 (e.g., using processor 212). Moreover, the functionality of image processing device 200 may be a standalone software application or may form part of a software application that performs additional tasks related to image processing device 200. The described functions may be considered as methods that a processing unit, e.g., processor 212 of circuit 210, is configured to perform. Also, while the described functions may be implemented in software, such functionality may likewise be performed via dedicated hardware or firmware, or some combination of hardware, firmware, and / or software.

[0071] Circuit 210 is configured to preprocess images of the image sequence by performing filtering function 231 and color correction function 232. Filtering function 231 is configured to filter the images by applying a privacy filter, which is configured to distort the images in a manner that prevents identification of people in the filtered images. Color correction function 232 is configured to color correct the filtered images by, for at least a subset of the filtered images, changing the colors of pixels of a plurality of scattering areas of the filtered images such that the color of each of the one or more pixels in each of the plurality of scattering areas represents the original color of the one or more pixels at the location of that area in the filtered image before filtering.

[0072] The camera 202 may further include local data storage (not shown) configured to store the video stream, and / or a transmitter (not shown) configured to transmit the video stream, e.g., wirelessly, for continuous transfer of the captured video stream to a remote site.

[0073] The circuit 210 may be further configured to perform an image sequence capture function 233. The image sequence capture function 233 is configured to acquire images of an image sequence captured by the camera 202.

[0074] The circuit 210 may further be configured to perform an image identification acquisition function 234. The image identification acquisition function 234 is configured to acquire an identification of which images in the image sequence should be color corrected after being filtered.

[0075] The circuit 210 may be further configured to perform a subarea receiving function 235. The subarea receiving function 235 is configured to receive, for each image of a sequence of images captured by the camera, one or more subareas of a filtered image corresponding to an object identified in the image, and for at least a subset of the filtered images, a plurality of scattering areas are arranged within the received subarea of ​​the filtered image.

[0076] The circuit 210 may further be configured to perform a low pass filter application function 236. The low pass filter application function 236 is configured to further preprocess each image of the image sequence by applying a low pass filter to the color corrected filtered image.

[0077] The circuit 210 may further be configured to perform an intensity reduction function 237. The intensity reduction function 237 is configured to further pre-process each image of the image sequence by reducing the intensity of color-modified pixels of a plurality of scattering areas of the color-corrected filtered image.

[0078] The functions performed by the encoder 220 and the circuit 210 may further be adapted as corresponding steps of the method 100 described with respect to FIG.

[0079] Those skilled in the art will understand that the present invention is not limited to the above-described embodiments. Rather, many modifications and variations are possible within the scope of the appended claims. Such modifications and variations can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. [Explanation of symbols]

[0080] 100 ways 410 objects / subareas 420 objects / subarea 310 vehicles 320 people 330 people 412 Small circular / scattered area 414 Small circular / scattered area 200 Image Processing Devices 208 Image Sensor 202 Camera 220 Encoder 210 circuits / camera circuits 212 processors 230 memory 231 Filter Processing Function 232 Color correction function 233 Image sequence acquisition function 234 Image Identification Acquisition Function 235 Sub-area reception function 236 Low-pass filter application function 237 Strength reduction function

Claims

1. 1. A method for encoding an image captured by a camera, comprising: For each image in the sequence of images captured by the camera: filtering the image by applying a privacy filter, the privacy filter being configured to distort the image in a manner that prevents identification of people in the filtered image, and wherein the color of pixels in the image is changed from the original color of the pixels by applying the privacy filter; and color-modifying the image by, for each image of at least one subset of images filtered by applying the privacy filter, changing the color of one or more pixels in each area of ​​a plurality of scattering areas of the image to a color representing the original color of the one or more pixels before the privacy filter was applied to the image, so that only the subset of pixels that were previously changed from their original color by applying the privacy filter are changed to a color representing the original color, wherein adjacent scattering areas have a distance between them and an area between them that is not part of any of the scattering areas such that identification of a person in the filtered image is hindered even after color modification, and wherein the size of each of the scattering areas is smaller than a size that would allow identification of a person by color-modifying the image; pre-processing the image; encoding the pre-processed images into an encoded video stream; A method comprising:

2. The method of claim 1 , wherein the plurality of scattering areas are arranged in a predetermined pattern in the at least one subset of the filtered image.

3. The method of claim 2 , wherein the predetermined pattern is fixed for all filtered images of the at least one subset of filtered images.

4. The method of claim 2 , wherein the predetermined pattern varies among the filtered images of the at least one subset of the filtered images.

5. The method of claim 1 , wherein the plurality of scattering areas are arranged in a sparse grid in the at least one subset of the filtered image.

6. The method of claim 1 , wherein applying a privacy filter comprises applying a gradient operator.

7. The method of claim 1 , wherein the privacy filter is configured to produce a monochrome image.

8. The method of claim 1 , wherein the privacy filter is an edge filter.

9. 9. The method of claim 1, wherein each of the respective colors of the one or more pixels in each area of ​​the plurality of scattering areas represents an original color of a selected pixel of the one or more pixels before filtering at a location of that area in the filtered image.

10. 9. The method of claim 1, wherein at least one of the respective colors of the one or more pixels in each area of ​​a plurality of areas represents an average of the one or more original colors of the one or more pixels before filtering at the location of that area in the filtered image.

11. For each image in the sequence of images captured by the camera: receiving one or more sub-areas of the filtered image corresponding to an object identified in the image; further comprising for the at least one subset of the filtered images, the plurality of scattering areas are arranged within the received sub-area of ​​the filtered images; 11. The method according to any one of claims 1 to 10.

12. The images of the sequence of images captured by the camera are reducing the intensities of the color-modified pixels of the plurality of scattering regions of the color-corrected filtered image. The method according to any one of claims 1 to 11, further pretreated by

13. A non-transitory computer-readable storage medium storing instructions for implementing the method of any one of claims 1 to 12 when executed on a device having processing capabilities.

14. Each image in the image sequence captured by the camera is a filtering function configured to filter the image by applying a privacy filter, the privacy filter configured to distort the image in a manner that prevents identification of people in the filtered image, and wherein the color of pixels in the image is changed from their original color by applying the privacy filter; a color correction function configured to, for each image of at least one subset of images filtered by applying the privacy filter, color correct the image by changing a color of one or more pixels in each area of ​​a plurality of scattering areas of the image to a color representing an original color of the one or more pixels before the privacy filter was applied to the image, so that only the subset of pixels previously changed from their original color by applying the privacy filter are changed to a color representing the original color, wherein there is a distance between adjacent scattering areas, and there is an area between the adjacent scattering areas that is not part of any of the scattering areas such that identification of a person in the filtered image is hindered even after color correction, and the size of each of the scattering areas is smaller than a size that allows identification of a person; and a circuit configured to preprocess by performing an encoder configured to encode the pre-processed images into an encoded video stream; An image processing device comprising:

15. A camera comprising an image processing device according to claim 14.

Images