Device and method of encoding images including privacy filter
Patent Information
- Application Number
- JP2021195461
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-12-07
- Filing Date
- 2021-12-01
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-12-01
AI Technical Summary
Existing image encoding methods that apply privacy filters compromise the ability to extract information from encoded images, as they distort pixel colors, making it difficult to identify objects like people or vehicles while maintaining privacy.
Applying a privacy filter to distort images and then selectively color-correcting a subset of pixels in scattering areas to represent original colors, allowing object identification without compromising privacy.
Enables the extraction of object color information from encoded images, facilitating identification of objects like hats, jackets, bicycles, and cars, while maintaining privacy by preventing facial recognition and vehicle number identification.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0004] ,
[0001] The present invention relates to encoding an image, and more particularly, to encoding an image preprocessed by applying a privacy filter.
Background Art
[0002] When using a surveillance camera to capture an image encoded in a video stream, in some scenarios, it may be necessary or at least desirable to apply a privacy filter to the captured image before encoding in order to achieve privacy. Privacy is typically achieved by distorting the captured image such that face recognition, identification of vehicle registration numbers, etc. related to the image are hindered. For example, such a scenario could be that the surveillance camera cannot be installed unless privacy is achieved. Another scenario could be that if privacy can be achieved, permission to install the surveillance camera is not required or is granted earlier. [[ID=According to a first aspect, a method for encoding images captured by a camera is provided. In this method, for each image in an image sequence captured by the camera, the image is preprocessed by filtering the image by applying a privacy filter, the privacy filter being configured to distort the image in such a way that privacy is achieved in the filtered image; and by color correcting the filtered image, for at least one subset of the filtered image, by changing the color of pixels in a plurality of scattered areas of the filtered image such that the color of one or more pixels in each of the plurality of scattered areas represents one or more original colors of one or more pixels in that area location in the filtered image before filtering. The preprocessed image is then encoded into an encoded video stream.
[0005] By applying a privacy filter to distort an image in a way that achieves privacy, and by encoding such an image into an encoded video stream, the encoded video stream alone should not be usable to identify people, vehicles, etc. However, if the police are conducting an investigation and people, vehicles, etc. related to the investigation can be found in the encoded video stream, such identification may be desirable.
[0006] The inventors found that the seemingly contradictory desires of maintaining privacy in an encoded video stream while simultaneously enabling later identification can be eliminated or at least mitigated by introducing information about the original color of pixels into each scattered area of the image filtered by the application of a privacy filter. Information about the original color can be extracted later for each image and for each scattered area by changing the color of the pixels in each of several scattered areas of the filtered image to represent the original color of the pixels at the location of each area. Thus, across several consecutive encoded images of an encoded video stream, a moving object such as a person or vehicle may be in a position such that the area of the scattered area whose pixels have been color-corrected matches the hat, jacket, bicycle, car, etc. Thus, at least an approximate color of the hat, jacket, bicycle, car, etc. can be determined by extracting the color of one or more pixels in that area of the multiple scattered areas. Such color information alone does not enable the identification of a person, vehicle, or other. However, such color information can be used by police in investigations, for example, in which people, vehicles, etc. related to the investigation can be found in the encoded video stream. For example, extracted colors from hats, jackets, bicycles, cars, etc., can be compared with information from other sources. Such comparisons can lead to the exclusion or inclusion of individuals, vehicles, or other items in an investigation.
[0007] A privacy filter is configured to distort an image in such a way that privacy is achieved in the filtered image. This means that after the application of the privacy filter, the image is distorted in such a way that it is impossible to identify a person, such as by recognizing facial features, vehicle registration numbers, or other features that could be linked to a particular person. The term privacy filter shall encompass image processing that achieves such impediment. One example of such processing is applying a filter that removes image details that are not related to high gradients in the image, such as a Sobel filter or other filters that use gradient operators. Other examples of such processing include, for example, a substantial reduction in resolution, blurring all or selected parts of the image, and pixelation of all or selected parts of the image by covering or removing selected objects in the image.
[0008] All images in an image sequence need to be filtered to achieve privacy, but only a subset of the filtered images need to be color-corrected; for example, every other image or some other subset of the filtered images may need to be color-corrected. Of course, all filtered images may also be color-corrected.
[0009] Scattered areas, here, mean that areas are separated in an image such that there is distance between adjacent areas and between adjacent areas that are not part of any of the scattered areas. By using information that can be extracted in the form of color-modified pixels representing the original color of the pixels in the scattered areas, the risk of privacy breaches can be eliminated or at least reduced.
[0010] Multiple scattering areas may be arranged in a predetermined pattern in at least one subset of the filtered image. This simplifies the decoder's identification of scattering areas containing pixels whose color has been altered.
[0011] A given pattern is fixed for all filtered images in at least one subset of filtered images, in other words, it can be the same in all color-corrected filtered images. In this way, the location of the scattering area can be known on the decoder side without the need to provide metadata pointing to the current pattern between the encoder and decoder sides.
[0012] The given pattern may also vary, in other words, differ, between at least one subset of filtered images. In this case, the pattern should preferably vary in a predetermined manner such that the location of the scattering area can be known on the decoder side without the need to provide metadata pointing to the current pattern between the encoder and decoder sides.
[0013] Multiple scattering areas may be arranged in a low-density grid in at least one subset of the filtered images. This is beneficial because, for moving objects captured in an image sequence, a portion of the moving object whose color is noteworthy may be positioned in at least one of the color-corrected filtered images such that at least one of the scattering areas lies on the portion.
[0014] A privacy filter may involve applying a gradient operator. Such a gradient operator allows the distortion of the image to enhance edges related to high gradients and reduce details related to low gradients. Such a privacy filter allows the identification of objects in the filtered image but prevents the identification of people, such as recognizing facial features. A privacy filter may be configured to alter the original colors of an image, and in particular, to produce a monochrome image. In particular, a privacy filter may be an edge filter, such as a Sobel filter.
[0015] Each of the colors of one or more pixels in each of the multiple scattering areas may represent the original color of a selected pixel from one or more pixels before filtering at that location in the filtered image. Thus, the color of each of one or more pixels may be changed to represent the original color of one pixel before filtering. Preferably, the colors of two or more pixels in each of the multiple scattering areas may be changed to represent the original color of one pixel before filtering. This is beneficial because changing the colors of two or more pixels in each area to the original color of one pixel reduces the risk that the colors will be altered by encoding the color-corrected filtered image. This is especially true when two or more pixels are adjacent pixels, for example, k × k pixels, where k is an integer.
[0016] At least one of the colors of one or more pixels in each of the multiple areas may represent the average or median of one or more original colors of one or more pixels in that area's location in the filtered image before filtering. Preferably, the color of one or more pixels in each of the multiple areas may represent the average of the colors of two or more pixels in that area's location in the filtered image before filtering. This is beneficial because the average of the original colors of two or more pixels before filtering may provide a better estimate of the color of an object or part of an object in the image over which the scattering area lies than the color of a single pixel. For example, an image may be subject to noise, and if the average of the colors of two or more pixels is used, the noise will have less impact. The average or median of the colors of two or more pixels before filtering may be determined by a median filter.
[0017] The images in the image sequence can be further preprocessed by applying a low-pass filter to the color-corrected and filtered images. This is beneficial because the coding cost is reduced when encoding the color-corrected and filtered images.
[0018] The method may further include, for each image in an image sequence captured by a camera, receiving one or more sub-areas of a filtered image corresponding to an object identified in the image, and for at least one subset of the filtered image, a plurality of scattered areas are arranged within the received sub-areas of the filtered image. Identified objects in an image may generally correspond to objects for which color morphology information can be focused on for subsequent extraction. Therefore, arranging all or at least a majority of the plurality of scattered areas within the sub-areas of the filtered image corresponding to an object identified in the image increases the likelihood that such information can be extracted.
[0019] Images from a sequence of images captured by a camera may be further preprocessed by reducing the intensity of color-changed pixels in multiple scattered areas of the color-corrected and filtered images. By reducing the intensity of color-changed pixels, they become less conspicuous and therefore less distracting when viewing the decoded version of the encoded video. This is beneficial because reducing the intensity of color-changed pixels reduces distraction when the video is viewed for reasons other than extracting color information.
[0020] According to a second embodiment, a non-temporary computer-readable storage medium is provided that stores instructions for implementing the method according to the first embodiment when executed on a device having processing capabilities.
[0021] The features described above of the method according to the first embodiment apply similarly to this second embodiment, where applicable. References to the above are made to avoid excessive repetition.
[0022] According to a third aspect, an image processing device is provided. The image processing device comprises circuitry configured to preprocess each image in an image sequence captured by a camera by performing filtering and color correction functions. The filtering function is configured to filter an image by applying a privacy filter, which is configured to distort the image in such a way that the identification of people in the filtered image is hindered. The color correction function is configured to color correct a filtered image by changing the color of pixels in a plurality of scattered areas of the filtered image for at least one subset of the filtered image such that the color of one or more pixels in each of the plurality of scattered areas represents the original color of one or more pixels in that area of the filtered image before filtering. The image processing device further comprises an encoder configured to encode the preprocessed images into an encoded video stream.
[0023] The features described above of the method according to the first embodiment also apply to this third embodiment, where applicable. References to the above are made to avoid excessive repetition.
[0024] According to a fourth aspect, a camera is provided that includes an image processing device according to a third aspect.
[0025] Further scope of the applicability of the present invention will become apparent from the following embodiments for carrying out the invention. However, it should be understood that while the embodiments for carrying out the invention and specific examples point to preferred embodiments of the present invention, various modifications and variations within the scope of the invention will become apparent to those skilled in the art from these embodiments, and are therefore given only as examples.
[0026] Therefore, it should be understood that the present invention is not limited to the specific component parts of the described device or the acts of the method described as such a device, and the method may vary. It should also be understood that the terms used herein are for the purpose of describing particular embodiments only and are not limiting. It should be noted that, as used in this specification and the appended claims, the articles "a", "an", "the", and "said" mean one or more elements, unless the context clearly dictates otherwise. Thus, for example, reference to "a unit" or "the unit" may include several devices and the like. Moreover, the words "comprising", "including", "containing" and similar expressions do not exclude other elements or steps.
[0027] Next, the above and other aspects of the present invention will be described in more detail with reference to the accompanying drawings. The drawings should not be regarded as limiting, but rather are used for illustration and understanding.
Brief Description of the Drawings
[0029] Next, the present invention will be described below with reference to the accompanying drawings illustrating current preferred embodiments of the invention. However, the present invention can be embodied in many different forms and should not be construed as being limited to the embodiments described herein.
[0030] The present invention is applicable in scenarios where privacy is desired or required to be achieved with respect to an encoded video stream, in other words, where the identification of people, vehicles, or other entities in the image should not be possible from the encoded video stream on its own. Such privacy can be achieved by image distortion, for example, by the application of a privacy filter.
[0031] Next, embodiments of method 100 for encoding images captured by a camera will be described with reference to Figure 1, and further with reference to Figures 3a, 3b and 4. An image sequence consisting of n images captured by a camera is acquired for preprocessing and encoding into an encoded video stream S102. The camera may be, for example, a surveillance camera or monitoring camera used to monitor an area. Each image i, i=1→n, is then to be preprocessed and encoded into a video stream. The first image of the image sequence is preprocessed by filtering the first image by applying a privacy filter S110, where the variable i is set to 1 S106. The privacy filter is configured to distort the image in such a way that privacy is achieved in the filtered image.
[0032] Privacy is achieved when an image is distorted in a way that prevents the identification of a person, such as recognizing facial features, vehicle registration numbers, or other features that could be linked to a specific person. Such distortion can be achieved by applying a filter that removes image details that are not related to high gradients in the image, such as an edge filter using a gradient operator (e.g., a Sobel filter) or any other filter. Such an edge filter is a kernel-based filter that gives a strong response to rising and falling gradients in all directions. The absolute value of such a gradient operator, along with a gain coefficient, can be used to produce a filtered image that includes all edges but omits or reduces all other details of the image. While it is possible to edge filter the color of an image, generally, luminance-based edge filtering is sufficient to provide a filtered image in which objects can be identified at a general level, but the identification of a specific person is prevented. Preprocessing an image by applying a luminance-based edge filter produces a monochrome image. References to Figures 3a and 3b are given for an example of a filtered image after applying a Sobel filter. For the images in Figures 3a and 3b, a negative gain coefficient was used to produce filtered images where edges are darker and areas without edges are brighter.
[0033] Other examples of processes that achieve privacy include, for example, a substantial reduction in resolution, blurring all or selected parts of an image, and pixelation of all or selected parts of an image by covering or removing selected objects in the image. An example of an image after processing to remove selected objects in an image is referenced to Figure 4. In Figure 4, two objects 410 and 420, corresponding to people, have been removed from the image and replaced with a blurred representation of the background. For clarity of the illustration, the outlines of the two objects 410 and 420 are identified by solid lines in Figure 4.
[0034] A common feature of identified privacy-achieving processing techniques is that the color of pixels in an image is altered by a method in which the color of the processed pixel may differ significantly from the original color of the pixel before processing. Furthermore, it is not possible to identify the original color of the pixel before processing from the processed pixel.
[0035] Next, C114 determines whether the filtered first image should be color corrected.
[0036] Whether the images in an image sequence should be color corrected after initial filtering can be predetermined, for example, so that all images are color corrected, or so that only a predetermined subset or some other predetermined subset is color corrected, such as every other image or every two images. In alternative identification, the identification of which images in an image sequence should be color corrected after filtering can be obtained for the image sequence such that this can vary over time, for example, so that a larger portion of the image sequence or even all images are color corrected at one time, and a smaller portion of the images in the image sequence are color corrected at another time S104. This can be determined, for example, based on an identified amount of movement.
[0037] If only a subset of filtered images should be color-corrected, the decoder needs to determine which of the filtered images have been color-corrected to avoid attempting to identify color-corrected pixels in the uncolor-corrected image sequence by any means. The subset is preferably predetermined, such as by alternating images, by every other image, or by any other given pattern starting from a given point in time. In this case, the pattern can be synchronized with the picture group (GOP) structure of the encoded video stream so that the decoder knows which of the filtered images have been color-corrected. If the pattern is not predetermined, metadata indicating whether the image has been color-corrected should be provided to the decoder for each image, preferably, for example, in a side channel.
[0038] If the filtered first image does not need color correction, the first image is encoded into an encoded video stream.
[0039] If the filtered first image is to be color corrected, the first image is further preprocessed by color correcting the filtered first image S120, which modifies the colors of pixels in multiple scattered areas of the filtered image by changing the color of one or more pixels in each of the multiple scattered areas to represent one or more original colors of one or more pixels in that area's location in the filtered image before filtering.
[0040] The color of one or more pixels in each of several scattered areas, representing the original color of one or more pixels before filtering at the location of an area in the filtered image, can later be extracted from the decoded image of the encoded video stream. Thus, across several consecutive encoded images of the encoded video stream, a moving object, such as a person or vehicle, may be located such that at least one of the scattered areas whose pixels have been color-corrected is positioned on the hat, jacket, bicycle, car, etc. Thus, at least an approximate color of the hat, jacket, bicycle, car, etc., can be determined by extracting the color of one or more pixels in that area of the multiple scattered areas. Such color information alone does not enable the identification of a person, vehicle, or other. However, such color information can be used by the police in an investigation, for example, in which a person, vehicle, etc., related to the investigation can be found in the encoded video stream. For example, the extracted color of the hat, jacket, bicycle, car, etc., can be compared with information from other sources. From such a comparison, a person, vehicle, or other may be excluded or included in the investigation.
[0041] Scattered areas are separated in the image such that there is distance between adjacent areas of the scattering area; in other words, there are areas between adjacent areas that are not part of any of the scattering areas.
[0042] Multiple scattering areas may be arranged in a predetermined pattern in a color-corrected and filtered image. This simplifies the identification of scattering areas on the decoder side, as the predetermined pattern may also be known on the decoder side. In particular, scattering areas may be of low density and relate to only one or a few pixels. If the location of each area of the scattering area is unknown on the decoder side, where the encoded video stream is decoded and analyzed for extracting the original color of the pixels, identifying the pixels whose color has been altered about it may be possible, but difficult and time-consuming.
[0043] The given pattern is fixed, or in other words, it may be the same for all color-corrected and filtered images. In this way, the location of the scattering area can be known on the decoder side without the need to provide metadata pointing to the current pattern between the encoder and decoder sides.
[0044] Alternatively, a given pattern may vary among filtered images of at least one subset of filtered images. If the pattern varies, it should preferably vary in a predetermined manner such that the location of the scattering area can be known at the decoder side without the need to provide metadata pointing to the current pattern between the encoder and decoder sides. If the pattern varies in an undetermined manner, metadata pointing to the current pattern must be provided to the decoder side.
[0045] Instead of arranging the scattering areas in a predetermined pattern, metadata pointing to a pattern can be provided to the decoder for each color-corrected filtered image. Such metadata may be provided, for example, in the side channels.
[0046] The multiple scattering areas are preferably arranged to be evenly spread throughout each image of the image sequence such that, for an object placed at any location in the images of the image sequence, at least one of the scattering areas may be positioned over some part of that object. Furthermore, if the object is moving, each part of the moving object in at least one image of the image sequence, whose color could be noticed, may be positioned such that at least one of the scattering areas is positioned over that part.
[0047] Multiple scattering areas may be arranged in a low-density grid in at least one subset of the filtered image, as illustrated in Figure 3b, which is, for example, an enlarged portion of the filtered image in Figure 3a having a scattered area with color-corrected pixels. In Figure 3b, the scattered areas with color-corrected pixels are in the form of circular areas arranged such that each has eight columns, each containing five rows. The color correction is indicated by different shading of the circular areas. As can be seen in Figure 3b, the scattered areas are close enough that several scattered areas are located on the vehicle 310. Furthermore, even if the scattered areas are not close enough so that at least one of the scattered areas is located on each part of the person in the image, such as a hat, helmet, jacket, and trousers, as the person 320, 330 moves between images in the image sequence, at least one of the scattered areas may be located on one of these parts in one of the images.
[0048] Alternatively, multiple scattering areas may be arranged in a pseudo-random pattern. The pseudo-random pattern may preferably be predetermined and fixed or may vary in a predetermined manner between images in an image sequence.
[0049] Multiple scattering areas may also be arranged only within selected portions of each image in an image sequence. For example, the selected portion may relate to a part of the scene captured in the image sequence that is of interest, such as a road or a building entrance. The selected portion may also relate to an area of the scene captured in the image sequence for which additional information, such as information about the original color of a pixel, is made available in the encoded video stream. The selected portion may relate to a restricted area of the scene, for example, an area inside the premises of a person or group where the camera capturing the image sequence is set up, while the other portion relates to an area outside those premises for which information about the original color of a pixel is not made available in the encoded video stream.
[0050] Each area of the scattering area is preferably small relative to the image, for example, a 1-pixel, 3x3 pixel, or 5x5 pixel square. It may also have a rectangular, circular, or other shape. While the size may be larger, it is beneficial that each area is small relative to the image so that the areas do not distract to anyone viewing the video stream for reasons other than extracting the original color information. Furthermore, since the color of the pixels in the scattering area is changed to represent the original color of the pixels at the scattering area's location, the size of each scattering area should not be large enough to compromise privacy.
[0051] Each of the colors of one or more pixels in each of the multiple scattering areas may represent the original color of one of the one or more pixels in that area's location in the filtered image before filtering. For example, if each of the multiple scattering areas comprises multiple pixels, each of the multiple pixels in each area may represent or be modified to represent the original color of one of the pixels in that area before filtering. Preferably, the pixel is predetermined so that it can be known to the decoder without the need to provide metadata to the decoder, for example in a side channel, indicating which pixels in the image had the original color. Changing the colors of multiple pixels in each area to the original color of a selected pixel is beneficial because it reduces color distortion caused by encoding the color-corrected filtered image.
[0052] For example, each area of a scattering area may consist of a 3x3 pixel square. Each of the colors of each 3x3 pixel in each area of multiple scattering areas can then be changed to the original color of one of the pixels in that 3x3 square. One of the pixels could be, for example, the center pixel. Other examples are possible, such as a 5x5 pixel square, a circular shape of pixels, or a rectangular shape of pixels.
[0053] Furthermore, at least one of the colors of one or more pixels in each area of a group of areas may represent the average or median of one or more original colors of one or more pixels in that area's location in the filtered image before filtering. For example, if each area of a scattering area has multiple pixels, then one or more of the colors of the pixels in each area of the group of areas may represent the average of the colors of those pixels in that area's location in the filtered image before filtering. This is beneficial because the average of the original colors of multiple pixels may provide a better estimate of the color of an object or part of an object in the image over which the scattering area is located than the color of a single pixel. For example, an image may be subject to noise, and if the average of the colors of two or more pixels is used, the noise will have less impact. The average or median of the colors of two or more pixels before filtering may be determined by a median filter applied to the multiple pixels.
[0054] For example, each area of a scattering area may consist of a 3x3 pixel square. One or more of the colors of each 3x3 pixel in each area of multiple scattering areas may, in that case, be changed to the average or median of the original colors of the 3x3 pixels. Other examples are possible, such as a 5x5 pixel square, a circular shape of pixels, or a rectangular shape of pixels.
[0055] Next, the pre-processed first image is encoded into an encoded video stream S130.
[0056] Next, the variable i is incremented to i=i+1, and it is checked in C134 whether i>n. If not, the method is repeated, starting with the next image i, by preprocessing the next image by filtering the next image in S110. In this case, i=2 and n>1, and therefore the method is repeated by preprocessing the second image i=2 of the image sequence by applying a privacy filter.
[0057] If check C134 indicates that i > n, this means that all n images in the image sequence have already been preprocessed and encoded into the video stream, and therefore the method is complete.
[0058] The images of the image sequence captured by the camera may be further preprocessed by S122, which reduces the intensity of color-changed pixels in multiple scattered areas of the color-corrected and filtered image. By reducing the intensity of color-changed pixels, the color-changed pixels become less conspicuous and therefore less distracting when the decoded version of the encoded video is viewed by a person for reasons other than extracting the original color information of the pixels. The color gain may be reduced, for example, from the normal 100% to 30%. The amount of reduction may be predetermined so that the amount of reduction is known to the decoder side, so that the reduction is compensated for if the image is to be used to extract information about the original color of the pixels. Alternatively, metadata indicating the amount of reduction may be provided to the decoder side, for example, in a side channel.
[0059] The images in the image sequence may be further preprocessed by applying a low-pass filter to the color-corrected and filtered images (S124). By applying a low-pass filter to the color-corrected and filtered images, the color information is "blurred" relative to adjacent pixels so that the color gradient is reduced. Thus, some of the additional coding cost due to the high-frequency changes caused by the added pixels of the original color can be avoided.
[0060] In some scenarios, a sub-area of a filtered image representing an object in each image of an image sequence may be identified. This may be done, for example, based on an object tracking algorithm for tracking an object in an image sequence. Information relating to the identified sub-area may be used to filter the image by adding an opaque block covering the sub-area in each image of the image sequence, or information relating to the identified sub-area may be used to remove the object from each image of the image sequence. An example of the latter is shown in Figure 4, in which sub-areas 410, 420 corresponding to two people are identified and removed from the filtered image and replaced with a blurred representation of the background behind each of the two people. Method 100 may then further include S112 receiving one or more sub-areas of a filtered image corresponding to an object identified in the image for each image of an image sequence captured by the camera. For each filtered image, a plurality of scattered areas may then be arranged within the received sub-areas of the filtered image. One example of this is shown in Figure 4, where all the scattering areas are represented as small circles 412 and 414 within the left sub-area 410, and as similar circles within the right sub-area 420. Identified sub-areas in the filtered image may generally correspond to objects for which color morphology information can be focused for subsequent extraction. Therefore, arranging all or at least most of multiple scattering areas within filtered image sub-areas corresponding to objects identified in the image increases the likelihood that such information can be extracted.
[0061] Since the sub-areas corresponding to identified objects generally vary between images in an image sequence in an unpredictable manner, the scattering area pattern also varies in an unpredictable manner. Therefore, metadata pointing to the current pattern needs to be provided to the decoder, for example, via a side channel, so that the decoder can easily identify the locations of pixels whose color has been changed to represent the original color before filtering.
[0062] As can be seen in Figure 4, the scattering areas 412 and 414 in the left sub-area 410, and the corresponding scattering areas in the right sub-area 420, are subsets of multiple scattering areas of a general pattern of low-density grid morphology similar to the low-density grid illustrated in Figure 3b. These subsets of multiple scattering areas are the scattering areas located within sub-area 410 and sub-area 420, respectively. Therefore, the metadata provided to the decoder side may be metadata that identifies sub-areas 410 and 420. If the general pattern of low-density grid morphology is predetermined, the scattering areas within sub-areas 410 and 420 can be determined by the decoder side based on the predetermined pattern and the received metadata that identifies sub-areas 410 and 420.
[0063] Method 100 has described preprocessing and encoding n images in an image sequence from the first image, in other words, from image 1 to the last image, in other words, image n. However, it should be noted that the images do not need to be preprocessed in the order of the image sequence, in other words, in the order in which the images were captured by the camera. The images can be preprocessed in any order, and then encoded into a video stream such that the video stream contains the preprocessed images encoded in the order corresponding to the order in which the images in the image sequence were captured by the camera.
[0064] Furthermore, please note that the sizes of the scattering areas in Figure 3b and Figure 4 are for illustrative purposes only. The actual size of the scattering areas may be considerably smaller.
[0065] Figure 2 is a schematic diagram relating to an embodiment of an image processing device 200 and an embodiment of a camera 202 comprising an image sensor 208 configured to capture images of an image sequence. Image sensors and image capturing are well known to those skilled in the art and will not be discussed in further detail in this disclosure. The camera 202 may be a surveillance camera. The camera 202 may be a separate unit or it may be incorporated into another unit, such as a helmet, glasses, etc. The camera 202 may be used to capture video in relation to monitoring an area without the special need to identify people, vehicles, or others in the video. However, the captured data may then be used as evidence, for example, when investigating a crime and prosecuting a suspect in the crime. To store the captured data, an external data management system, such as a video management system or evidence management system, may be used for the camera 202. Such a data management system generally provides storage of the captured data and also provides viewing of the captured data, either in real time or as playback of recorded data.
[0066] The image processing device 200 includes an encoder 220 and a circuit 210.
[0067] The encoder 220 is configured to encode images captured by the image sensor 208 of the camera 202, for example, images of an image sequence, into a video stream. The video stream provided by the encoder 220 may be called the encoded video stream.
[0068] Circuit 210 is configured to perform the functions of the image processing device 200. Circuit 210 may include a processor 212, such as a central processing unit (CPU), microcontroller, or microprocessor. The processor 212 is configured to execute program code. The program code may be configured, for example, to perform the functions of the image processing device 200.
[0069] The image processing device 200 may further include a memory 230. The memory 230 may be one or more of a buffer, flash memory, hard drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or other suitable devices. In a typical configuration, the memory 230 may include non-volatile memory for long-term data storage and volatile memory serving as system memory for the circuit 210. The memory 230 may exchange data with the camera circuit 210 via a data bus. Accompanying control lines and an address bus between the memory 230 and the circuit 210 may also be present.
[0070] The functions of the image processing device 200 may be provided in the form of executable logical routines (e.g., lines of code, software programs, etc.) that are stored in a non-temporary computer-readable medium of the image processing device 200 (e.g., memory 230) and executed by the circuit 210 (e.g., using the processor 212). Furthermore, the functions of the image processing device 200 may be a standalone software application or form part of a software application that performs additional tasks related to the image processing device 200. The functions described may be considered as being performed in a manner configured for a processing unit, for example, the processor 212 of the circuit 210. Also, while the functions described may be implemented in software, such functionality may also be performed through dedicated hardware or firmware, or any combination of hardware, firmware and / or software.
[0071] The circuit 210 is configured to preprocess images in an image sequence by performing a filtering function 231 and a color correction function 232. The filtering function 231 is configured to filter the images by applying a privacy filter, which is configured to distort the images in such a way that the identification of people in the filtered images is hindered. The color correction function 232 is configured to color correct the filtered images by changing the color of pixels in multiple scattered areas of the filtered images for at least one subset of the filtered images, such that the color of one or more pixels in each of the multiple scattered areas represents the original color of one or more pixels in that area's location in the filtered image before filtering.
[0072] The camera 202 may further include local data storage (not shown) configured to store the video stream, and / or a transmitter (not shown) configured to transmit the video stream, for example, wirelessly, to continuously transfer the captured video stream to a remote site.
[0073] The circuit 210 may be further configured to perform an image sequence acquisition function 233. The image sequence acquisition function 233 is configured to acquire images of an image sequence captured by the camera 202.
[0074] The circuit 210 may be further configured to perform an image recognition acquisition function 234. The image recognition acquisition function 234 is configured to acquire identification of which images in the image sequence should be color corrected after filtering.
[0075] The circuit 210 may be further configured to perform a sub-area receiving function 235. The sub-area receiving function 235 is configured to receive, for each image in an image sequence captured by the camera, one or more sub-areas of a filtered image corresponding to an object identified in the image, and for at least one subset of the filtered image, a plurality of scattered areas are arranged within the received sub-areas of the filtered image.
[0076] The circuit 210 may be further configured to perform a low-pass filter application function 236. The low-pass filter application function 236 is configured to further preprocess each image in the image sequence by applying a low-pass filter to the color-corrected filtered image.
[0077] The circuit 210 may be further configured to perform an intensity reduction function 237. The intensity reduction function 237 is configured to further preprocess each image in the image sequence by reducing the intensity of color-changed pixels in multiple scattered areas of the color-corrected filtered image.
[0078] The functions performed by the encoder 220 and circuit 210 can be further adapted as corresponding steps of method 100 described with respect to Figure 1.
[0079] Those skilled in the art will understand that the present invention is not limited to the embodiments described above. Rather, many modifications and variations are possible within the scope of the appended claims. Such modifications and variations can be understood and performed by those skilled in the art in practicing the claimed invention by examining the drawings, this disclosure, and the appended claims. [Explanation of Symbols]
[0080] 100 ways 410 Objects / Sub-areas 420 Objects / Sub-areas 310 vehicles 320 people 330 people 412 Small circular / scattered area 414 Small circular / scattered area 200 Image Processing Devices 208 Image Sensors 202 Camera 220 encoders 210 Circuits / Camera Circuits 212 processors 230 memory 231 Filtering Function 232 Color correction function 233 Image Sequence Acquisition Function 234 Image Recognition Acquisition Function 235 Sub-area reception function 236 Low-pass filter application function 237 Strength reduction function
Claims
1. 1. A method for encoding an image captured by a camera, comprising the steps of: For each image in the sequence of images captured by the camera, filtering the image by applying a privacy filter, the privacy filter being configured to distort the image in a manner that prevents identification of people in the filtered image, and wherein a color of a pixel in the image is changed from the original color of the pixel by applying the privacy filter; and for each image of at least a subset of images filtered by applying the privacy filter, color correcting the images by changing a color of one or more pixels in each area of a plurality of scattering areas of the image to a color that represents an original color of the one or more pixels before the privacy filter was applied to the image. pre-processing the image; encoding the pre-processed images into an encoded video stream; A method comprising:
2. The method of claim 1 , wherein the plurality of scattering areas are arranged in a predetermined pattern in the at least one subset of the filtered image.
3. The method of claim 2 , wherein the predetermined pattern is fixed for all filtered images of the at least one subset of filtered images.
4. The method of claim 2 , wherein the predetermined pattern varies among the filtered images of the at least one subset of the filtered images.
5. The method of claim 1 , wherein the plurality of scattering areas are arranged in a sparse grid in the at least one subset of the filtered image.
6. The method of claim 1 , wherein applying a privacy filter comprises applying a gradient operator.
7. The method of claim 1 , wherein the privacy filter is configured to generate a monochrome image.
8. The method of claim 1 , wherein the privacy filter is an edge filter.
9. 9. The method of claim 1, wherein each of the respective colors of the one or more pixels in each of the plurality of scattering areas represents an original color of a selected pixel of the one or more pixels before filtering at a location of that area in the filtered image.
10. 9. The method of claim 1, wherein at least one of the respective colors of the one or more pixels in each of a plurality of areas represents an average of the one or more original colors of the one or more pixels prior to filtering at that area's location in the filtered image.
11. For each image in the sequence of images captured by the camera, receiving one or more sub-areas of the filtered image corresponding to an object identified in the image; Further comprising: for the at least one subset of the filtered images, the plurality of scattering areas are arranged within the received sub-area of the filtered images; 11. The method according to any one of claims 1 to 10.
12. The images of the sequence of images captured by the camera are reducing the intensities of the color-modified pixels of the plurality of scattering areas of the color-corrected filtered image. The method according to claim 1 , further comprising pretreating the surface of the substrate by a method according to claim 1 .
13. A non-transitory computer readable storage medium storing instructions for implementing the method of any one of claims 1 to 12 when executed on a device having processing capabilities.
14. For each image in the image sequence captured by the camera, a filtering function configured to filter the image by applying a privacy filter, the privacy filter configured to distort the image in a manner that prevents identification of people in the filtered image, and wherein a color of a pixel in the image is changed from its original color by applying the privacy filter; a color correction function configured to, for each image of at least a subset of images filtered by applying the privacy filter, color correct the image by changing a color of one or more pixels in each area of a plurality of scattering areas of the image to a color that represents an original color of the one or more pixels before the privacy filter was applied to the image; and a circuit configured to preprocess by performing an encoder configured to encode the pre-processed images into an encoded video stream; An image processing device comprising:
15. A camera comprising an image processing device according to claim 14.
