Method and device for obscuring objects

Abstract

The invention provides a method and a device for shielding an object. An input image is reduced to an object detection image having a resolution lower than a resolution of the input image and lower than a resolution of an output image. The object detection image is input to an object detection module, and a confidence score for a pixel or pixel region of the object detection image is received from the object detection module. Each confidence score indicates a respective probability that a pixel or pixel region correlates with an object of a category to be masked. On the basis of an input image, an intermediate image having a higher resolution than an object detection image resolution is generated, and an adaptive mask threshold value is set such that the greater the ratio between the output image resolution and the object detection image resolution is, the lower the mask threshold value is. An output image is then generated by obscuring pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the adaptive obscuring threshold.

Description

Technical Field

[0001] The present invention relates to image processing and, in particular, to generating an output image with occlusions of objects of a class that need to be occluded. Background Art

[0002] In video applications, it may be desirable to be able to detect objects of selected categories so that they can be anonymized in an image or video. For example, if an operator is viewing a video stream from a surveillance camera, people, license plates, or other objects should be detected and masked in the video image frames of the video stream so that the operator cannot identify them when viewing the video stream. By developing an object detection module (e.g., using a neural network), it is now possible to automatically detect and mask objects of selected categories of objects in the video image frames of a video in real time. However, since it is very important to maintain high accuracy in object detection to avoid objects that should be masked but are not, there are challenges in using automatic object detection because such automatic object detection may result in reduced accuracy in some cases. Summary of the invention

[0003] An object of the present invention is to overcome or at least alleviate the problems and disadvantages of the prior art.

[0004] According to a first aspect, a method for generating an output image having a masking of an object of a category to be masked is provided. The method includes reducing an input image to an object detection image having a resolution lower than the resolution of the input image and lower than the resolution of the output image, and inputting the object detection image to an object detection module. The method further includes receiving confidence scores of pixels or pixel regions of the object detection image from the object detection module. Each confidence score indicates a corresponding probability that the pixel or pixel region is associated with an object of a category to be masked. The method further includes generating an intermediate image based on the input image. The intermediate image has a resolution higher than the resolution of the object detection image. The method further includes setting an adaptive masking threshold so that the greater the ratio between the output image resolution and the object detection image resolution, the lower the adaptive masking threshold, and generating an output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the adaptive masking threshold.

[0005] By setting an adaptive masking threshold based on a ratio between the output image resolution and the object detection image resolution, the masking threshold can be adjusted relative to the increased risk that the object detection module does not detect an object of a class that should be masked in the object detection image. At the same time, if the object is not masked due to a higher resolution in the output image than in the object detection image, the operator is able to identify the object in the output image. Specifically, by setting the adaptive masking threshold to a lower value when the ratio between the output image resolution and the object detection image resolution is larger, the adaptive masking threshold is set at an increasing ratio based on the increased risk, so that objects with lower confidence scores are masked. The ability of the object detection module to detect objects of a class that should be masked in the image depends on the size of the object in the image used for object detection and the resolution of the image. Similarly, the ability of the operator to identify objects in the image also depends on the size of the object in the image displayed to the operator and the resolution of the image.

[0006] Downscaling means any way of reducing the number of pixels in an image, and is not intended to encompass cropping.

[0007] The intermediate image may be generated by reducing the input image to an intermediate image, wherein the resolution of the intermediate image is lower than the input image resolution. Alternatively, the intermediate image may be generated by using the input image as an intermediate image, wherein the intermediate image resolution is equal to the input image resolution.

[0008] The method may further include setting a fixed masking threshold that is independent of the ratio between the output image resolution and the object detection image resolution. For pixels or pixel regions corresponding to objects closer than a preset distance, the output image is then generated by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score above the adaptive masking threshold. For pixels or pixel regions corresponding to objects at or further than a preset distance, the output image is generated by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score above the fixed masking threshold.

[0009] It should be noted that the "fixed" in the fixed occlusion threshold is intended to indicate that it is not adjusted with respect to the ratio between the output image resolution and the object detection image resolution. Adjustment of the fixed occlusion threshold with respect to other parameters is not excluded.

[0010] This enables the use of a fixed masking threshold for objects at or beyond a preset distance, for example, the masking threshold that would have been used if the masking threshold had not been adjusted relative to the ratio. The preset distance may, for example, be a distance at which an operator would (or at least be assumed to) be unable to identify an object. For example, at a given resolution of an image from a camera, an operator would not be able to identify a person at a certain distance from the camera because the person would be too small in the image, making the number of pixels on the face too small to enable the operator to recognize the features of the face. Using a fixed masking threshold for such a distance is beneficial because the adaptive masking threshold will reduce the masking threshold relative to the masking threshold that would have been used if the masking threshold had not been adjusted relative to the ratio, and thus will typically mask many objects that do not belong to the category that should be masked. Thus, unnecessary masking will be reduced. Furthermore, if the preset distance is set such that an operator would (or at least be assumed to) be unable to identify objects at a distance equal to or beyond the preset distance, then it does not matter if such objects are not masked, even if they belong to the category that should be masked. Alternatively, masking may even be avoided altogether for objects at a distance equal to or beyond the preset distance.

[0011] In an embodiment where the method further includes setting a fixed masking threshold that is independent of the ratio between the output image resolution and the object detection image resolution, two preset distances may be set, namely a preset distance and another preset distance, wherein the other preset distance is closer to the preset distance. For pixels or pixel regions corresponding to objects that are closer to the preset distance and farther than the other preset distance, the output image is generated by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the adaptive masking threshold. For pixels or pixel regions corresponding to objects that are equal to or farther than the preset distance, or equal to or close to the other preset distance, the output image is generated by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the fixed masking threshold.

[0012] This enables the use of a fixed masking threshold for objects at or beyond a preset distance, for example, the masking threshold that would have been used if the masking threshold had not been adjusted relative to the ratio. The preset distance may, for example, be a distance at which an operator would (or at least assume) be unable to identify an object. For example, at a given resolution of the output image, an operator would not be able to identify a person at a certain distance from a camera that captured an image on which the output image is based, because the person would be too small in the image, making the number of pixels on the person's face too small to enable the operator to identify the features of the person's face. Using a fixed masking threshold for such a distance is beneficial because the adaptive masking threshold will reduce the masking threshold relative to the masking threshold that would have been used if the masking threshold had not been adjusted relative to the ratio, and thus will typically mask many objects that do not belong to the category that should be masked. Thus, unnecessary masking can be reduced. Furthermore, if the preset distance is set so that an operator would (or at least assume) be unable to identify objects at a distance equal to or beyond the preset distance, then it does not matter if such objects are not masked, even if they belong to the category that should be masked. Alternatively, for objects at a distance equal to or greater than a preset distance, occlusion may even be avoided completely.

[0013] In addition, this enables the use of a fixed masking threshold for objects that are equal to or close to another preset distance, for example, the masking threshold that would have been used if the masking threshold was not adjusted relative to the ratio. The other preset distance may, for example, be a distance at which the object detection module will (or at least is assumed to be) able to detect the object. For example, at a given resolution of the object detection image, the object detection module will be able to detect a person at a certain distance from the camera that captured the image on which the object detection image is based, because the person will be large enough in the image so that the number of pixels on the person's face is large enough to enable the object detection module to recognize that the object is a person that should be masked. Using a fixed masking threshold for such a distance is beneficial because the adaptive masking threshold will reduce the masking threshold relative to the masking threshold that would have been used if the masking threshold was not adjusted relative to the ratio, and therefore will typically mask many objects that do not belong to the category that should be masked. Therefore, unnecessary masking can be reduced.

[0014] In an embodiment where the method further includes setting a fixed masking threshold that is independent of the ratio between the output image resolution and the object detection image resolution, a preset pixel density may be set. For pixels or pixel regions corresponding to objects in the intermediate image having a pixel density higher than the preset pixel density, the image is output by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the adaptive masking threshold. For pixels or pixel regions corresponding to objects in the intermediate image having a pixel density equal to or lower than the preset pixel density, the output image is generated by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the fixed masking threshold.

[0015] Here, pixel density is defined as the number of horizontal pixels across the width of an object or the number of pixels per unit length (e.g., centimeters), where the length is the actual length of the object in real life. In the latter case, it is necessary to determine the actual size of the object. This can be determined, for example, by a preliminary classification of the object.

[0016] This enables the use of a fixed masking threshold for objects having a pixel density equal to or lower than a preset pixel density, for example, the masking threshold that would have been used if the masking threshold was not adjusted relative to the ratio. The preset pixel density may, for example, be a pixel density at which an operator would (or at least be assumed to) be unable to identify the object. For example, if the pixel density is so small that the operator cannot recognize the features of a face, the operator will not be able to identify the person. Using a fixed masking threshold for such distances is beneficial because the adaptive masking threshold will reduce the masking threshold relative to the masking threshold that would have been used if the masking threshold was not adjusted relative to the ratio, and thus will typically mask many objects that do not belong to the category that should be masked. Therefore, unnecessary masking will be reduced. Furthermore, if the preset pixel density is set so that an operator will (or at least be assumed to) be unable to identify objects having a pixel density equal to or lower than the preset pixel density, then it does not matter if such objects are not masked, even if they belong to the category that should be masked. Alternatively, masking may even be avoided completely for objects having a pixel density equal to or lower than the preset pixel density.

[0017] In addition, in an embodiment where the method further includes setting a fixed masking threshold value that is independent of the ratio between the output image resolution and the object detection image resolution, two preset pixel densities may be set, namely a preset pixel density and another preset pixel density, wherein the other preset pixel density is higher than the preset pixel density. For pixels or pixel regions corresponding to an object having a pixel density higher than the preset pixel density and lower than another preset pixel density in the object detection image, an output image is generated by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the adaptive masking threshold. For pixels or pixel regions corresponding to an object having a pixel density equal to or lower than the preset pixel density, or equal to or higher than another preset pixel density in the object detection image, an output image is generated by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the fixed masking threshold.

[0018] This enables the use of a fixed masking threshold for objects having a pixel density equal to or lower than a preset pixel density, for example, the masking threshold that would have been used if the masking threshold was not adjusted relative to the ratio. The preset pixel density may, for example, be a pixel density at which an operator would (or at least be assumed to) be unable to identify the object. For example, if the pixel density is so small that an operator cannot identify features of a face, the operator will not be able to identify the person. Using a fixed masking threshold for such distances is beneficial because the adaptive masking threshold will reduce the masking threshold relative to the masking threshold that would have been used if the masking threshold was not adjusted relative to the ratio, and thus will typically mask many objects that do not belong to the category that should be masked. Therefore, unnecessary masking will be reduced. Furthermore, if the preset pixel density is set so that an operator will (or at least be assumed to) be unable to identify objects having a pixel density equal to or lower than the preset pixel density, then it does not matter if such objects are not masked, even if they belong to the category that should be masked. Alternatively, masking may even be avoided entirely for objects having a pixel density equal to or lower than the preset pixel density.

[0019] In addition, this enables the use of a fixed masking threshold for objects having a pixel density equal to or higher than another preset pixel density, for example, the masking threshold that would have been used if the masking threshold was not adjusted relative to the ratio. The other preset pixel density may, for example, be a pixel density at which the object detection module will (or at least is assumed to be) able to detect the object. For example, at a pixel density equal to or higher than another preset pixel density, the number of pixels on a face will be large enough for the object detection module to recognize that the object is a person that should be masked. Using a fixed masking threshold for such distances is beneficial because the adaptive masking threshold will reduce the masking threshold relative to the masking threshold that would have been used if the masking threshold was not adjusted relative to the ratio, and therefore will typically mask many objects that do not belong to the category that should be masked. Therefore, unnecessary masking can be reduced.

[0020] The method may further include downscaling the input image to an intermediate image having a resolution lower than the resolution of the input image. Generating the output image then includes generating the output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score above an adaptive masking threshold.

[0021] According to a second aspect, there is provided a non-transitory computer-readable storage medium having stored thereon instructions for implementing the method according to the first aspect when the instructions are executed in a device having processing capabilities.

[0022] Where applicable, the further features of the method of the first aspect mentioned above also apply to the non-transitory computer-readable storage medium according to the second aspect. To avoid unnecessary repetitions, reference is made to the above.

[0023] According to a third aspect, an image processing device for generating an output image having a masking of an object of a category to be masked is provided. The image processing device includes a circuit configured to perform a reduction function, an input function, a receiving function, a first generating function, a setting function, and a second generating function. The reduction function is configured to reduce the input image to an object detection image having a resolution lower than the input image resolution and lower than the output image resolution. The input function is configured to input the object detection image to an object detection module. The receiving function is configured to receive confidence scores of pixels or pixel regions of the object detection image from the object detection module, each confidence score indicating a corresponding probability that the pixel or pixel region is associated with an object of a category to be masked. The first generating function is configured to generate an intermediate image having a resolution higher than the object detection image resolution based on the input image. The setting function is configured to set an adaptive masking threshold so that the greater the ratio between the output image resolution and the object detection image resolution, the lower the masking threshold. The second generating function is configured to generate an output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the adaptive masking threshold.

[0024] Where applicable, the additional features of the method of the first aspect mentioned above are also applicable to the image processing apparatus according to the third aspect. To avoid unnecessary repetition, reference is made to the above.

[0025] Further scope of applicability of the present invention will become apparent from the detailed description given below. However, it should be understood that while indicating preferred embodiments of the present invention, the detailed description and specific examples are given by way of illustration only, because various changes and modifications within the scope of the present invention will become apparent to those skilled in the art from the detailed description.

[0026] Therefore, it should be understood that the present invention is not limited to the particular components of the described device or the actions of the described method, because such device and method can vary. It should also be understood that the terms used herein are only used for the purpose of describing a particular embodiment and are not intended to be restrictive. It must be noted that the articles "one", "an", "the" and "said" as used in the specification and the appended claims are intended to mean that there are one or more elements, unless the context clearly indicates otherwise. Therefore, for example, references to "unit" or "the unit" can include multiple devices, etc. In addition, words such as "comprises" and "including" do not exclude other elements or steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other aspects of the present invention will now be described in more detail with reference to the accompanying drawings.The drawings should not be considered limiting but should be used for explanation and understanding.

[0028] Figure 1 A flow chart is shown relating to an embodiment of a method of generating an output image with occlusions of objects of a class to be occluded.

[0029] Figure 2 A schematic diagram related to an embodiment of an image processing apparatus for generating an output image with masking of an object of a category to be masked is shown. DETAILED DESCRIPTION

[0030] The invention will now be described hereinafter with reference to the accompanying drawings, in which currently preferred embodiments of the invention are illustrated. However, the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0031] Now combine Figure 1 The flowchart in describes an embodiment of a method 100 of generating an output image with occlusions of objects of a class to be occluded.

[0032] Embodiments of method 100 are applicable to scenarios where an operator is viewing an image where one or more classes of objects need to be masked (e.g., to maintain privacy or for other reasons). The image may be an image frame of a video stream. The class of objects may be, for example, a person, a vehicle including a license plate, or other class of objects that, if identified in the image, may reveal the identity of the person. The image may be provided in real time from a camera that captures the image for viewing by the operator, or the image may be provided in any other manner such that objects that need to be masked in the image need to be detected in real time or within a limited time before a masked version of the image is provided for viewing by the operator.

[0033] The method 100 includes reducing S110 the input image to an object detection image having a resolution lower than the input image resolution. The input image may be, for example, an image captured by a camera, or may be a processed version of an image captured by a camera. For example, the reduction may be performed so that the object detection module can process the object detection image within a required time period. For example, if the input image has a high resolution such as 4K or above, the object detection module may not be able to process such a high resolution image within the required time period. Therefore, this will introduce a delay, which may be undesirable or even harmful in a specific application.

[0034] The object detection image is then input S120 to the object detection module, and confidence scores for pixels or pixel regions of the object detection image are received S130 from the object detection module. Each confidence score indicates a respective probability that the pixel is within an object of the class to be masked or that the pixel region corresponds to an object of the class to be masked. Alternative pixels and pixel regions relate to whether the object detection module provides a confidence score for each pixel of the image, or whether the object detection module provides a confidence score for each of a set of pixel regions of the image (such as a bounding box associated with a possible object detection).

[0035] The object detection module may be based on any type of object detection algorithm that provides confidence scores for pixels or pixel regions of an image. For example, the object detection module may be based on a neural network such as a convolutional neural network. In another example, object detection may be based on a transformer, see e.g.

[0036] https: / / en.wikipedia.org / wiki / Transformer_(machine_learning_model).

[0037] An intermediate image having a higher resolution than the object detection image is generated S140 based on the input image. The intermediate image is used to generate an output image (eg, for display to an operator), in which objects of the category to be masked are masked.

[0038] The intermediate image may be generated by downscaling the input image. The intermediate image resolution is then lower than the input image resolution. This may be the case if the resolution of the image that should be provided to the operator for viewing (such as due to display resolution, transmission bit rate, storage space or other reasons) should be lower than the resolution of the input image.

[0039] Alternatively, the input image may be the same as the intermediate image, or a processed version of the input image without changing the resolution, such that the intermediate image resolution is equal to the input image resolution.

[0040] Furthermore, the intermediate image may be generated by cropping of the input image. This may be done in the case where the object detection image is a downscaled and cropped version of the input image, and in the case where no cropping of the input image is used to generate the object detection image. In both cases, the output image will correspond to the cropped portion of the input image, and the resolution of the output image should be the resolution of the cropped portion compared to the resolution of the corresponding portion of the object detection image.

[0041] In addition, an adaptive masking threshold is set S150 so that the greater the ratio between the output image resolution and the object detection resolution, the lower the masking threshold. Then, an output image is generated S160 by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the adaptive masking threshold. Because the output image is generated by masking pixels or pixel regions of the intermediate image, the output image resolution is the same as the intermediate image resolution.

[0042] Setting the adaptive masking threshold defined by method 100 is based on the understanding that as the ratio increases, the operator will likely identify more objects of the class that should be masked at the output image resolution, while at the same time, these objects will not be detected by the object detection module at the object detection image resolution. Here, "detected by the object detection module" means that the confidence score of the pixel or pixel region associated with the object is higher than the non-adaptive masking threshold set without considering the ratio.

[0043] The adaptive occlusion threshold T may be a linear function of the ratio r between the output image resolution and the object detection image resolution. For example, the adaptive occlusion threshold T may be determined as the default occlusion threshold T d Subtract the constant α and multiply the ratio r, that is, T = T d -α·r (Equation 1). The constant α will vary between different object detection modules, as some modules are better than others at detecting objects at lower resolutions.

[0044] If for a given object detection module, we want to set the default masking threshold T to 1 when the output image resolution is 3840×2160 and the object detection image resolution is 1024×576 (i.e., r=14.0625). d Going down 20 steps (on a scale of 1-100), Equation 1 gives an α of approximately 1.42.

[0045] If the adaptive occlusion threshold T is also to be adaptive with respect to the size of the object, then the adaptive occlusion threshold T of the object o The default masking threshold T d Subtract the constant α times the ratio r divided by the constant β times the size of the object s o , that is, T o =T d -α·r / β·s o (Equation 2). The constant β will vary between different classes of objects because the ability of the object detection module to detect an object of a given size will be different for different classes of objects.

[0046] The method 100 may further include setting S155 a fixed occlusion threshold that is independent of the ratio between the output image resolution and the object detection image resolution. This fixed occlusion threshold will be used when the adaptive occlusion threshold is not used. The fixed occlusion threshold may correspond to the default occlusion threshold in Equation 1 and Equation 2 above. In this case, the adaptive occlusion threshold will be according to a corresponding one of Equation 1 and Equation 2, and the fixed occlusion threshold will be the default occlusion threshold. It should be noted that the "fixed" in the fixed occlusion threshold is intended to indicate that it is not adjusted relative to the ratio between the output image resolution and the object detection image resolution. Adjustment of the fixed occlusion threshold relative to other parameters is not excluded.

[0047] For example, the adaptive masking threshold may be used only for objects that are close to a preset distance. In this case, the S160 output image may be generated differently for pixels or pixel areas corresponding to the object depending on whether the object is close to the preset distance. Specifically, for pixels or pixel areas corresponding to objects that are close to the preset distance, the output image is generated by masking the pixels or pixel areas of the intermediate image corresponding to the pixels or pixel areas of the object having a confidence score higher than the adaptive masking threshold. For pixels or pixel areas corresponding to objects that are equal to or farther than the preset distance, the output image is generated by masking the pixels or pixel areas of the intermediate image corresponding to the pixels or pixel areas of the object having a confidence score higher than the fixed masking threshold. Here, "close to the preset distance" means that the distance from the camera capturing the image to the object is closer than the preset distance. For example, the distance from the camera may be determined by a distance sensor such as a laser radar or radar or any other type of distance sensor arranged on the camera. In another example, the distance from the camera may be determined based on the focal length used when capturing the image, the size of the object in the image, or the size of the bounding box of the object, and the assumed category of the object. Other examples are the use of neural networks (see, e.g., https: / / github.com / mrharicot / monodepth) and the use of perspective transforms, where the distance along the ground can be determined based on the camera's mounting height and field of view (see, e.g., https: / / www.tutorialspoint.com / dip / perspective_transformation.htm).

[0048] The preset distance may be, for example, a distance at which the operator will (or at least is assumed to) be unable to identify an object. For example, at a given resolution of the image from the camera, the operator will be unable to identify a person at a certain distance from the camera because the person will be too small in the image and the number of pixels on the person's face will be too small to enable the operator to recognize features of the person's face.

[0049] In addition, the adaptive masking threshold can be used only for objects at a location farther than another preset distance, wherein the other preset distance is shorter than the preset distance. In this case, the S160 output image can be generated differently for pixels or pixel areas corresponding to the object depending on whether the object is farther than another preset distance. Specifically, for pixels or pixel areas corresponding to objects farther than another preset distance, the S160 output image is generated by masking the pixels or pixel areas of the intermediate image corresponding to the pixels or pixel areas of the object detection image having a confidence score higher than the adaptive masking threshold. For pixels or pixel areas corresponding to objects equal to or close to another preset distance, the S160 output image is generated by masking the pixels or pixel areas of the intermediate image corresponding to the pixels or pixel areas of the object detection image having a confidence score higher than the fixed masking threshold.

[0050] Another preset distance may be, for example, a distance at which the object detection module will (or at least is assumed to) be able to detect an object. For example, at a given resolution of the object detection image, the object detection module will be able to detect a person at a certain distance from the camera that captured the image on which the object detection image is based, because the person will be large enough in the image so that the number of pixels on the person's face is large enough to enable the object detection module to recognize that the object is a person that should be shielded. The ability of the object detection module to detect an object will also vary in different situations based on other parameters (such as lighting conditions, the pose of the object, etc.). Therefore, the difficulty of object detection will vary from low difficulty to high difficulty for different situations. Another preset distance may, for example, be set so that the object detection module is at least able to detect an object in a situation where the difficulty of object detection is medium.

[0051] The preset distance and another preset distance can be used alone or together. In the latter case, for pixels or pixel areas corresponding to objects closer to the preset distance and farther than another preset distance, the S160 output image is generated by masking the pixels or pixel areas of the intermediate image corresponding to the pixels or pixel areas of the object detection image having a confidence score higher than the adaptive masking threshold. For pixels or pixel areas corresponding to objects equal to or farther than the preset distance, or equal to or close to another preset distance, the S160 output image is generated by masking the pixels or pixel areas of the intermediate image corresponding to the pixels or pixel areas of the object detection image having a confidence score higher than the fixed masking threshold.

[0052] In another example, the adaptive masking threshold may be used only for objects having a pixel density higher than a preset pixel density. In this case, the S160 output image may be generated differently for pixels or pixel regions corresponding to the object depending on whether the object has a pixel density higher than the preset pixel density. Specifically, for pixels or pixel regions corresponding to objects having a pixel density higher than the preset pixel density in the intermediate image, the S160 output image is generated by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the adaptive masking threshold. For pixels or pixel regions corresponding to objects having a pixel density equal to or lower than the preset pixel density in the intermediate image, the S160 output image is generated by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the fixed masking threshold.

[0053] The preset pixel density can be, for example, a pixel density at which an operator will (or at least is assumed to) be unable to identify an object. For example, if the pixel density is too small, so that the operator cannot recognize the features of a person's face, the operator will not be able to identify a person.

[0054] In addition, the adaptive masking threshold can be used only for objects with a pixel density lower than another preset pixel density, wherein the other preset pixel density is higher than the preset pixel density. In this case, the S160 output image can be generated differently for the pixels or pixel areas corresponding to the object depending on whether the object has a pixel density lower than the preset pixel density. Specifically, for pixels or pixel areas corresponding to objects having a pixel density higher than the preset pixel density and lower than another preset pixel density in the object detection image, the S160 output image is generated by masking the pixels or pixel areas of the intermediate image corresponding to the pixels or pixel areas of the object detection image having a confidence score higher than the adaptive masking threshold. For pixels or pixel areas corresponding to objects having a pixel density equal to or lower than the preset pixel density or equal to or higher than another preset pixel density in the object detection image, the S160 output image is generated by masking the pixels or pixel areas of the intermediate image corresponding to the pixels or pixel areas of the object detection image having a confidence score higher than the fixed masking threshold.

[0055] Another preset pixel density may be, for example, a pixel density at which the object detection module can detect the object. For example, at a pixel density equal to or higher than another preset pixel density, the number of pixels on the face will be large enough to enable the object detection module to identify that the object is a person to be shielded.

[0056] The preset pixel density and another preset pixel density may be used alone or together. In the latter case, for pixels or pixel regions corresponding to an object having a pixel density in the object detection image that is higher than the preset pixel density and lower than another preset pixel density, an output image S160 is generated by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than an adaptive masking threshold, wherein the other preset pixel density is higher than the preset pixel density. For pixels or pixel regions corresponding to an object having a pixel density in the object detection image that is equal to or lower than the preset pixel density, or equal to or higher than another preset pixel density, an output image S160 is generated by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than a fixed masking threshold.

[0057] If the object detection module is based on a convolutional neural network, the convolutional neural network may be any type of convolutional neural network that can be trained and used to classify objects in an image into different object categories such as cars, people, bicycles, etc. For example, the Google Bodypix trained on the COCO dataset may be used.

[0058] (See https: / / github.com / tensorflow / tfjs-models / tree / master / body-pix, and https: / / blog.tensorflow.org / 2019 / 11 / updated-bodypix-2.html).

[0059] Figure 2 A block diagram related to an embodiment of an image processing apparatus for generating an output image with occlusion of an object of a class to be occluded is shown. The image processing apparatus 200 may be, for example, a camera, included in a camera or connected to a camera for capturing images.

[0060] The image processing device 200 includes a circuit 210. The circuit 210 is configured to perform the functions of the image processing device 200. The circuit 210 may include a processor 212 such as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a microcontroller, or a microprocessor. The processor 212 is configured to execute program code. The program code may be configured to perform the functions of the image processing device 200, for example.

[0061] The image processing apparatus 200 may further include a memory 220. The memory 220 may be a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, a random access memory (RAM), or another suitable device. In a typical arrangement, the memory 220 may include a non-volatile memory for long-term data storage and a volatile memory used as a device memory for the circuit 210. The memory 220 may exchange data with the circuit 210 via a data bus. There may also be accompanying control lines and an address bus between the memory 220 and the circuit 210.

[0062] The functions of the image processing device 200 may be embodied in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored on a non-transitory computer-readable medium (e.g., memory 220) of the image processing device 200 and executed by the circuit 210 (e.g., using the processor 212). In addition, the functions of the sending device 200 may be independent software applications or form part of a software application that performs additional tasks related to the image processing device 200. The described functions may be viewed as methods that a processing unit (e.g., the processor 212 of the circuit 210) is configured to perform. Likewise, although the described functions may be implemented in software, such functions may also be performed via dedicated hardware or firmware, or some combination of hardware, firmware and / or software.

[0063] The circuit 210 is configured to perform a reduction function 221 , an input function 222 , a reception function 223 , a first generation function 224 , a setting function 225 , a second generation function 226 and optionally a further setting function 227 .

[0064] The downscaling function 221 is configured to downscale the input image to an object detection image having a resolution lower than that of the input image and lower than that of the output image.

[0065] The input function 222 is configured to input the object detection image to the object detection module.

[0066] The receiving function 223 is configured to receive confidence scores of pixels or pixel regions of the object detection image from the object detection module, each confidence score indicating a corresponding probability that the pixel or pixel region is associated with an object of a class to be masked.

[0067] The first generation function 224 is configured to generate an intermediate image having a higher resolution than the object detection image based on the input image.

[0068] The setting function 225 is configured to set the adaptive occlusion threshold such that the larger the ratio between the output image resolution and the object detection image resolution, the lower the occlusion threshold.

[0069] The second generation function 226 is configured to generate an output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score above an adaptive masking threshold.

[0070] The first generation function 224 can be configured to generate an intermediate image by reducing the input image to an intermediate image. The intermediate image then has a resolution lower than the input image resolution. Alternatively, the first generation function 224 can be configured to generate an intermediate image by using the input image as an intermediate image. The intermediate image then has a resolution equal to the input image resolution.

[0071] An optional further setting function 227 is configured to set a fixed occlusion threshold that is independent of the ratio between the output image resolution and the object detection image resolution.

[0072] In an embodiment comprising another setting function 227, the second generating function may be configured to:

[0073] generating an output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score above an adaptive masking threshold for pixels or pixel regions corresponding to objects closer than a preset distance, and

[0074] An output image is generated by masking pixels or pixel regions of the intermediate image that correspond to pixels or pixel regions of the object detection image having a confidence score above a fixed masking threshold for pixels or pixel regions corresponding to objects at or beyond a preset distance.

[0075] In an embodiment comprising another setting function 227, the second generating function may be configured to:

[0076] outputting an image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having confidence scores above an adaptive masking threshold, wherein for pixels or pixel regions corresponding to objects that are closer than a preset distance and farther than another preset distance, the another preset distance is shorter than the preset distance, and

[0077] An image is output by masking pixels or pixel areas of an intermediate image corresponding to pixels or pixel areas of an object detection image having a confidence score higher than a fixed masking threshold for pixels or pixel areas corresponding to objects that are equal to or farther than a preset distance, or equal to or closer than another preset distance.

[0078] In an embodiment comprising another setting function 227, the second generating function may be configured to:

[0079] generating an output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image, the object detection image having a confidence score above an adaptive masking threshold for pixels or pixel regions corresponding to objects in the intermediate image having a pixel density higher than a preset pixel density, and

[0080] An output image is generated by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image, wherein the object detection image has a confidence score higher than a fixed masking threshold for pixels or pixel regions corresponding to objects in the intermediate image having a pixel density equal to or lower than a preset pixel density.

[0081] In an embodiment comprising another setting function 227, the second generating function may be configured to:

[0082] generating an output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score above an adaptive masking threshold, wherein for pixels or image regions corresponding to objects in the intermediate image having a pixel density higher than a preset pixel density and lower than another preset pixel density, the other preset pixel density is higher than the preset pixel density, and

[0083] An output image is generated by masking pixels or pixel areas of the intermediate image corresponding to pixels or pixel areas of the object detection image, wherein the object detection image has a confidence score higher than a fixed masking threshold for pixels or corresponding pixel areas corresponding to objects in the intermediate image having a pixel density equal to or lower than a preset pixel density, or equal to or higher than another preset pixel density.

[0084] Combination of the above Figure 1 The detailed description of the actions of the method 100 described above also applies to the corresponding functions of the image processing device 200. In addition, when applicable, the above combined Figure 1 The optional further features described for method 100 also apply to image processing device 200 .

[0085] Those skilled in the art will recognize that the present invention is not limited to the above-described embodiments. On the contrary, many modifications and variations are possible within the scope of the appended claims. Those skilled in the art will understand and implement such modifications and variations when practicing the claimed invention by studying the accompanying drawings, this disclosure, and the appended claims.

Claims

1. A method for generating an output image, wherein the output image has a masking of an object of a class to be masked, the method comprising: downscaling an input image to an object detection image having a resolution lower than a resolution of the input image and lower than a resolution of the output image; inputting the object detection image into an object detection module; receiving confidence scores for pixels or pixel regions of the object detection image from the object detection module, each confidence score indicating a respective probability that the pixel or pixel region is associated with an object of a class to be masked; generating an intermediate image having a resolution higher than a resolution of the object detection image based on the input image; Setting an adaptive occlusion threshold such that the greater the ratio between the output image resolution and the object detection image resolution, the lower the adaptive occlusion threshold; as well as The output image is generated by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having confidence scores above the adaptive masking threshold.

2. The method according to claim 1, wherein: The intermediate image is generated by one of the following: reducing the input image to the intermediate image, wherein the intermediate image has a lower resolution than the input image, and The input image is used as the intermediate image, wherein the intermediate image resolution is equal to the input image resolution.

3. The method according to claim 1, further comprising: setting a fixed occlusion threshold that is independent of the ratio between the output image resolution and the object detection image resolution, Wherein, generating the output image comprises: generating the output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having confidence scores above the adaptive masking threshold for pixels or pixel regions corresponding to objects closer than a preset distance; and For pixels or pixel regions corresponding to objects that are equal to or farther than the preset distance, the output image is generated by masking pixels or pixel regions of the intermediate image that correspond to pixels or pixel regions of the object detection image that have confidence scores higher than the fixed masking threshold.

4. The method according to claim 1, further comprising: setting a fixed occlusion threshold that is independent of the ratio between the output image resolution and the object detection image resolution, Wherein, generating the output image comprises: generating the output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having confidence scores above the adaptive masking threshold for pixels or pixel regions corresponding to objects closer than a preset distance and farther than another preset distance, wherein the another preset distance is shorter than the preset distance; and For pixels or pixel areas corresponding to objects that are equal to or farther than the preset distance, or equal to or close to the other preset distance, the output image is generated by masking the pixels or pixel areas of the intermediate image corresponding to the pixels or pixel areas of the object detection image that have a confidence score higher than the fixed masking threshold.

5. The method according to claim 1, further comprising: setting a fixed occlusion threshold that is independent of the ratio between the output image resolution and the object detection image resolution, Wherein, generating the output image comprises: generating the output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the adaptive masking threshold, for pixels or pixel regions of the intermediate image corresponding to objects having a pixel density higher than a preset pixel density; and For pixels or pixel areas corresponding to objects in the intermediate image having a pixel density equal to or lower than the preset pixel density, the output image is generated by masking the pixels or pixel areas of the intermediate image corresponding to pixels or pixel areas of the object detection image having a confidence score higher than the fixed masking threshold.

6. The method according to claim 1, further comprising: setting a fixed occlusion threshold that is independent of the ratio between the output image resolution and the object detection image resolution, Wherein, generating the output image comprises: generating the output image by masking pixels or pixel regions of the input image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the adaptive masking threshold, for pixels or pixel regions corresponding to objects in the object detection image having a pixel density higher than a preset pixel density and lower than another preset pixel density, wherein the another preset pixel density is higher than the preset pixel density; and For pixels or pixel regions corresponding to objects in the object detection image having a pixel density equal to or lower than the preset pixel density, or equal to or higher than another preset pixel density, the output image is generated by masking the pixels or pixel regions of the input image corresponding to the pixels or pixel regions of the object detection image having a confidence score higher than the fixed masking threshold.

7. A non-transitory computer-readable storage medium having stored thereon instructions for implementing the method according to claim 1 when the instructions are executed in a device having processing capabilities.

8. An image processing apparatus for generating an output image having an occlusion of an object of a class to be occluded, the image processing apparatus comprising circuitry configured to: a downscaling function configured to downscale an input image to an object detection image having a resolution lower than a resolution of the input image and lower than a resolution of the output image; an input function configured to input the object detection image into an object detection module; a receiving function configured to receive confidence scores of pixels or pixel regions of the object detection image from the object detection module, each confidence score indicating a corresponding probability that the pixel or pixel region is associated with an object of a class to be masked; a first generating function configured to generate an intermediate image having a higher resolution than a resolution of an object detection image based on the input image; a setting function configured to set an adaptive occlusion threshold such that the larger the ratio between the output image resolution and the object detection image resolution, the lower the adaptive occlusion threshold; and A second generation function is configured to generate the output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score above the adaptive masking threshold.

9. The image processing apparatus according to claim 8, wherein: The first generating function is configured to generate the intermediate image by one of: reducing the input image to the intermediate image, wherein the intermediate image has a lower resolution than the input image, and The input image is used as the intermediate image, wherein the intermediate image resolution is equal to the input image resolution.

10. The image processing device according to claim 8, wherein: The circuit is further configured to perform: another setting function configured to set a fixed occlusion threshold independent of the ratio between the output image resolution and the object detection image resolution, Wherein, the second generation function is configured as follows: generating the output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having confidence scores above the adaptive masking threshold for pixels or pixel regions corresponding to objects closer than a preset distance; and For pixels or pixel regions corresponding to objects that are equal to or farther than the preset distance, the output image is generated by masking pixels or pixel regions of the intermediate image that correspond to pixels or pixel regions of the object detection image that have confidence scores higher than the fixed masking threshold.

11. The image processing apparatus according to claim 8, wherein: The circuit is further configured to perform: another setting function configured to set a fixed occlusion threshold independent of the ratio between the output image resolution and the object detection image resolution, Wherein, the second generation function is configured as follows: generating the output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having confidence scores above the adaptive masking threshold for pixels or pixel regions corresponding to objects that are closer than a preset distance and farther than another preset distance that is shorter than the preset distance; and For pixels or pixel areas corresponding to objects that are equal to or farther than the preset distance, or equal to or close to the other preset distance, the output image is generated by masking the pixels or pixel areas of the intermediate image corresponding to the pixels or pixel areas of the object detection image that have a confidence score higher than the fixed masking threshold.

12. The image processing apparatus according to claim 8, wherein: The circuit is further configured to perform: another setting function configured to set a fixed occlusion threshold independent of the ratio between the output image resolution and the object detection image resolution, Wherein, the second generation function is configured as follows: generating the output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the adaptive masking threshold, for pixels or pixel regions of the intermediate image corresponding to objects having a pixel density higher than a preset pixel density; and For pixels or pixel areas corresponding to objects in the intermediate image having a pixel density equal to or lower than the preset pixel density, the output image is generated by masking the pixels or pixel areas of the intermediate image corresponding to pixels or pixel areas of the object detection image having a confidence score higher than the fixed masking threshold.

13. The image processing apparatus according to claim 8, wherein: The circuit is further configured to perform: another setting function configured to set a fixed occlusion threshold independent of the ratio between the output image resolution and the object detection image resolution, Wherein, the second generation function is configured as follows: generating the output image by masking pixels or pixel regions of the intermediate image corresponding to pixels or pixel regions of the object detection image having a confidence score higher than the adaptive masking threshold, for pixels or pixel regions of the input image corresponding to objects having a pixel density higher than a preset pixel density and lower than another preset pixel density, wherein the another preset pixel density is higher than the preset pixel density; and For pixels or pixel areas corresponding to objects in the input image having a pixel density equal to or lower than the preset pixel density, or equal to or higher than another preset pixel density, the output image is generated by masking the pixels or pixel areas of the intermediate image corresponding to the pixels or pixel areas of the object detection image having a confidence score higher than the fixed masking threshold.

Images