Methods, apparatus, systems, and programs for applying masks to objects in an image.

Abstract

This disclosure provides a method, apparatus, system, and program for applying a mask to an object in an image. The method includes a processor determining whether to apply a mask to an object in an image based on a comparison with another object in the image, and the processor applying the mask to the object in the image based on the determination.

Description

Technical Field

[0001] The present invention generally relates to, but is not limited to, methods, apparatuses, systems, and programs for applying masks to objects in images.

Background Art

[0002] Video surveillance systems are widely used around the world for law enforcement and security for crime monitoring and protection. With the spread of video surveillance systems, concerns about public privacy regarding video surveillance are increasing. For example, there is a risk that the identity of an individual in a public area may be revealed by a security guard while monitoring a surveillance camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] To address the privacy concerns mentioned above, privacy masking is applied to surveillance camera footage to protect sensitive and personal information, such as individual identities, before it is displayed on the user's device. Object detection is often used to detect such sensitive information before selecting object areas for privacy masking. This is because it is fast and can detect both people facing forward and those with their backs turned. To further optimize the privacy masking process, some solutions may choose to skip privacy masking for smaller objects that may not contain enough data to reveal sensitive information. However, methods for detecting small objects may inadvertently skip privacy masking for objects that may still contain sensitive information.

[0005] This specification discloses embodiments of methods, apparatus, systems, and programs for applying masks to objects in an image, addressing one or more of the above-mentioned problems.

[0006] Furthermore, other desirable features and characteristics will become apparent from the following detailed description and the attached claims, in conjunction with the attached drawings and this background of the disclosure. [Means for solving the problem]

[0007] In a first aspect, the Disclosure provides a method for applying a mask to an object in an image, the method comprising: a processor determining whether to apply a mask to an object in an image based on a comparison with another object in an image; and the processor applying a mask to the object in the image based on the determination.

[0008] In a second aspect, the Disclosure provides an apparatus for applying a mask to an object in an image, comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code cause the apparatus to perform, at least, the following actions using the at least one processor: determine whether to apply a mask to an object in an image based on a comparison with another object in the image, and apply a mask to the object in the image based on the determination.

[0009] In a third aspect, the Disclosure provides a system for applying a mask over an object in an image, comprising the apparatus of the second aspect and one or more imaging devices configured to capture one or more images or video frames, wherein one or more images or video frames include images of a person, and the person is the object. [Effects of the Invention]

[0010] Further benefits and advantages of the disclosed embodiments will become apparent from the specification and drawings. Benefits and / or advantages can be obtained individually from the various embodiments and features of the specification and drawings, and not all of them are required to obtain one or more of such benefits and / or advantages. [Brief explanation of the drawing]

[0011] The accompanying drawings, in which similar reference numerals refer to identical or functionally similar elements through separate drawings, are incorporated herein by reference to the following detailed description and form part thereof, illustrating various embodiments only as non-limiting examples, and serving to illustrate various principles and advantages of embodiments of the present invention.

[0012] Embodiments of the present invention will be readily apparent to those skilled in the art, as will be understood and made clear to those skilled in the art from the following description, in conjunction with the drawings, as merely examples. [Figure 1]Figure 1 shows an example of the object detection and privacy masking process on a sample video. [Figure 2] Figure 2 shows an example of object detection and privacy masking on an image. [Figure 3] Figure 3 is an illustrative diagram of the setting of body proportions for object detection and privacy masking according to various embodiments of the present disclosure. [Figure 4] Figure 4 is an illustrative diagram for setting up a two-dimension (2D) plane for object detection and privacy masking according to various embodiments of the present disclosure. [Figure 5] Figure 5 is a flowchart illustrating methods for applying a mask to an object in an image according to various embodiments of the present disclosure. [Figure 6] Figure 6 is an exemplary diagram of object detection and privacy masking on an image according to an embodiment of the present disclosure. [Figure 7] Figure 7 is an illustrative diagram of object detection and privacy masking on an image according to another embodiment of the present disclosure. [Figure 8] Figure 8 is an illustrative diagram of object detection and privacy masking on an image according to another embodiment of the present disclosure. [Figure 9] Figure 9 is an illustrative diagram of object detection and privacy masking on an image according to another embodiment of the present disclosure. [Figure 10] Figure 10 is a flowchart of object detection and privacy masking according to various embodiments of the present disclosure. [Figure 11] Figure 11 is an illustrative image of an object detected and a privacy mask applied according to an embodiment of the present disclosure. [Figure 12] Figure 12 is a schematic diagram of an exemplary computing device suitable for use in carrying out the methods of Figures 5 and 10. [Modes for carrying out the invention]

[0013] Embodiments of the present invention will be described by way of example only and with reference to the drawings. Like reference numerals and signs in the drawings refer to like elements or equivalents.

[0014] Unless otherwise specified, and as will be apparent hereinafter, throughout this specification, discussions using terms such as "detect", "estimate", "compare", "receive", "calculate", "determine", "update", "generate", "initialize", "output", "cause to receive", "recover", "identify", "distribute", "authenticate", etc. refer to actions and processes of a computer system or similar electronic device that manipulate and transform data represented as physical quantities within the computer system into other data similarly represented as physical quantities within the computer system, or other information storage devices, transmission devices, or display devices.

[0015] This specification also discloses an apparatus for performing the operations of the method. Such an apparatus may be specially constructed for the required purposes or may comprise a computer or other device selectively activated or reconfigured by a computer program stored in the computer. The algorithms and displays presented in this specification are not inherently related to any particular computer or other device. Various machines may be used with the programs according to the teachings of this specification. Alternatively, construction of more specialized apparatus for performing the required method steps may be appropriate. The structure of the computer will become apparent from the following description.

[0016] In addition, this specification also implicitly discloses a computer program in that it will be apparent to those skilled in the art that the individual steps of the methods described herein can be implemented by computer code. The computer program is not intended to be limited to any particular programming language and its implementation. It will be understood that various programming languages and their codings can be used to implement the teachings of the disclosure contained herein. Further, the computer program is not intended to be limited to any particular control flow. There are many other variations of computer programs that can use different control flows without departing from the spirit or scope of the invention.

[0017] Furthermore, one or more of the steps of the computer program can be implemented in parallel rather than sequentially. Such a computer program can be stored on any computer-readable medium. The computer-readable medium can include a storage device such as a magnetic disk or optical disk, a memory chip, or other storage devices suitable for interfacing with a computer. The computer-readable medium can also include a hardwired medium as exemplified in an Internet system, or a wireless medium as exemplified in a GSM mobile phone system. When the computer program is loaded and executed on such a computer, it effectively provides an apparatus for implementing the steps of the preferred method.

[0018] (Embodiment) Various embodiments of the present disclosure relate to a method and apparatus for applying a mask on an object in an image.

[0019] A privacy mask (also referred to herein as a mask) is applied to a specific area within an image, video, or other similar medium to blur or pixelate that area, so that the image, video, or similar medium can be viewed without compromising the privacy of entities associated with that area. For example, using object detection to detect people or objects, such as license plates in dashcam or security camera video footage for privacy masking, can be resource-intensive and time-consuming. Each individual frame of the video footage requires an object detection engine to detect objects of interest to be privacy masked. The privacy masking algorithm is then applied to each detected object of interest before merging them back into a final frame that is re-encoded into privacy-masked dashcam video footage for sharing. It will be understood that objects can refer to people, license plates, address plates, or other similar objects to which a privacy mask may need to be applied to protect against privacy infringement.

[0020] Referring to Figure 100 in Figure 1, performing privacy masking on one detected object in the input image 102 typically takes about 50 ms (milliseconds). The input image 102 can typically be a video frame from FHD (full high definition) video (e.g., recorded via a security camera) operating at 20 fps (frames per second). Therefore, the input image 102 may be one of 20 video frames per second of FHD video. Thus, one second of 20 fps FHD video requires (total frames) * (processing time for privacy masking per frame) = (1 second * 20 fps) * (50 ms) = 20 * 50 ms = 1000 ms to perform privacy masking on one person per second of FHD video. Running at 20fps, one minute (e.g., 60 seconds) of FHD video with five people in it would require 60 seconds * (5 people * 1000ms) = 60 * (5000ms) = 60 * 5 seconds = 300 seconds = 5 minutes to complete object detection of all five people (e.g., as shown in image 104) and privacy masking of all five people (e.g., as shown in image 106), which is not feasible with real-time video.

[0021] One way to optimize privacy masking is to skip unnecessary privacy masking based on the detected height of a person (for example, if the detected height of an object in the image is shorter than a certain threshold, it may be determined that the object in the image is too small for privacy masking to be required), using object and pose detection, for example. Since pose detection is a resource-intensive process, two types of pre-trained machine learning models can be used. The first type is based on single pose detection, which can achieve high-accuracy detection based on more keypoints (e.g., points on the image of the person corresponding to the location of the person's body parts) for the pose detection of a single person. The second type is based on multiple pose detection, which is limited to detecting 4 to 6 people with an average number of keypoints to balance detection accuracy and speed. A pre-trained pose estimation machine learning (ML) model can provide multiple keypoints between a person's eyes and toes that can be used to identify a person and estimate their height. For example, referring to image 200 in Figure 2, privacy masking may be skipped for detected people whose height in image 200 is less than 25px (pixels). Since three people (e.g., people 202, 204, and 206) with a height of less than 25 pixels are detected in image 200, privacy masking is skipped for these three people. Pose estimation allows for a better estimation of the height of person 206, for example, who is crouching and whose actual height may not actually be less than 25 pixels, but it requires higher computational power and is limited to detecting only 4-6 poses in the image.

[0022] This disclosure proposes a method for improving object detection accuracy to skip privacy masking of selected individuals detected in an image, based on a pre-configured minimum detection height (e.g., threshold height) and further based on the detection of the tallest person in a 2D (two-dimensional) plane within the image (e.g., the person with the highest height among one or more people in a 2D plane within the image). For example, referring to Figure 302 in Figure 3, the average body proportions may be determined to be 1:7 for the head-to-body ratio and 1:4 for the head-to-arm ratio. Therefore, as shown in Figure 304, based on a standing height of 7 units and a width of 5 units (e.g., based on a head-to-arm ratio of 1 unit:4 units, the width including the head and arms is set to 1+4=5 units), the minimum standing height-to-width ratio may be determined to be 7 / 5=1.4.

[0023] Further assumptions are as follows: The minimum height of a detected person in the image for applying a privacy mask may be set to 25px (25 pixels). The upper 2D plane distance may be set to 3px. The lower 2D plane distance may be set to 2px. The search on the Y-axis of the image may be configured to start from the detected person's height minus the upper 2D plane distance and end from the detected person's height minus the lower 2D plane distance. For example, referring to Figure 400 in Figure 4, person 402 can be detected from pixel 1200 on the X-axis and pixel 510 on the Y-axis of the image. Person 402 may be below the minimum height of 25px (e.g., has a detected height of 24px) and may also be below the minimum height-to-width ratio (e.g., threshold ratio) of 1.4 for applying a privacy mask. This triggers a search for the person with the highest height within the 2D plane distance of person 402 (e.g., the 2D plane distance between pixels 512 and 507 on the Y-axis of the image, 406). Person 404 is detected as the tallest person within a 2D plane distance from Person 402, based on pixels 1550 on the X axis and 511 on the Y axis in the image, and has a height of 35px, exceeding the minimum height of 25px. Therefore, the privacy mask flag for Person 402 is set to "true", and for example, a privacy mask is applied to Person 402. In this embodiment, one or more algorithms can be used to determine the privacy mask flag for all detected persons in the image. The privacy mask flag can carry an initial value of "?" and then be set to either "TRUE" or "FALSE" based on the detected height and ratio, as well as a comparison with one or more other detected persons located in the 2D plane as described above. Advantageously, it is possible to skip privacy masking of detected persons without resource-intensive processes such as pose detection and without the limitations of keypoints during pose detection. It will be understood that the threshold height and ratio can be adjusted based on applications and factors such as image or video resolution, the field of view of the camera from which the video or image was taken, the scene captured in the image or video, and other similar factors.

[0024] Figure 5 is a flowchart 500 illustrating a method for applying a mask to an object in an image according to various embodiments of the present disclosure. In step 502, the processor determines whether to apply a mask to an object in the image based on a comparison with another object in the image. In step 504, the processor applies a mask to the object in the image based on the determination.

[0025] In one embodiment, determining whether to apply a mask may include comparing the height of an object in the image to a threshold height, and if the height of the object is lower than the threshold height, further comparing the ratio of the height to the width of the object in the image to the threshold ratio, and determining whether to apply a mask based on the further comparison. The method may further include identifying another object from one or more other objects in the image if the ratio is lower than the threshold ratio, wherein the other object has the highest height among the object and one or more other objects, and one or more other objects are located within a two-dimensional plane distance from the object, and determining whether a mask should be applied to the other object. The method may further include applying a mask to the object if it is determined that a mask should be applied to the other object.

[0026] In embodiments, the method may further include applying a mask to an object if the object's height is greater than or equal to a threshold height. The method may further include determining whether to apply a mask to an object if the ratio is greater than a threshold ratio.

[0027] In one embodiment, the method may further include applying a mask to an object in multiple frames of a video, where the image is a frame from the multiple frames of the video.

[0028] In one embodiment, the apparatus may include at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code cause the apparatus to perform at least method 500 using the at least one processor. In one embodiment, the system for applying a mask to an object in an image may include the apparatus and one or more image and video capture devices configured to capture one or more images or video frames, wherein one or more images or video frames include images of a person, and the person is the object.

[0029] Figure 6 is an illustrative diagram of object detection and privacy masking on image 600 according to an embodiment of the present disclosure. Two people, 602 and 604, are detected in image 600. Table 608 provides various detected dimensions of people 602 and 604 in the first and second rows, respectively. Referring to Table 608, person 602 is detected in image 600 from pixel 1200 on the X axis and pixel 485 to pixel 1223 on the X axis and pixel 509 on the Y axis, with a height of 24px, a width of 23px, a height-to-width ratio of 1.0434, a corresponding 2D plane 606 in pixels 506 to 511 on the Y axis of image 600, and a privacy mask flag "?". Furthermore, person 604 is detected in image 600 from pixel 1550 on the X axis and pixel 472 on the Y axis to pixel 1570 on the X axis and pixel 507 on the Y axis, with a height of 35px, a width of 20px, a height-to-width ratio of 1.75, a corresponding 2D plane in pixels 504 to 509 on the Y axis of image 600, and a privacy mask flag "?". It should be understood that there may be one or more other persons detected in image 600 but not shown for simplification.

[0030] The detected individuals 602 and 604 may be processed one after another in a continuous sequence to determine each individual's privacy mask flag. For example, it is checked whether the privacy mask flag for individual 602 is "?". If not (for example, the privacy mask flag is "TRUE" or "FALSE", which means that the detected individual associated with the privacy mask flag has already been processed and it has been determined that a privacy mask should be applied or masking should be skipped), individual 602 is not processed (because a privacy mask flag that is not "?" indicates that the individual has already been processed). Otherwise, the height of individual 602 is compared to the threshold height (e.g., 25px). As shown in Table 608 of Figure 6, the height of individual 602 is 24px, which is less than the threshold height. Therefore, the height-to-width ratio of individual 602 is also compared to the threshold ratio (e.g., 1.4). Since the ratio of person 602, 1.0434, is also smaller than the threshold ratio, a search is performed for one or more people in the same 2D plane as person 602 (for example, the corresponding 2D plane 606 within pixels 506 to 511 on the Y axis of image 600). The one or more people found in the search are further compared to determine the tallest person among them, and this tallest person is used as a criterion for further comparison with person 602 (for example, to determine whether the privacy mask flag for person 602 should be updated to "TRUE" or "FALSE"). Since person 604 is the only person in the same 2D plane 606 as person 602 in image 600, the search returns person 604 as a search result. If there are one or more other people in the same 2D plane 606 besides person 604, and person 604 is determined to have the tallest height among the one or more other people, it will be understood that the search returns person 604 as a search result.

[0031] Since the privacy mask flag for person 604 is "?" (for example, it has not yet been determined whether a privacy mask should be applied to person 604), the height of person 604 is checked. Based on Table 608, since the height of person 604 is 35px, which is higher than the threshold height of 25px, the privacy mask flag for person 604 is updated to "TRUE" (for example, a privacy mask should be applied to person 604). Also, the privacy mask flag for person 602 is updated to "TRUE" to be the same as person 604 (person 604 is the only person 604 detected in the same 2D plane 606 as person 602) (for example, a privacy mask should be applied to person 602).

[0032] Figure 7 is an exemplary diagram of object detection and privacy masking on image 700 according to an embodiment of the present disclosure. Two people, 702 and 704, are detected in image 700. Table 708 provides various detected dimensions of people 702 and 704 in the first and second rows, respectively. Referring to Table 708, person 702 is detected in image 700 from pixel 1550 on the X axis and pixel 472 on the Y axis to pixel 1570 on the X axis and pixel 507 on the Y axis, with a height of 35px, a width of 20px, a height-to-width ratio of 1.75, a corresponding 2D plane in pixels 504 to 509 on the Y axis of image 700, and a privacy mask flag "TRUE" (for example, person 702 has already been processed and determined to have a privacy mask applied). Furthermore, person 604 is detected in image 700 from pixel 1200 on the X axis and pixel 485 to pixel 1223 on the X axis and pixel 509 on the Y axis, with a height of 24px, a width of 23px, a height-to-width ratio of 1.0434, a corresponding 2D plane in image 700 from pixel 506 to 511 on the Y axis, and a privacy mask flag "?". It will be understood that there may be one or more other persons (not shown in image 700 for simplification) that are detected in image 700 but not in the 2D plane 706 with person 702.

[0033] The detected individuals 702 and 704 may be processed one after another in a continuous sequence to determine each individual's privacy mask flag. For example, it is checked whether the privacy mask flag for individual 702 is "?". If not (for example, the privacy mask flag is "TRUE" or "FALSE", which means that the detected individual associated with the privacy mask flag has already been processed and it has been determined that a privacy mask should be applied or masking should be skipped), individual 702 is not processed (because a privacy mask flag that is not "?" indicates that the individual has already been processed). Otherwise, the height of individual 702 is compared to the threshold height (e.g., 25px). Since the privacy mask flag for individual 702 already indicates "TRUE", processing for individual 702 is skipped, and individual 704 is processed next. As shown in Table 708, the height of individual 704 is 24px, which is less than the threshold height. Therefore, the height-to-width ratio of individual 704 is also compared to the threshold ratio (e.g., 1.4). Since the ratio of person 704, 1.0434, is also smaller than the threshold ratio, a search is performed for one or more people in the same 2D plane as person 704 (for example, the corresponding 2D plane within pixels 506 to 511 on the Y axis of image 700). The one or more people found in the search are further compared to determine the tallest person among them, and this tallest person is used as a criterion for further comparison with person 704 (for example, to determine whether the privacy mask flag for person 704 should be updated to "TRUE" or "FALSE"). Since person 702 is the only person in the same 2D plane 706 as person 704 in image 700, the search returns person 702 as a search result.

[0034] Since the privacy mask flag for person 702 is "TRUE" (for example, it has already been determined that a privacy mask should be applied to person 702), the privacy mask flag for person 704 is also updated to "TRUE" to be the same as person 702 (person 702 is the only person detected in the same 2D plane 706 as person 704) (for example, a privacy mask should be applied to person 704).

[0035] Figure 8 is an illustrative diagram of object detection and privacy masking on image 800 according to an embodiment of the present disclosure. Two people, 802 and 804, are detected in image 800. Table 808 provides various detected dimensions of people 802 and 804 in the first and second rows, respectively. Referring to Table 808, person 802 is detected in image 800 from pixel 300 on the X axis and pixel 250 on the Y axis to pixel 324 on the X axis and pixel 274 on the Y axis, with a height of 24px, a width of 24px, a height-to-width ratio of 1.0, a corresponding 2D plane 806 in pixels 271 to 276 on the Y axis of image 800, and a privacy mask flag "?". Furthermore, person 804 is detected in image 800 from pixel 450 on the X axis and pixel 248 on the Y axis to pixel 473 on the X axis and pixel 271 on the Y axis, with a height of 23px, a width of 23px, a height-to-width ratio of 1.0, a corresponding 2D plane in image 800 from pixel 268 to 273 on the Y axis, and a privacy mask flag "?". It will be understood that there may be one or more other persons (not shown in image 800 for simplification) that are detected in image 800 but not in the 2D plane 806 with person 802.

[0036] The detected individuals 802 and 804 may be processed one after another in a sequential manner to determine each individual's privacy mask flag. For example, it is checked whether the privacy mask flag for individual 802 is "?". If not (for example, the privacy mask flag is "TRUE" or "FALSE", which means that the detected individual associated with the privacy mask flag has already been processed and it has been determined that a privacy mask should be applied or masking should be skipped), individual 802 is not processed (because a privacy mask flag that is not "?" indicates that the individual has already been processed). Otherwise, the height of individual 802 is compared to the threshold height (e.g., 25px). As shown in Table 808, the height of individual 802 is 24px, which is less than the threshold height. Therefore, the height-to-width ratio of individual 802 is also compared to the threshold ratio (e.g., 1.4). Since person 802's ratio of 1.0 is also smaller than the threshold ratio, a search is performed for one or more people in the same 2D plane as person 802 (for example, the corresponding 2D plane within pixels 271 to 276 on the Y axis of image 800). The one or more people found in the search are further compared to determine the tallest person among them, and this tallest person is used as a criterion for further comparison with person 802 (for example, to determine whether person 802's privacy mask flag should be updated to "TRUE" or "FALSE"). Since person 804 is the only person in the same 2D plane 806 as person 802 in image 800, the search returns person 804 as a search result.

[0037] Since the privacy mask flag for person 804 is "?" (for example, it has not yet been determined whether a privacy mask should be applied to person 804), the height of person 804 is checked. According to Table 808, the height of person 804 is 23px, which is less than the threshold height of 25px. Therefore, a search is performed for one or more people in the same 2D plane as person 804 (for example, the corresponding 2D plane within pixels 268 to 273 on the Y axis of image 800). The one or more people found in the search are further compared to determine the tallest person among them, and the tallest person is used as the criterion for further comparison with person 804 (for example, to determine whether the privacy mask flag for person 804 should be updated to "TRUE" or "FALSE"). Since person 802 is the only person in the same 2D plane as person 804 in image 800, the search returns person 802 as a search result. Since there are no other individuals available for comparison, the privacy mask flag for person 804 is updated to the default value of "TRUE" (e.g., a privacy mask should be applied to person 804). Similarly, the privacy mask flag for person 802 is also updated to "TRUE" to match person 804 (the only person 804 detected in the same 2D plane 806 as person 802) (e.g., a privacy mask should be applied to person 802). It should be understood that the default value may be set to "FALSE" (e.g., no privacy mask is needed) depending on the application.

[0038] Figure 9 is an illustrative diagram of object detection and privacy masking on image 900 according to an embodiment of the present disclosure. Person 902 is detected in image 900. Table 906 provides various detected dimensions of person 902. Referring to Table 906, person 902 is detected in image 900 from pixel 1200 on the X axis and pixel 485 to pixel 1223 on the X axis and pixel 509 on the Y axis, with a height of 24px, a width of 23px, a height-to-width ratio of 1.0434, a corresponding 2D plane 904 in image 900 from pixel 506 to 511 on the Y axis, and a privacy mask flag "?". It will be understood that there may be one or more other persons (not shown in image 900 for simplification) that are detected in image 900 but not in the 2D plane 904 with person 902.

[0039] The detected person 902 may be processed one after another in a continuous sequence (for example, together with one or more other people in image 900 but not shown for simplification) to determine the privacy mask flag for each person. For example, it is checked whether the privacy mask flag for person 902 is "?". If not (for example, the privacy mask flag is "TRUE" or "FALSE", which means that the detected person associated with the privacy mask flag has already been processed and it has been determined that a privacy mask should be applied or masking should be skipped), person 902 is not processed (because a privacy mask flag that is not "?" indicates that the person has already been processed). Otherwise, the height of person 902 is compared to the threshold height (e.g., 25px). As shown in Table 906, the height of person 902 is 24px, which is less than the threshold height. Therefore, the height-to-width ratio of person 902 is also compared to the threshold ratio (e.g., 1.4). Since the ratio of person 902, 1.0434, is also smaller than the threshold ratio, a search is performed for one or more people within the same 2D plane as person 902 (for example, the corresponding 2D plane 904 within pixels 506 to 511 on the Y axis of image 900). The one or more people found in the search are further compared to determine the tallest person among them, and this tallest person is used as a criterion for further comparison with person 902 (for example, to determine whether the privacy mask flag for person 902 should be updated to "TRUE" or "FALSE"). Since no other people exist within the same 2D plane 904 in image 900, the privacy mask flag for person 902 is also updated to the default value of "TRUE" (for example, the privacy mask should be applied to person 902).

[0040] Figure 10 is a flowchart 1000 of object detection and privacy masking according to various embodiments of the present disclosure. In step 1002, a video file can be uploaded to, for example, an apparatus or system for applying masks to objects in an image, according to various embodiments of the present disclosure. In step 1004, the video file can be decoded into a plurality of video frames, each video frame being an image in which one or more objects may be detected, and masks may or may not be applied to the detected objects. In step 1006, it is determined whether the plurality of video frames contains more video frames, for example, new video frames different from previously processed video frames. If it is determined that the plurality of video frames does not contain more video frames, the process proceeds to step 1034, where one or more privacy masks are applied to the plurality of video frames according to the results obtained from previously processed video frames. For example, the results for one or more detected persons and their corresponding privacy mask flags may be obtained from previously processed video frames and stored, for example, in a database, and these results may be reused when the same video frame is encountered. This advantageously saves processing time and resources that would otherwise be required for the plurality of video frames. Subsequently, in step 1036, multiple processed video frames (for example, having a privacy mask that may have been applied in step 1034) are encoded into a video file and generated as output, and the process terminates.

[0041] If a new video frame is determined in step 1006, the process proceeds to step 1008 instead, where object detection is performed on each of the multiple video frames. In step 1010, it is determined whether there are any detected persons in the video frame that have a privacy mask flag with an initial value of "?" (for example, indicating that the detected person has not been processed to determine whether a privacy mask should be applied). If it is determined that no persons with a privacy mask flag with an initial value of "?" have been detected, the process returns to step 1006. Otherwise, the process proceeds to step 1012, where it is determined whether the height of the detected person is less than a threshold height. If the height is greater than or equal to the threshold height, the process proceeds to step 1030, where the privacy mask flag of the detected person is set to "TRUE". In step 1032, the identifier (ID) of the video frame being processed, as well as the processing and detection results (for example, the results for all detected persons in the video frame and their corresponding privacy mask flags), are stored in a database, which can be retrieved for use, for example, during the processing of other video frames in step 1010. The process then proceeds to steps 1032 and 1034, and thus terminates.

[0042] If the height is determined to be less than the threshold height in step 1012, the process proceeds to step 1014 instead, where it is determined whether the height-to-width ratio of the detected person is less than the threshold ratio. If the height-to-width ratio of the detected person is determined to be greater than or equal to the threshold ratio, the process proceeds to step 1026, where the privacy mask flag for the detected person is set to "FALSE" (for example, no privacy mask is applied to the detected person). The process then proceeds to steps 1032, 1034, and 1036 as described above, and then the process terminates. If the height-to-width ratio of the detected person is determined to be less than the threshold ratio in step 1014, the process proceeds to step 1016, where the Y-coordinate search range is calculated to determine the 2D plane corresponding to the detected person.

[0043] In step 1018, it is determined whether the tallest person within the Y-coordinate search range (for example, the tallest person among one or more other people in the same 2D plane as the detected person) has been found. If no such person is found (for example, no other people are in the same 2D plane as the detected person), the process proceeds to step 1028, where the privacy mask flag for the detected person is set to the default value of "TRUE," and then proceeds to steps 1032, 1034, and 1036 as described earlier, after which the process stops. Otherwise, the process proceeds from step 1018 to step 1020, where the privacy mask flag for the tallest person is checked. If the privacy mask flag is checked to be "?", the process proceeds to step 1022, where it is determined whether the privacy mask flag for the tallest person should be "TRUE" or "FALSE" (for example, whether a privacy mask should be applied to the person based on a comparison of the person's height with a threshold height), and then proceeds to step 1024, where the privacy mask flag is updated according to the determination in step 1022. On the other hand, if step 1020 checks that the privacy mask flag is not "?", step 1024 is skipped. After step 1024, the process proceeds to steps 1030, 1032, 1034, and 1036 as described above, and then the process terminates.

[0044] Figure 11 shows an exemplary image of the resulting image 1100 after an object has been detected and a privacy mask has been applied. For example, person 1102 may have a height below the threshold height and a height-to-width ratio below the threshold ratio. This allows for a Y-coordinate search (e.g., searching for the person with the highest height among one or more other people in the same 2D plane 1104 as person 1102), from which person 1110 is discovered. Since the height of the detected person 1110 is above the threshold height, the corresponding privacy mask flag for person 1110 is set to "TRUE" (e.g., the privacy mask flag is applied to person 1110 to protect their identity). Also, because the corresponding privacy mask flag for person 1110 is set to "TRUE", the privacy mask flag for person 1102 is also set to "TRUE" (e.g., privacy masking is required for person 1102). Furthermore, since the heights of detected individuals 1106, 1108, and 1112 are above the threshold height, the corresponding privacy mask flags for these individuals are set to "TRUE" (for example, the privacy mask flag is applied to each of these individuals to protect their identity). Additionally, since the heights of detected individuals 1114 and 1116 are below the threshold height, no privacy mask is applied to them.

[0045] Figure 12 shows an exemplary computing device 1200, which is referred to below as interchangeable with the computer system 1200, and which can be used to perform the methods of Figures 5 and 10. The exemplary computing device 1200 can be used to implement an apparatus for applying a mask to an object in an image. In one embodiment, the system for applying a mask to an object in an image may comprise an apparatus and one or more image and video capture devices configured to capture one or more images or video frames, where one or more images or video frames include images of people. The following description of the computing device 1200 is provided only as an example and is not intended to limit it.

[0046] As shown in Figure 12, the exemplary computing device 1200 includes a processor 1204 for executing software routines. Although a single processor is shown for clarity, the computing device 1200 may also include a multiprocessor system. The processor 1204 is connected to a communication infrastructure 1206 for communicating with other components of the computing device 1200. The communication infrastructure 1206 may include, for example, a communication bus, a crossbar, or a network.

[0047] The computing device 1200 further includes main memory 1208, such as Random Access Memory (RAM), and secondary memory 1210. The secondary memory 1210 may include a storage drive 1212, which may be a hard disk drive, a solid-state drive, or a hybrid drive, and / or a removable storage drive 1214, which may include a magnetic tape drive, an optical disc drive, a solid-state storage drive (e.g., a USB flash drive, a flash memory device, a solid-state drive, or a memory card). The removable storage drive 1214 reads from and / or writes to a removable storage medium 1218 in a well-known manner. The removable storage medium 1218 may include a magnetic tape, an optical disc, a non-volatile memory storage medium, etc., which is read from and written to by the removable storage drive 1214. As one or more persons skilled in the art will understand, the removable storage medium 1618 includes a computer-readable storage medium storing computer executable program code instructions and / or data.

[0048] In alternative embodiments, the secondary memory 1210 may further or alternatively include other similar means for enabling computer programs or other instructions to be loaded into the computing device 1200. Such means may include, for example, a removable storage unit 1222 and interface 1220. Examples of removable storage units 1222 and interface 1220 include program cartridges and cartridge interfaces (such as those found in video game console devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, removable solid-state storage drives (such as USB flash drives, flash memory devices, solid-state drives, or memory cards), and other removable storage units 1222 and interface 1220 that enable the transfer of software and data from the removable storage unit 1222 to the computer system 1200.

[0049] The computing device 1200 also includes at least one communication interface 1224. The communication interface 1224 enables software and data to be transferred between the computing device 1200 and an external device via a communication path 1226. In various embodiments of the present invention, the communication interface 1224 enables data to be transferred between the computing device 1200 and a data communication network, such as a public data or private data communication network. The communication interface 1224 may also be used to exchange data between different computing devices 1200, such computing devices 1200 forming part of an interconnected computer network. Examples of the communication interface 1224 include a modem, a network interface (such as an Ethernet card), a communication port (such as serial, parallel, printer, GPIB, IEEE1394, RJ45, USB), an antenna with associated circuitry, and the like. The communication interface 1224 may be wired or wireless. The software and data transferred via the communication interface 1224 may be in the form of signals, which may be electronic signals, electromagnetic signals, optical signals, or other signals that can be received by the communication interface 1224. These signals are provided to the communication interface via the communication path 1226.

[0050] As shown in Figure 12, the computing device 1200 further includes a display interface 1202 for performing operations to render images or videos to an associated display 1230, and an audio interface 1232 for performing operations to play audio content through one or more associated speakers 1234.

[0051] As used herein, the term “computer program product” may in part refer to the removable storage medium 1218, the removable storage unit 1222, the hard disk installed in the storage drive 1212, or the carrier wave that carries the software to the communication interface 1224 via the communication path 1226 (wireless link or cable). Computer-readable storage medium refers to any non-temporary, non-volatile, tangible storage medium that provides recorded instructions and / or data to the computing device 1200 for execution and / or processing. Examples of such storage mediums include magnetic tape, CD-ROM, DVD, Blu-ray disc, hard disk drive, ROM or integrated circuit, solid-state storage drive (such as USB flash drive, flash memory device, solid-state drive, or memory card), hybrid drive, magneto-optical disk, or computer-readable card such as a PCMCIA card, whether such device is inside or outside the computing device 1200. Examples of temporary or intangible computer-readable transmission media that may also be involved in providing software, application programs, instructions, and / or data to the computing device 1200 include wireless or infrared transmission channels, as well as network connections to other computers or networked devices, and the Internet or intranets, including email transmissions and information recorded on websites.

[0052] The computer program (also referred to as computer program code) is stored in main memory 1208 and / or secondary memory 1210. The computer program may also be received via the communication interface 1224. When such a computer program is executed, it enables the computing device 1200 to implement one or more features of the embodiments discussed herein. In various embodiments, when the computer program is executed, it enables the processor 1204 to implement the features of the embodiments described above. Thus, such a computer program represents the controller of the computer system 1200.

[0053] The software is stored in a computer program product and can be loaded onto the computing device 1200 using the removable storage drive 1214, storage drive 1212, or interface 1220. The computer program product may be a non-temporary computer-readable medium. Alternatively, the computer program product may be downloaded to the computer system 1200 via the communication path 1226. Once executed by the processor 1204, the software causes the computing device 1200 to perform the actions necessary to carry out the methods shown in Figures 5 and 10.

[0054] It should be understood that the embodiment shown in Figure 12 is presented merely as an example to illustrate the operation and structure of a device for applying a mask to an object in an image. Therefore, in some embodiments, one or more features of the computing device 1200 may be omitted. Also, in some embodiments, one or more features of the computing device 1200 may be combined together. Furthermore, in some embodiments, one or more features of the computing device 1200 may be divided into one or more components.

[0055] Those skilled in the art will understand that numerous variations and / or modifications can be made to the invention as shown in specific embodiments without departing from the spirit or scope of the invention as broadly described. Therefore, these embodiments should be considered illustrative and not restrictive in all respects.

[0056] Each drawing or figure is merely an example to illustrate one or more embodiments. Each figure does not have to be associated with only one specific embodiment, but may be associated with one or more other embodiments. As those skilled in the art will understand, various features or steps described with reference to any one of the figures can be combined with features or steps shown in one or more other figures to create embodiments that are not explicitly illustrated or described. Not all features or steps shown in any one of the figures to illustrate an embodiment are necessarily essential, and some features or steps may be omitted. The order of steps described in any of the figures may be changed as necessary.

[0057] All or part of the embodiments disclosed above may be described as follows, but are not limited to these. (Note 1) A method for applying a mask to an object in an image, The processor determines whether to apply a mask to an object in the image based on a comparison with another object in the image, A method that includes applying a mask to objects in an image based on a determination by a processor. (Note 2) Determining whether or not to apply a mask is Comparing the height of an object in an image to a threshold height, If the object's height is lower than the threshold height, the ratio of the object's height to its width in the image is further compared to the threshold ratio. The method according to Appendix 1, which includes determining whether to apply a mask based on further comparison. (Note 3) If the object's height is greater than or equal to a threshold height, this further includes applying a mask to the object. The method described in Appendix 2. (Note 4) A comparison with another object, Identifying another object from one or more other objects in an image when the ratio is lower than the threshold ratio, wherein the other object has the highest height among the object and one or more other objects, and one or more other objects are located within a two-dimensional plane distance from the object. The method described in Appendix 2, which includes determining whether a mask should be applied to another object. (Note 5) The method according to Appendix 4, further including applying the mask to an object if it is determined that the mask should be applied to another object. (Note 6) The method described in Appendix 2, further comprising determining whether to apply the mask to the object if the ratio is higher than a threshold ratio. (Note 7) The method according to any one of the appendices 1 to 6, further comprising applying a mask to an object in multiple frames of a video, wherein the image is a frame from the multiple frames of the video. (Note 8) A device for applying a mask to an object in an image, At least one processor, A memory containing computer program code, wherein the memory and the computer program code are used by at least one processor to provide the device with at least one The decision of whether to apply a mask to an object in an image is based on a comparison with another object in the image, A device comprising at least one memory and a function that performs the following actions: applying a mask to an object in an image based on a determination. (Note 9) Determining whether or not to apply a mask is Comparing the height of an object in an image to a threshold height, If the object's height is lower than the threshold height, the ratio of the object's height to its width in the image is further compared to the threshold ratio. The apparatus as described in Appendix 8, including determining whether to apply a mask based on further comparison. (Note 10) It is further configured to apply a mask to an object if the object's height is greater than or equal to a threshold height. The apparatus described in Appendix 9. (Note 11) A comparison with another object, Identifying another object from one or more other objects in an image when the ratio is lower than the threshold ratio, wherein the other object has the highest height among the object and one or more other objects, and one or more other objects are located within a two-dimensional plane distance from the object. The apparatus described in Appendix 9, which includes determining whether a mask should be applied to another object. (Note 12) The apparatus described in Appendix 11, further configured to apply a mask to an object if it is determined that the mask should be applied to another object. (Note 13) The apparatus described in Appendix 9, further configured to determine whether to apply the mask to an object if the ratio is higher than a threshold ratio. (Note 14) The apparatus described in any one of appendices 8 to 13, further configured to apply a mask to an object in multiple frames of a video, wherein the image is a frame from among the multiple frames of the video. (Note 15) A system for applying a mask to an object in an image, comprising: an apparatus described in any one of appendices 8 to 14; and one or more image and video capture devices configured to capture one or more images or video frames, wherein one or more images or video frames include images of a person, and the person is the object. (Note 16) A program for applying a mask to an object in an image, which allows a computer to The decision of whether to apply a mask to an object in an image is based on a comparison with another object in the image, A program that applies a mask to objects in an image based on a determination. (Note 17) Determining whether or not to apply a mask is Comparing the height of an object in an image to a threshold height, If the object's height is lower than the threshold height, the ratio of the object's height to its width in the image is further compared to the threshold ratio. The program described in Appendix 16, which includes determining whether to apply a mask based on further comparison. (Note 18) If the object's height is greater than or equal to a threshold height, this further includes applying a mask to the object. The program described in Appendix 17. (Note 19) A comparison with another object, Identifying another object from one or more other objects in an image when the ratio is lower than the threshold ratio, wherein the other object has the highest height among the object and one or more other objects, and one or more other objects are located within a two-dimensional plane distance from the object. The program described in Appendix 17, which includes determining whether a mask should be applied to another object. (Note 20) The program described in Appendix 19 further includes applying the mask to another object if it is determined that the mask should be applied to that object. (Note 21) The program described in Appendix 2 further includes determining whether to apply the mask to an object if the ratio is higher than a threshold ratio. (Note 22) A program as described in any one of the appendices 1 to 6, further comprising applying a mask to an object in multiple frames of a video, wherein the image is a frame from the multiple frames of the video.

[0058] Some or all of the elements specified in any of the appendices may apply to various types of hardware, software, and recording means for recording software, systems, and methods.

[0059] This application is based on and claims priority to Singapore Patent Application No. 10202300828P, filed on 27 March 2023, the disclosure thereof being incorporated herein by reference in its entirety. [Explanation of Symbols]

[0060] 102 Input Images 104 images 106 images 200 images 302 Description 400 descriptions 402 People 404 People 406 2D Plane Distance 500 ways 502 steps 504 steps 600 images 608 table 700 images 708 table 800 images 808 table 900 images 906 table

Claims

1. A method for applying a mask to an object in an image, The processor determines whether to apply a mask to an object in the image based on a comparison with another object in the image, A method comprising applying the mask to the object in the image based on the determination by the processor.

2. Determining whether to apply the aforementioned mask is The height of the object in the aforementioned image is compared with a threshold height, If the height of the object is lower than the threshold height, the ratio of the height to the width of the object in the image is further compared with the threshold ratio. Based on the aforementioned further comparison, a determination is made as to whether to apply the mask, The method according to claim 1, including the method described in claim 1.

3. If the height of the object is greater than or equal to the threshold height, the further includes applying the mask to the object. The method according to claim 2.

4. The comparison with the aforementioned other object, If the ratio is lower than the threshold ratio, the identification of the other object from one or more other objects in the image is such that the other object has the highest height among the object and the one or more other objects, and the one or more other objects are located within a two-dimensional plane distance from the object. Determining whether a mask should be applied to the aforementioned other object, The method according to claim 2, including the method described in claim 2.

5. The method according to claim 4, further comprising applying the mask to the other object if it is determined that the mask should be applied to the other object.

6. The method according to claim 2, further comprising determining that the mask is not applied to the object if the ratio is higher than the threshold ratio.

7. The method according to any one of claims 1 to 6, further comprising applying the mask to the object in multiple frames of a video, wherein the image is a frame among the multiple frames of the video.

8. A device for applying a mask to an object in an image, At least one processor, A memory containing computer program code, wherein the at least one memory and the computer program code are used by at least one processor to provide the device with at least, The determination of whether to apply a mask to an object in an image is made based on a comparison with another object in the same image, Applying the mask to the object in the image based on the determination, A device having at least one memory and for executing [the function].

9. Determining whether to apply the aforementioned mask is The height of the object in the aforementioned image is compared with a threshold height, If the height of the object is lower than the threshold height, the ratio of the height to the width of the object in the image is further compared with the threshold ratio. Based on the aforementioned further comparison, a determination is made as to whether to apply the mask, The apparatus according to claim 8, including the apparatus described in claim 8.

10. If the height of the object is greater than or equal to the threshold height, the following configuration is further configured to apply the mask to the object: The apparatus according to claim 9.

11. The comparison with the aforementioned other object, If the ratio is lower than the threshold ratio, the identification of the other object from one or more other objects in the image is such that the other object has the highest height among the object and the one or more other objects, and the one or more other objects are located within a two-dimensional plane distance from the object. Determining whether a mask should be applied to the aforementioned other object, The apparatus according to claim 9, including

12. The apparatus according to claim 11, further configured to apply the mask to the other object if it is determined that the mask should be applied to the other object.

13. The apparatus according to claim 9, further configured to determine whether the mask should not be applied to the object if the ratio is higher than the threshold ratio.

14. The apparatus according to any one of claims 8 to 13, further configured to apply the mask to the object in multiple frames of a video, wherein the image is a frame among the multiple frames of the video.

15. A system for applying a mask to an object in an image, comprising: an apparatus according to any one of claims 8 to 14; and one or more image and video capture devices configured to capture one or more images or video frames, wherein the one or more images or video frames include images of a person, and the person is the object.

16. A program for applying a mask to an object in an image, which allows a computer to The determination of whether to apply a mask to an object in an image is made based on a comparison with another object in the same image, Applying the mask to the object in the image based on the determination, A program that executes something.

17. Determining whether to apply the aforementioned mask is The height of the object in the aforementioned image is compared with a threshold height, If the height of the object is lower than the threshold height, the ratio of the height to the width of the object in the image is further compared with the threshold ratio. The program according to claim 16, comprising determining whether to apply the mask based on the further comparison.

18. If the height of the object is greater than or equal to the threshold height, the further includes applying the mask to the object. The program according to claim 17.

19. The comparison with the aforementioned other object, If the ratio is lower than the threshold ratio, the identification of the other object from one or more other objects in the image is such that the other object has the highest height among the object and the one or more other objects, and the one or more other objects are located within a two-dimensional plane distance from the object. The program according to claim 17, comprising determining whether a mask should be applied to the other object.

20. The program according to claim 19, further comprising applying the mask to the other object if it is determined that the mask should be applied to the other object.

Images