Content-based image privacy control

Abstract

The invention relates to a method and device for image processing. An example method includes receiving a request to access an image from an application, wherein the application is enabled to access one or more categories of image content; processing the image to modify one or more portions of the image that include content that is not within the one or more categories; and providing the processed image to the application.

Description

Background Art Technical Field

[0001] The present disclosure relates generally to processing systems and, more particularly, to one or more techniques for image processing and privacy control.

[0002] Related technologies

[0003] Many computing devices include one or more image sensors (e.g., a charge coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor, or other camera sensor) for capturing still images and / or video. Examples of such computing devices include dedicated digital cameras, tablet computers, laptop computers, smartphones, other mobile devices, wearable devices, IoT devices, automotive devices, etc.

[0004] Applications running on a computing device (e.g., a camera application, a video calling application, etc.) can utilize the image sensor of the computing device to capture one or more images. For privacy purposes, in order to operate the image sensor, the application may first need to request permission to access the image sensor from the user of the computing device. For example, the user can be prompted through a dialog on the computing device to "always allow", "allow when in use", or "deny" image sensor permission for the application. Explicit permissions requested from the user help prevent applications from gaining access to private data without the user's consent, and provide the user with additional context and visibility into the type of permission the application is seeking or has been granted. Summary of the invention

[0005] A simplified summary of one or more aspects is presented below in order to provide a basic understanding of these aspects. This summary is not an extensive overview of all contemplated aspects, and is neither intended to identify key elements of all aspects nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description presented later.

[0006] In certain aspects, a method for image processing is disclosed. The method includes: receiving a request from an application to access an image, wherein the application is enabled to access one or more categories of image content; processing the image to modify one or more portions of the image including content that is not within the one or more categories; and providing the processed image to the application.

[0007] Other aspects provide: an apparatus operable to, configured to, or otherwise adapted to perform the aforementioned method and those described elsewhere herein; a non-transitory computer-readable medium comprising instructions that, when executed by a processor of the apparatus, cause the apparatus to perform the aforementioned method and those described elsewhere herein; a computer program product embodied on a computer-readable storage medium, the computer-readable storage medium comprising code for performing the aforementioned method and those described elsewhere herein; and an apparatus comprising means for performing the aforementioned method and those described elsewhere herein. As an example, an apparatus may include a computing device, a processing system, etc.

[0008] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will become apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram illustrating an example computing device including an image sensor in accordance with certain aspects of the present disclosure.

[0010] Figure 2 is an example of some aspects of the present disclosure Figure 1 A block diagram of an example software stack for a computing device.

[0011] Figure 3 is a call flow diagram illustrating example image processing according to certain aspects of the present disclosure.

[0012] Figure 4 Example methods for content-based image processing in accordance with certain aspects of the present disclosure are illustrated.

[0013] Figure 5 Example image processing for an application requesting permission to access an image with Quick Response (QR) code content in accordance with certain aspects of the present disclosure is illustrated.

[0014] Figure 6 is a flow chart illustrating example operations for image processing according to certain aspects of the present disclosure. DETAILED DESCRIPTION

[0015] The present disclosure provides techniques for image processing, and more particularly, provides image processing based on content that is requested and authorized for use by an application.

[0016] In some aspects, an application running on a computing device (e.g., a camera application, a video calling application, a camera scanner application, etc.) can utilize an image sensor of the computing device to capture one or more images. However, in conventional designs, an application can utilize the image sensor only if a runtime permission to use the image sensor is granted to the application.

[0017] Permissions are used to protect the privacy of the user. Runtime permissions cover areas where applications running on the computing device seek data or resources that involve the user's private information or may potentially affect the user's stored data or the operation of other applications running on the computing device. For example, the ability of an application to access a feature (such as an image sensor of a computing device) may first require the application to request permission for the application to use the image sensor at runtime and require the user of the device to grant such permission. Only after such permission is granted can the application issue an image sensor open call to an API of an application programming interface (API) framework to capture one or more images using the image sensor of the computing device. Granting permission to an application for use of an image sensor can be unlimited, as the application can have access to all content captured by the image sensor.

[0018] While runtime permissions help prevent applications from gaining access to features of a computing device without user consent, such unlimited permissions granted may be excessive. In practice, an application may be seeking to use an image sensor for a specific purpose. For example, a plant identification application may request access to an image sensor of a computing device for capturing images of plants for identification (e.g., obtaining an image with plants), while a payment application may request access to an image sensor for scanning a QR code for payment (e.g., obtaining an image with the content of a QR code). Regardless of the purpose for which these applications utilize the image sensor, each application that is granted permission to use the image sensor may have access to all content contained within an image captured by the image sensor, including redundant content that may, in some cases, include private and / or sensitive information of a user of the computing device.

[0019] Thus, certain aspects herein provide techniques for image processing, and more specifically, image processing based on content that is requested and authorized for use by an application. The image processing described herein may include detection of content in an image requested by an application, as well as techniques for de-emphasis and / or removal of other redundant information present in the image (e.g., information not requested by the application). Removal of non-essential information may allow improved privacy control by eliminating such non-essential information in the image before providing the image to one or more applications for use. The improved user privacy protection provided by the techniques described herein may help improve the overall user experience when using a computing device.

[0020] The various aspects of the system, device, computer program product and method will be described more fully below for image processing. However, the present disclosure can be embodied in many different forms and should not be interpreted as being limited to any specific structure or function presented throughout the present disclosure. On the contrary, these aspects are provided so that the present disclosure will be thorough and complete, and the scope of the present disclosure will be fully conveyed to those skilled in the art. Based on the teachings of this article, it should be understood by those skilled in the art that the scope of the present disclosure is intended to cover any aspect of the system, device, computer program product and method disclosed herein, whether it is implemented independently of other aspects of the present disclosure or implemented in combination with other aspects of the present disclosure. For example, any number of aspects set forth herein can be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods, which are practiced using other structures, functionality or structures and functionality as a supplement to the various aspects of the present disclosure set forth herein or in addition to the various aspects of the present disclosure set forth herein. Any aspect disclosed herein can be embodied by one or more elements of the claims.

[0021] Although various aspects are described herein, many variations and permutations of these aspects fall within the scope of the present disclosure. Although some potential benefits and advantages of various aspects of the present disclosure are mentioned, the scope of the present disclosure is not intended to be limited to specific benefits, uses or objectives. Specifically, various aspects of the present disclosure are intended to be broadly applicable to different graphics technologies, system configurations, etc., some of which are illustrated by way of example in the drawings and the description below. The specific embodiments and drawings merely illustrate the present disclosure and do not limit the present disclosure. The scope of the present disclosure is defined by the attached claims and their equivalent technical solutions.

[0022] Several aspects are presented with reference to various devices and methods. These devices and methods are described in the following detailed description and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively referred to as "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether these elements are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0023] For example, an element or any part of an element or any combination of elements may be implemented as a "processing system" including one or more processors (which may also be referred to as a processing unit). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), general-purpose GPUs, central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on chip (SOCs), baseband processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gating logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described in this disclosure. One or more processors in a processing system may execute software. Software may be broadly interpreted as instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software components, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, processes, functions, etc., whether referring to software, firmware, middleware, microcode, hardware description languages, or other names. The term "application" may refer to software. As described herein, one or more technologies may refer to applications configured to perform one or more functions, i.e., software. In such examples, the application may be stored in a memory (e.g., an on-chip memory of a processor, a system memory, or any other memory). The hardware described herein (such as a processor) may be configured to execute the application. For example, an application may be described as including code that, when executed by hardware, causes the hardware to perform one or more techniques described herein. As an example, the hardware may access code from a memory and execute the code accessed from the memory to perform one or more techniques described herein. In some examples, components are identified in the present disclosure. In such examples, the components may be hardware, software, or a combination thereof. Each component may be a separate component or a subcomponent of a single component.

[0024] Therefore, in one or more examples described herein, the functions described can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored or encoded on a computer-readable medium as one or more instructions or codes. Computer-readable media include computer storage media. Storage media can be any available media that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, a combination of computer-readable media of the above types, or any other medium that can be used to store computer executable code in the form of instructions or data structures that can be accessed by a computer.

[0025] Figure 11 is a block diagram illustrating an example computing device 104 configured to implement one or more techniques of the present disclosure. The computing device 104 may include one or more components or circuits for performing the various functions described herein. In some examples, one or more components of the computing device 104 may be components of a system on a chip (SOC) or an integrated circuit (IC). The computing device 104 may include one or more components configured to perform one or more techniques of the present disclosure. In the example shown, the device 104 may include a GPU 120, a CPU 122, a system memory 124, and one or more image sensors 134 (e.g., a CCD sensor, a CMOS sensor, etc.). In some aspects, the computing device 104 may include multiple optional components, such as a communication interface 126, a transceiver 132 (e.g., including a receiver 138 and a transmitter 136), and one or more displays 131. In some aspects, these components are communicatively connected to each other via one or more buses 150. It should be noted that although in Figure 1 1 and 2. Although some connections between components are shown in FIG. 1 , the connections between components may be different from those shown. For example, all components may be coupled to a shared bus 150, different components may be coupled to different buses, and so on.

[0026] References to display 131 may refer to one or more displays 131. For example, display 131 may include a single display or multiple displays. GPU 120 may be configured to perform graphics processing. One or more displays 131 may be configured to display or otherwise present frames processed by GPU 120 and / or CPU 122. In some examples, one or more displays 131 may include one or more of the following: a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, a projection display device, an augmented reality display device, a virtual reality display device, a head mounted display, or any other type of display device.

[0027] Memory external to GPU 120 and CPU 122, such as system memory 124 (described in more detail below), may be accessed by GPU 120 and CPU 122. For example, GPU 120 and CPU 122 may be configured to read from and / or write to external memory, such as system memory 124. GPU 120 and CPU 122 may be communicatively coupled to system memory 124 via a bus. In some examples, GPU 120 and CPU 122 may be communicatively coupled to each other via a bus or different connections.

[0028] In some aspects, CPU 122 and / or GPU 120 may be configured to receive images (e.g., still images, videos, etc.) from one or more image sensors 134. For example, CPU 122 and / or GPU 120 may be configured to receive images directly from one or more image sensors 134. In some aspects, one or more image sensors 134 may be configured to write images to an external memory, such as system memory 124. In some aspects, CPU 122 and / or GPU 120 may be configured to receive (e.g., read) images from system memory 124.

[0029] The GPU 120 may be a general purpose GPU or any other processing unit that may be configured to perform graphics processing. In some examples, the GPU 120 may be integrated into the motherboard of the device 104. In some examples, the GPU 120 may be present on a graphics card installed in a port in the motherboard of the device 104, or may otherwise be incorporated into a peripheral device configured to interoperate with the device 104. The GPU 120 may include one or more processors, such as one or more microprocessors, GPUs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If these techniques are partially implemented in software, the GPU 120 may store instructions for the software in a suitable non-transitory computer-readable storage medium (e.g., system memory 124), and may use one or more processors to execute instructions in hardware to perform the techniques of the present disclosure. Any of the above (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors.

[0030] CPU 122 may be any processing unit configured to transmit instructions to GPU 120 and perform general computing processing (e.g., non-graphics processing). In some examples, CPU 122 may be integrated into the motherboard of device 104. CPU 122 may include one or more processors, such as one or more microprocessors, ASICs, FPGAs, ALUs, DSPs, video processors, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If these techniques are partially implemented in software, CPU 122 may store instructions for the software in a suitable non-transitory computer-readable storage medium (e.g., system memory 124), and may use one or more processors to execute instructions in hardware to perform the techniques of the present disclosure. Any of the above (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors.

[0031] System memory 124 may include one or more volatile or nonvolatile memory or storage devices. In some examples, system memory 124 may include RAM, static RAM (SRAM), dynamic RAM (DRAM), erasable programmable ROM (EPRAM), EEPROM, flash memory, magnetic data media, or optical storage media, or any other type of memory.

[0032] According to some examples, system memory 124 may be a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagating signal. However, the term "non-transitory" should not be interpreted to mean that system memory 124 is non-removable or that its contents are static. As an example, system memory 124 may be removed from device 104 and moved to another device. As another example, system memory 124 may not be removable from device 104.

[0033] In certain aspects, the system memory 124 may include one or more content detection models 128 (e.g., individually and collectively herein referred to as content detection models 128). The content detection model 128 may be an artificial intelligence (AI) model, such as a machine learning model, capable of detecting one or more content categories in still images, videos, and the like from one or more image sensors 134. The one or more content categories may include QR codes, identification (ID) images, plants, files, people, places, pictures, and the like.

[0034] In some aspects, one content detection model 128 may be stored in the system memory 124, wherein the content detection model 128 is an AI model trained to detect multiple content categories (e.g., trained to detect QR codes, ID images, plants, files, etc. in images). In some other aspects, multiple content detection models 128 may be stored in the system memory 124, wherein each content detection model 128 is an AI model trained to detect a specific category of content (e.g., the first content detection model 128 is trained to detect QR codes in images, the second content detection model 128 is trained to detect ID images in images, the third content detection model 128 is trained to detect plants in images, etc.).

[0035] The content detection model 128 may be trained using training data, which may include data associated with one or more images. More specifically, data associated with one or more images may be used as input (e.g., training data) for training a model to predict (e.g., detect) coordinates of at least one content category within the one or more images.

[0036] In some aspects, the content detection model 128 may be trained only at the cloud 142, where the cloud 142 may be a public cloud or a private cloud. Specifically, the content detection model 128 may be trained at the cloud 142 via the training module 144, and after such training, the content detection model may be deployed at the computing device 104 for detecting one or more categories of image content. In the case where the content detection model 128 is trained at the cloud 142, the training data may be stored in the storage device 146 and used by the training module 144 as input for training the content detection model 128. The storage device 146 may be implemented as any type of data storage, such as a relational database, a non-relational database, a key-value data storage, a file system including a hierarchical file system, and the like. In some aspects, the content detection model 128 may be trained only at the computing device 104 (and may not involve the cloud 142). In some aspects, the content detection model 128 may be trained (initially and / or updated) at both the computing device 104 and the cloud 142. In some aspects, the training of the content detection model 128 may be federated such that the training occurs at different computing devices 104. Each different computing device 104 may send updates to the cloud 142, which may be aggregated by the cloud 142 to update the model parameters for the content detection model 128 on each computing device 104. The cloud 142 may then send the updated model parameters for the content detection model 128 on each computing device 104 to each computing device 104.

[0037] In some aspects, the list of content categories that the content detection model 128 is capable of detecting may be stored in a file in the system memory 124 and / or in a storage device (e.g., Figure 1 The content category may change as a new content detection model 128 is created and trained to detect a content category that an existing content detection model 128 is not capable of detecting, an existing content detection model 128 is updated to detect a content category that an existing content detection model 128 is not capable of detecting, and / or an existing content detection model 128 is updated so that the existing content detection model 128 is no longer capable of detecting a previously detectable content category. An application requesting access to the image sensor 134 for obtaining an image may select a content category from a list of content categories.

[0038] Content categories may include, for example, QR codes, ID images, plants, documents, people, places, pictures, etc. For example, a payment application may request access to the image sensor 134 for scanning a QR code for payment (e.g., obtaining an image with QR code content), wherein at least one content detection model 128 is capable of detecting a portion of the image containing the QR code content. As another example, a banking application may request access to the image sensor 134 for scanning an ID image for registration (e.g., obtaining an image with ID image content), wherein at least one content detection model 128 is capable of detecting a portion of the image containing the ID image content. As another example, a plant recognition application may request access to the image sensor 134 for capturing an image of a plant for identification (e.g., obtaining an image with a plant), wherein at least one content detection model 128 is capable of detecting a portion of the image containing the plant content.

[0039] In certain aspects, computing device 104 may include an optional communication interface 126. Computing device 104 may use communication interface 126 to communicate with cloud 142 connected by network 148. Network 148 may be, for example, a direct link, a local area network (LAN), a wide area network (WAN) such as the Internet, another type of network, or any combination thereof.

[0040] The communication interface 126 may include a receiver 138 and a transmitter 136. The receiver 138 may be configured to perform any receiving functions described herein with respect to the computing device 104. Additionally, the receiver 138 may be configured to receive information from another device (or cloud 142). The transmitter 136 may be configured to perform any sending functions described herein with respect to the computing device 104. For example, the transmitter 136 may be configured to send information to another device (e.g., such as cloud 142). The receiver 138 and the transmitter 136 may be combined into a transceiver 132. In such examples, the transceiver 132 may be configured to perform any receiving functions and / or sending functions described herein with respect to the computing device 104.

[0041] The CPU 122 may be configured to run a content detection model 128. In some aspects, the CPU 122 may be configured to run a content detection model 128 to detect one or more portions of the image content that include a particular content category (e.g., detecting a portion of a still image that includes a QR code). In some aspects, the CPU 122 may be configured to run a content detection model 128 to detect one or more portions of one or more images that do not include a particular content category (e.g., detecting a portion of a still image that does not include a QR code). In some aspects, the CPU 122 may be configured to run a content detection model 128 to detect one or more portions of one or more images that include and / or do not include a particular content category. In some aspects, another processor (such as a dedicated neural processor) may be configured to run a content detection model 128 for detecting one or more portions of one or more images that include and / or do not include a particular content category.

[0042] As described in more detail below, CPU 122 can use the detected portion of one or more images that includes a specific content category to process the remaining portion of the image for privacy purposes. For example, based on detecting a QR code in a static image, in addition to the portion of the image that includes the QR code, CPU 122 can also process other portions of the image to mask (e.g., blur, remove, cover, etc.) the remaining portion of the image. In another example, CPU 122 can use the detected portion of the image that does not include a specific content category to process such portions of the image for privacy purposes. For example, based on detecting a portion that does not contain QR code content in a static image (e.g., a background area of ​​a static image and / or other images), CPU 122 can process such portions to mask the content in these portions of the image. In either example, such masking can allow the QR code to remain visible while keeping other content in the static image (e.g., invisible to an application).

[0043] In certain aspects, CPU 122 may be configured to pass the processed image to an application running on computing device 104 that requested the image. As opposed to the raw image captured by image sensor 134, transmitting the processed image to the application may provide the application with necessary information requested by the application without providing the application with non-essential and / or private information (e.g., of a user using computing device 104).

[0044] As described herein, a device such as device 104 may refer to any device, apparatus or system configured to perform one or more technologies described herein. For example, a device may be a server, a base station, a user equipment, a client device, a station, an access point, a computer (e.g., a personal computer, a desktop computer, a laptop computer, a tablet computer, a computer workstation or a mainframe computer), a final product, an apparatus, a phone, a smart phone, a server, a video game platform or console, a handheld device (e.g., a portable video game device or a personal digital assistant (PDA)), a wearable computing device (e.g., a smart watch, an augmented reality device or a virtual reality device), a non-wearable device, a display or display device, a television, a television set-top box, an intermediate network device, a digital media player, a video streaming device, a content streaming device, an onboard computer, any mobile device, any device configured to generate graphics content, or any device configured to perform one or more technologies described herein. The process herein may be described as being performed by a specific hardware component (e.g., a GPU), but in further embodiments, it may be performed using other hardware components (e.g., a CPU) consistent with the disclosed embodiments.

[0045] Figure 2 is exemplified in Figure 1 104. It should be noted that the software stack 200 may also be executed on other suitable computing devices. In addition, in some aspects, the computing device 104 may execute other similar software (including fewer or additional components) to perform the techniques discussed herein. For example, the functionality of one or more components discussed may be combined in a single component or may be divided into additional components. In some aspects, the code for the software stack 200 may be stored in a memory (such as the system memory 124 of the computing device 104). In addition, in some aspects, the software stack 200 may be executed by one or more processors (such as the CPU 122 of the computing device 104).

[0046] As shown, the software stack 200 includes one or more applications 202 (individually referred to herein as application 202 and collectively referred to herein as application 202), a framework 206 (e.g., an Android framework) including one or more modules 204 (individually referred to herein as module 204 and collectively referred to herein as module 204), a hardware abstraction layer (HAL) 208, and a kernel 210 (e.g., a Linux kernel for Android) including an image sensor driver 212. In some aspects, the one or more applications 202, the framework 206 including the one or more modules 204, and / or the HAL 208 run in user space 214, meaning that they have access to memory and / or resources defined as accessible in user space 214, which may be less than the amount of memory and / or resources defined as accessible in kernel space 216. In some aspects, the kernel 210 runs in kernel space 216.

[0047] In some aspects, one or more applications 202 may capture one or more images (e.g., still images, videos, etc.) utilizing one or more image sensors 134 of the computing device 104. For example, one or more applications 202 may include one or more of a camera application, a video calling application, and the like.

[0048] In some aspects, framework 206 is an API framework that provides APIs (e.g., a set of Java framework APIs) that expose functionality of native libraries, system components, and / or services to applications, such as application 202. For example, framework 206 provides an API to expose functionality of one or more image sensors 134 (such as camera functionality) to application 202.

[0049] In some aspects, the framework 206 includes one or more modules 204, such as (e.g. Figure 3 ) image sensor privacy control module 204(1), image sensor manager module 204(2) and image processing module 204(3).

[0050] Image sensor privacy control module 204(1) may be configured to manage application 202 regarding (e.g. Figure 1 For example, image sensor privacy control module 204(1) may (1) determine whether the content category requested by application 202 is a content category that content detection model 128 is capable of detecting, and (2) verify that the user has granted the application authorization to access this particular content category of one or more images.

[0051] Image sensor manager module 204(2) may act as a gateway between applications 202 and image sensor 134. In certain aspects, image sensor manager module 204(2) may communicate with image sensor privacy control module 204(1) to determine whether an application requesting a particular content category actually has permission to access that content category.

[0052] Image processing module 204(3) may be configured to (1) receive one or more raw images from image sensor 134, and (2) process the one or more raw images to modify portions of the one or more raw images that include content that is not within a content category that application 202 has requested and has access to. Figure 4 Described in more detail, processing the one or more original images may involve detecting portions of the one or more images that include content that is not within the requested content category or portions of the one or more original images that include content within the requested content category using the content detection model 128. In addition, processing may include masking portions of the one or more original images that do not include content within the requested content category.

[0053] The operations performed by the image sensor privacy control module 204(1), the image sensor manager module 204(2), and the image processing module 204(3) may be described below with respect to Figure 3 In addition, it should be noted that although aspects are described herein with respect to the use of these three modules, in certain other aspects, the operations performed by each of these modules may be performed by a single module or different modules implemented in software stack 200.

[0054] In some aspects, the HAL 208 provides a standard interface that exposes device hardware capabilities to the framework 206. For example, in some aspects, the HAL 208 includes library modules that implement interfaces for hardware components, such as one or more image sensors 134. In some aspects, when the framework 206 makes a call to access a hardware component, the HAL 208 loads the library module for the hardware component. For example, when the framework 206 issues an image sensor open call to access one or more image sensors 134 (e.g., for obtaining a raw image), the HAL 208 communicates with an image sensor driver 212 in the kernel 208, as discussed further herein.

[0055] In some aspects, the kernel 210 interfaces with hardware components, such as adjusting the operating mode (e.g., operating frequency and / or voltage) of one or more components (e.g., CPU 122, GPU 120, and / or bus 150). In some aspects, the kernel 2010 includes one or more drivers, each driver for interfacing with a specific hardware component. For example, the kernel 210 includes an image sensor driver 212 for interfacing with one or more image sensors 134.

[0056] In some aspects, the application 202 may issue an image sensor on call to an API of the framework 206 to obtain one or more images using an image sensor. The call may be a request to access one or more content categories contained within an image (e.g., captured using the image sensor 134 or retrieved from storage). The image sensor on call may be handled based on the one or more content categories requested by the application 202 according to the techniques discussed herein.

[0057] Figure 3 is a call flow diagram 300 illustrating example image processing according to certain aspects of the present disclosure. The call flow diagram 300 is described with respect to the software stack 200. However, it should be understood that the call flow diagram 300 or techniques similar thereto may be performed by any suitable software running on any suitable computing device.

[0058] At operation 304, application 202 transmits a request to access one or more categories of image content. For example, application 202 may issue a call to an API of framework 206 (and more specifically, to image sensor privacy control module 204(1)) to access one or more categories of image content. As an illustrative example, as previously mentioned, application 202 may be a payment application and at operation 304 may transmit a request to image sensor privacy control module 204(1) to access QR code content within an image for scanning the QR code for payment.

[0059] In some aspects, prior to operation 304, at operation 302, the image sensor privacy control module 204(1) sends a list of content categories to the application 202. The image sensor privacy control module 204(1) may send this list to the application 202 periodically, when new content categories become available, when content categories are removed from the list, etc., in response to a request from the application 202 and / or another trigger. As mentioned, the list sent to the application 202 may be a list of content categories that the content detection model 128 is capable of detecting. Thus, the application 202 may select one or more content categories from the list and provide an indication of the selected one or more content categories that the application 202 is requesting access to in the application's API call to the image sensor privacy control module 204(1).

[0060] At operation 306, based on receiving the request from the application 202, the image sensor privacy control module 204(1) determines whether the one or more content categories requested by the application 202 are valid content categories. In other words, the image sensor privacy control module 204(1) determines whether the one or more content categories are categories within the list of content categories that the content detection model 128 is capable of detecting. For example, in the case where the application 202 requests access to an image with a QR code, at operation 306, the image sensor privacy control module 204(1) checks whether the QR code content is a content category that the content detection model 128 is capable of detecting. For purposes of explanation, it may be assumed that the content categories requested by the application 202 are content categories that the content detection model 128 is capable of detecting.

[0061] Additionally, at operation 306, based on receiving the request from application 202, image sensor privacy control module 204(1) determines whether the user has granted application 202 permission to access the requested one or more categories of image content. In some cases, this determination may involve image sensor privacy control module 204(1) prompting the user to grant application 202 permission to access the requested one or more categories of image content. Prompting the user to grant permission may, for example, involve prompting the user to grant permission via a dialog (e.g., a pop-up) on computing device 104 to “always allow,” “allow while in use,” or “deny” application 202 permission for the requested one or more categories of image content.

[0062] At operation 308, application 202 transmits a request to access an image having one or more categories of image content. For example, application 202 may issue an image sensor open call to an API of framework 206 (and more specifically, to image sensor manager module 204(2)) to obtain an image having one or more categories of image content (e.g., to be captured or previously captured by image sensor 134).

[0063] Based on receiving the image sensor open call, the image sensor manager module 204(2) determines whether the application 202 has permission to access images having the requested one or more categories of image content. Thus, at operations 310 and 312, the image sensor manager module 204(2) communicates with the image sensor privacy control module 204(1) (e.g., which has knowledge of the permissions of the application 202) to determine whether the application 202 has been granted permission. For purposes of explanation, it may be assumed that the application 202 is authorized to access images having the requested one or more categories of image content.

[0064] At operation 314, based on the image sensor privacy control module 204(1) verifying the permission of the application 202 at operation 312, the image sensor manager module 204(2) issues an image sensor on event to the HAL 208. Thus, at operation 316, based on receiving the image sensor on event, the HAL 208 issues an operating mode instruction to the kernel 210 (e.g., to the image sensor driver 212). In some aspects, the operating mode instructions include instructions for capturing an image using the image sensor 134. In some aspects, the operating mode instructions include instructions for retrieving an image from a storage device. The captured or retrieved image can be a raw image that contains minimally processed data (or data that has not yet been processed) from the image sensor 134. The term "raw" can refer to not being processed to remove content that is not in one or more categories, but in some aspects, other image processing may have been performed.

[0065] At operation 320, the raw image may be transferred from kernel 210 to HAL 208, and then from HAL 208 to an API of framework 206, and more specifically, to image processing module 204(3), at operation 322. As previously mentioned, image processing module 204(3) processes the raw image to modify portions of the raw image that include content that is not within the content categories that application 202 has requested and has access permission to. Thus, prior to processing the raw image received at operation 322, at operation 324, image processing module 204(3) may request image sensor privacy control module 204(1) to identify one or more categories of image content to which application 202 is requesting access. At operation 326, in response to the request received from image processing module 204(3), image sensor privacy control module 204(1) may indicate to image processing module 204(3) the one or more categories of image content requested by application 202. The indication may also communicate to the image processing module 204 ( 3 ) that the application 202 has permission to access such content category and / or that such content category is a valid content category (eg, a content category that the content detection model 128 is able to detect).

[0066] At operation 328, the image processing module 204(3) processes the original image to modify one or more portions of the image that include content that is not within one or more categories (e.g., categories requested by the application 202, to which the application 202 has access permissions and that are valid categories). Figure 4 The processing performed by the image processing module 204 ( 3 ) at operation 328 may be better understood.

[0067] Figure 4 An example method 400 for content-based image processing according to certain aspects of the present disclosure is illustrated. Figure 4 As shown in Figure 3 At operation 322, the image processing module 204(3) may receive the raw image from the HAL 208. As a first step, in a processor (e.g. Figure 1 The content detection model 128 running on the CPU 122 in the image processing module 204 (3), or another processor such as a neural processor, can use the original image received at the image processing module 204 (3) as input to the content detection model 128 to detect one or more content categories within the original image. Figure 4 As shown in , the original image may contain a QR code and a figure (eg, a person), and the content detection model 128 may be used to predict (eg, detect) coordinates of at least one of the QR code or the figure.

[0068] After detecting the QR code and the graph in the original image, as a second step, similar to Figure 3 In operations 324 and 326 illustrated in FIG. 1 , the image processing module 204(3) may request the image sensor privacy control module 204(1) to identify one or more categories of image content to which the application 202 is requesting access, and in response to the request received from the image processing module 204(3), the image sensor privacy control module 204(1) may indicate to the image processing module 204(3) the one or more categories of image content requested by the application 202.

[0069] although Figure 3 This request / response is illustrated as occurring prior to processing (e.g., at least prior to using content detection model 128 to detect one or more content categories in the original image), but in some aspects, e.g., Figure 4 As shown in , the request / response can occur after the content detection model 128 is used to detect one or more content categories in the original image. In the case where the request / response occurs after the use of the content detection model 128, the content detection model 128 can detect all the content in the original image that these content detection models are configured to detect. On the other hand, in some aspects, in the case where the request / response occurs before the use of the content detection model 128, the content detection model 128 can be configured to detect (1) only the portion of the image that includes content that is not within one or more categories indicated in the response or (2) only the portion of the image that includes content within one or more categories indicated in the response. In certain other aspects, in the case where the request / response occurs before the use of the content detection model 128, the content detection model 128 can be configured to detect (1) at least the portion of the image that includes content that is not within one or more categories indicated in the response or (2) at least the portion of the image that includes content within one or more categories indicated in the response.

[0070] In response to receiving an indication of one or more categories of image content requested by application 202, image processing module 204(3) may be configured to process the original image by masking one or more portions of the original image that contain content not requested by application 202. For example, Figure 4 As shown in , in response to receiving an indication that the application 202 requests QR code content, the image processing module 204 (3) can be configured to mask the portion of the original image that does not include the QR code content. In some aspects, the image processing module 204 (3) can blur the portion of the original image that does not include the QR code content, remove / crop the portion of the original image that does not include the QR code content, and / or mask (e.g., fill with color) the portion of the original image that does not include the QR code content. The image processing module 204 (3) can also use other similar techniques for masking the portion of the original image that does not include the QR code content to protect the privacy of other content in the image before providing the image to the application 202.

[0071] return Figure 3 After processing the original image to create the processed image, the processed image is returned to the application 202 that made the initial request. Visible content in the processed image may include the image content requested by the application 202. According to various aspects described herein, other content within the processed image may be obscured.

[0072] Figure 5 An example 500 of image processing for an application requesting permission to access an image having QR code content is illustrated in accordance with certain aspects of the present disclosure.

[0073] like Figure 5 , the QR scanning application 202 may issue a camera open call to an API of the framework 206 to capture one or more images with QR code content using the image sensor 134. Accordingly, the framework 206 may issue a camera open event to the HAL 208. The HAL 208 may load a library module for the image sensor 134 and issue instructions to the kernel 210 and the image sensor driver 212 to start a camera session, including allocating resources to the image sensor 134 and / or the application 202, such as: memory resources of the system memory 124 for buffering or storing images captured by the image sensor 134, bus resources of the bus 150 for moving data between components such as the CPU 122, the GPU 120, and / or the system memory 124, and / or processor resources of the CPU 122 and / or the GPU 120 for running processes / library modules that support the operation of the image sensor 134.

[0074] The image sensor 134 can capture an image containing the QR code content and pass the raw image to the image processing module 204 (3). Figure 4 In more detail, the image processing module 204(3) may process the original image to modify the portion of the original image that does not include the QR code content. In other words, the image processing module 204(3) may use the content detection model 128 to (1) detect all the content within the original image that the content detection model 128 is configured to detect, (2) detect the portion of the original image that includes the QR code content, and / or (3) detect the portion of the original image that does not include the QR code content. In addition, the processing may include masking the portion of the original image that does not include the QR code content.

[0075] although Figure 5 The QR code content within the captured image is illustrated, but in some other cases, the requested content may not be within the original image and / or may not be detectable by the content detection model 128. In this case, the entire image may be obscured (e.g., the entire image may be blurred).

[0076] After processing, the processed image can be provided to the QR scanning application 202 for use. Figure 5 As shown in , the processed image can include QR code content while blurring the image present in the original image for privacy protection.

[0077] Figure 6 is a flow diagram illustrating example operations 600 for image processing in accordance with certain aspects of the present disclosure.

[0078] Operation 600 may be performed, for example, by a computing device (eg, Figure 1 Operation 600 may be performed by executing one or more processors (e.g., Figure 1 A software component (e.g., software stack 200) that is executed and runs on a CPU 122 or GPU 120 or another processor).

[0079] Operations 600 may begin at a first block 602 by receiving a request from an application to access an image, wherein the application is enabled to access one or more categories of image content.

[0080] The operation 600 may continue at a second block 604 by processing the image to modify one or more portions of the image that include content that is not within the one or more categories.

[0081] The operations 600 may continue at a third block 606 by providing the processed image to an application.

[0082] In some aspects, operation 600 also includes at least one of capturing an image using an image sensor or retrieving an image from a storage device.

[0083] In some aspects, operation 600 also includes receiving a request for permission to access one or more categories of image content from the application. In some aspects, operation 600 also includes prompting a user to grant permission to access one or more categories of image content. In some aspects, operation 600 also includes providing a list of multiple categories of image content to the application.

[0084] In certain aspects of operation 600 , the one or more categories include one or more of: a QR code, an ID image, a plant, or a document.

[0085] In some aspects of operation 600, processing the image to modify the one or more portions includes masking the one or more portions. In some aspects of operation 600, masking the one or more portions includes at least one of blurring, removing, or masking the one or more portions.

[0086] In certain aspects of operation 600, processing the image to modify the one or more portions includes detecting at least one of the one or more portions of the image that includes content that is not within the one or more categories or at least one portion of the image that includes content within the one or more categories. In certain aspects of operation 600, the detection is performed using an object detection artificial intelligence model.

[0087] Sample Clauses

[0088] Specific implementation examples are described in the following numbered clauses:

[0089] Clause 1. A method for image processing, the method comprising: receiving a request to access an image from an application, wherein the application is enabled to access one or more categories of image content; processing the image to modify one or more portions of the image including content that is not within the one or more categories; and providing the processed image to the application.

[0090] Clause 2. The method of clause 1, further comprising at least one of: capturing the image using an image sensor; or retrieving the image from a storage device.

[0091] Clause 3. The method of any one of clauses 1 or 2, further comprising: receiving a request from the application for permission to access the one or more categories of image content.

[0092] Clause 4. The method of clause 3, further comprising: prompting a user to grant the permission to access the one or more categories of image content.

[0093] Clause 5. The method according to any one of clauses 3 or 4, further comprising: providing a list of multiple categories of image content to the application.

[0094] Clause 6. The method of any one of clauses 1 to 5, wherein processing the image to modify the one or more portions comprises masking the one or more portions.

[0095] Clause 7. The method of clause 6, wherein masking the one or more portions comprises at least one of: blurring the one or more portions; removing the one or more portions; or masking the one or more portions.

[0096] Clause 8. A method according to any one of clauses 1 to 7, wherein processing the image to modify the one or more portions comprises: detecting at least one portion of the one or more portions of the image that includes content that is not within the one or more categories or at least one portion of the image that includes content that is within the one or more categories.

[0097] Clause 9. The method of clause 8, wherein the detection is performed using an object detection artificial intelligence model.

[0098] Clause 10. An apparatus comprising: a memory including executable instructions; and a processor configured to execute the executable instructions and cause the apparatus to perform the method according to any one of clauses 1 to 9.

[0099] Clause 11. An apparatus comprising means for performing the method according to any one of clauses 1 to 9.

[0100] Clause 12. A non-transitory computer-readable medium comprising executable instructions that, when executed by a processor of a device, cause the device to perform the method of any one of clauses 1 to 9.

[0101] Clause 13. A computer program product embodied on a computer-readable storage medium, comprising code for performing the method according to any one of clauses 1 to 9.

[0102] Additional considerations

[0103] According to the present disclosure, the term "or" may be interpreted as "and / or" if the context does not indicate otherwise. In addition, although phrases such as "one or more" or "at least one" may have been used for some features disclosed herein and not for other features, features not using such language may be understood to have such implicit meanings if the context does not dictate otherwise.

[0104] In one or more examples, the functions described herein may be implemented in hardware, software, firmware, or any combination thereof. For example, although the term "processing unit" is used throughout this disclosure, such processing unit may be implemented in hardware, software, firmware, or any combination thereof. If any function, processing unit, technique, or other module described herein is implemented in software, the function, processing unit, technique, or other module described herein may be stored on a computer-readable medium or sent on a computer-readable medium as one or more instructions or codes. Computer-readable media may include computer data storage media and communication media, including any media that facilitates the transfer of a computer program from one place to another. In this way, a computer-readable medium may generally correspond to (1) a non-transient tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to extract instructions, codes, and / or data structures for implementing the technology described in this disclosure. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media. A computer program product may include a computer-readable medium.

[0105] The code may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), arithmetic logic units (ALUs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the above structures or any other structure suitable for implementing the techniques described herein. In addition, these techniques may be fully implemented in one or more circuits or logic elements.

[0106] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in the present disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units may be combined in any hardware unit or provided by a collection of interoperable hardware units, including one or more processors as described above, together with appropriate software and / or firmware.

[0107] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for image processing, the method comprising: receiving a request from an application to access an image, wherein the application is enabled to access one or more categories of image content; processing the image to modify one or more portions of the image that include content not within the one or more categories; as well as The processed image is provided to the application.

2. The method according to claim 1, further comprising at least one of the following: capturing the image using an image sensor; or The image is retrieved from a storage device.

3. The method according to claim 1, further comprising: A request is received from the application for permission to access the one or more categories of image content.

4. The method according to claim 3, further comprising: A user is prompted to grant the permission to access the one or more categories of image content.

5. The method according to claim 3, further comprising: The application is provided with a list of multiple categories of image content. The method of claim 1 , wherein processing the image to modify the one or more portions comprises masking the one or more portions.

7. The method of claim 6, wherein shielding the one or more portions comprises at least one of: blurring the one or more portions; removing the one or more parts; or The one or more portions are masked.

8. The method of claim 1 , wherein processing the image to modify the one or more portions comprises: At least one of the one or more portions of the image that includes content that is not within the one or more categories or at least one portion of the image that includes content that is within the one or more categories is detected.

9. The method of claim 8, wherein the detection is performed using an object detection artificial intelligence model.

10. A computing device comprising: Memory; as well as a processor configured to cause the computing device to: receiving a request to access an image from an application running on the computing device, wherein the application is enabled to access one or more categories of image content; processing the image to modify one or more portions of the image that include content not within the one or more categories; as well as The processed image is provided to the application.

11. The computing device of claim 10, wherein the computing device further comprises an image sensor; and The processor is further configured to cause the computing device to perform at least one of the following: capturing the image using the image sensor; or The image is retrieved from a storage device.

12. The computing device of claim 10, wherein the processor is further configured to cause the computing device to: A request is received from the application for permission to access the one or more categories of image content.

13. The computing device of claim 12, wherein the processor is further configured to cause the computing device to: A user is prompted to grant the permission to access the one or more categories of image content.

14. The computing device of claim 12, wherein the processor is further configured to cause the computing device to: The application is provided with a list of multiple categories of image content.

15. The computing device of claim 10, wherein the processor is configured to cause the computing device to process the image to modify the one or more portions by masking the one or more portions.

16. The computing device of claim 15, wherein the processor is configured to cause the computing device to obscure the one or more portions by at least one of: blurring the one or more portions; removing the one or more parts; or The one or more portions are masked.

17. The computing device of claim 10, wherein the processor is configured to cause the computing device to process the image to modify the one or more portions by: At least one of the one or more portions of the image that includes content that is not within the one or more categories or at least one portion of the image that includes content that is within the one or more categories is detected.

18. The computing device of claim 17, wherein the detecting is performed using an object detection artificial intelligence model.

19. A computing device comprising: means for receiving a request to access an image from an application running on the computing device, wherein the application has permission to access one or more categories of image content; means for processing the image to modify one or more portions of the image that include content not within the one or more categories; as well as Means for providing the processed image to the application.

20. The computing device of claim 19, further comprising at least one of the following: means for capturing said image using an image sensor; or Means for retrieving said image from storage.

21. The computing device of claim 19, further comprising: Means for receiving a request from the application for the permission to access the one or more categories of image content.

22. The computing device of claim 21, further comprising: Means for prompting a user to grant the permission to access the one or more categories of image content.

23. The computing device of claim 19, wherein the means for processing the image to modify the one or more portions comprises means for masking the one or more portions.

24. The computing device of claim 19, wherein the means for processing the image to modify the one or more portions comprises: Means for detecting at least one of the one or more portions of the image that includes content not within the one or more categories or at least one portion of the image that includes content within the one or more categories.

25. A non-transitory computer-readable medium storing instructions that, when executed by a computing device, cause the computing device to perform operations for image processing, the operations comprising: receiving a request to access an image from an application running on the computing device, wherein the application has permission to access one or more categories of image content; processing the image to modify one or more portions of the image that include content not within the one or more categories; as well as The processed image is provided to the application.

26. The non-transitory computer readable medium of claim 25, wherein the operations further comprise at least one of: capturing the image using an image sensor; or The image is retrieved from a storage device.

27. The non-transitory computer readable medium of claim 25, further comprising: A request is received from the application for the permission to access the one or more categories of image content.

28. The non-transitory computer readable medium of claim 27, wherein the operations further comprise: A user is prompted to grant the permission to access the one or more categories of image content.

29. The non-transitory computer readable medium of claim 25, wherein processing the image to modify the one or more portions comprises masking the one or more portions.

30. The non-transitory computer readable medium of claim 25, wherein processing the image to modify the one or more portions comprises: At least one of the one or more portions of the image that includes content that is not within the one or more categories or at least one portion of the image that includes content that is within the one or more categories is detected.

Images