Privacy protection for personal computing devices based on spectator detection and classification

Abstract

The present disclosure provides methods, devices, and systems for protecting user privacy when operating a personal computing device. The present implementation more particularly relates to privacy protection techniques for personal computing devices based on spectator detection and classification. In some aspects, a computing device may include a display, one or more sensors, and a privacy controller that receives sensor data from the one or more sensors and controls the display based on the presence or classification of bystanders associated with the received sensor data. In some implementations, the privacy controller may compare any spectators detected from the sensor data to a database of contacts associated with the user, and classify each spectator as "trusted spectators" or "untrusted spectators" based on the comparison. More specifically, the privacy controller may selectively activate a privacy protection mechanism associated with the display based on whether an untrusted bystander is detected to arrive.

Description

Technical Field

[0001] The present implementation generally relates to personal computing devices, and more particularly to privacy protection of personal computing devices based on bystander detection and classification. Background Art

[0002] Personal computing devices (such as desktop computers, laptop computers, smart phones, and tablet computers) are commonly used to store and access information that is personal or private to the user of the device (such as financial records, medical records, intellectual property, or other confidential or proprietary information). As such, many personal computing devices have security features (such as various user authentication mechanisms) designed to prevent unauthorized access. However, existing user authentication mechanisms do not protect the privacy of information displayed on a personal computing device when the personal computing device is controlled or operated by an authenticated user.

[0003] As personal computing devices become more portable and feature-rich, and with the growing trend towards remote work, individuals are increasingly using their personal computing devices to access personal or private information in public spaces and other environments (such as cafes, libraries, or airplanes, among other examples) with potential bystanders or passersby. Thus, there is a need to protect the privacy of information displayed on a personal computing device in the presence of bystanders or passersby without significantly hindering the user's access to such information on the personal computing device. Summary of the Invention

[0004] This Summary of the Invention is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary of the Invention is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] One innovative aspect of the subject matter of the present disclosure can be implemented as a method executed by a controller for a computing device. The method includes: receiving sensor data from one or more sensors associated with the computing device; identifying an authorized user of the computing device based on the received sensor data; detecting one or more objects of interest associated with the received sensor data, wherein the one or more objects of interest are different from the authorized user; classifying each object of interest among the one or more objects of interest at least in part based on a database of contacts associated with the authorized user; and controlling a display associated with the computing device based on the classification of each object of interest.

[0006] Another innovative aspect of the subject matter of the present disclosure can be implemented in a controller for a computing device, the controller including a processing system and a memory. The memory stores instructions that, when executed by the processing system, cause the controller to: receive sensor data from one or more sensors associated with the computing device; identify an authorized user of the computing device based on the received sensor data; detect one or more objects of interest associated with the received sensor data, wherein the one or more objects of interest are different from the authorized user; classify each of the one or more objects of interest at least in part based on a database of contacts associated with the authorized user; and control a display associated with the computing device based on the classification of each object of interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] This implementation is illustrated by way of example and is not intended to be limited by the figures of the drawings.

[0008] Figure 1 An example environment for viewing or operating a personal computing device is shown.

[0009] Figure 2 A block diagram of an example computing device according to some implementations is shown.

[0010] Figure 3 A block diagram of an example bystander classification system for a personal computing device according to some implementations is shown.

[0011] Figure 4A An example operating environment of a personal computing device with privacy protection based on bystander detection and classification according to some implementations is shown.

[0012] Figure 4B Another example operating environment of a personal computing device with privacy protection based on bystander detection and classification according to some implementations is shown.

[0013] Figure 4C Another example operating environment of a personal computing device with privacy protection based on bystander detection and classification according to some implementations is shown.

[0014] Figure 5 Another block diagram of an example bystander classification system for a personal computing device according to some implementations is shown.

[0015] Figure 6 Another example operating environment of a personal computing device with privacy protection based on bystander detection and classification according to some implementations is shown.

[0016] Figure 7A block diagram of an example privacy controller for a personal computing device is shown in accordance with some implementations.

[0017] Figure 8 An illustrative flowchart depicting example operations for protecting the privacy of a user of a personal computing device in the presence of onlookers is shown in accordance with some implementations. DETAILED DESCRIPTION

[0018] In the following description, numerous specific details are set forth, such as examples of specific components, circuits, and processes, to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected to one or more intermediate components or circuits or connected through one or more intermediate components or circuits. The terms "electronic system" and "electronic device" may be used interchangeably to refer to any system capable of electronically processing information. Also, in the following description, and for purposes of explanation, specific terms are set forth to provide a thorough understanding of aspects of the present disclosure. However, it will be apparent to those skilled in the art that these specific details are not required to practice the example embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the present disclosure. Some portions of the following detailed description are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory.

[0019] These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. In the present disclosure, procedures, logic blocks, processes, etc. are considered to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. However, it should be borne in mind that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities.

[0020] Unless specifically stated otherwise, as will be apparent from the following discussion, it is appreciated that throughout this application, discussions using terms such as "access," "receive," "send," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," etc., refer to actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the registers and memories of the computer system into other data similarly represented as physical quantities within the computer system memory or registers or other such information storage, transmission, or display devices.

[0021] In the figures, a single block may be described as performing one or more functions; however, in actual practice, one or more of the functions performed by that block may be performed in a single component or across multiple components, and / or may be performed using hardware, using software, or using a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described generally herein in terms of their functionality. Whether such functionality is implemented as hardware or as software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each particular application, but such implementation decisions should not be construed as causing a departure from the scope of the present disclosure. Also, example input devices may include components other than those shown, including well-known components such as processors, memories, and the like.

[0022] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a specific manner. Any feature described as a module or component may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be at least partially implemented by a non-transitory processor-readable storage medium that includes instructions that, when executed, perform one or more of the methods described above. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include packaging materials.

[0023] The non-transitory processor-readable storage medium may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, other known storage media, and the like. Additionally or alternatively, the techniques may be at least partially implemented by a processor-readable communication medium that carries or transmits code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer or other processor.

[0024] The various illustrative logical blocks, modules, circuits, and instructions described in connection with the embodiments disclosed herein may be executed by one or more processors (or processing systems). As used herein, the term "processor" may refer to any general-purpose processor, special-purpose processor, conventional processor, controller, microcontroller, and / or state machine that is capable of executing one or more software programs, scripts, or instructions stored in a memory.

[0025] As described above, individuals are increasingly using their personal computing devices to access personal or private information in public spaces and other environments (such as cafes, libraries, or airplanes, among other examples) with potential onlookers or passersby. To protect user privacy, some personal computing devices implement a bystander detection system that can obfuscate the information on the display or notify the user when a potential bystander is detected in the environment. For example, many personal computing devices include a camera capable of capturing images or video of the surrounding environment. Such cameras are typically coplanar with the display to capture images of the person using the device (such as for authentication or video conferencing applications). The bystander detection system can use such images to detect potential bystanders in the surrounding environment (such as anyone other than the user of the personal computing device).

[0026] Existing bystander detection systems are configured to activate privacy protection mechanisms (such as by obfuscating the display or outputting a warning) if any potential bystander is detected in the background. However, in some instances, a user may wish to show the content displayed on the personal computing device to certain nearby people (such as colleagues, friends, or family). Since existing bystander detection systems impede the user's ability to collaborate with colleagues or share the displayed content, users typically disable this feature on their personal computing devices. Aspects of the present disclosure recognize that many personal computing devices store (or have access to) a database of contacts that can be used to distinguish between unwanted bystanders and trusted individuals known to or otherwise associated with the user. Example suitable databases include an address book or contact list associated with a phone or email client, or an organizational chart or directory of a business or organization, among other examples.

[0027] Various aspects generally relate to personal computing devices, and more particularly to privacy protection techniques for personal computing devices based on bystander detection and classification. In some aspects, a computing device may include a display, one or more sensors, and a privacy controller that receives sensor data from the one or more sensors and controls the display based on the presence or classification of bystanders associated with the received sensor data. In some implementations, the privacy controller may compare any bystanders detected from the sensor data with a database of contacts associated with the user and classify each bystander as a "trusted bystander" or an "untrusted bystander" based on the comparison. If one or more untrusted bystanders are detected in the received image, the privacy controller may activate a privacy protection mechanism associated with the display (such as by obfuscating the display or outputting a warning). If only trusted bystanders (or no bystanders) are detected in the received image, the privacy controller may avoid activating the privacy protection mechanism.

[0028] Certain implementations of the subject matter described in this disclosure can achieve one or more of the following potential advantages. By detecting and classifying onlookers as "trusted" or "untrusted", aspects of this disclosure can provide finer-grained control over privacy protection and a significantly improved user experience. For example, a user can share the content displayed on their personal computing device with trusted individuals (such as friends, family, or colleagues) without interference from the privacy controller. As a result, the user may be more inclined to enable the privacy controller during normal operation. By using an existing contact database for onlooker classification, aspects of this disclosure can distinguish between trusted and untrusted onlookers in a pre-configured (or automated) manner. As a result, the privacy controller can be used without any settings or input by the user of the personal computing device.

[0029] Figure 1 An example environment 100 for viewing or operating a personal computing device 110 is shown. The personal computing device 110 includes a display 112 and one or more sensors 114. The sensors 114 can implement any sensing technology capable of detecting objects of interest (such as humans) in the environment 100. In some implementations, the sensors 114 can include a camera or microphone capable of capturing images or recording audio from the environment 100. Other suitable sensing technologies can include thermal, time-of-flight (ToF), and radio frequency (RF) sensing (such as using Wi-Fi, Bluetooth, or other RF signals), among other examples. The example environment 100 includes a user 120 of the computing device 110 and an onlooker 130 standing or sitting behind the user 120. In Figure 1 the example, both the user 120 and the onlooker 130 are facing or otherwise looking at the display 112. Although only one onlooker 130 is shown in Figure 1 the example, in an actual implementation, the environment 100 can include fewer or more onlookers.

[0030] As Figure 1As shown in [Fig.], when user 120 is looking at display 112, bystander 130 is outside the user's field of view (FOV). As such, user 120 may not be aware of the presence of bystander 130. In some instances, bystander 130 may be a friend, colleague, or other known acquaintance of user 120. In some other instances, bystander 130 may be an unknown passerby with a line of sight to display 112. In some aspects, computing device 110 may periodically or continuously scan the environment 100 for bystanders while user 120 is viewing or operating computing device 110, and may perform one or more operations to protect the user's privacy based on the detection of bystanders in environment 100. In some implementations, computing device 110 may perform object detection operations for detecting humans based on sensor data captured via sensor 114.

[0031] In some implementations, computing device 110 may classify each person detected in environment 100 as an "authorized user" or a "bystander". An authorized user is a person (such as user 120) who is registered with or otherwise authorized to use computing device 110. In contrast, a bystander may be anyone who is not an authorized user of computing device 110 (such as bystander 130). In some implementations, computing device 100 may identify the face or voice of an authorized user based on data provided by the user during a registration or enrollment process. For example, user 120 may provide an image of the user's face or a voice recording as biometric features for authentication. Thus, computing device 110 may utilize existing biometric data (such as data for face or voice recognition) to determine whether a person is an authorized user. In some implementations, computing device 110 may classify anyone not identified as an authorized user as a bystander.

[0032] In some implementations, computing device 110 may determine the relationship between user 120 and bystander 130, and may further classify bystander 130 as "trusted" or "untrusted" based on the determined relationship. As used herein, the term "trusted bystander" refers to anyone who has an eligible association or relationship with an authorized user of computing device 110 (such as a friend, family member, colleague, or other known acquaintance). However, not all known acquaintances of an authorized user may be eligible as trusted bystanders. For example, a colleague who is employed by the same company as user 120 but belongs to a different department or group within the company may not have permission to view some confidential information that may be stored on or otherwise accessible via computing device 110. Any bystander who does not meet the criteria for a trusted bystander may instead be classified as an untrusted bystander.

[0033] Aspects of the present disclosure recognize that many personal computing devices store or have access to databases of contacts that include object recognition data (such as images or voice samples) of one or more individuals having a known association or relationship with an authorized user of the computing device. Example suitable databases include address books or contact lists associated with a phone or email client, or an organizational chart or directory of a business or other organization, among other examples. Such databases may be managed by the user or by the organization to which the user belongs. In some aspects, computing device 110 may determine the relationship between user 120 and bystander 130 based on a database of contacts associated with user 120. For example, computing device 110 may compare an image or audio recording of bystander 130 captured via sensor 114 with an image or voice sample of a contact stored in the database. Computing device 110 may further determine whether bystander 130 is trusted (or untrusted) based at least in part on whether the image or audio recording of bystander 130 matches an image or voice sample in the database.

[0034] In some aspects, computing device 110 may control display 112 to protect the privacy of user 120 based on the classification of bystander 130. For example, if bystander 130 is classified as a trusted bystander, computing device 110 may continue to operate normally (without interrupting or otherwise altering the content of display 112). However, if bystander 130 is classified as an untrusted bystander, computing device 110 may modify or adjust the content of display 112 to protect the privacy of user 120. In some implementations, computing device 110 may protect the privacy of user 120 by blurring or otherwise obscuring the content of display 112. In some other implementations, computing device 110 may protect the privacy of user 120 by presenting a notification (such as a pop-up window) on display 112 warning user 120 of the presence of bystander 130.

[0035] Figure 2 A block diagram of an example computing device 200 is shown in accordance with some implementations. Computing device 200 is configured to protect the privacy of a user viewing or operating computing device 200 in the presence of a bystander. In some implementations, computing device 200 may be Figure 1 an example of personal computing device 110.

[0036] The computing device 200 includes a display 210, one or more sensors 220, and a privacy controller 230. The display 210 can be any electronic display capable of rendering or displaying digital content. The sensors 220 can include any suitable sensors capable of detecting an object of interest proximate to the display 210. In some implementations, the sensors 220 can include a camera or a microphone capable of capturing an image or recording audio from the environment. For example, the camera or the microphone can be positioned to capture an image of the user or an audio recording when the user is operating the computing device 200 or otherwise viewing the display 210 (such as Figure 1 as shown). Referring to Figure 1 , the display 210 can be an example of the display 112, and the sensors 220 can be an example of the sensors 114.

[0037] The privacy controller 230 continuously or periodically collects sensor data 204 from the sensors 220 and controls the display 210 at least in part based on the sensor data 204. More specifically, the privacy controller 230 can analyze the sensor data 204 to detect an untrusted bystander and selectively activate one or more privacy protection mechanisms based on whether any untrusted bystander is detected based on the sensor data 204. In some implementations, the privacy controller 230 can include an object detection component 232, a user identification component 234, and a bystander classification component 236. The object detection component 232 is configured to analyze the sensor data 204 for an object of interest (such as a face or voice). For example, the object detection component 232 can perform object detection operations on the received sensor data 204.

[0038] Object detection encompasses various techniques for detecting objects belonging to known classes (such as humans, cars, or text) based on images, audio recordings, or other sensor data. For example, the presence and location of an object can be detected or inferred from an image by scanning the image for a set of features (such as eyes, nose, and lips) that are unique to a particular class of object (such as humans). Object detection models are generally used for feature extraction and localizing objects in images, audio recordings, or other sensor data. In some implementations, the object detection component 232 can use one or more statistical models for object detection. In some other implementations, the object detection component 232 can use one or more machine learning models for object detection.

[0039] Machine learning is a technology used to improve the ability of a computer system or application to perform a specific task. During the training phase, a machine learning system is provided with multiple "answers" and a large amount of raw input data. For example, the input data may include images depicting objects of interest, and the answers may include labels or bounding boxes indicating the presence or location of the objects of interest. The machine learning system analyzes the input data to learn a set of rules (also known as a "machine learning model") that can be used to map the input data to the answers. During the inference phase, the machine learning system uses the trained model to infer answers (such as object detection or bounding boxes) from new input data.

[0040] The user identification component 234 is configured to classify each object of interest detected by the object detection component 232 as an authorized user or a bystander. In some implementations, the computing device 200 may capture or collect sensor data associated with a user (such as an image of the user's face or a recording of the user's voice) during a registration or enrollment process, and extract various features of the user (such as facial features or voice features) from the sensor data. The user identification component 234 may compare the features of the user with the features extracted from each object of interest to determine whether to classify the object as an authorized user (such as when a match is detected) or a bystander (such as when no match is detected). In some implementations, the user identification component 234 may perform the classification at least in part based on a machine learning model.

[0041] The bystander classification component 236 is configured to further classify each bystander classified by the user identification component 234 as a trusted bystander or an untrusted bystander. In some implementations, the bystander classification component 236 may perform the classification at least in part based on the contact information 206 stored in the contact database 240. As referenced Figure 1 and described, the contact database 240 may be an address book or contact list associated with a phone or email client or an organizational chart or directory of a business or other organization, among other examples. As Figure 2 shown, the contact database 240 may be stored locally on the computing device 200. In some other implementations, at least a portion of the contact database 240 may be stored on an external server.

[0042] In some aspects, the contact database 240 may store object recognition data from which features can be extracted and used to identify each contact. Example suitable object recognition data includes contact photos and voice samples (such as previously recorded audio containing the voice of the contact) and other examples. For example, many businesses and organizations assign or otherwise require contact photos as a means of identifying employees. Thus, each contact photo may include the face of the corresponding contact represented in the contact database 240. In some implementations, the bystander classification component 236 may retrieve object recognition data (such as one or more contact photos) from the contact database 240 as contact information 206, and extract various features from the object recognition data. The bystander classification component 236 may further compare the extracted features of each contact with the features extracted from each object of interest classified as a bystander to determine whether the bystander is classified as a trusted bystander or an untrusted bystander.

[0043] In some implementations, the bystander classification component 236 may use a machine learning model to compare the features of the bystander with the features of the contacts. For example, the machine learning model may indicate whether the features of the bystander match the features of the contacts. If no match is detected between a given bystander and any of the contacts in the contact database 240, the bystander classification component 236 may classify the bystander as an untrusted bystander. In some implementations, as long as a match is detected between the bystander and one of the contacts in the contact database 240, the bystander classification component 236 may classify the bystander as a trusted bystander. In such implementations, it may be assumed that the contact database 240 stores only the contact information 206 of trusted contacts or that the bystander classification component 236 retrieves only the contact information 206 of trusted contacts in the contact database 240.

[0044] In some other implementations, if a match is detected between the bystander and a contact in the contact database 240, the bystander classification component 236 may analyze additional contact information 206 to determine whether to classify the bystander as a trusted (or untrusted) bystander. Example suitable contact information may also include, among other examples, the collaboration or communication history with the matching contact or the relationship with the matching contact. For example, a contact who collaborates more frequently with the user is more likely to be classified as a trusted bystander than a contact who collaborates less frequently with the user. Additionally, a contact who belongs to the same group or department as the user is more likely to be classified as a trusted bystander than a contact who belongs to a different group or department from the user.

[0045] In some aspects, the privacy controller 230 may selectively modify or augment the display data 202 provided to the display 210 based on whether an untrusted bystander is detected from the sensor data 204. The display data 202 may include any content viewed or accessed by a user of the computing device 200. For example, the privacy controller 230 may receive the display data 202 from an application executing on the computing device 200 via the application interface 250. In some implementations, the privacy controller 230 may blur the display data 202 (such that the content on the display 210 appears blurry, textured, or opaque) in response to detecting an untrusted bystander. In some other implementations, the privacy controller 230 may output a bystander notification 208 to the display 210 (as a pop-up window or an overlay on top of the display data 202) in response to detecting an untrusted bystander.

[0046] In some aspects, at least a portion of the privacy controller 230 may be implemented (or executed on) an application specific integrated circuit (ASIC) separate from the central processing unit of the computing device 200 that executes various other applications (such as an operating system). In some implementations, the ASIC may provide a trusted execution environment (TEE) for object detection and classification such that the sensor data 204 received via the sensors 220 is inaccessible to software or hardware external to the TEE. For example, the TEE may include an object detection component 232, a user identification component 234, and a bystander classification component 236. In such implementations, the ASIC may generate a binary signal indicating whether an untrusted bystander is detected, and the privacy controller 230 may control the display 210 based on the state of the binary signal.

[0047] Figure 3 A block diagram of an example bystander classification system 300 for a personal computing device is shown according to some implementations. The bystander classification system 300 is configured to receive a set of features 302 representative of a bystander and classify the bystander as a trusted bystander or an untrusted bystander based on contact information stored in a database of contacts associated with an authorized user of the personal computing device. In some implementations, the bystander classification system 300 may be Figure 2 an example of the bystander classification component 236. Referring Figure 2 to

[0048] In Figure 3 the example of Figure 2As described, the object recognition data 303 can include various media or sensor data (such as an image of the face of the corresponding contact or an audio recording of the voice of the corresponding contact) for each contact in the database from which features can be extracted and used to identify the contact. The communication history 305 can include a record of previous communications (including phone calls, email exchanges, or meetings) between the user and the corresponding contact or an indication of the frequency with which the user communicates with the contact. The descriptor 307 can include the role or title of the corresponding contact within the organization, team, or department to which the contact belongs, or the personal relationship (such as parent, child, or sibling) between the contact and the user.

[0049] The bystander classification system 300 includes a feature analyzer 310, a collaboration detector 320, and a classifier 330. The feature analyzer 310 is configured to determine whether the bystander features 302 match any of the object recognition data in the object recognition data 303 stored in the contact database. In some aspects, the feature analyzer 310 can extract a set of features from the object recognition data 303 associated with each contact and compare the bystander features 302 with the features of each contact. In some implementations, the feature analyzer 310 can use a machine learning model to compare the bystander features 302 with the object recognition data 303. The feature analyzer 310 further outputs matching information 304 indicating whether the bystander features 302 match any of the object recognition data in the object recognition data 303 stored in the database. In some implementations, in the case where a match is detected, the matching information 304 can also identify the matching contact.

[0050] The collaboration detector 320 is configured to determine, based on the communication history 305 associated with the contact, whether the matching contact (indicated by the matching information 304) is a frequent collaborator of the user. For example, if the communication history 305 indicates that the contact has had at least a threshold number of previous communications with the user, the collaboration detector 320 can determine that the contact is a frequent collaborator. In some implementations, certain types of communications (such as phone calls, email exchanges, or meetings) can be weighted more heavily in this determination than other communications. The weights and thresholds used to determine whether a contact is a frequent collaborator can be pre-configured, configured by the user, or configured by a system administrator of the organization.

[0051] In some aspects, the collaboration detector 320 may utilize an existing list of frequent contacts in the process of determining whether a contact is a frequent collaborator. For example, some email clients may store a list of "frequent contacts" based on the history of email exchanges between the user and various contacts in the user's contact list. Thus, the communication history 305 may indicate whether the corresponding contact is included in the list of frequent contacts. In some implementations, if the contact is included in the list of frequent contacts, the collaboration detector 320 may determine that the contact is a frequent collaborator. The collaboration detector 320 further outputs collaboration information 306 indicating whether the matching contact is a frequent collaborator of the user.

[0052] The classifier 330 is configured to determine the classification 308 of the bystander at least in part based on the matching information 304. For example, if the matching information 304 indicates that the bystander does not match any of the contacts in the database, the classifier 330 may classify the bystander as an untrusted bystander. If the matching information 304 indicates that the bystander matches a contact in the database, the classifier 330 may further determine the classification 308 of the bystander based on the collaboration information 306 and the descriptor 307 associated with the matching contact. In some implementations, if the collaboration information 306 indicates that the matching contact is a frequent collaborator of the user, the classifier 330 may classify the bystander as a trusted bystander.

[0053] If the collaboration information 306 indicates that the matching contact is not a frequent collaborator, the classifier 330 may compare the descriptor 307 of the contact with a set of criteria associated with trusted bystanders. For example, contacts who are managers or supervisors (according to the organizational chart or hierarchy) or members of the same team as the user may be whitelisted as trusted bystanders. In some implementations, the collaboration information 306 may be weighted with respect to the descriptor 307 during the process of determining the classification 308. The weights and criteria for determining whether a contact is a trusted bystander can be pre-configured, configured by the user, or configured by the system administrator of the organization.

[0054] Figure 4A An example operating environment 400 of a personal computing device 401 with privacy protection based on bystander detection and classification according to some implementations is shown. In some implementations, the personal computing device 401 may be an example of any of the computing devices 110 or 200 respectively Figure 1 and Figure 2 The example environment 400 includes an authorized user 402 of the computing device 401 and a bystander 403 standing or sitting behind the user 402.

[0055] The computing device 401 includes a display 404 and one or more sensors 405. In some aspects, the computing device 401 may use the sensors 405 to continuously or periodically scan the surrounding environment of bystanders and control the display 404 based on the detection and classification of one or more bystanders (such as described with reference to Figures 1-3 . For example, the computing device 401 may identify the user 402 (as an authorized user) based on sensor data captured via the sensors 405 and may also detect a bystander 403 from the sensor data. The computing device 401 may further determine that the bystander 403 matches a known contact stored in a database of contacts associated with the user 402 based on feature analysis. In Figure 4A example, the computing device 401 may classify the bystander 403 as a trusted bystander based on the matching contact.

[0056] Since the bystander 403 is classified as a trusted bystander, the computing device 401 may avoid activating any privacy protection mechanisms associated with the display 404. In some aspects, the computing device 401 may allow the user 402 and the trusted bystander 403 to continue viewing the collaborative content 406 presented on the display 404 without interruption (shown as "collaborative content" in Figure 4A . In some other aspects, the computing device 401 may display a notification 407 on the display 404 indicating that a trusted bystander has been detected. In some implementations, the notification 407 may include the name or other identification information associated with the trusted bystander 403 (such as "colleague detected" as shown in Figure 4A .

[0057] Figure 4B FIG. 410 shows another example operating environment of the personal computing device 401 according to some implementations. The example environment 410 includes an authorized user 402, a bystander 403, and a second bystander 412.

[0058] In Figure 4B example, the computing device 401 may identify the user 402 (as an authorized user) based on sensor data captured via the sensors 405 and may also detect the bystanders 403 and 412 from the sensor data. In Figure 4B example, the computing device 401 may classify the bystander 403 as a trusted bystander (such as described with reference to Figure 4Aand described) and may classify the second bystander 412 as an untrusted bystander. In some implementations, the computing device 401 may determine, based on feature analysis, that the bystander 412 does not match any known contacts stored in a database of contacts associated with the user 402. In some other implementations, the computing device 401 may determine that the bystander 412 matches a known contact, but the matching contact does not meet the criteria for a trusted bystander (such as those referenced Figure 3 and described).

[0059] In response to classifying the bystander 412 as an untrusted bystander, the computing device 401 may activate one or more privacy protection mechanisms associated with the display 404. In Figure 4B the example, the computing device 401 is shown to blur the display 404 such that the collaborative content 414 presented on the display is difficult (if not impossible) for the untrusted bystander 412 to read. For example, the blurred content 414 may appear blurry, textured, or opaque. Aspects of the present disclosure recognize that in some instances, the user 402 may not be viewing private or confidential content on the computing device 401. In such instances, the computing device 401 may provide a poor user experience by blurring the display 404 in response to detecting the untrusted bystander 412. Thus, in some aspects, the computing device 401 may implement different privacy protection mechanisms depending on the circumstances, user preferences, or security threat level.

[0060] Figure 4C Another example operating environment 420 of a personal computing device 401 according to some implementations is shown. The example environment 420 includes an authorized user 402 and bystanders 403 and 412.

[0061] The computing device 401 may identify the user 402 (as the authorized user) based on sensor data captured via the sensor 405 and may also detect the bystanders 403 and 412 from the sensor data. In Figure 4C the example, the computing device 401 may classify the bystander 403 as a trusted bystander (such as those referenced Figure 4A and described) and may classify the bystander 412 as an untrusted bystander (such as those referenced Figure 4B and described). In response to classifying the bystander 412 as an untrusted bystander, the computing device 401 may activate one or more privacy protection mechanisms associated with the display 404.

[0062] In Figure 4C the example, the computing device 401 is shown to display a notification 422 on the display 404 indicating that an untrusted bystander has been detected. Associated with Figure 4BIn contrast to the blurred content 414, the notification 422 does not significantly impede the ability of the user 402 to view the collaborative content on the display 404. However, the notification 422 also does not provide the same level of privacy protection against untrusted bystanders 412 as the blurred content 414. Thus, in some aspects, the computing device 401 may implement a combination of privacy protection mechanisms described with reference to Figure 4B and Figure 4C . For example, when an untrusted bystander 412 is first detected, the computing device 401 may blur the display 404, and if the user 402 rejects or vetoes the privacy protection mechanism, the computing device 401 may display only the notification 422 in response to a subsequent detection of the untrusted bystander 412.

[0063] Aspects of the present disclosure further recognize that the level of trust between an authorized user and a bystander may also depend on the type of content presented on the display. For example, a user may not wish to share certain types of information (such as personal bank records) with any bystander, regardless of their relationship with the user. However, in some instances, a user may inadvertently trigger the presentation of such personal information on the display while collaborating with a bystander (such as by accidentally switching to the wrong application or pop-up window). Thus, in some aspects, the bystander classification system may further classify bystanders at least in part based on the type of content presented on the display.

[0064] Figure 5 Another block diagram showing an example bystander classification system 500 for a personal computing device according to some implementations. The bystander classification system 500 is configured to receive a set of features 501 representing a bystander and, based on contact information stored in a database of contacts associated with an authorized user of the personal computing device, classify the bystander as a trusted bystander or an untrusted bystander for a particular type of application data 507 presented on the display. In some implementations, the bystander classification system 500 may be Figure 2 an example of the bystander classification component 236. Referring to Figure 2 , the application data 507 may be an example of the display data 202, and the set of features 501 may be extracted from the sensor data 204 by the object detection component 232.

[0065] In Figure 5 the example, the contact information is shown to include object recognition data 502, a communication history 504 with the user, and one or more descriptors 505 for each contact in the database. As referred to Figure 1As described, the object recognition data 502 can be various media or sensor data (such as an image of a contact's face or an audio recording of a contact's voice) for each contact in a database from which features can be extracted and used to identify the contact. The communication history 504 can include a record of previous communications (including phone calls, email exchanges, or meetings) between the user and the corresponding contact or an indication of the frequency of communication between the user and the contact. The descriptor 505 can include the role or title of the corresponding contact within the organization, team, or department to which the contact belongs, or the personal relationship between the contact and the user (such as parent, child, or sibling).

[0066] The bystander classification system 500 includes a feature analyzer 510, a relationship analyzer 520, and a classifier 530. The feature analyzer 510 is configured to determine whether the bystander features 501 match any of the object recognition data 502 stored in the contact database. In some aspects, the feature analyzer 510 can extract a set of features from the object recognition data 502 associated with each contact and compare the bystander features 501 with the features of each contact. In some implementations, the feature analyzer 510 can use a machine learning model to compare the bystander features 501 with the object recognition data 502. The feature analyzer 510 further outputs matching information 503 indicating whether the bystander features 501 match any of the object recognition data 502 stored in the database. In some implementations, in the case where a match is detected, the matching information 503 can also identify the matching contact.

[0067] The relationship analyzer 520 is configured to determine the relationship between the user and the matching contact (indicated by the matching information 503) based on the descriptor 505 and the communication history 504 associated with the contact (such as described with reference to Figure 3 ). In some implementations, the relationship analyzer 520 can determine relationship information 506 indicating whether the matching contact is a frequent collaborator of the user based on the communication history 504 associated with the contact. In some other implementations, the relationship analyzer 520 can determine relationship information 506 indicating whether the matching contact and the user belong to the same team or group within the organization based on the descriptor 505 of the contact.

[0068] The classifier 530 is configured to determine the classification 508 of the bystander based on the matching information 503, the relationship information 506, and the application data 507. For example, if the matching information 503 indicates that the bystander does not match any of the contacts in the database, the classifier 530 may classify the bystander as an untrusted bystander for the application data 507. If the matching information 503 indicates that the bystander matches a contact in the database, the classifier 530 may further determine the classification 508 of the bystander based on the application data 507 and the relationship information 506 associated with the matching contact. In some aspects, the classifier 530 may determine whether the relationship information 506 meets a set of criteria associated with a trusted bystander for the application data 507. For example, the criteria may include a pre-determined list of applications (or websites) or may be configurable by a user or a system administrator of an organization.

[0069] In some implementations, a user may specify one or more applications that are private to the user and should not be viewed by any bystander. In such implementations, the classifier 530 may classify all bystanders as untrusted bystanders for the application data 507 associated with the private applications. In some other implementations, a user or a system administrator may authorize individual contacts in the contact database or contacts having a specific relationship with the user to view certain types of application data. For example, a user may authorize a partner or a spouse to view the user's personal banking information. As another example, a system administrator of an organization may authorize members of the organization having certain roles or titles (such as the chief financial officer of the finance department or the manager of the user) to view the organization's corporate banking information.

[0070] The classifier 530 may classify the bystander as a trusted bystander only if the matching contact meets a set of criteria associated with a trusted bystander for the specific type of application data 507 presented on the display. Thus, the bystander classification system 500 may classify bystanders with a finer granularity compared to the bystander classification system 300. In some aspects, in the case where multiple applications are presented on the display, the classifier 530 may determine the corresponding classification 508 for each of the applications (for each detected bystander). As a result, a bystander may be classified as a trusted bystander for the application data associated with one of the applications and may be classified as an untrusted bystander for the application data associated with another of the applications. In some implementations, a privacy controller (such as Figure 2 the privacy controller 230) may obfuscate only the content for which an untrusted bystander is detected.

[0071] Figure 6Another example operating environment 600 of a personal computing device 601 with privacy protection based on bystander detection and classification is shown. In some implementations, the personal computing device 601 can be an example of any of the computing devices 110 or 200, respectively, of Figure 1 and Figure 2 The example environment 600 includes an authorized user 602 of the computing device 601 and a bystander 603 standing or sitting next to the user 602.

[0072] The computing device 601 includes a display 604 and one or more sensors 605. In some aspects, the computing device 601 can use the sensors 605 to continuously or periodically scan the surrounding environment of the bystander and control the display 604 based on the detection and classification of one or more bystanders (such as described with reference to Figures 1-3 and Figure 5 For example, the computing device 601 can identify the user 602 (as an authorized user) based on the sensor data captured via the sensors 605 and can also detect the bystander 603 from the sensor data. The computing device 601 can further determine that the bystander 603 matches a known contact stored in a database of contacts associated with the user 602 based on feature analysis. In the example of Figure 6 the computing device 601 can classify the bystander 603 as a trusted bystander for collaborative content presented on the right side of the display 604 (shown as "Collaborative Content" in Figure 6 and classify the bystander 603 as an untrusted bystander for private content presented on the left side of the display 604 (shown as "Private Content" in Figure 6 Since the bystander 603 is classified as a trusted bystander for the collaborative content, the computing device 601 can avoid activating any privacy protection mechanisms for the right side of the display 604. In other words, the computing device 601 can allow the user 602 and the bystander 603 to continue viewing the collaborative content without interruption. However, since the bystander 603 is classified as an untrusted bystander for the private content, the computing device 601 can blur the left side of the display 604. For example, the private content can appear blurred, textured, or opaque. Although the collaborative content and the private content are presented side by side in the example of

[0073] various other presentations are possible. For example, in some implementations, the collaborative content can be presented as a pop-up window or an overlay on top of the private content. In such implementations, only the background portion of the display 604 that contains the private content can be blurred. Figure 6 In such implementations, only the background portion of the display 604 that contains the private content can be blurred.

[0074] Figure 7FIG. 0 is a block diagram showing an example privacy controller 700 for a personal computing device. The privacy controller 700 is configured to: detect and classify bystanders based on sensor data captured by one or more sensors associated with the personal computing device; and control a display associated with the personal computing device based on the detection and classification of the bystanders. In some implementations, the privacy controller 700 can be Figure 2 an example of the privacy controller 230.

[0075] The privacy controller 700 includes a device interface 710, a processing system 720, and a memory 730. The device interface 710 is configured to communicate with one or more components of the personal computing device. In some implementations, the device interface 710 can include a display interface (I / F) 712 and a sensor interface (I / F) 714. The display interface 712 is configured to communicate with a display associated with the personal computing device (such as Figure 2 the display 210). The sensor interface 714 is configured to communicate with one or more sensors associated with the personal computing device (such as Figure 2 the sensor 220). In some implementations, the sensor interface 714 can receive sensor data from one or more sensors associated with the personal computing device.

[0076] The memory 730 can include a contact data store 731 configured to store a database of contacts associated with authorized users of the personal computing device (such as Figure 2 the contact database 240). The memory 730 can also include a non-transitory computer-readable medium (including one or more non-volatile memory elements such as EPROM, EEPROM, flash memory, or a hard disk drive and other examples) that can store at least the following software (SW) modules: · A user identification SW module 732 for identifying an authorized user of the computing device based on the received sensor data; · An object detection SW module 734 for detecting one or more objects of interest associated with the received sensor data, where the one or more objects of interest are different from the authorized user; · An object classification SW module 736 for classifying each of the one or more objects of interest at least in part based on a database of contacts associated with the authorized user; and · A display control SW module 738 for controlling a display associated with the computing device based on the classification of each object of interest. Each software module includes instructions that, when executed by the processing system 720, cause the privacy controller 700 to perform a corresponding function.

[0077] The processing system 720 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in the privacy controller 700, such as in the memory 730. For example, the processing system 720 may execute the user identification SW module 732 to identify an authorized user of the computing device based on the received sensor data. The processing system 720 may also execute the object detection SW module 734 to detect one or more objects of interest associated with the received sensor data, where the one or more objects of interest are different from the authorized user. In addition, the processing system 720 may execute the object classification SW module 736 to classify each object of interest among the one or more objects of interest at least in part based on a database of contacts associated with the authorized user. Still further, the processing system 720 may execute the display control SW module 738 to control a display associated with the computing device based on the classification of each object of interest.

[0078] Figure 8 An illustrative flowchart showing an example operation 800 for protecting the privacy of a user of a personal computing device in the presence of bystanders is shown. In some implementations, the example operation 800 may be performed by a privacy controller for a personal computing device, such as any privacy controller among the privacy controllers 230 or 700 of Figure 2 and Figure 7 respectively.

[0079] The privacy controller receives sensor data from one or more sensors associated with the computing device (810). The privacy controller identifies an authorized user of the computing device based on the received sensor data (820). The privacy controller also detects one or more objects of interest associated with the received sensor data, where the one or more objects of interest are different from the authorized user (830). The privacy controller classifies each object of interest among the one or more objects of interest at least in part based on a database of contacts associated with the authorized user (840). The privacy controller further controls a display associated with the computing device based on the classification of each object of interest (850).

[0080] In some aspects, the database may store object recognition data associated with one or more contacts. In some implementations, the object recognition data may include one or more images associated with one or more contacts. In some other implementations, the object recognition data may include one or more voice recordings associated with one or more contacts. In some aspects, the classification of each detected object of interest may include determining whether the object of interest matches any of the object recognition data stored in the database. In some implementations, the control of the display may include outputting a notification to the display in response to determining that at least one of the one or more objects of interest does not match any of the object recognition data stored in the database.

[0081] In some aspects, the privacy controller may, in response to determining that the object of interest matches the object recognition data associated with a first contact among one or more contacts, further determine the relationship between the authorized user and the first contact, wherein the object of interest is classified at least in part based on the relationship between the authorized user and the first contact. In some implementations, the database may further store information indicating the organizational role or title associated with each of the one or more contacts, and the relationship between the authorized user and the first contact may be determined at least in part based on the organizational role or title associated with the first contact. In some other implementations, the database may further store information indicating the communication history between the authorized user and each of the one or more contacts, and the relationship between the authorized user and the first contact may be determined at least in part based on the communication history between the authorized user and the first contact.

[0082] In some aspects, the privacy controller may further determine the type of content presented on the display, wherein the object of interest is further classified at least in part based on the type of content presented on the display. In some implementations, the object of interest may be classified as a trusted bystander for the type of content presented on the display in response to determining that the relationship between the authorized user and the first contact meets a set of criteria associated with the type of content. In some other implementations, the object of interest may be classified as an untrusted bystander for the type of content presented on the display in response to determining that the relationship between the authorized user and the first contact does not meet a set of criteria associated with the type of content. In some implementations, the control of the display may include blurring the content presented on the display in response to classifying at least one of the one or more objects of interest as an untrusted bystander for the type of content.

[0083] Those skilled in the art will appreciate that information and signals can be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0084] In addition, those skilled in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithmic steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and the design constraints imposed on the overall system. The skilled person may implement the described functionality in a different manner for each particular application, but such implementation decisions should not be construed as causing a departure from the scope of the present disclosure.

[0085] The methods, sequences, or algorithms described in connection with the aspects disclosed herein may be implemented directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integrated with the processor.

[0086] In the foregoing specification, embodiments have been described with reference to specific examples thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader scope of the disclosure as set forth in the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method performed by a controller for a computing device, comprising: receiving sensor data from one or more sensors associated with the computing device; identifying an authorized user of the computing device based on the received sensor data; detecting one or more objects of interest associated with the received sensor data, the one or more objects of interest being different from the authorized user; classifying each of the one or more objects of interest based at least in part on a database of contacts associated with the authorized user; as well as A display associated with the computing device is controlled based on the classification of each object of interest.

2. The method of claim 1, wherein: The database stores object identification data associated with one or more contacts.

3. The method of claim 2, wherein: The object recognition data includes one or more images associated with the one or more contacts.

4. The method of claim 2, wherein: The object recognition data includes one or more voice recordings associated with the one or more contacts.

5. The method of claim 2, wherein: The classification of each object of interest includes: A determination is made as to whether the object of interest matches any of the object identification data stored in the database.

6. The method of claim 5, wherein: The controlling of the display includes: In response to determining that at least one of the one or more objects of interest does not match any of the object identification data stored in the database, a notification is output to the display.

7. The method of claim 5, further comprising: In response to determining that the object of interest matches the object identification data associated with a first contact of the one or more contacts, a relationship between the authorized user and the first contact is determined, and the object of interest is classified based at least in part on the relationship between the authorized user and the first contact.

8. The method of claim 7, wherein: The database further stores information indicating an organizational role or title associated with each of the one or more contacts, the relationship between the authorized user and the first contact being determined based at least in part on the organizational role or title associated with the first contact.

9. The method of claim 7, wherein: The database further stores information indicating a communication history between the authorized user and each of the one or more contacts, the relationship between the authorized user and the first contact being determined at least in part based on the communication history between the authorized user and the first contact.

10. The method of claim 7, further comprising: A type of content presented on the display is determined, and the object of interest is further classified based at least in part on the type of content presented on the display.

11. The method of claim 10, wherein: In response to determining that the relationship between the authorized user and the first contact satisfies a set of criteria associated with a type of content, the object of interest is classified as a trusted bystander for the type of content presented on the display.

12. The method of claim 10, wherein: In response to determining that the relationship between the authorized user and the first contact does not satisfy a set of criteria associated with the type of content, classifying the object of interest as an untrusted bystander for the type of content presented on the display.

13. The method of claim 12, wherein: The controlling of the display includes: In response to classifying at least one of the one or more objects of interest as an untrusted bystander of the type for content, the content presented on the display is obfuscated.

14. A controller for a computing device, comprising: Processing systems; as well as a memory storing instructions that, when executed by the processing system, cause the controller to: receiving sensor data from one or more sensors associated with the computing device; identifying an authorized user of the computing device based on the received sensor data; detecting one or more objects of interest associated with the received sensor data, the one or more objects of interest being different from the authorized user; classifying each of the one or more objects of interest based at least in part on a database of contacts associated with the authorized user; as well as A display associated with the computing device is controlled based on the classification of each object of interest.

15. The controller of claim 14, wherein: The database stores object identification data associated with one or more contacts.

16. The controller of claim 15, wherein: The classification of each object of interest includes: A determination is made as to whether the object of interest matches any of the object identification data stored in the database.

17. The controller of claim 16, wherein: Execution of the instructions further causes the controller to: In response to determining that the object of interest matches the object identification data associated with a first contact of the one or more contacts, a relationship between the authorized user and the first contact is determined, and the object of interest is classified based at least in part on the relationship between the authorized user and the first contact.

18. The controller of claim 17, wherein: The database further stores information indicating an organizational role or title associated with each of the one or more contacts, the relationship between the authorized user and the first contact being determined based at least in part on the organizational role or title associated with the first contact.

19. The controller of claim 17, wherein: The database further stores information indicating a communication history between the authorized user and each of the one or more contacts, the relationship between the authorized user and the first contact being determined at least in part based on the communication history between the authorized user and the first contact.

20. The controller of claim 17, wherein: Execution of the instructions further causes the controller to: A type of content presented on the display is determined, and the object of interest is further classified based at least in part on the type of content presented on the display.

Images