Privacy control for shared embeddings for searching and indexing media content

Abstract

This document describes techniques and systems that implement privacy controls for searching and indexing shared embeddings of media content. A set of images of a user's face is obtained and a machine learning model (122) is applied to the set of images to generate a user-specific dataset (124) of face embeddings for the user. Media content stored in a media store (116, 136) is indexed by applying the machine learning model (122) to the media content to provide indexed media information (118) that identifies one or more faces shown in the media content. Access to the indexed media information (118) by other users querying the media content for images or videos depicting the user is controlled based on a digital key (112) shared by the user with the other users, where the digital key (112) is associated with the user-specific dataset (124) that can be used to identify images or videos that depict the user.

Description

Background Technology

[0001] Deep learning models can generate embeddings for different types of input—such as multiple input images of a given user's face. An embedding is an n-dimensional vector representing an image and can be used by machine learning techniques to represent the visual appearance of a person based on multiple images. Embeddings can be used in a variety of applications to create models that can be used to identify people in media content such as images, videos, or audio content. Depending on the application, embeddings can be used to capture, for example, a user's face, body, fingerprints, or even voice. While these embeddings are a powerful tool, they can also be used to identify sensitive user data. Summary of the Invention

[0002] This document describes techniques and systems for implementing privacy controls using shared embeddings for searching and indexing media content. The techniques described herein allow users to grant access to search and indexing features based on embeddings that map user faces to a limited whitelist of users, and subsequently revoke access from one or more of these users.

[0003] The technology and system include a method for controlling the privacy of shared embeddings used for searching and indexing media content. The method includes obtaining a set of images of a first user's face and applying a machine learning model to the image set to generate a user-specific dataset for the first user's facial embeddings. The media content stored in media storage is then indexed based on the user-specific dataset to provide indexed information identifying one or more faces shown in the media content. Access to the indexed information by the second user to query media content for images or videos depicting the first user is controlled based on a digital key shared by the first user and a second user, wherein the digital key is associated with the user-specific dataset, and the user-specific dataset can be used to identify images or videos depicting the first user.

[0004] Digital keys can be stored "in the cloud," such as at the service provider's system or in other secure online storage associated with the service provider's system, rather than on the user's device. The service provider's system can provide users with online storage of personalized media collections. These personalized media collections can be account-based and securely encrypted. In this implementation, users can provide user credentials to log into their accounts and then initiate search queries through the service provider's system based on their accounts. In this way, the shared digital key is not shared with any actual user device, thus providing greater security against forgery and duplication. Furthermore, personalized media collections searchable for a specific user can be linked to that user's account, thereby protecting other media collections from unauthorized searches.

[0005] Users' personalized media collections can be indexed, and the service provider system can store the indexing information for each media collection separately. Keeping the indexing information for different users stored separately can provide an additional level of security against unauthorized access (e.g., granting access to the indexing information of a first user increases the risk of unauthorized access to the indexing information of a second user).

[0006] This invention is provided to introduce simplified concepts of privacy controls for shared embeddings used in searching and indexing media content, which will be further described in detail below with accompanying drawings. This invention is not intended to identify essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter. Attached Figure Description

[0007] Details of one or more aspects of the privacy controls for shared embeddings used to search and index media content are described in this document with reference to the following figures. The same numbering is used throughout the figures to refer to similar functions and components:

[0008] Figure 1 The illustration shows an example environment in which technologies for sharing embedded privacy controls for searching and indexing media content can be implemented.

[0009] Figure 2 The illustration shows an example implementation of an electronic device that can implement shared, embedded privacy controls for searching and indexing media content.

[0010] Figure 3 The illustration shows an example implementation of privacy controls for shared embeddings used for searching and indexing media content.

[0011] Figure 4 Example search results depicting user-specific search queries based on the techniques described in this article.

[0012] Figure 5 Describe an example method for controlling the privacy of shared embeddings used for searching and indexing media content.

[0013] Figure 6 Described Figure 5 Additional details of the method include a method for controlling access to media content for a user's specific search query.

[0014] Figure 7 Described Figure 5 Additional details of the method include a method for updating a user-specific dataset based on additional data.

[0015] Figure 8 Described Figure 5Additional details about the methods in the document include methods for updating user-specific datasets based on data removal.

[0016] Figure 9 Describe an example method for controlling the privacy of shared embeddings used for searching and indexing media content.

[0017] Figure 10 The diagram can be implemented as shown in the reference. Figures 1-9 The aforementioned example computing system, comprising any type of client, server, and / or electronic device, implements privacy controls for shared embeddings used for searching and indexing media content, or an example computing system in which technologies for implementing privacy controls for shared embeddings used for searching and indexing media content can be implemented. Detailed Implementation

[0018] Overview

[0019] This document describes techniques and systems for implementing privacy controls on shared embeddings used for searching and indexing media content. Shared embeddings of users—such as facial embeddings—can allow another user to detect a user's face in arbitrary data, which may be undesirable. For example, a leaked embedding could allow anyone using it to detect a user's face in a public video stream. The techniques described in this document provide software-based solutions to control search access to specific users described in media content. In particular, the techniques allow users to grant access to search and indexing features based on embeddings that map a user's face to a limited whitelist of users, and subsequently revoke access to one or more of those users.

[0020] In various aspects, a method for privacy controls of facial embedding sharing for searching and indexing media content is described. The method is performed by a service provider system, which obtains a set of images of a first user's face and applies a machine learning model to the image set to generate a user-specific dataset for facial embeddings of the first user. The method also includes the service provider system indexing media content stored in a media storage device by applying the machine learning model to the media content to provide indexed information identifying one or more faces shown in the media content. Furthermore, the method includes the service provider system controlling, via an application programming interface, a second user's access to the indexed information in the media content for querying images or videos depicting the first user. Access is controlled based on a digital key shared between the first and second users. The digital key is associated with the user-specific dataset. The user-specific dataset can be used in conjunction with the indexed information to identify images or videos depicting the first user in the media content.

[0021] These are just a few examples of how the described technologies and devices can be used to implement privacy controls for face embedding sharing or for searching and indexing media content. Other examples and implementations are also described in this document. This document now turns to example operating environments and then describes example devices, methods, and systems.

[0022] Operating environment

[0023] Figure 1 The illustration shows an example environment 100 in which privacy controls for face-embedded sharing, used for searching and indexing media content, can be implemented. Example environment 100 includes an electronic device 102 (e.g., a user device) configured to communicate with a service provider system 104 via a network 106. Electronic device 102 includes a privacy control module 108, an embedding module 110, one or more digital keys 112, and media content 114.

[0024] Service provider system 104 is shown as including media storage 116, indexed media information 118, storage service module 120, machine learning model 122, one or more user-specific datasets 124, and search manager module 126. Storage service module 120 can apply machine learning model 122 to media content 114 (e.g., image data, video data, audio data, etc.) provided by electronic device 102. Machine learning model 122 is trained using machine learning techniques (e.g., convolutional neural network) to generate embeddings from media content 114, thereby creating user-specific datasets 124 for the user depicted in media content 114. For example, media content 114 may include a collection of images of a user—such as the face of a user of electronic device 102. Storage service module 120 can run a general face detection algorithm to detect facial boundaries in the images. Then, using machine learning model 122, storage service module 120 computes embeddings (also called embedding vectors) of substantially all faces found in the images to create user-specific datasets 124 for the user. User-specific datasets 124 are one or more n-dimensional vectors representing the user's face. The user-specific dataset 124 is stored in a secure, encrypted "cloud" that is inaccessible to other devices.

[0025] The storage service module 120 can index the media storage 116 by identifying users present in the media storage 116—such as images or videos stored in the media storage 116—to generate indexed media information 118. For security reasons, the storage service module 120 only identifies those users who have consented to and chosen to store their user-specific datasets 124 in the cloud (e.g., at the service provider system 104). It is important to note that no other user or device is permitted to access the user-specific datasets 124 without the user's consent.

[0026] A user of electronic device 102 can initiate a search query to service provider system 104 to search for the user's image in media storage 116. Electronic device 102 invokes application programming interface (API) 128 at search manager module 126 to access indexed media information 118 and search for embedding information that substantially matches the user's user-specific dataset 124. Based on the matching embedding information, search manager module 126 can locate the corresponding image in media storage 116 depicting the user. The obtained image is then presented on electronic device 102 in a display application via display device 130.

[0027] A user of electronic device 102 can share access to indexed media information 118 corresponding to that user with one or more other users (e.g., family, friends, etc.). This allows other users to query media storage 116 using user-specific queries. In the example, user B initiates a user-specific query on their user device 132 to find images of user A (the user of electronic device 102) in media storage 116 of service provider system 104. However, neither machine learning model 122 nor user-specific dataset 124 is shared with user B. Instead, user B has a digital key corresponding to user A's user-specific dataset 124, previously provided by user A. User B provides the digital key to search manager module 126. API 128 of search manager module 126 uses the digital key to identify user A's user-specific dataset 124. API 128 compares the embeddings in user's user-specific dataset 124 with the indexed media information 118 to identify matching embeddings and index information corresponding to the matching embeddings. API 128 uses the identified index information to locate the corresponding image or video depicting user A from media storage 116. API 128 can then return the corresponding image to user B's user device 132 for display.

[0028] In some respects, user B may wish to use user A's user-specific query to query different image or video corpora. For example, user B can query media storage service 134 for user A's images in media storage 136. Alternatively, user B can query the local storage (not shown) of user device 132. In either case, user B initiates a query to API 128, which includes a digital key shared by user A and an identifier of the location of the specific image or video corpus to be searched (e.g., media storage 136 or local storage of user device 132). If these corpora have not yet been indexed by storage service module 120, API 128 applies (or invokes storage service module 120 to apply) machine learning model 122 to the specific image or video corpus to index the content. The storage service module 120 of service provider system 104 retains the obtained indexed information in indexed media information 118. The search manager module 126 can then compare the obtained indexed information with the user-specific dataset 124 to identify which images in a specific image or video corpus (e.g., media storage 136 or local storage of user device 132) include user A, and provide the results to user device 132 for use by user B.

[0029] In some aspects, electronic device 102 may include machine learning model 122, or instances of machine learning model 122. Embedding module 110 may apply machine learning model 122 to a set of images captured and / or stored by electronic device 102 in media content 114 to generate a user-specific dataset 124 (e.g., an embedding set). In some aspects, the image set may be images captured using a near-infrared camera, such as those captured by a facial authentication system (e.g., a face unlock application), which captures near-infrared image data of a user's face and generates an embedding (an n-dimensional numerical vector) that digitally represents the user's unique facial features without including personally identifiable information. Electronic device 102 may then transmit (e.g., upload) the embedding (e.g., user-specific dataset 124) to service provider system 104 for storage "in the cloud." In one example, electronic device 102 or service provider system 104 may create a three-dimensional (3D) representation of the face to be used as input data for facial embedding.

[0030] Privacy control module 108 is configured to generate a digital key 112 for sharing with authorized users. Only authorized users with the shared key (e.g., digital key 112) can invoke API 128 to access indexed media information 118 to identify specific images or videos depicting the user from an image or video corpus.

[0031] Because the user-specific dataset 124 is not directly exposed to other users (the only access granted to other users is for indexed content used for search queries), the granted access can be revoked. User A of electronic device 102 can opt to revoke shared access by deleting a digital key previously shared with a specific user (e.g., user B). Then, the next time user B initiates a search for an image or video depicting user A, no results will be provided.

[0032] Throughout this disclosure, it is described that a computing system (e.g., electronic device 102, client device, server device, service provider system 104, computer, or other type of computing system) can analyze information associated with a user (e.g., radar, inertial, and facial recognition sensor data, images), such as the facial image just mentioned. However, the computing system can be configured to use the information only after receiving explicit permission from the user of the computing system to use the data. Individual users may have constant control over what the sensor data can or cannot be done. For example, before electronic device 102 shares sensor data with other devices (e.g., to train models executed on other devices), electronic device 102 may preprocess the sensor data to ensure the removal of any user identification or device identification information embedded in the data. Thus, the user can control whether information about the user and the user's device is collected, and if so, how this information is used by the computing device and / or remote computing system.

[0033] While the examples described herein pertain to image data depicting a user's face, the techniques described herein can also be implemented on other types of data to generate corresponding embeddings, which can be used to search for media content corresponding to the user. Some other exemplary types of input may include voice data, large numbers of full-body images, fingerprint data, iris scan data, video data, etc. Embeddings generated from voice data can be used to locate video or audio files of a user speaking, or audio segments within those video or audio files. In one example, these techniques can be used to identify when a specific person speaks during a recorded meeting or who is speaking at different times during the meeting. Embeddings generated from a user's full-body image can be used to identify the user based on their gait or how they move. These techniques can also be used for facial authentication. For example, if a company uses facial authentication and has a user-specific dataset for its users, it can provide users with a digital key to access an API using facial scanning to enter a secure building. In yet another example, using these techniques, it is not necessary to require users to train new electronic devices for facial unlock applications. Instead, a digital key can be provided to new electronic devices, allowing them to call APIs to access user-specific datasets to authenticate users for facial unlock.

[0034] In more detail, consider Figure 2 The illustration shows an example implementation 200 of an electronic device 102 for implementing privacy controls for face-embedded sharing, used for searching and indexing media content. Figure 2 The electronic device 102 is shown to include various exemplary devices, including a smartphone 102-1, a tablet computer 102-2, a laptop computer 102-3, a desktop computer 102-4, a computing watch 102-5, computing glasses 102-6, a gaming system 102-7, a home automation control system 102-8, and a microwave oven 102-9. The electronic device 102 may also include other devices such as televisions, entertainment systems, audio systems, automobiles, drones, trackpads, drawing tablets, netbooks, e-readers, home security systems, and other home appliances. Note that the electronic device 102 may be wearable, non-wearable but mobile, or relatively fixed (e.g., desktop and appliance).

[0035] Electronic device 102 also includes one or more computer processors 202 and one or more computer-readable media 204 including memory media and storage media. An application and / or operating system 206, implemented as computer-readable instructions on the computer-readable medium 204, can be executed by the computer processor 202 to provide some or all of the functions described herein. For example, the computer-readable medium 204 may include a privacy control module 108, an embedding module 110, media content 114, a machine learning model 122, and a secure storage unit 208. The embedding module 110 can invoke the machine learning model 122. The privacy control module 108 can control (e.g., authorize, revoke) access to a digital key 112 that authorizes user-specific searches in the media storage.

[0036] Secure storage unit 208 is configured to store security data (e.g., user credentials) (including facial authentication data, password / password information, fingerprint data, etc.) for privacy controls—such as controls for unlocking electronic device 102. While this security data can be used to authenticate a user using facial authentication, password / password authentication, fingerprint authentication, voice authentication, etc., to unlock electronic device 102, it does not acquire personal information about the user. Specifically, the user cannot be identified by the security data. Instead, the security data is used to simply determine whether data received from a user attempting to unlock the phone matches stored profile data (e.g., user-specific dataset 124) representing a user who has set up security on electronic device 102. In the example, the embeddings generated from a captured image of the user's face are numerical vector representations of the user's facial features. These embeddings are only used to compare with new embeddings generated from images captured during a facial authentication attempt to locate a match. In other embodiments, these embeddings are used to compare with new embeddings generated from images depicting the user's face captured by the camera of electronic device 102 or stored in media content 114 (obtained from another device).

[0037] Electronic device 102 may also include network interface 210. Electronic device 102 may use network interface 210 for communicating data via wired, wireless, or optical networks. By way of example and not limitation, network interface 210 may communicate data via local area network (LAN), wireless local area network (WLAN), personal area network (PAN), wide area network (WAN), intranet, Internet, peer-to-peer network, point-to-point network, or mesh network.

[0038] Various implementations of the authentication system 212 may include a system-on-a-chip (SoC), one or more integrated circuits (ICs), a processor having embedded processor instructions or processor instructions configured to access processor instructions stored in memory, hardware with embedded firmware, a printed circuit board with various hardware components, or any combination thereof. In the example, the authentication system 212 may, in secure mode, compare authentication data received from a user with security data stored in a secure storage unit 208 to authenticate the user to unlock the electronic device 102. In some aspects, the authentication system 212 uses image data obtained from a camera system to generate authentication data and provides the authentication data to the secure storage unit 208 so that the secure storage unit 208 can compare the authentication data with the stored security data and determine if a match exists.

[0039] Electronic device 102 also includes a camera system 214 implemented to capture image data. The image data can be used to generate a three-dimensional depth map of an object—such as a user's face. Any suitable camera system can be used, including a color camera (e.g., a red-green-blue (RGB) camera or a near-infrared (NIR) camera). Camera system 214 can be integrated into electronic device 102 or otherwise associated with electronic device 102. In various respects, camera system 214 can be wirelessly connected to electronic device 102.

[0040] Electronic device 102 may also include one or more sensors 216. The one or more sensors 216 may include any of a variety of sensors such as an audio sensor (e.g., a microphone), a touch input sensor (e.g., a touch screen), an image capture device (e.g., a camera or camcorder), a proximity sensor (e.g., a capacitive sensor), or an ambient light sensor (e.g., a photodetector).

[0041] Electronic device 102 may also include display device 130. Display device 130 may include any suitable display device, such as touch screen, liquid crystal display (LCD), thin film transistor (TFT) LCD, in-plane switching (IPS) LCD, capacitive touch screen display, organic light-emitting diode (OLED) display, active matrix organic light-emitting diode (AMOLED) display, super AMOLED display, etc.

[0042] The electronic device 102 may also include a face detector 218. The face detector 218 can detect the facial boundaries of each of one or more faces in an image. By detecting facial boundaries, the face detector 218 limits the amount of image data processed by the embedding module 110. In an embodiment, the face detector 218 may be implemented as computer-readable instructions on a computer-readable medium 204 and executed by a computer processor 202 to detect the facial boundaries of individual faces in an image.

[0043] These and other capabilities and configurations, as well as Figure 1 and Figure 2 The actions and interactions of these entities will be described in more detail below. These entities can be further divided, combined, etc. Figure 1 Environment 100 Figures 2 to 4 Implementation methods 200, 300, and 400 and Figures 5 to 9 The detailed method illustrates some of the many possible environments and devices in which the technology can be employed.

[0044] Figure 3 The illustration shows an example implementation 300 of privacy controls for shared embeddings used for searching and indexing media content. Figure 3The diagram illustrates a service provider system 104 and multiple electronic devices 102, such as device A 302, device B 304, and device C 306. In the example shown, each of the devices corresponds to a different user; for example, device A 302 corresponds to user A 308, device B corresponds to user B 310, and device C corresponds to user C 312.

[0045] Each of the devices can be implemented using an instance of privacy control module 108 to control access to image or video searches for the corresponding user. In this exemplary implementation 300, user A 308 of device A 302 has selected to store user-specific dataset 124 (e.g., dataset A 314) in service provider system 104. Dataset A 314 is generated based on machine learning model 122 applied to a set of images of user A 308, or provided by device A 302, or stored in media storage 116 of service provider system 104, or both.

[0046] Service provider system 104 provides online storage for personalized media collections for users. These personalized media collections may be account-based and securely encrypted in media storage 116. Each user may have one or more personalized media collections, such as images and videos organized in different folders or subfolders. Because personalized media collections are account-based, users can log in to their accounts using any suitable electronic device and upload images and / or videos to create personalized media collections. For simplicity, the example shown depicts each user with one electronic device and one personalized media collection. For example, media storage 116 includes media collection A 318 uploaded by user A 308, media collection B 320 uploaded by user B 310, and media collection C 322 uploaded by user C 312.

[0047] To share access to a user-specific dataset 124 used for searching media content describing a user, a corresponding privacy control module 108 can provide a corresponding digital key 112. A new key is created for each user sharing access (e.g., 10 different keys are generated for 10 users). To revoke access for a specific user, the privacy control module 108 deletes its corresponding key. In the example shown, device A 302 and / or service provider system 104 store a set of digital keys 112 for user A 308 (e.g., owned keys 324, including keys A1, A2, ..., An). Furthermore, a set of digital keys 112 (e.g., owned keys 326, including keys B1, B2, ..., Bn) is stored by device B 304 and service provider system 104. In some implementations, only service provider system 104 stores each user's digital key 112, and the management of the digital key 112 is account-based, allowing users to log in to their accounts at service provider system 104 to manage the shared keys.

[0048] As further illustrated, user A 308 has shared a first key A1 (shown by shared key 328) with device B 304 and a second key A2 (shown by shared key 330) with device C 306. Similarly, device B 304 shares key B1 with device C 306 but does not provide any key to device A 302. In the example shown, device C 306 does not possess any key because user C 312 did not choose to store user-specific datasets in service provider system 104, therefore, no digital key is generated for device C 306. Alternatively, digital key 112 may simply be stored "in the cloud," such as in service provider system 104 or other secure online storage associated with service provider system 104, and not on the user's device. In such an implementation, a user can provide user credentials to log in to their account and then initiate search queries through service provider system 104 based on their account. In this way, the shared digital key 112 is not shared with any user device, thus providing greater security against forgery and duplication. In addition, media collections that can be searched for a specific user can be linked to that user's account, thus protecting other media collections from unauthorized searches.

[0049] Suppose user B 310 wants to search for user A 308's personalized media collection B320 in their images and / or videos. User B 310 inputs a search query to device B 304. Device B 304 then invokes API 128 at service provider system 104, providing API 128 with a shared key A1 and an indication of media collection B320. If media collection B 320 is not yet indexed, API 128 can apply machine learning model 122 to media collection B 320 to generate corresponding indexed information. Service provider system 104 can store the corresponding indexed information separately from other indexed information corresponding to other media collections in indexed media information 118. Separately storing the indexed information of different users can provide an additional level of security to prevent unauthorized access (e.g., authorized access to the indexed information of the first user grants unauthorized access to the indexed information of the second user).

[0050] API 128 uses shared key A1 to access user A 308's dataset A 314 and compares the embeddings in dataset A 314 with the indexed media information 118 corresponding to media collection B 320. If the API locates a matching embedding in the indexed media information 118 corresponding to media collection B 320, API 128 identifies the corresponding image and / or video in media collection B 320 and returns those images and / or videos as results of the search query to device B 304. The results do not identify which face or person in the images and / or videos belongs to user A 308. Without such identification information, user B 310 cannot construct a copy of dataset A 314. The results only indicate images and / or videos depicting user A 308.

[0051] In some implementations, service provider system 104 may use the results of user B 310's search query to update user A 308's dataset A 314, even if user A 308 may not have access to user B 310's media collection B 320. The larger the user-specific dataset 124, the more accurate the results of the user-specific search query may be.

[0052] Because shared keys 330A2 and B1 are already shared with device C 306, user C 312 can initiate a search query for images containing one or both of user A 308 and user B 310. Assume user C 312 initiates a search query for images of either user A 308 or user B 310. Privacy control module 108 provides both keys A2 and B1 to API 128, and API 128 identifies the corresponding datasets (e.g., dataset A 314 and dataset B 316). API 128 compares these datasets with indexed media information 118 corresponding to user C 312's media collection C 322. Based on matching embeddings, service provider system 104 returns search results identified from media collection C 322 that show images of one or both of user A 308 and user B 310.

[0053] Note that user B 310 does not share a digital key with user A 308 (e.g., one of the keys 326 they possess). Therefore, user A 308 is not permitted to use dataset B 316 for search queries. More specifically, if user A 308 initiates a search query 310 for user B 310's images in media collection A 318, service provider system 104 does not return any results. Because device A 302 does not provide the appropriate digital key to API 128, API 128 cannot identify which user-specific dataset 124 is used for comparison with indexed media information 118.

[0054] Similarly, if user A 308 chooses to revoke access to dataset A from user C 312, user A 308 can enter a command to cause privacy control module 108 to delete the previously shared key A2 at service provider system 104 (and at device A 302). Then, when device C 306 uses key A2 to perform a subsequent query search on the media collection C 322 that depicts user A 308's images or videos 308, service provider system 104 does not return results, regardless of whether similar search queries have previously yielded results. Thus, device C is essentially prevented from performing user-specific searches on images or videos depicting user A 308.

[0055] Furthermore, in the example shown, user C 312 did not choose to store user-specific dataset 124 in service provider system 104. Therefore, neither device A 302 nor device B 304 can obtain results for search queries on user C 312's images and / or videos.

[0056] continue Figure 3 The example shown, Figure 4 Example search results for user-specific search queries based on the techniques described herein are depicted.Figure 4 In the index 400, there is example indexing information (e.g., indexed media information 118) indicating that image A 402 contains user A 308 and user B 310, image B contains user B 310, and video C 404 includes user A 308 in frame XY and user B 310 in frame QZ.

[0057] User C 312 of device C 306 enters a search query for "images and / or videos of user A" in search bar 406. Device C 306 combines the search query with an appropriate shared key (e.g., ...). Figure 3 The shared key A2 shown in the shared key 330 is transmitted to API 128. API 128 uses the shared key A2 to identify the corresponding user-specific dataset 124 (e.g., dataset A 314) for comparison with index 400. Based on this comparison, API 128 returns search results 408 including image A 402 and video C 404.

[0058] In another example, user A 308 of device A 302 enters a search query for "videos with user B" in search bar 410. No results are provided in search pane 412 because device A 302 cannot provide a copy of one of the shared keys, such as one of the keys 326 owned by user B 310, corresponding to user B 310's dataset B 316, along with the search query.

[0059] Example Method

[0060] Figures 5 to 8 Example methods 500, 600, 700, and 800 are described for controlling the privacy of shared embeddings used for searching and indexing media content. These methods can be executed by service provider system 104, which uses search manager module 126 to control access to search image or video corpora using user-specific queries. Figure 5 A method for indexing media content based on user-specific datasets is described. Figure 6 Described Figure 5 Additional details of method 500 include method 600 for controlling access to media content used for a user-specific search query. Figure 7 Described Figure 5 Additional details of method 500, which includes method 700 for updating a user-specific dataset based on additional data. Figure 8 Described Figure 5 Additional details of method 500 include method 800 for updating a user-specific dataset based on data removal.

[0061] Methods 500, 600, 700, and 800 are shown as sets of blocks specifying operations to be performed, but are not necessarily limited to the order or combination of operations shown for performance by the individual blocks. Furthermore, any one or more operations may be repeated, combined, rearranged, or linked to provide a wide variety of additional and / or alternative methods. References may be made in the sections discussed below. Figure 1 Example operating environment 100 or Figures 2-4 The entities or processes detailed herein are for illustrative purposes only. These techniques are not limited to the performance of one or more entities running on a single device.

[0062] At 502, the service provider system obtains a set of images of the first user's face. For example, service provider system 104 may obtain the image set from electronic device 102.

[0063] At 504, the service provider system applies a machine learning model to the image set to generate a user-specific dataset with facial embeddings for the first user. For example, service provider system 104 may apply machine learning model 122 to the image set to generate a user-specific dataset 124 for the user of electronic device 102.

[0064] At 506, the service provider system indexes the media content stored in the media storage by applying a machine learning model to the media content to provide indexed information that identifies one or more faces shown in the media content. For example, service provider system 104 can index the media storage 116 stored in service provider system 104 by applying a user-specific dataset 124 to the media content in the media storage 116 and identifying images or videos depicting the user of the electronic device 102, which provides indexed media information 118.

[0065] At 508, the service provider system controls access to indexed information in media content queried by a second user for an image or video depicting a first user. For example, service provider system 104 may implement a search manager module 126 for controlling access to indexed media information 118. The search manager module includes an API 128, which acts as an intermediary between user devices (e.g., electronic device 102, user device 132) and secure data stored in service provider system 104. If the second user provides an authorization digital key, API 128 accesses the indexed media information 118 and uses a user-specific dataset 124 corresponding to the digital key to identify matching embeddings in the indexed media information 118 that indicate a specific image or video depicting the first user in media storage 116. Method 500 may optionally proceed to reference... Figures 6-8 The method described is any one of 600, 700 or 800.

[0066] Figure 6 Described Figure 5 Additional details of the described method 500 include a method 600 for controlling access to media content used for a user-specific search query. At 602, the service provider system receives a search query for an image or video depicting a first user from media content. For example, service provider system 104 receives a search query from requesting user device 132 for an image or video depicting a first user of electronic device 102. The search query includes a digital key previously shared by the first user of electronic device 102 and a second user of requesting user device 132. The search query also includes an indication of a specific media storage (e.g., a personalized media collection in media storage 116, media storage 136 at media storage service 134, or local storage at user device 132).

[0067] In 604, the service provider system uses a search manager module with an application programming interface (API) to determine whether a search query has an authorized digital key. For example, API 128 determines whether the digital key provided by the search query matches one of the digital keys 112 associated with the first user.

[0068] If the API determines that the digital key is not authorized ("No" at 604), then at 606, the service provider system does not return any results to the search query. For example, digital key 112 may have been deleted based on input from the first user, causing digital key 112 to no longer be authorized for use in the search. In some cases, the search query may not include the digital key.

[0069] If the API determines that the digital key is authorized ("Yes" at 604), then at 608, the API uses the digital key to identify a user-specific dataset. For example, API 128 uses digital key 112 to locate the first user's user-specific dataset 124, which provides an indication of which face to search for in media storage 116.

[0070] At 610, the API accesses indexed information to identify which images or videos from the media content depict the first user based on a user-specific dataset. For example, API 128 uses facial embeddings from user-specific dataset 124 to compare with indexed media information 118. Indexed media information 118 includes embeddings associated with faces in media storage 116. Accordingly, the embedding in indexed media information 118 that matches the facial embedding in user-specific dataset 124 directs API 128 to the images or videos in media storage depicting the first user.

[0071] At 612, the service provider system provides search results including images or videos depicting the identification of the first user from media content. For example, service provider system 104 transmits images or videos depicting the identification of the first user to requesting user equipment 132 as search results for a search query.

[0072] Figure 7 Described Figure 5 Additional details of method 500 include method 700 for updating a user-specific dataset based on additional data. At 702, the service provider system receives one or more additional images from the first user's electronic device. For example, electronic device 102 may upload one or more new images depicting the user's face, such as close-up photos or full-body photos.

[0073] In method 704, the service provider system applies a machine learning model to one or more additional images to generate one or more new facial embeddings. In method 706, the service provider system adds one or more new facial embeddings to the user-specific dataset used for the first user's facial embeddings to update the user-specific dataset. The additional embeddings can improve the user-specific dataset to achieve more accurate results compared to fewer embeddings (e.g., more accurate user identification in images or videos). Then, method 700 returns to... Figure 5 508 in the code controls access to indexed information.

[0074] Figure 8 Described Figure 5 Additional details of method 500 include method 800 for updating a user-specific dataset based on data removal. At 802, the service provider system receives input from a first user to remove one or more images from a set of images previously used to generate the user-specific dataset. For example, the first user may choose to remove or delete one or more images used to generate the user-specific dataset 124. The user may dislike a particular image, or the image may be an older image that no longer represents a good representation of the user's current facial features.

[0075] At 804, the service provider system deletes one or more images from the image collection based on input from the first user to provide a subset of images. For example, the storage service module 120 of service provider system 104 deletes the images selected by the user for removal.

[0076] In step 806, the service provider system updates the user-specific dataset for the face embeddings of the first user by applying a machine learning model to a subset of images to provide an updated user-specific dataset. When the image set is reduced, to update the user-specific dataset 124, the storage service module 120 can recreate the user-specific dataset 124. Alternatively, the storage service module 120 can delete the embeddings corresponding to the deleted images from the user-specific dataset 124. For multiple deleted images, multiple corresponding embeddings can be deleted from the user-specific dataset 124. Then, method 700 returns to... Figure 5 Method 508 controls access to the indexed information. In at least some aspects, methods 700 and 800 can be combined to allow user-specific datasets to be updated based on the addition and removal of images used to generate the user-specific datasets.

[0077] Figure 9 An example method 900 is described for controlling the privacy of shared embeddings of media content used for searching and indexing. This method 900 can be executed by an electronic device 102 that uses a privacy control module 108 to control access to search an image or video corpus using a user-specific query.

[0078] Method 900 is shown as a set of blocks that specify the operations to be performed, but is not necessarily limited to the shown order or combination of operations performed by the individual blocks. Furthermore, any one or more operations may be repeated, combined, rearranged, or linked to provide a wide range of additional and / or alternative methods. References may be made in the sections discussed below. Figure 1 Example operating environment 100 or Figures 2-4 The entities or processes detailed herein are for illustrative purposes only. These techniques are not limited to the performance of one or more entities running on a single device.

[0079] At 902, the electronic device captures a set of images of the first user of the electronic device. For example, the first user of the electronic device 102 may use camera system 214 to capture images (e.g., "selfies"). The images may be captured using a color camera of camera system 214. Alternatively, images may be captured using a near-infrared camera of camera system 214, such as during facial authentication in authentication system 212.

[0080] According to one option at 904, the electronic device transmits the image set to a service provider system for cloud storage and to generate a user-specific dataset for facial embedding for the first user. For example, electronic device 102 uploads the image set to service provider system 104 for secure storage in media storage 116.

[0081] As an alternative to transmitting the image set to the service provider system at 904, at 906, the electronic device may optionally apply a machine learning model to the image set to generate a user-specific dataset of facial embeddings for a first user. For example, electronic device 102 may apply a machine learning model 122 stored at electronic device 102 to the image set. Machine learning model 122 may be part of a facial authentication system (e.g., authentication system 212) used to create facial embeddings from captured user images to authenticate the user against the registered embeddings to unlock electronic device 102. Machine learning model 122 may generate a user-specific dataset 124 of facial embeddings based on the image set.

[0082] At 908, the electronic device transmits a user-specific dataset to a service provider system for cloud storage. For example, electronic device 102 may transmit a user-specific dataset 124, rather than a collection of images, to service provider system 104.

[0083] At 910, whether from 904 or 908, the electronic device shares a digital key associated with a user-specific dataset with the second user, enabling the second user to invoke a user-specific image search at the service provider's system for images or videos depicting the first user. The user of electronic device 102 can share a digital key 112 with a friend, enabling the friend to search media storage 116 for images or videos describing the user. A different digital key is provided for each of the user's friends.

[0084] In 912, the electronic device revokes access to user-specific image searches by requesting the service provider system to delete a digital key previously shared with a second user. For example, a user of electronic device 102 can revoke access previously granted to a friend by deleting a digital key shared with a friend. Therefore, the friend is no longer authorized to use the user's user-specific dataset 124 to search for images or videos depicting the user.

[0085] Generally, any of the components, modules, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Some operations of the example methods can be described in the general context of executable instructions of a local and / or remote computer processing system stored on a computer-readable storage medium, and implementations may include software applications, programs, functions, etc. Optionally or additionally, any functionality described herein may be performed at least in part by one or more hardware logic components such as, but not limited to, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0086] Example computing system

[0087] Figure 10 The illustration shows various components of a computing system 1000, which can be implemented as any type of client, server, and / or electronic device, as shown in the previous illustration. Figures 1-9 The aforementioned implementation of privacy controls for shared embeddings used for searching and indexing media content.

[0088] The computing system 1000 includes a communication device 1002 that enables wired and / or wireless communication of device data 1004 (e.g., radar data, authentication data, reference data, received data, data being received, data scheduled for broadcast, and data packets). Device data 1004 or other device content may include device configuration settings, media content stored on the device, and / or information associated with the device's user (e.g., the identity of a person within the radar field or customized air gesture data). Media content stored on the computing system 1000 may include any type of radar, biometric, audio, video, and / or image data. The computing system 1000 includes one or more data inputs 1006 through which any type of data, media content, and / or input can be received, such as human speech, interaction with the radar field, touch input, user-selectable input or interaction (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and / or image data received from any content and / or data source.

[0089] The computing system 1000 also includes a communication interface 1008, which can be implemented as one or more of a serial and / or parallel interface, a wireless interface, any type of network interface, a modem, and any other type of communication interface. The communication interface 1008 provides a connection and / or communication link between the computing system 1000 and a communication network, through which other electronic, computing, and communication devices communicate data with the computing system 1000.

[0090] The computing system 1000 includes one or more processors 1010 (e.g., any of a microprocessor, controller, or other controller) capable of processing various computer-executable instructions to control the operation of the computing system 1000 and to implement technologies or technologies capable of implementing shared embedded privacy controls for searching and indexing media content. Optionally or additionally, the computing system 1000 may be implemented using any one or a combination of hardware, firmware, or fixed logic circuitry, which is combined with processing and control circuitry typically identified at 1012. Although not shown, the computing system 1000 may include a system bus or data transfer system coupling various components within the device. The system bus may include any one or a combination of different bus structures such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any one of various bus architectures.

[0091] The computing system 1000 also includes a computer-readable medium 1014, such as one or more memory devices capable of persistent and / or non-transitory data storage (compared to simple signal transmission), examples of which include random access memory (RAM), non-volatile memory (e.g., any one or more of read-only memory (ROM), flash memory, EPROM, EEPROM, etc.), and disk storage devices. The disk storage device can be implemented as any type of magnetic or optical storage device, such as a hard disk drive, a recordable and / or rewritable optical disc (CD), any type of digital versatile optical disc (DVD), etc. The computing system 1000 may also include a mass storage medium device (storage medium) 1016.

[0092] Computer-readable medium 1014 provides a data storage mechanism for storage device data 1004, as well as various device applications 1018 and any other types of information and / or data related to the operation of computing system 1000. For example, operating system 1020 may be maintained as a computer application having computer-readable medium 1014 and executing on processor 1010. Device application 1018 may include device managers, such as any form of control application, software application, signal processing and control module, device-specific native code, abstraction module, air gesture recognition module, and other modules. Device application 1018 may also include system components, engines, modules, or managers to implement privacy controls for shared embeddings used to search and index media content, such as storage service module 120 or search manager module 126. Computing system 1000 may also include or have access to one or more machine learning systems.

[0093] Some examples are described below:

[0094] Example 1. A method for controlling the privacy of shared embeddings used for searching and indexing media content, the method performed by a service provider system: obtaining a set of images of a first user's face; applying a machine learning model to the set of images to generate a user-specific dataset for embedding the face of the first user; indexing the media content stored in media storage by applying the machine learning model to the media content to provide indexed information identifying one or more faces shown in the media content; and controlling access to the indexed information by a second user via an application programming interface to query the media content for images or videos depicting the first user, the access being controlled based on a digital key shared by the first user and the second user, the digital key being associated with the user-specific dataset, the user-specific dataset being usable for comparison with the indexed information to identify images or videos depicting the first user in the media content.

[0095] Example 2. The method according to Example 1, wherein the user-specific dataset is securely encrypted and the second user cannot access it.

[0096] Example 3. The method according to any of the preceding examples, wherein the image set comprises a plurality of color images.

[0097] Example 4. The method according to any of the foregoing examples further includes: receiving a search query for images or videos depicting the first user from the media content, the search query including a digital key associated with the user-specific dataset; using the digital key through the application programming interface to access the indexed information to identify which images or videos from the media content depict the first user; and providing search results including the identified images or videos depicting the first user from the media content.

[0098] Example 5. The method according to Example 4 further includes: identifying the user-specific dataset based on the digital key; and searching the indexed information for one or more embeddings that match one or more facial embeddings in the user-specific dataset.

[0099] Example 6. The method according to Example 4 or 5, wherein the identified video includes one or more frames in the identified video depicting the face of the first user.

[0100] Example 7. The method according to any of the foregoing examples, wherein the media storage is owned by a third-party entity.

[0101] Example 8. The method according to any of the foregoing examples further includes: receiving one or more additional images from the first user's electronic device; updating the user-specific dataset for the face embedding for the first user by: applying the machine learning model to the one or more additional images to generate one or more new face embeddings; and adding the one or more new embeddings to the user-specific dataset for the face embedding for the first user.

[0102] Example 9. The method according to any of the foregoing examples further comprises: deleting one or more images from the image set previously used to generate the user-specific dataset based on user input from the first user, the deletion providing a subset of images; and updating the user-specific dataset for the face embedding of the first user by applying the machine learning model to the subset of images to generate an updated user-specific dataset.

[0103] Example 10. The method according to any of the foregoing examples further includes: using a face detector to detect the facial boundaries of each face in the image set.

[0104] Example 11. The method according to any of the foregoing examples further includes: maintaining a set of digital keys shared by the first user and other users, each digital key in the set being shared with different users.

[0105] Example 12. The method according to any of the preceding examples, wherein the media content includes a media collection associated with the second user; the method further includes: searching for the media collection associated with the second user for an image or video depicting the first user.

[0106] Example 13. The method according to any of the foregoing examples further includes: receiving a user selection to delete a digital key shared by the first user and the second user; receiving a subsequent search query from the second user's electronic device, the subsequent search query including the digital key shared by the first user; and not returning any results to the search query based on determining that the digital key is not included in the set of digital keys associated with the user-specific dataset used by the first user.

[0107] Example 14. The method according to any of the foregoing examples further includes: receiving a second search query from a third user for an image or video from the media content depicting the first user, the second search query not including a digital key associated with the user-specific dataset; and not returning any results to the second search query.

[0108] Example 15. A service provider system comprising: media storage; a storage service module for managing data stored in the media storage; a machine learning model for generating a user-specific dataset for facial embedding for a particular user; and a processor and a memory for implementing the method described in any of the preceding examples.

[0109] in conclusion

[0110] Although embodiments and apparatuses for implementing privacy controls for shared embeddings used in searching and indexing media content have been described in language specific to features and / or methods, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, specific features and methods are disclosed as exemplary embodiments of privacy controls for shared embeddings used in searching and indexing media content.

Claims

1. A method for controlling the privacy of shared embeddings used for searching and indexing media content, the method being performed by a service provider system: Obtain a set of facial images of the first user; A machine learning model is applied to the image set to generate a user-specific dataset for facial embedding of the first user; By applying the machine learning model to the media content, the media content stored in the media storage is indexed to provide indexed information for identifying one or more faces shown in the media content; as well as The second user's access to the indexed information for querying media content based on an image or video depicting the first user is controlled via an application programming interface. This access is based on a digital key shared by the first and second users and is associated with a user-specific dataset that can be used to compare with the indexed information to identify images or videos depicting the first user in the media content. Receive a search query for an image or video depicting the first user from the media content, the search query including a digital key associated with a user-specific dataset; Using the digital key through the application programming interface, the indexed information is accessed to identify which images or videos from the media content depict the first user; as well as Provide search results including images or videos from the media content that depict the first user's identified images or videos.

2. The method according to claim 1, wherein, The user-specific dataset is securely encrypted and inaccessible to the second user.

3. The method according to claim 1, wherein, The image set includes multiple color images.

4. The method according to claim 1, further comprising: The user-specific dataset is identified based on the digital key; as well as The indexed information is searched for one or more embeddings that match one or more facial embeddings in the user-specific dataset.

5. The method according to claim 1, wherein, The identified video includes one or more frames in the identified video that depict the face of the first user.

6. The method according to claim 1, wherein, The media storage is owned by a third-party entity.

7. The method according to claim 1, further comprising: Receive one or more additional images from the first user's electronic device; as well as The user-specific dataset used for the face embedding of the first user is updated as follows: The machine learning model is applied to the one or more additional images to generate one or more new facial embeddings; as well as Add the one or more new embeddings to the user-specific dataset used for the facial embedding of the first user.

8. The method according to claim 1, further comprising: Based on user input from the first user, one or more images are deleted from the image set previously used to generate the user-specific dataset, the deletion providing a subset of images; as well as By applying the machine learning model to the subset of images, the user-specific dataset used for the facial embedding of the first user is updated to generate an updated user-specific dataset.

9. The method according to claim 1, further comprising: The facial boundaries of each face in the image set are detected using a face detector.

10. The method according to claim 1, further comprising: Maintain a set of digital keys shared by the first user and other users, wherein each digital key in the set is shared with different users.

11. The method according to claim 1, wherein: The media content includes a collection of media associated with the second user; as well as The method further includes: searching for the media collection associated with the second user for images or videos depicting the first user.

12. The method according to claim 1, further comprising: The user who receives the deletion request is selected by the first user and the second user to delete the digital key shared by the first user and the second user; Receive subsequent search queries from the second user's electronic device, the subsequent search queries including a digital key shared by the first user; as well as Based on the determination that the digital key is not included in the set of digital keys associated with the user-specific dataset used by the first user, no results are returned to the search query.

13. The method according to claim 1, further comprising: Receive a second search query from a third user for an image or video from the media content that depicts the first user, wherein the second search query does not include a digital key associated with the user-specific dataset; as well as No results are returned for the second search query.

14. A service provider system, comprising: Media storage; Storage service module, which is used to manage the data stored in the media storage; A machine learning model, wherein the machine learning model is used to generate a user-specific dataset for facial embeddings for a specific user; and A processor and a memory, said processor and processor memory being used to implement the method of any one of claims 1-13.

Images