Neural network inference method and device for protected data

Abstract

A method and apparatus for performing inferences on protected data. A user device retrieves the protected data from a secure memory, generates inferences about the protected data using one or more neural network models stored on the user device, and updates a user interface of the user device based at least in part on the inferences. The secure memory is inaccessible to applications executing in a rich environment of the user device. Thus, in some aspects, the inferences may be generated at least in part by a neural network application executing in a trusted environment of the user device.

Description

Technical Field

[0001] The present embodiments generally relate to systems and devices for machine learning. Technical Background

[0002] Machine learning is a technique for improving the ability of a computer system or application to perform a task. Machine learning can be broken down into two components: training and inference. During the training phase, the machine learning system is provided with an "answer" and a large amount of raw data associated with the answer. For example, a machine learning system can be trained to recognize felines by providing the system with a large number of feline photos and / or videos (e.g., raw data) and an indication that the provided media contains "felines" (e.g., answers). The machine learning system can then analyze the raw data to "learn" a set of rules that can be used to describe the answer. For example, the system can perform statistical analysis on the raw data to determine a common set of rules that can be associated with the term "feline" (e.g., whiskers, claws, fur, four legs, etc.). The set of rules can be referred to as a neural network model. During the inference phase, the machine learning system can apply rules to new data to generate answers or inferences about the data. For example, the system can analyze family photos and determine that the photos include images of felines based on the learned rules. Summary of the invention

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] A method and apparatus for performing inference on protected data is disclosed. One innovative aspect of the subject matter of the present disclosure can be implemented in a method of inference by a user device. In some embodiments, the method can include the following steps: retrieving protected data from a secure memory, wherein the secure memory is inaccessible to applications executing in a rich environment of the user device; generating inferences about the protected data using one or more neural network models stored on the user device; and updating a user interface of the user device based at least in part on the inferences.

[0005] Another innovative aspect of the subject matter of the present disclosure can be implemented in a user device. The user device includes a processing circuit configured to operate in a secure state or an unsecure state and a memory having a secure partition and an unsecure partition. The secure partition stores instructions that, when executed by the processing circuit when operating in the secure state, cause the user device to retrieve protected data from the secure partition, wherein the secure partition is inaccessible to the processing circuit when operating in the unsecure state; retrieve one or more neural network models from the secure partition; and generate inferences about the protected data using the one or more neural network models. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present embodiments are illustrated by way of example and are not intended to be limited by the figures of the accompanying drawings.

[0007] Figure 1 A block diagram of a machine learning system according to some embodiments is shown.

[0008] Figure 2 A block diagram of a user equipment according to some embodiments is shown.

[0009] Figure 3 A sequence diagram depicting an example process for performing machine learning on protected data in accordance with some embodiments is shown.

[0010] Figure 4 A block diagram of a processing system is shown in accordance with some embodiments.

[0011] Figure 5 Another block diagram of a user equipment according to some embodiments is shown.

[0012] Figure 6 is an illustrative flow diagram depicting operations for inference by a user device in accordance with some embodiments.

[0013] Figure 7 is an illustrative flow chart depicting example image capture operations in accordance with some embodiments.

[0014] Figure 8 is an illustrative flow chart depicting example media playback operations in accordance with some embodiments.

[0015] Fig. 9 is an illustrative flow chart depicting an example authentication operation in accordance with some embodiments. DETAILED DESCRIPTION

[0016] In the following description, many specific details, such as examples of specific components, circuits and processes, are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupling" means being directly connected to or connected by one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific terms are set forth to provide a thorough understanding of the various aspects of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be needed to practice the example embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid making the present disclosure unclear. Some parts of the following detailed description are presented according to other symbolic representations of processes, logic blocks, processing and operations on data bits in computer memory. The interconnection between circuit elements or software blocks may be shown as a bus or as a single signal line. Each of the bus may alternatively be a single signal line, and each of the single signal line may alternatively be a bus, and a single line or bus may represent any one or more of a large number of physical or logical mechanisms for communication between components.

[0017] Unless otherwise specifically stated as apparent from the following discussion, it is understood that throughout this application, discussions utilizing terms such as "access," "receive," "send," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," and the like refer to actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission, or display devices.

[0018] Unless explicitly described as being implemented in a particular manner, the techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Any features described as modules or components may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transitory computer-readable storage medium comprising instructions that, when executed, perform one or more of the methods described above. The non-transitory computer-readable storage medium may form part of a computer program product, which may include packaging materials.

[0019] Non-transitory processor-readable storage media may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, other known storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a processor-readable communication medium that carries or transmits code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer or other processor.

[0020] The various illustrative logical blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein may be executed by one or more processors. The term "processor" as used herein may refer to any general purpose processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0021] Figure 1 A block diagram of a machine learning system 100 is shown in accordance with some embodiments. System 100 includes a network environment 101 and a user device 110. In some embodiments, network environment 101 can communicate with user device 110 to perform machine learning on private and / or protected content stored on or otherwise accessible by user device 110.

[0022] The network environment 101 can be configured to generate one or more neural network models 102 through deep learning. Deep learning is a special form of machine learning in which a training phase is performed on multiple layers, generating a more abstract set of rules in each successive layer. Due to the way in which information is processed, deep learning architectures are often referred to as artificial neural networks (e.g., similar to biological nervous systems). For example, each layer of a deep learning architecture can be composed of multiple artificial neurons. The neurons can be interconnected across the layers so that input data (e.g., raw data) can be passed from one layer to another. More specifically, each layer of neurons can perform different types of transformations on the input data that will ultimately result in a desired output (e.g., an answer). The interconnected framework of neurons can be referred to as a neural network model. Therefore, the neural network model 102 can include a collection of rules that can be used to describe a particular object or feature (such as a cat).

[0023] In some aspects, the network environment 101 is a cloud computing platform that can receive raw data from multiple sources (including user devices 110 and / or other content sources in communication with the network environment 101). The network environment 101 can be trained to recognize a set of rules associated with the raw data (e.g., certain objects, features, quality of service, such as the quality of received signals or pixel data, and / or other detectable attributes). For example, in some aspects, the network environment 101 can be trained to recognize one or more logos (e.g., used in corporate branding and / or advertising). During the training phase, the network environment 101 can receive a large number of photos and / or videos containing such logos from one or more content sources communicatively coupled to the network environment 101. The network environment 101 can also receive an indication that the provided media contains a logo (e.g., in the form of user input from a user or operator who examines the media and / or utilizes the data or metadata provided by the media). The network environment 101 can then perform statistical analysis on the received photos and / or videos to determine a common set of features associated with (one or more) logos. In some aspects, the determined features (or rules) can form an artificial neural network across multiple abstraction layers. The network environment 101 may provide the set of rules (eg, as a neural network model 102 ) to the user device 110 for inference.

[0024] User device 110 may be any end user or edge device. User device 110 may interact with a user, for example, by receiving user input and / or outputting content to the user. In some aspects, user device 110 may be any device capable of providing a customizable user experience (such as a personalized user interface) based on a given user's preferences, activities, or habits. In some other aspects, user device 110 may be any device capable of capturing, storing, and / or playing back media content. Example user devices may include, but are not limited to, set-top boxes (STBs), computers, mobile phones, tablet computers, televisions (TVs), and the like.

[0025] The user device 110 may include a neural network application 112, a content storage 114, and a user interface 116. The content storage 114 may store or buffer media content (e.g., thermal images, optical images, video, audio, etc.) for playback and / or display on the user device 110 or a display device (not shown) coupled to the user device 110. In some aspects, at least some of the media content displayed by the user device 110 may correspond to premium media content 122 received (e.g., streamed) from one or more content delivery networks (CDNs) 120. For example, the premium media content 122 may include television programs, movies, and / or media content created by third-party content creators or providers (e.g., television networks, production studios, streaming services, etc.). In some implementations, the user device 110 may store or buffer the premium media content 122 in the content storage 114 for playback. For example, the content memory 114 may operate as a decoded video frame buffer that stores or buffers (decoded) full-frame pixel data associated with premium media content 122 to be rendered or displayed by the user device 110 .

[0026] The neural network application 112 may be configured to generate one or more inferences about the media content stored in the content storage 114. For example, in some aspects, the neural network application 112 may analyze the media content to infer or identify objects of interest contained therein (e.g., faces, landmarks, destinations, etc.). In some embodiments, the neural network application 112 may generate inferences based on the neural network model 102 received from the network environment 101. For example, during an inference phase, the machine learning system may apply the neural network model 102 to new media content stored in the content storage 114 (e.g., by traversing artificial neurons in an artificial neural network) to infer information about the new media content (e.g., whether the media content includes one or more known landmarks).

[0027] Aspects of the present disclosure recognize that it may be undesirable (if not impossible) to send certain media content to network environment 101, e.g., to further refine neural network model 102 and / or generate additional neural network models based on media content stored on user device 110. For example, content providers and / or creators may restrict the sharing or publication of premium media content 122 (e.g., in accordance with "premium content protection" or digital rights management (DRM) laws and regulations). Additionally, users may not wish to send their personal information to the cloud, where it may be accessible to others. Accordingly, embodiments described herein may be used to perform machine learning on media content in a manner that protects user privacy and the rights of content providers.

[0028] In some embodiments, the user device 110 may selectively send the filtered feedback data 104 back to the network environment. In some aspects, the filtered feedback data 104 may include limited information about the inference generated from the media content stored or buffered in the content memory 114. Specifically, the limited information may be selected or filtered so that it does not violate the privacy rights of the user of the user device 110 and / or the owner of the content. For example, the filtered feedback data 104 may not contain any pixel data or other information that can be used to reconstruct the original media content (e.g., images and / or videos) stored in the content memory 114. In addition, the filtered feedback data 104 may not disclose any personally identifiable information (e.g., name, age, gender, location, etc.) about the user of the user device 110 or any information that can be used to derive such personally identifiable information.

[0029] In some embodiments, neural network application 112 may perform additional training on neural network model 102 using content stored or buffered in content memory 114. For example, neural network application 112 may refine neural network model 102 and / or generate a new neural network model based on media content stored or buffered in content memory 114. In some aspects, neural network application 112 may provide updated neural network model to network environment 101 (e.g., as filtered feedback data 104) to further refine the deep learning architecture. In this way, network environment 101 may further refine its neural network model 102 based on media content stored on user device 110 (e.g., as well as media content stored on various other user devices) without receiving or accessing raw data corresponding to the actual media content.

[0030] Some media content, such as premium media content 122, may be stored in a secure repository on the user device 110 (e.g., in a trusted environment). The secure repository may be virtually and / or physically partitioned from the rest of the user device 110 so that only applications and / or hardware residing within the trusted environment may access data stored in the secure repository. In some aspects, the secure repository may be formed at least in part within the content memory 114. Thus, the premium media content 122 (and other protected content) may be stored within the secure repository of the content memory 114. Any hardware and / or applications operating outside of the trusted environment (e.g., in a rich environment) may be restricted from accessing data stored in the secure repository, while hardware and / or applications within the trusted environment may have very limited (if any) communication with the outside world.

[0031] Some neural network applications may operate in a rich environment rather than a trusted environment in order to communicate with the outside world (e.g., with network environment 101). However, neural network applications operating in a rich environment may not be able to access protected media content (such as premium media content 122 or other media content protected under DRM, copyright, privacy laws, etc.), and therefore may not be able to perform machine learning (or deep learning) on ​​such content.

[0032] In some embodiments, neural network application 112 may reside at least partially within a trusted environment of user device 110. Placing neural network application 112 within a trusted environment enables neural network application 112 to perform machine learning on protected media content (such as premium media content 122) that would otherwise be inaccessible to a neural network application operating in a rich environment. For example, neural network application 112 may access full-frame pixel data that may be used to render or display various television programs and / or movies that the user watches on user device 110. Thus, neural network application 112 may generate inferences about the interests and / or viewing habits of a user of user device 110 based on the television programs and / or movies that the user watches. In some embodiments, neural network application 112 (or another application executing on mobile device 110) may also provide recommendations and / or additional content to the user based on inferences about the user's viewing habits.

[0033] The user interface 116 may provide an interface or feature through which a user may operate, interact with, or otherwise use the device 110 or an electronic system (not shown for simplicity) coupled to the user device 110. In some aspects, the user interface 116 may include one or more input sources for receiving user input. Example input sources may include, but are not limited to, a camera, a microphone, a button, a keyboard, a mouse, a touchpad, a fingerprint scanner, a photoelectric sensor, an accelerometer, a gyroscope, and the like. In some other aspects, the user interface 116 may include one or more output sources for outputting content to the user. Example output sources may include, but are not limited to, a speaker, a display, a light, and the like. In some embodiments, the user interface 116 may display, present, or otherwise visualize content or information on the user device 110 based at least in part on the inferences generated by the neural network application 112.

[0034] Figure 2 A block diagram of a user device 200 according to some embodiments is shown. The user device 200 may be Figure 1The user device 200 includes a hardware platform 230 and a software execution environment 201. The hardware platform 230 may include any hardware of the user device 200 (e.g., a processor, a memory, a communication interface, etc.). The software execution environment 201 includes any software or instructions (e.g., a kernel, an operating system, an application, etc.) executed on the hardware platform 230.

[0035] In some embodiments, the software execution environment 201 may be partitioned into a rich environment 210 and a trusted environment 220. The rich environment 210 may include one or more user applications 212, a rich neural network application 214, and a trusted execution environment (TEE) client application programming interface (API) 216. The trusted environment 220 may include one or more trusted applications 222, a trusted neural network application 224, and a TEE kernel 226. As described above, the trusted environment 220 may be physically or virtually partitioned (e.g., separated or walled off) from the rich environment 210. More specifically, only software or instructions executed in the trusted environment 220 may access secure hardware (HW) resources 232 residing on the hardware platform 230. Communication between the rich environment 210 and the trusted environment 220 is only possible through the TEE client API 216. In this way, the TEE client API 216 can ensure that the secure hardware resources 232 are inaccessible to any software or instructions executed in the rich environment 210.

[0036] In some embodiments, secure hardware resources 232 may include a secure repository or memory (such as Figure 1 content storage 114 or at least a portion thereof). For example, in some aspects, the protected data may include premium content (e.g., television programs, movies, etc.) or other media content that may be protected under DRM, copyright, or other laws and / or regulations. In some other aspects, the protected data may include a biometric signature (e.g., an image of a user's face, a fingerprint, a retinal scan, a recording of a user's voice, etc.) or other media that may be used for authentication purposes. Due to the confidentiality and / or sensitivity of the protected data, the secure hardware resources 232 may be inaccessible to software and / or hardware outside of the trusted environment 220. In addition, applications within the trusted environment 220 (such as trusted applications 222 and trusted neural network applications 224) may be restricted from transmitting information associated with the protected data to the rich environment 210.

[0037] In some embodiments, the user device 200 can perform machine learning on data stored on the hardware platform 230. In some aspects, the user device 200 can obtain information from a network environment (such as Figure 1The network environment 101 of the present invention receives one or more neural network models that can be used to generate inferences about the data stored on the hardware platform 230. In some other aspects, the user device 200 (e.g., the rich neural network application 214 and / or the trusted neural network application 224) can locally learn or generate at least some of the neural network models based on the data stored on the hardware platform 230 (e.g., as described above with respect to Figure 1 described).

[0038] Rich neural network application 214 may apply a neural network model to unprotected data stored on hardware platform 230 (e.g., in rich environment 210). However, rich neural network application 214 may not be able to access protected data stored in secure hardware resources 232. In some embodiments, trusted neural network application 224 may also receive a neural network model (e.g., via TEE client API 216) and may apply the neural network model to protected data stored in secure hardware resources 232. In some aspects, trusted neural network application 224 may also access unprotected data stored on hardware platform 230 (e.g., in rich environment 210).

[0039] Because trusted neural network application 224 resides within trusted environment 220, neural network application 224 can perform machine learning on full-frame pixel data of protected data stored in secure hardware resources 232. Therefore, neural network application 224 can generate inferences about protected data. Although neural network application 224 may not send any raw data (e.g., pixel data) from secure hardware resources 232 outside of trusted environment 220, neural network application 224 can provide inferences about the raw data to rich environment 210. As described in more detail below, inferences can be used to provide an enhanced user experience to a user of user device 200. For example, because the inferences can indicate the interests, preferences, and / or behaviors of the user, user device 200 can use the inferences to output recommendations and / or additional content that may be relevant to the user.

[0040] In some aspects, protected data may include premium media content, such as television programs or movies, that may be presented or displayed on user device 200 or a display device (not shown for simplicity) coupled to user device 200. In some embodiments, neural network application 224 may infer objects of interest (e.g., people, places, signs, etc.) from the premium content. For example, neural network application 224 may identify a logo or symbol associated with a content provider (e.g., a television broadcast network, a movie production studio, etc.). Neural network application 224 may determine, based on the identified logo or symbol, that a user of user device 200 has a preference for television programs broadcast on a particular television station or network. As a result, neural network application 224 may recommend other programs from the same television network to the user.

[0041] In another example, neural network application 224 may identify a logo or symbol associated with a product brand and / or advertisement. In some aspects, neural network application 224 may determine that the user of user device 200 has a preference or interest in a particular brand or type of product based on the identified logo or symbol. Thus, neural network application 224 may present targeted advertisements for a particular brand or type of product (or related brands and / or products) to the user. In some other aspects, neural network application 224 may determine which advertisements and / or product brandings are most likely to have been viewed by the user of user device 200 based on the identified logo or symbol. Thus, neural network application 224 may provide attribution to a television network broadcasting media content for advertising or product placement.

[0042] In some other aspects, the protected data may include biometric data, such as a face, fingerprint, or voice recording, that can be used to authenticate a user of user device 200. In some embodiments, neural network application 224 may generate one or more neural network models for the user based on the biometric data. Neural network application 224 may use the neural network model to infer the identity of the user from images and / or videos subsequently captured via a camera or other image capture device resident on or coupled to user device 200. In some aspects, neural network application 224 may further display or recommend content to the user based on the identity of the user. For example, neural network application 224 may display television programs and / or movies that match the known interests of the user.

[0043] In some embodiments, other applications (such as user application 212) executing on user device 200 can also leverage inferences generated by trusted neural network application 224 to enhance their user experience. For example, some applications can be configured to detect and respond to voice input. Once an object of interest has been detected by trusted neural network application 224, the user can verbally instruct user device 200 to retrieve additional information about the detected object (e.g., without any physical interaction with user device 200). In addition, many voice-based applications use automatic speech recognition (ASR) to detect and respond to specific voice patterns. For example, an ASR application can look up specific phrases in one or more dictionaries or libraries to determine how to process or respond to voice input. Once an object of interest has been detected by trusted neural network application 224, user device 200 can retrieve relevant ASR dictionaries (e.g., related to the detected object) to prepare a response to the user's voice input.

[0044] In some embodiments, trusted neural network application 224 may generate additional neural network models based on data stored on secure hardware resources 232, or refine existing neural network models. In some aspects, rich neural network application 214 may receive updated neural network models and / or filtered feedback data (e.g., from trusted neural network application 224) via TEE client API 216. Rich neural network application 214 may also transmit the updated neural network models and / or filtered feedback data to a cloud or network environment (such as a Figure 1 In this way, the network environment can further refine the neural network model based on data stored on the media device 200 (e.g., and content from various other content sources) without receiving or accessing the original data stored in the secure hardware resources 232 of the media device 200.

[0045] Figure 3 A sequence diagram depicting an example process 300 for performing machine learning on protected data according to some embodiments is shown. Figure 1 , process 300 may be performed by machine learning system 100 to generate and / or apply a neural network model to data stored in a secure repository (e.g., in a trusted environment). Figure 3 In an embodiment of the present invention, process 300 is performed between secure content memory 312, neural network application 314, and network resources 322. Secure content memory 312 and neural network application 314 reside in a trusted execution environment 310 (such as a Figure 2 , while network resources 322 operate in a non-secure environment 320 (eg, the outside world).

[0046] The network resource 322 may be configured to generate one or more neural network models 302 through machine learning (e.g., deep learning). In some aspects, the network resource 322 may be a cloud computing platform that may receive raw data from multiple content sources. The network resource 322 may be trained to recognize a set of rules associated with the raw data (e.g., certain objects, features, quality of service, and / or other detectable attributes). During the training phase, the network resource 322 may receive a large amount of raw data from one or more content sources that are communicatively coupled to the network resource 322. The network resource 322 may also receive an indication of one or more rules associated with the raw data (e.g., in the form of user input from a user or operator who inspects the media and / or utilizes the data or metadata provided by the media). The network resource 322 may then perform statistical analysis on the raw data to determine a common set of attributes associated with the rules. In some aspects, the determined attributes (or rules) may form an artificial neural network across multiple abstraction layers. The network resource 322 may provide the set of rules (e.g., as a neural network model 302) to the neural network application 314 for inference.

[0047] Neural network application 314 may receive neural network model 302 from network resource 322 (e.g., via a TEE client API provided on a user device), and may use neural network model 302 to generate one or more inferences about data stored in secure content memory 312. Secure content memory 312 may store protected data (e.g., premium media content, biometric data, etc.) that may be displayed or presented on a user device or a corresponding display device. More specifically, the protected data may be inaccessible to hardware and / or applications outside of trusted execution environment 310. However, because neural network application 314 resides within trusted execution environment 310, neural network application 314 may have full access to the original data stored in secure content memory 312 (including full-frame pixel data of images and / or videos to be displayed on the user device).

[0048] Neural network application 314 may receive protected data 304 from secure content storage 312. In some aspects, protected data 304 may correspond to media content being streamed and played back by a user device. For example, protected data 304 may include images and / or videos currently being viewed by a user of the user device. In some other aspects, protected data 304 may include biometric data stored in secure content storage 312 during a previous enrollment or authentication process.

[0049] In some embodiments, neural network application 314 may perform machine learning 306 on protected data 304. In some aspects, neural network application 314 may generate inferences about protected data 304 based on one or more neural network models, such as neural network model 302 received from network resource 322. In some other aspects, neural network application 314 may generate new neural network models (or rules) based on protected data 304. Furthermore, in some aspects, neural network application 314 may use protected data 304 to update one or more existing neural network models, such as neural network model 302 received from network resource 322.

[0050] In some embodiments, neural network application 314 can provide filtered feedback data 308 (e.g., as described above with respect to Figure 2 ). In some aspects, the filtered feedback data 308 may include limited information about inferences generated from the protected data 304. Specifically, the limited information may be filtered so that it does not violate the privacy rights of the user of the user device and / or the owner of the content. For example, the filtered feedback data 308 may not contain any original data (e.g., pixel data) or other information that can be used to reconstruct the original data 304. In addition, the filtered feedback data 308 may not disclose any personally identifiable information (e.g., name, age, gender, location, etc.) about the user of the user device or any information that can be used to derive such personally identifiable information. In some other aspects, the filtered feedback data 308 may include a new or updated neural network model generated by the neural network application 314. In this way, the network resource 322 can further refine its neural network model 302 without receiving or accessing the protected data 304.

[0051] Figure 4 4 shows a block diagram of a processing system 400 according to some embodiments. The processing system 400 may be Figure 1 User equipment 110 and / or Figure 2 The processing system 400 includes an application processing unit (ACPU) 410 , a neural network processing unit (NPU) 420 , an input processing unit (IPU) 430 , and a storage device 440 .

[0052] ACPU 410 may include one or more general purpose processors configured to execute one or more applications and / or operating systems. ACPU 410 may include a rich execution environment (REE) 412 and a trusted execution environment (TEE) 414. REE 412 and TEE 414 may be consistent with rich environment 210 and trusted environment 220 of software execution environment 201, respectively. Figure 4In an embodiment of the present invention, the ACPU 410 may execute a rich neural network (NN) application 413 in the REE 412 and may execute a trusted neural network (NN) application 415 in the TEE 414. In some embodiments, the ACPU 410 may be configured to operate in a secure state and a non-secure state. For example, when executing an application and / or process from the TEE 414, the ACPU 410 may operate in a secure state, and when executing an application and / or process from the REE 412, the ACPU 410 may operate in a non-secure state.

[0053] NPU 420 may include one or more processors configured to accelerate neural network inference. For example, the hardware architecture of NPU 420 may be specifically designed to traverse a neural network faster and / or more efficiently than a general-purpose processor such as ACPU 410. In some embodiments, ACPU 410 may at least partially request NPU 420 to execute rich neural network application 413 or trusted neural network application 415. Therefore, in some embodiments, NPU 420 may also be configured to operate in a secure state and / or a non-secure state. When operating in a secure state, NPU 420 may communicate with software and / or hardware resources (such as secure HW resources 232) residing in a trusted environment, and may access software and / or hardware resources (such as secure HW resources 232) residing in a trusted environment.

[0054] The IPU 430 may include hardware resources configured to process (e.g., by filtering, analyzing, encoding, etc.) user input 405 to be stored or otherwise used by the processing system 400. The user input 405 may include text-based input, selection-based input, and / or biometric input provided by the user. The user input 405 may be received and / or detected by one or more input devices (not shown for simplicity). Example input devices may include, but are not limited to, keyboards, mice, joysticks, cameras, capacitive sensors, touchpads, fingerprint sensors, microphones, and the like. In some embodiments, the ACPU 410 may configure the IPU 430 to process the user input 405 in conjunction with a trusted neural network application. Therefore, the IPU 430 may also be configured to operate in a secure state. When operating in a secure state, the IPU 430 may communicate with and access software and / or hardware resources (such as secure HW resources 232) residing in a trusted environment.

[0055] The ACPU 410 may initiate a non-secure memory access service 402 from the REE 412 and a secure memory access service 404 from the TEE 414. The NPU 420 and the IPU 430 may respectively utilize the storage device 440 to perform NPU services 406 and IPU services 408. For example, each of the services 402-408 may include a read service (e.g., to read data from the storage device 440) or a write service (e.g., to write data to the storage device 440). The initiator of the service may be referred to as a "master", and the recipient of the service may be referred to as a "slave". Therefore, for the purposes of discussion, the ACPU 410, the NPU 420, and the IPU 430 may generally be referred to as multiple masters in this document. Although the processing system 400 is shown as including three masters 410-430, in some embodiments, the processing system 400 may include more than one master. Figure 4 Those depicted in the main part are less or more numerous.

[0056] The storage device 440 includes a non-secure partition 450 and a secure partition 460. The secure partition 460 and the non-secure partition 450 may be physically and / or virtually separated from each other. In some embodiments, the storage partitions 450 and 460 may each include different address spaces of a shared storage device (e.g., DRAM). In some other embodiments, the storage partitions 450 and 460 may be implemented on separate storage devices. The non-secure partition 450 permanently resides in a rich environment and can therefore be configured to store any data that needs to be operated by REE 412 and other software and / or hardware resources to form a rich environment. In contrast, the secure partition 460 permanently resides in a trusted environment and can therefore be configured to store data that is accessible only to TEE 414 and other software and / or hardware resources (such as NPU 420 and / or IPU 430) operating from a trusted environment or in a secure state.

[0057] exist Figure 4 In an embodiment, the non-secure partition 450 stores unprotected data 452, and the secure partition 460 stores a neural network model 462, an inference result 464, and protected data 466. In some embodiments, the storage device 440 may filter the storage services 402-408 based at least in part on the security status of the host that initiates the service. More specifically, the storage device 440 may ensure that software and / or hardware operating in a rich environment can access the non-secure partition 450 but not the secure partition 460. For example, the storage device 440 may allow the non-secure ACPU service 402 to access the unprotected data 452, but deny any non-secure ACPU service 402 access to the neural network model 462, the inference result 464, or the protected data 466.

[0058] The storage device 440 can also ensure that software and / or hardware operating in a secure environment can access the secure partition 460. For example, when the IPU 430 initiates the service 408 from the secure state, the storage device 440 can allow the IPU service 408 to write protected data 466 (e.g., user data, premium media content, etc.) to the secure partition 460. When the NPU 420 initiates the service 406 from the secure state, the storage device 440 can also allow the NPU service 406 to read the neural network model 462 and / or protected data 466 from the secure partition 460, and write the inference result 464 to the secure partition 460. In addition, the storage device 440 can allow the secure ACPU service 404 to read the inference result 464 from the secure partition.

[0059] Figure 5 Another block diagram of a user device 500 according to some embodiments is shown. The user device 500 may be Figure 1 User equipment 110 and / or Figure 2 The user equipment 500 includes a device interface 510, a processor 520 and a memory 530.

[0060] Device interface 510 may be used to communicate with one or more input sources and / or output sources coupled to user device 500. Example input sources may include, but are not limited to, cameras, microphones, buttons, keyboards, mice, touchpads, fingerprint scanners, photoelectric sensors, accelerometers, gyroscopes, and the like. For example, sensor interface 510 may transmit signals to and receive signals from input sources to receive user input from a user of device 500. Example output sources may include, but are not limited to, speakers, displays, lights, and the like. For example, sensor interface 510 may transmit signals to and receive signals from output sources to output information and / or media content to a user of device 500.

[0061] Memory 530 may include a secure partition 531 and a non-secure partition 535. Secure partition 531 may include a protected data store 532 configured to store protected data (such as, for example, user data, premium media content, etc.). Secure partition 531 may also include a non-transitory computer-readable medium (e.g., one or more non-volatile memory elements such as EPROM, EEPROM, flash memory, hard drive, etc.) that may store a neural network software (SW) module 534 to generate inferences about the protected data stored in protected data store 532. Similarly, non-secure partition 535 may include a non-transitory computer-readable medium that may store a user interface SW module 536 to output information and / or media content to a user based at least in part on inferences about the protected data.

[0062] Each software module includes instructions that, when executed by processor 520, cause user device 500 to perform a corresponding function. The non-transitory computer-readable medium of memory 530 thus includes instructions for executing the following regarding Figure 6-9 Processor 520 may be any suitable processor or processors capable of executing scripts or instructions of one or more software programs stored in user device 500. For example, processor 520 may execute neural network SW module 534 to generate inferences about protected data stored in protected data storage 532. Processor 520 may also execute user interface SW module 536 to output information and / or media content to a user based at least in part on inferences about protected data.

[0063] Figure 6 is an illustrative flow chart depicting operations 600 for inferring by a user device according to some embodiments. Figure 1 , operation 600 may be performed by a user device 110 having a secure memory or data repository.

[0064] User device 110 may retrieve protected data from secure storage (610). Example protected data may include, but is not limited to, premium media content, user data, etc. In some implementations, secure storage may correspond to a secure partition of a storage device (such as Figure 4 More specifically, data stored in the secure partition may be inaccessible to hardware and / or applications executing in the rich environment of the user device 110. Figure 4 , protected data 466 stored in secure partition 460 may be inaccessible to applications (such as rich neural network application 413) executed from REE 412 of ACPU 410. Therefore, in some embodiments, trusted neural network application 415 may retrieve protected data 466 from secure partition 460 of storage device 440.

[0065] User device 110 may use one or more neural network models stored on the device to generate inferences about the protected data (620). In some embodiments, the neural network models may also be stored in secure memory. See, e.g. Figure 4, ACPU 410 may perform inference on protected data 466 using neural network model 462 stored in secure partition 460 when executing trusted neural network application 415. In some other aspects, ACPU 410 may instruct NPU 420 to perform inference on protected data 466 using neural network model 462 when executing trusted neural network application 415. Inference may identify objects of interest in the protected protected data. Example objects of interest may include, but are not limited to, signs, symbols, toys, and / or biometric features of one or more users of user device 110.

[0066] The user device 110 may then update the user interface (630) based at least in part on the inference. For example, the inference may indicate the interests, preferences, and / or behaviors of the user. Thus, in some embodiments, the inference may be used to provide recommendations and / or additional content via the user interface (e.g., to enhance or enhance the user experience). In some aspects, the user device 110 may recommend programs from a particular content source or television network to the user. In some other aspects, the user device 110 may provide attributes for a particular advertisement to a television network or content source that broadcasts or streams media content including advertisements. Additionally, in some aspects, the user device 110 may display television programs and / or movies that match the interests of a known user.

[0067] Figure 7 is an illustrative flow chart depicting an example image capture operation 700 according to some embodiments. Figure 2 , operation 700 may be performed by the user device 200 to provide an augmented reality (AR) interface for a camera-based application.

[0068] User device 200 may retrieve one or more neural network models from a training module (710). In some aspects, the training module may be located in a cloud or network environment external to user device 200 (e.g., Figure 1 In some other aspects, the training module may reside locally on user device 200 (e.g., rich neural network application 214 or trusted neural network application 224). Thus, the neural network model may be trained and / or retrained (e.g., to detect new objects) using only data or information that resides locally on user device 200. In some embodiments, the neural network model may correspond to a set of rules that describe certain objects of interest detectable by the AR interface (e.g., dolls, acting characters, or other toys that may have interactive features or aspects when presented in an AR environment such as a game).

[0069] User device 200 may also capture a scene using a camera (720). For example, user application 212 may include a camera-based application that may interface with a camera or other image capture device resident on or coupled to user device 200. In some embodiments, the scene may include one or more users of user device 200. In addition, user application 212 may present or display the captured scene on a display of user device 200 and / or a display device coupled to user device 200.

[0070] User device 200 may use one or more neural network models to generate inferences about a scene (730). For example, neural network application 224 may detect and / or identify objects of interest in a scene based on a set of rules associated with the neural network model. In some aspects, the identified objects may correspond to toys (such as dolls, performing characters, etc., which may have applications in interactive environments (such as AR games)) or other interactive objects that may be engaged by camera-based applications (e.g., user application 212). In some embodiments, the inferences may be performed in a trusted execution environment (TEE). As described above, the trusted execution environment may allow only limited information to be shared with the outside world. Therefore, by generating inferences within a trusted execution environment, trusted neural network application 224 may ensure that the user's privacy and / or personal information is protected.

[0071] The user device 200 may also output content associated with the inference (740). For example, the user application 212 may receive the inference from the neural network application 224 and may provide the user with supplemental content associated with the object inferred in the scene. In some aspects, the user application 212 may display a description of the object (e.g., as an AR overlay on top of the scene). In some other aspects, the user application 212 may play back audio associated with the object (such as the name of the object) via a speaker on the user device 200 or coupled to the user device 200. For example, the audio may be played back in a variety of different languages. In addition, in some aspects, the user application 212 may provide the user with the option of viewing images and / or videos (such as movies or television shows) associated with the identified object.

[0072] Figure 8 is an illustrative flow chart depicting example media playback operations 800 according to some embodiments. Figure 2 , operations 800 may be performed by user device 200 to provide a dynamic video interface for playback of media content based on the interests or preferences of a user.

[0073] User device 200 may retrieve one or more neural network models from a training module (810). In some aspects, the training module may be located in a cloud or network environment external to user device 200 (e.g., Figure 1 In some other aspects, the training module may reside locally on user device 200 (e.g., rich neural network application 214 or trusted neural network application 224). In some embodiments, the neural network model may correspond to a set of rules that describe certain objects of interest (e.g., logos, symbols, and other brand identification marks) that are detectable in media content displayed or played back by user device 200.

[0074] The user device 200 may initiate playback of premium content (820). For example, premium content may include any media content (such as television programs or movies) protected by DRM, copyright, trademark, and various other laws and / or regulations. In some aspects, premium content may be received from (e.g., streamed or otherwise provided by) one or more content delivery networks (CDNs). As described above, such premium content may be stored or buffered in a secure repository (e.g., secure hardware resources 232) residing within a trusted execution environment (TEE) of the user device 200. More specifically, the secure hardware resources 232 may be inaccessible to software and / or hardware outside of the trusted environment.

[0075] User device 200 may use one or more neural network models to generate inferences about premium content (830). For example, neural network application 224 may detect and / or identify objects of interest in premium content based on a set of rules associated with the neural network model. In some aspects, the identified objects may correspond to logos, symbols, and other brand identification marks that may indicate a production studio, television network, or advertiser involved in the premium content. As described above, the trusted execution environment may be inaccessible to the outside world. However, because trusted neural network application 224 resides within the trusted execution environment, trusted neural network application 224 can access full-frame pixel data used to display or present premium content on user device 200. Therefore, neural network application 224 can generate inferences about premium content (e.g., to be presented or displayed by user device 200) based on actual pixel data stored or buffered in a decoded video frame buffer.

[0076] User device 200 may also output content and / or recommendations based on the inferences (840). For example, user application 212 may include a video playback application that may stream or playback premium content stored in secure hardware resource 232. User application 212 may receive inferences from neural network application 224 and may provide supplemental content associated with objects inferred in the premium content to the user. In some embodiments, user application 212 may determine preferences and / or interests of the user based on inferences (e.g., signs, symbols, etc.) received from neural network application 224.

[0077] For example, in some aspects, user application 212 may determine that a user of user device 200 has a preference for television programs broadcast on a particular television station or network based on the identified logo or symbol. Accordingly, user application 212 may recommend other programs from the same television network or production studio to the user. In some embodiments, the owner or creator of the media content (e.g., which may be a broadcast television network or production studio) may be different from the content delivery network (CDN) that streams or transmits the media content to user device 200 (e.g., which may be a streaming video on demand (VoD) or pay-per-view (PPV) service). Accordingly, in some aspects, user application 212 may provide the user with the option to view additional content from a preferred content creator or owner (e.g., based on the inference generated by neural network application 224), where such additional content may be available for streaming (or delivery) by the same CDN that streams (or otherwise delivers) the premium content from which the inference was made.

[0078] In some other aspects, the user application 212 may determine, based on inference (e.g., a recognized logo or symbol), that the user of the user device 200 has a preference or interest in a particular brand or type of product. Accordingly, the user application 212 may present targeted advertisements for the particular brand or type of product (or related brands and / or products) to the user. For example, the targeted advertisement may include an option to purchase the recognized product.

[0079] Additionally, in some embodiments, the user device 200 may send filtered feedback data to the content provider based on the inference (850). For example, in some other aspects, the user application 212 may determine which advertisements and / or product brandings are most likely to have been viewed by the user of the user device 200 based on the inference. Thus, the user application 212 may provide attributes to a television network that broadcasts media content for advertisements or product placement. For example, the user application 212 may provide analytics about the advertisement (such as the timing of the advertisement, the network on which the advertisement was displayed, the location in which the advertisement was viewed, etc.) to a company that created the product and / or advertisement.

[0080] Fig. 9 is an illustrative flow chart depicting an example authentication operation 900 according to some embodiments. Figure 2 , operations 900 may be performed by user device 200 to provide a dynamic video interface for playing back media content based on user classifications.

[0081] User device 200 may capture a scene using a camera (910). For example, user application 212 may include a camera-based application that may interface with a camera or other image or video capture device resident on or coupled to user device 200. In some embodiments, the scene may include one or more users of user device 200. In some aspects, user application 212 may present or display the captured scene on a display of user device 200 and / or a display device coupled to user device 200.

[0082] User device 200 may retrieve one or more neural network models from the training module (920). In some embodiments, the neural network model may correspond to a set of rules that describe biometric indicators (e.g., facial features, voice patterns, fingerprints, etc.) that can be used to authenticate a user of user device 200. The neural network model may be generated based on biometric data acquired during a registration process (e.g., when a user registers his or her biometric signature for detection by user device 200). To enhance security and to protect the privacy of the user, the biometric data may be stored in a secure repository (e.g., secure hardware resource 232) residing within a trusted execution environment (TEE) of user device 200. Therefore, in some embodiments, the training module (e.g., trusted neural network application 224) may also reside within a trusted execution environment on user device 200.

[0083] The user device 200 may use one or more neural network models to generate inferences about the scene (930). For example, the neural network application 224 may detect and / or identify objects of interest in the scene based on a set of rules associated with the neural network model. In some aspects, the identified object may correspond to a biometric feature (such as a user's face) that may indicate the identity of (one or more) users of the user device 200. For example, the neural network application 224 may classify one or more users of the user device 200 (e.g., one or more people in a scene captured by a camera) based on the identified biometric features. As described above, the trusted execution environment may be inaccessible to the outside world. However, because the trusted neural network application 224 resides within the trusted execution environment, the neural network application 224 may access biometric data (and neural network models) associated with each user. In some embodiments, the neural network application 224 may use a captured image or video of the scene to further train or retrain a neural network model for the user (such as the last layer or last few layers of the neural network model).

[0084] User device 200 may also output content and / or recommendations based on the inferences (940). For example, user application 212 may receive inferences from neural network application 224 and may provide targeted recommendations for (one or more) users of user device 200 to (one or more) users. More specifically, user application 212 may determine which recommendations to display via user device 200 based at least in part on the classification of one or more users of user device 200. As described above with respect to Figure 5 As described, user application 212 may know the user's interest and / or preference for media content. Therefore, user application 212 may provide (one or more) users with the option of viewing images and / or videos (such as movies or television programs) that match the interests of one or more users of user device 200. In some aspects, user application 212 may generate a customized user interface (e.g., a program guide, a launcher, etc.) curated for the interests or preferences of one or more users. For example, a customized user interface (UI) may only include the types of media content (e.g., sports, reality shows, dramas, etc.) that match the interests of one or more users. In some aspects, the customized UI may be dynamically updated whenever a new user is detected by user device 200.

[0085] In some aspects, where multiple users are detected by neural network application 224, user application 212 may display recommendations for only one of the users. For example, if it is detected by neural network application 224 that an adult has one or more children, user application 212 may display recommendations that are only suitable for children (e.g., cartoons, educational programs, and other child-friendly content). In some other aspects, where multiple users are detected by neural network application 224, user application 212 may display recommendations for multiple users. For example, if a husband and wife are detected by neural network application 224, user application 212 may display recommendations for the husband and recommendations for the wife. Further, in some aspects, neural network application 224 may display recommendations for a group of users. For example, if a group of friends is detected by neural network application 224, user application 212 may display recommendations based on the group's common interests (e.g., game shows, sports, and other shows or programming that are known to be enjoyed by the group as a whole).

[0086] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0087] In addition, those skilled in the art will appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the aspects disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above in general terms according to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician may implement the described functionality in a varying manner for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0088] The methods, sequences, or algorithms described in conjunction with the aspects disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integral to the processor.

[0089] In the foregoing description, embodiments have been described with reference to specific examples thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader scope of the present disclosure as set forth in the appended claims. Therefore, the description and the drawings are to be regarded as being in an illustrative rather than a restrictive sense.

Claims

1. A method for inference by a user device, comprising: retrieving protected data from a secure memory, wherein the secure memory is inaccessible to applications executing in a rich environment of the user device, the protected data comprising pixel data associated with one or more images or videos to be displayed via a user interface of the user device; generating inferences about one or more signs or symbols contained in the one or more images or videos using one or more neural network models stored on the user device; updating the user interface of the user device based at least in part on the inference; as well as Transmitting filtered feedback data based at least in part on the inference to an external network resource including a cloud or network environment external to the user device, wherein the filtered feedback data does not include any pixel data stored in the secure memory.

2. The method of claim 1, wherein the one or more neural network models are stored in the secure memory.

3. The method of claim 1 , wherein the inference is generated at least in part by a neural network application executing in a trusted environment of the user device.

4. The method of claim 1 , wherein the inference is generated at least in part by a neural network processing unit (NPU). 5 . The method of claim 1 , wherein the protected data comprises pixel data of premium media content to be displayed or played back via the user interface.

6. A method according to claim 1, wherein the protected data includes user input data or personal information about a user of the user device, the user input data including text-based input, selection-based input and / or biometric input provided by the user.

7. The method of claim 1, wherein the updating comprises: Content or recommendations are output via the user interface based at least in part on the inference.

8. The method according to claim 1, further comprising: The one or more neural network models are updated based at least in part on the inferences.

9. The method according to claim 7, further comprising: The updated neural network model is sent to an external network resource configured for training the neural network model.

10. A user equipment, comprising: User interface; Processing circuit; as well as a secure memory storing instructions that, when executed by the processing circuit, cause the user device to: retrieving protected data from the secure memory, wherein the secure memory is inaccessible to applications executing in a rich environment of the user device, the protected data comprising pixel data associated with one or more images or videos to be displayed via a user interface of the user device; generating inferences about one or more signs or symbols contained in the one or more images or videos using one or more neural network models stored on the user device; updating the user interface based at least in part on the inference; as well as Transmitting filtered feedback data based at least in part on the inference to an external network resource including a cloud or network environment external to the user device, wherein the filtered feedback data does not include any pixel data stored in the secure memory.

11. A user device according to claim 10, wherein the one or more neural network models are stored in the secure memory.

12. The user device of claim 10, wherein the instructions are executed in a trusted environment of the user device.

13. The user device of claim 10, wherein the processing circuit comprises a neural network processing unit (NPU).

14. The user device of claim 10, wherein the protected data comprises pixel data of premium media content to be displayed or played back via the user interface.

15. A user device according to claim 10, wherein the protected data includes user input data or personal information about a user of the user device, the user input data including text-based input, selection-based input and / or biometric input provided by the user.

16. The user device of claim 10, wherein execution of the instructions for updating the user interface causes the user device to: Content or recommendations are output via the user interface based at least in part on the inference.

17. The user equipment of claim 10, wherein execution of the instructions further causes the user equipment to: The one or more neural network models are updated based at least in part on the inferences.

18. A user equipment, comprising: processing circuitry configured to operate in a secure state or an unsecure state; as well as a memory having a secure partition and a non-secure partition, the secure partition storing instructions that, when executed by the processing circuitry when operating in the secure state, cause the user device to: retrieving protected data from the secure partition, wherein the secure partition is inaccessible to the processing circuitry when operating in the unsecure state, the protected data comprising pixel data associated with one or more images or videos to be displayed via a user interface of the user device; retrieving one or more neural network models from the secure partition; generating, using the one or more neural network models, inferences about one or more signs or symbols contained in the one or more images or videos; as well as Transmitting filtered feedback data based at least in part on the inference to an external network resource including a cloud or network environment external to the user device, wherein the filtered feedback data does not include any pixel data stored in the secure partition.

19. The user device of claim 18, wherein the protected data comprises pixel data of premium media content to be displayed or played back via the user interface.

20. A user device according to claim 18, wherein the protected data includes user input data or personal information about a user of the user device, the user input data including text-based input, selection-based input and / or biometric input provided by the user.

21. The user equipment of claim 18, wherein execution of the instructions further causes the user equipment to: Content or recommendations are output based at least in part on the inference.

22. The user equipment of claim 18, wherein execution of the instructions further causes the user equipment to: The one or more neural network models are updated based at least in part on the inferences.

23. The user device of claim 22, wherein the non-secure partition stores instructions that, when executed by the processing circuit, further cause the user device to: The updated neural network model is transmitted to an external network resource configured for training the neural network model.

Images