System and method for controlling access to content on an electronic device

Abstract

A method and system for controlling a frame unit on an electronic device for accessing content are described. The method includes: capturing interactions indicative of user interactions with each type of content; extracting a set of attribute values ​​based on the content and interactions, wherein the set of attribute values ​​indicates user behavior patterns for each content type; classifying the content based on the extracted set of attribute values, wherein the classification determines the content's sensitivity as either interesting or uninteresting; and controlling access to the content for one or more applications associated with the electronic device based on whether the classified content is of interest or uninteresting to the user.

Description

Technical Field

[0001] This disclosure relates to content monitoring, and for example, to systems and methods for controlling access to content on electronic devices to restrict application access to content. Background Technology

[0002] In the digital age, users are using multiple apps on their electronic devices. To utilize these multiple apps, users need to grant various permissions and enable different settings for each app on their device. In many cases, these permissions and settings allow multiple apps on an electronic device to access user data. In other words, users face the challenge of managing multiple apps on their electronic devices and their respective privacy settings to access their user data.

[0003] By leveraging permissions and settings, apps can access and read users' personal and non-personal data. Furthermore, apps can capture user interests as they interact with various content on electronic devices. Users may not intend to share data and interaction information with the app; however, the app can still access this data and interaction information intentionally or unintentionally. This data and information can then be shared with third parties, and thus, targeted advertising can be provided. Additionally, the data and information may also be sold to other parties.

[0004] Existing technologies include reporting on privacy breaches through applications. Furthermore, existing technologies include monitoring which applications are capturing and accessing user information, and notifying users if any application uses permissions that are particularly important or outside the scope of its normal operation. However, original equipment manufacturers (OEMs) lack mechanisms for automatically controlling access to user data. Applications can freely access user data at any time, provided the user grants permission. Data access is controlled based on static permission sets. Moreover, existing technologies do not protect privacy when users interact with content through applications. Therefore, users' interest information is easily leaked, and privacy is compromised.

[0005] Therefore, systems and methods are needed to overcome at least some of the aforementioned limitations. Summary of the Invention

[0006] Solution to the problem

[0007] According to an example embodiment of this disclosure, a method is provided for controlling a framework unit on an electronic device for accessing content. The method includes: capturing interactions indicative of user interactions with each type of content; extracting a set of attribute values ​​based on the content and interactions, wherein the set of attribute values ​​indicates user behavior patterns for each type of content; classifying the content based on the extracted set of attribute values, wherein the classification determines the sensitivity of the content as either interesting or uninteresting; and controlling access to the content for one or more applications associated with the electronic device based on whether the classified content is of interest or uninteresting to the user.

[0008] According to an example embodiment of this disclosure, a system is provided for controlling frame units on an electronic device for accessing content. The system includes: a memory configured to store a plurality of modules including executable instructions; and at least one processor including processing circuitry communicatively coupled to the memory, wherein the at least one processor is individually and / or collectively configured to execute the executable instructions associated with the plurality of modules. The plurality of modules includes: an interaction estimator module configured to capture interactions indicative of user interactions with each type of content; an attribute extractor module configured to extract a set of attribute values ​​based on content and interactions, wherein the set of attribute values ​​indicates user behavior patterns with respect to each type of content; an interest estimator module configured to classify content based on the extracted set of attribute values, wherein the classification determines the sensitivity of the content as either interesting or uninteresting; and a privacy controller module configured to control access to the content for one or more applications associated with the electronic device based on whether the classified content is of interest or uninteresting to the user.

[0009] According to an example embodiment of this disclosure, a method for dynamically restricting application access to content in an electronic device is provided. The method includes: capturing user interactions with content types, the captured user interactions forming user behavior patterns about attributes indicative of user responses to each content type; training an on-device machine learning model based on the content types and the captured interactions to extract user-specific attributes; classifying the content based on the extracted attributes, wherein the classification determines the content's sensitivity as interest or disinterest; capturing user behavior based on interactions with new content on the electronic device; using the trained on-device machine learning model to predict whether interactions with new content result in user interest or disinterest in the content; and controlling application access to the content on the electronic device.

[0010] To further illustrate the advantages and features of this disclosure, a more specific description of various exemplary embodiments will be presented with reference to the accompanying drawings. It should be understood that these drawings depict only exemplary embodiments and should not be considered as limiting its scope. This disclosure will be described and explained with additional features and details in conjunction with the accompanying drawings. Attached Figure Description

[0011] These and other features, aspects, and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, wherein the same characters denote the same parts throughout the drawings, and wherein:

[0012] Figure 1 This is a block diagram illustrating an example environment including an electronic device and a system associated with the electronic device according to various embodiments;

[0013] Figure 2 This is a block diagram illustrating example configurations of a system according to various embodiments;

[0014] Figure 3 This is a block diagram illustrating example operational flows associated with determining the optimal interaction according to various embodiments;

[0015] Figure 4A This is a block diagram illustrating example operational flows associated with extracting a set of attributes according to various embodiments;

[0016] Figure 4B and Figure 4C An example set of attribute values ​​extracted by an attribute extractor according to various embodiments is shown;

[0017] Figure 5 This is a block diagram illustrating example operational flows associated with content classification according to various embodiments;

[0018] Figure 6 This is a block diagram illustrating an exemplary hybrid model deployment according to various embodiments;

[0019] Figure 7 This is a block diagram illustrating example operational flows associated with controlling access to content according to various embodiments;

[0020] Figure 8 This is a flowchart illustrating example methods for a control frame unit on an electronic device to access content according to various embodiments;

[0021] Figure 9 This is a flowchart illustrating example methods for dynamically restricting application access to content on an electronic device according to various embodiments; and

[0022] Figure 10 , Figure 11 , Figure 12 and Figure 13 This is a diagram illustrating various example use case scenarios for system control of access to content according to various embodiments.

[0023] Furthermore, those skilled in the art will understand that the elements in the accompanying drawings are shown for simplicity and may not necessarily be drawn to scale. For example, flowcharts illustrate example methods in terms of the steps involved to aid in understanding various aspects of this disclosure. Additionally, regarding the construction of the device, one or more components of the device may have been represented by conventional symbols in the drawings, and the drawings may show details relevant to understanding various embodiments of this disclosure so as not to obscure the drawings with details readily understood by those skilled in the art from the description herein. Detailed Implementation

[0024] Various exemplary embodiments will now be referenced, and specific language will be used to describe these exemplary embodiments. However, it will be understood that this is not intended to limit the scope of the disclosure, and such changes and further modifications to the systems shown, as well as such further applications of the principles of the disclosure as illustrated herein, are considered to be common knowledge of those skilled in the art to which this disclosure pertains.

[0025] Those skilled in the art will understand that the foregoing general description and the following detailed description are illustrative of this disclosure and are not intended to limit it.

[0026] References to “one aspect,” “another aspect,” or similar language throughout this disclosure may refer to a specific feature, structure, or characteristic described, for example, in connection with an embodiment included in at least one embodiment of this disclosure. Therefore, the phrases “in an embodiment,” “in another embodiment,” and similar language throughout this disclosure may, but not necessarily, refer to the same embodiment.

[0027] The term "comprise" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process or method that includes a list of steps includes not only those steps but may also include other steps not expressly listed or inherent to such process or method. Similarly, without further limitation, a list of one or more devices, subsystems, elements, structures, or components beginning with "comprise..." does not exclude the presence of other devices or subsystems or elements or structures or components, or additional devices or subsystems or elements or structures or components.

[0028] Figure 1This is a block diagram illustrating an example overview of an environment 100 including an electronic device 110 and a system 120 associated with the electronic device 110 according to various embodiments. The system 120 may be communicatively coupled to the electronic device 110. In embodiments, the system 120 may be integrated within the electronic device 110. In embodiments, the system 120 may be provided in a distributed manner, because one or more components of the system 120 may be provided on the electronic device 110, and one or more components of the system 120 may be provided on a remote, cloud-based unit.

[0029] Electronic device 110 may include one or more applications 111, application framework 113, library 115, kernel 117, and a set of framework units 119 (framework unit 1, framework unit 2, and framework unit N). The one or more applications 111 may include software applications (e.g., including executable program instructions) that can be downloaded from, for example, the Internet or pre-installed on electronic device 110. Electronic device 110 may also include a user interface and hardware buttons. The set of framework units 119 may refer to a monitoring service that runs in conjunction with one or more applications 111 and / or electronic device 110. In embodiments, the monitoring service may include a first monitoring service that runs in conjunction with one or more applications 111 to monitor user interactions and a second monitoring service that runs in conjunction with electronic device 110 to monitor user interactions. Furthermore, for the sake of brevity, details regarding application framework 113, library 115, kernel 117, and the set of framework units 119 are not provided herein.

[0030] In non-limiting instances, electronic device 110 may include (e.g., not limited to) mobile devices, smartwatches, tablet computers, smart TVs, wearable controllers, laptop computers, extended reality (AR) headsets, virtual reality (VR) headsets, and the like.

[0031] System 120 can be configured to control a set of frame units on electronic device 110 for accessing content. In an embodiment, system 120 may be integrated within application framework 113. System 120 can be configured to perform operations and achieve technical advantages by performing one or more operations, as at least referred to below. Figure 8 and Figure 9 As explained in more detail.

[0032] refer to Figure 2 , Figure 2 This is a block diagram illustrating an example configuration of system 120 according to various embodiments. System 120 may include multiple modules (e.g., including various circuits and / or executable program instructions) 201, a processor (e.g., including processing circuitry) 202, an input / output (I / O) interface (e.g., including I / O circuitry) 203, a memory 204, and a transceiver 205.

[0033] In an example embodiment, processor 202 may be operatively coupled to each of I / O interface 203, multiple modules 201, transceiver 205, and memory 204. In an embodiment, processor 202 may include a graphics processing unit (GPU) and / or an artificial intelligence engine (AIE). In an embodiment, processor 202 may include at least one data processor for performing processes in a virtual memory area network. Processor 202 may include dedicated processing units such as an integrated system (bus) controller, a memory management control unit, a floating-point unit, a graphics processing unit, a digital signal processing unit, etc. In an embodiment, processor 202 may include a central processing unit (CPU), a graphics processing unit (GPU), or both. Processor 202 may be one or more general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), servers, networks, digital circuits, analog circuits, combinations thereof, or other devices now known or hereafter developed for analyzing and processing data. Processor 202 may execute software programs, such as manually generated (e.g., programmed) code, to perform desired operations. Processor 202 may include various processing circuits and / or multiple processors. For example, as used herein, including the claims, the term "processor" can include various processing circuitry, including at least one processor, wherein one or more of the at least one processor can be configured individually and / or collectively in a distributed manner to perform the various functions described herein. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform a number of functions, these terms cover, for example, but not limited to, cases where one processor performs some of the functions and another processor performs other functions, and cases where a single processor can perform all of the functions. Additionally, at least one processor can include, for example, a combination of processors performing the various described / disclosed functions in a distributed manner. At least one processor can execute program instructions to implement or perform the various functions.

[0034] Processor 202 can be configured to communicate with one or more input / output (I / O) devices via I / O interface 203. In various embodiments, processor 202 can communicate with electronic device 110 using I / O interface 203. In various embodiments, the I / O interface can be implemented within electronic device 110. I / O interface 203 may include various I / O circuits and employs communication technologies such as Code Division Multiple Access (CDMA), High-Speed ​​Packet Access (HSPA+), Global System for Mobile Communications (GSM), Long Term Evolution (LTE), and WiMax.

[0035] Using I / O interface 203, system 120 can communicate with one or more I / O devices. For example, input devices can be antennas, microphones, touchscreens, touchpads, storage devices, transceivers, video devices / sources, etc. Output devices can be video displays (e.g., cathode ray tube (CRT), liquid crystal displays (LCD), light-emitting diodes (LEDs), plasma displays, plasma display panels (PDP), organic light-emitting diode displays (OLEDs), etc.), audio speakers, etc.

[0036] Processor 202 can be configured to communicate with a communication network via a network interface. In an embodiment, the network interface may be I / O interface 203. The network interface can be connected to the communication network to enable system 120 to connect to electronic device 110 and / or the external environment. The network interface may employ connectivity protocols, including but not limited to direct connection, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), Transmission Control Protocol / Internet Protocol (TCP / IP), Token Ring, IEEE 802.11a / b / g / n / x, etc. The communication network may include, but is not limited to, direct interconnect, local area network (LAN), wide area network (WAN), wireless network (e.g., using Wireless Application Protocol), Internet, etc. Using the network interface and the communication network, system 120 can communicate with other devices.

[0037] In various embodiments, memory 204 may be communicatively coupled to processor 202. Memory 204 may be configured to store data and instructions executable by processor 202. In embodiments, memory 204 may be located within electronic device 110. In embodiments, memory 204 may be provided via a cloud-based unit. In embodiments, memory 204 may communicate with processor 202 via a bus within system 120. In embodiments, memory 204 may be located remotely from processor 202 and may communicate with processor 202 via a network. Memory 204 may include, but is not limited to, non-transitory computer-readable storage media, such as various types of volatile and non-volatile storage media, including but not limited to random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media, etc. In one example, memory 204 may include a cache or random access memory for processor 202. In various embodiments, memory 204 may be decoupled from processor 202, such as processor cache memory, system memory, or other memory. Memory 204 may be an external storage device or database for storing data. Memory 204 may be operable to store instructions executable by processor 202. The functions, actions, or tasks shown or described in the figures may be executed by programmable processor 202 to execute the instructions stored in memory 204. Functions, actions, or tasks are independent of a particular type of instruction set, storage medium, processor, or processing strategy, and may be executed by software, hardware, integrated circuits, firmware, microcode, etc., operating individually or in combination. Similarly, processing strategies may include multiprocessing, multitasking, parallel processing, etc.

[0038] In various embodiments, multiple modules 201 may be included within memory 204. Memory 204 may also include a database for storing data. Multiple modules 201 may include a set of instructions (e.g., executable program instructions) that can be executed to cause system 120 (including, for example, processor 202 of system 120) to perform any one or more of the methods / processes disclosed herein. Multiple modules 201 may be configured to perform steps of this disclosure using data stored in the database. For example, multiple modules 201 may be configured to perform the steps described below (reference to...) Figure 8 and Figure 9 More detailed description of the steps. In an embodiment, each of the plurality of modules 201 may include hardware that may be external to memory 204. Furthermore, memory 204 may include an operating system, such as one executed by a general-purpose operating system, for performing one or more tasks of system 120.

[0039] Transceiver 205 can be configured to receive signals from and / or transmit signals to electronic device 110. In an embodiment, the database can be configured to store information required by multiple modules 201 and processor 202 to perform functions such as... Figures 8 to 9 One or more functions disclosed in the document.

[0040] The multiple modules 201 may include, but are not limited to, a content recognizer 210, an interaction estimator 220, an attribute extractor 230, an interest estimator 240, and a privacy controller 250. The multiple modules 201 may be implemented through suitable hardware and / or software applications.

[0041] In various embodiments, at least one of the multiple modules 201 may use an AI model. AI-related functions may be performed via non-volatile memory, volatile memory, and processor 202.

[0042] Processor 202 may include one or more processors. At this time, the one or more processors may be general-purpose processors (such as central processing unit (CPU), application processor (AP), etc.), graphics-only processing units (such as graphics processing unit (GPU), vision processing unit (VPU)) and / or AI-specific processors (such as neural processing unit (NPU)).

[0043] One or more processors control the processing of input data according to predefined operating rules stored in non-volatile memory, or may employ a suitable artificial intelligence (AI) model executed from a server or local memory module.

[0044] AI models can include multiple neural network layers. Each layer has multiple weight values, and layer operations are performed by computing the previous layer and operating on the multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks.

[0045] Learning techniques can refer to, for example, methods used to train a predetermined target device (e.g., a robot) using multiple learning data sets to enable, allow, or control the target device to make determinations or predictions. Examples of learning techniques include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, active learning, and reinforcement learning. Processor 202 can perform preprocessing operations on the data to transform it into a form suitable for use as input to an artificial intelligence (AI) model.

[0046] Reasoning and prediction can refer to techniques for logical reasoning and prediction by determining information, and includes, for example, knowledge-based reasoning, optimization prediction, preference-based planning or recommendation.

[0047] Furthermore, this disclosure contemplates a non-transitory computer-readable medium that includes instructions or instructions for receiving and executing in response to a propagating signal. Additionally, instructions can be sent or received via a network through a communication port or interface or using a bus (not shown). The communication port or interface can be part of processor 202 or can be a separate component. The communication port can be created in software or can be a physical connection in hardware. The communication port can be configured to connect to a network, external media, a display, or any other component or combination thereof in the system. The connection to the network can be a physical connection, such as a wired Ethernet connection, or can be established wirelessly. Similarly, additional connections to other components of system 120 can be physical or can be established wirelessly. The network can alternatively be directly connected to the bus. For brevity, the architecture and standard operation of memory 204, processor 202, transceiver 205, and I / O interface 203 are not discussed in detail.

[0048] Figure 2 Example operational flows associated with the multiple modules 201 are further illustrated. It should be understood that the details described with reference to the multiple modules 201 can be executed in conjunction with the processor 202 and memory 204.

[0049] Content recognizer 210 can be configured to identify content associated with a user using electronic device 110. Content can be associated with various content types, such as, but not limited to, images, music, video, text, links, audio, and documents. Content can refer to one or more of the content types that the user will interact with. In various embodiments, electronic device 110 may include multiple view components, such as ImageView, TextView, VideoView, etc. View components may correspond to content types. In various embodiments, electronic device 110 may include an event listener associated with the view component to listen for (e.g., monitor) user interactions. Therefore, the type of content the user is currently interacting with can be determined.

[0050] Interaction estimator 220 can be configured to capture the optimal interaction that indicates the user's interaction with each content type. In embodiments, the optimal interaction may indicate the interaction with electronic device 110 and may be simply referred to as "interaction". The optimal interaction can be determined based on a continuous stream of interactions performed by the user. In non-limiting embodiments, the optimal interaction may include one or more of touch, physical buttons, gestures, voice commands, text input, navigation, application interaction, media control, camera and multimedia, communication, notifications, security, settings, customization, file management, payment, accessibility, smart device control, gaming, screen recording, search, battery, power management, and emergency services.

[0051] In an embodiment, the optimal interaction can be determined from the interaction stream captured by a set of frame units 119, as shown in the reference. Figure 3 The above, Figure 3 This is a block diagram illustrating an example operation flow 300 for determining the optimal interaction. As shown in block 302, a set of frame units 119 can be provided, which includes multiple frame units such as audio, network, location, biometrics, sensors, hardware, touch, video, etc. As shown in block 304, a set of listening components can be created and registered with the set of frame units 119. A set of listening components can correspond to a set of frame units 119. For example, a set of listening components may include an audio listening component, a touch listening component, a user interface (UI) listening component, etc.

[0052] A set of listening components can capture an interactive stream based on data callbacks from a set of frame units 119. Therefore, an interactive stream can be received from a set of frame units 119 via data callbacks. Table 1 shows a non-limiting example of an interactive stream captured by a set of listening components and a set of frame units 119.

[0053] Table 1

[0054]

[0055] At box 306, a composite interaction can be generated based on the content from the content recognizer 210 and the received interaction stream. In an embodiment, the interaction estimator 220 may include a data mixer submodule configured to generate the composite interaction.

[0056] At box 308, the composite interaction can be provided to the interaction classifier model. The interaction classifier model processes the composite interaction into meaningful data that can be used to identify user interactions. In an embodiment, the interaction classifier model can be a trained mathematical model. In an embodiment, the interaction classifier model can be stored in memory 204. As shown in box 308, the processing of the composite interaction can include preprocessing 308A, data analysis 308B, and feature calculation and selection 308C. In an embodiment, preprocessing can include data cleaning, time correction, and decomposition of the raw stream.

[0057] At box 310, the optimal interaction (or interaction) can be selected based on the analysis of the composite interactions. The optimal interaction can refer to a relevant and meaningful interaction that occurs in real time. As shown in box 310A, the output from box 308 can be processed to select the optimal interaction. In various embodiments, as shown in 310B, a feedback loop can be provided to enhance the accuracy of the selection and classification of the optimal interaction. In various embodiments, the optimal interaction may include interaction with the camera of the electronic device 110, touchscreen interaction, hardware buttons (volume buttons, power buttons, home buttons, etc.), electronic device orientation, microphone, and other connected units (wearable controllers, styluses, etc.).

[0058] Refer again Figure 1 and Figure 2 The attribute extractor 230 can be configured to extract a set of attribute values ​​based on content from the content recognizer 210 and the best interaction from the interaction estimator 220. This set of attribute values ​​can indicate user behavior patterns for each content type. In a non-limiting example, the set of attribute values ​​includes values ​​associated with interaction frequency, interaction context, content type frequency, content engagement time, interaction intensity, button interaction rate, historical data, user feedback, demographics, etc.

[0059] In an embodiment, the attribute extractor 230 may be configured to apply one or more pattern recognition techniques to the content from the content recognizer 210 and the optimal interaction from the interaction estimator 220. (See reference...) Figure 4A The document describes an example workflow 400 for extracting a set of attribute values. At box 402, content from content recognizer 210 and optimal interaction from interaction estimator 220 can be received and filtered by attribute extractor 230. In an embodiment, filtered data can be generated at box 402. Filtering may include cleaning the received data (content and optimal interaction), detecting anomalies, removing outliers, handling missing values, and removing corrupted data. In an embodiment, statistical techniques such as the Z-squared method can be used for anomaly detection.

[0060] At box 404, attribute extractor 230 can be configured to convert filtered data into a predefined format suitable for processing by pattern recognition techniques and other mathematical models (machine learning and artificial intelligence models), as described below.

[0061] At box 406, pattern recognition techniques can be applied to filtered data in a predefined format to determine a set of attribute values. Pattern recognition techniques can include, for example, convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

[0062] In various embodiments, at block 408, data normalization techniques can be applied to filtered data in a predefined format to reduce computation for pattern recognition techniques. Therefore, a set of attribute values ​​can be extracted by attribute extractor 230. Figure 4B and Figure 4C A set of attribute values ​​for an example extracted by attribute extractor 230 is shown. Attribute values ​​for audio content and associated interactions are depicted by spectrogram frame 410, while attribute values ​​for directional interactions are depicted by table 412 of directional frame data.

[0063] Refer again Figure 1 and Figure 2 Interest estimator 240 can be configured to classify content from content recognizer 210 based on a set of attribute values ​​extracted from attribute extractor 230. By means of classification, interest estimator 240 determines the sensitivity of content to whether a user interacting with the content is interested or not. Interest estimator 240 can classify content based on one or more of a plurality of estimation models. In an embodiment, the plurality of estimation models can be trained mathematical models stored in memory 204. In an embodiment, at least one of the plurality of estimation models is a trained mathematical model.

[0064] refer to Figure 5 The document describes an example operation flow 500 for classifying content. At block 502, the interest estimator 540 receives an extracted set of attribute values ​​and determines the data type associated with that set of attribute values. At block 504, an estimation model among multiple estimation models is determined based on the determined data type. At block 506, the interest estimator 240 determines a first probability value related to sensitivity to content of interest and a second probability value related to sensitivity to content of disinterest. For example, the interest estimator 240 determines whether content has a higher probability of being of interest to the user or not. Based on the first and second probability values, the interest estimator 240 classifies the content as either of interest or disinterest to the user.

[0065] In an example embodiment, the content type may be audio, and system 120 may determine whether the user intends to be heard by the electronic device. In this example, intending to be heard means being interested, while not intending to be heard means being uninterested (or not interested). Non-limiting examples of audio that is not of interest may include the user talking to another user and the electronic device 110 being in the user's pocket, a remote speaker playing audio that is not intended for use by the electronic device 110, random ambient sounds in the surrounding environment, etc. Non-limiting examples of audio that is of interest may include the user talking to a smart assistant integrated with the electronic device 110, the user talking to another user via telephone, the user recording video, the user talking to a virtual user in an online game, etc.

[0066] Audio can be captured and converted into a spectrogram image by applying a Fast Fourier Transform (FFT) to the captured audio. The spectrogram image can indicate details of the frequency composition of the captured audio over time. The spectrogram image can be fed as input to an estimation model, such as a CNN. A CNN can include multiple layers, such as Conv2D layers, max-pooling 2D layers, fully connected layers, softmax, etc. In one embodiment, a Conv2D layer can learn features or patterns in the input by applying a set of learnable filters to the input. Thus, relevant features from the input can be learned. In another embodiment, a max-pooling 2D layer can learn features using a simple max function instead of a kernel. This layer takes activation blocks from the original feature map and replaces them with the maximum activations in those blocks. In another embodiment, a fully connected layer classifies the data into two classes. In yet another embodiment, a softmax function transforms the output of the fully connected layer into a probability distribution output in the range of 0 to 1, where the sum equals 1. Therefore, the probability of being interested in or not interested in the content can be determined. In various embodiments, a CNN can be trained based on backpropagation, where gradients are applied to the weights of layers in the CNN until convergence.

[0067] In an example embodiment, system 120 can determine whether a user's orientation pattern is of interest or not. In this example, a user could rotate electronic device 110 in landscape mode to watch video, thus indicating the user's interest in orientation. Orientation data capture may include accelerometer and magnetometer data. The orientation data can be preprocessed to convert it into a pandas dataframe, which depicts the shape of a 3D vector and uses PCA to reduce dimensionality. The data can be fed as input to a CNN model, which can classify the data as of or not of user interest.

[0068] In various embodiments, the estimation model can be trained based on unsupervised learning techniques (e.g., K-means clustering models). Training the estimation model in various embodiments involves providing attribute values ​​as input to the model. The input can be encoded using techniques such as label encoding, word2vec, etc. The input can be converted into a pandas dataframe for dimensionality reduction. In embodiments, the input can undergo principal component analysis (PCA) for dimensionality reduction. Furthermore, the number of clusters can be input (here, the number of clusters can be 2 - interested and uninterested). Further distances between data points calculated by PCA can be computed. Additionally, data points can be grouped based on minimum distances, and the stability of the clusters can be determined. In cases of cluster instability, centroids can be recalculated, and feedback can be provided. Through the training process described above, a trained estimation model can be obtained with the number of clusters (e.g., 2 clusters).

[0069] In various embodiments, at least one of the multiple estimation models can be an on-device model and can be trained based on a global estimation model stored on a cloud-based server. (See reference) Figure 6 The diagram illustrates a block diagram of an example hybrid model deployment 600. At least one estimation model 602 can be used as a distillation model relative to a global estimation model 604. At least one estimation model 602 can classify content as either of interest or disinterest to the user. At least one estimation model 602 can be configured to adjust appropriate weights and biases based on parameters provided by the global estimation model 604. The global estimation model 604 can be a pre-trained model, which is also configured to be continuously trained based on feedback and errors received from the electronic device 110 implementing at least one estimation model 602.

[0070] In use, at least one estimation model 602 can take input from the attribute extractor 230 and predict probability values ​​associated with content that the user is interested in or not interested in. At least one estimation model 602 can also provide a confidence score corresponding to the prediction. In cases of low confidence scores or contradictory feedback from the user, error feedback can be generated at box 606 and sent along with the relevant attribute values ​​to the global estimation model 604. The global estimation model 604 can be retrained based on the error feedback to readjust its weights and biases. This readjustment is then propagated to at least one estimation model to improve performance. Therefore, a hybrid model deployment can be implemented to provide a smaller-sized model on the device.

[0071] Refer again Figure 1 and 2Privacy controller 250 can be configured to control access to content by one or more applications 111 based on whether the categorized content is of interest to or not to the user. The one or more applications 111 may include third-party applications capable of tracking content and user interactions with it. However, in cases where the user is interested in the content's sensitivity, it may be necessary to restrict or completely block access to the content by one or more applications 111. Therefore, privacy controller 250 can determine possible tracking methods and control access to the content via frame unit 119, through which one or more applications 111 can track the content.

[0072] Figure 7 This is a block diagram illustrating example operational flows of access to content controlled by a privacy controller 250 according to various embodiments. The privacy controller 250 may receive content and optimal interaction from a content recognizer 210 and an interaction estimator 220, respectively. At block 702, the privacy controller 250 may be configured to determine, based on the content and the optimal interaction, multiple possible tracking methods to be used by the one or more applications 111 to track the content. Multiple possible tracking methods may refer to detection methods that the one or more applications 111 may employ to detect user interactions. In embodiments, information regarding multiple possible tracking methods may be pre-stored in memory 204.

[0073] At box 704, privacy controller 250 is configured to select one or more feasible tracking methods from a variety of possible tracking methods. For example, not all of the various possible tracking methods are feasible given the content and optimal interaction. Privacy controller 250 checks which tracking methods are likely feasible and operational on electronic device 110, thereby reducing the amount of access instructions generated to control access to content.

[0074] At box 706, privacy controller 250 can be configured to generate access instructions for frame unit 119. Privacy controller 250 can receive input from interest estimator 240 to generate the instructions. The access instructions can be based on content sensitivity, because if the content is determined to be of interest to the user, the access instructions can instruct blocking access to the content.

[0075] At box 708, privacy controller 250 can filter the generated instructions to generate a sublist associated with the corresponding frame unit 119. At box 710, privacy controller 250 can send the generated access instructions to the corresponding frame unit within a set of frame units 119 to control access to content by one or more applications. Therefore, when the content is sensitive content of interest to the user, a frame unit can block access from one or more applications 111.

[0076] Taking a screenshot captured by a user on electronic device 110 as an example, the content may include visible content that the user is consuming, such as social media content. The user can continue to capture screenshots indicating the user's interaction with the content. Interactions may include pressing a hardware key, long-pressing a touch, extending screen time, etc. Privacy controller 250 can determine a variety of possible tracking methods for tracking screenshots. In various embodiments, the variety of possible tracking methods may include one or more applications overriding an application programming interface (API) to track user interactions, monitoring notification channels, specially created callback functions, access to files and media on electronic device 110, clipboard access, etc.

[0077] Privacy controller 250 can select a feasible tracking method from a variety of tracking methods. For example, feasible tracking methods may include notification channels and key event callbacks. Based on the feasible tracking method and the sensitivity of the content that the user is interested in, privacy controller 250 can generate access instructions that can be sent to the relevant frame unit 119. For example, access instructions can be sent to the hardware frame unit, UI frame unit, and application frame unit. Access instructions may include blocking hardware button callbacks, sandboxing notifications, etc. Therefore, system 120 can prevent or block access to content captured via screenshots.

[0078] In another example, where a user captures the screen of electronic device 110 rotating, the content may include visible content that the user is consuming, such as playing games, watching videos, showing videos to friends, recording videos, etc. The user can continue rotating the electronic device, indicating interaction with the content. Interaction may include device movement or pressing or accessibility buttons. Privacy controller 250 can determine a variety of possible tracking methods to track changes in orientation. In various embodiments, the various possible tracking methods may include orientation callbacks, notification channels, rotation vector sensors, accessibility services, etc.

[0079] Privacy controller 250 can select a feasible tracking method from a variety of tracking methods. For example, feasible tracking methods may include directional callbacks and rotation vector sensors. Based on the feasible tracking method and the sensitivity of the content of interest to the user, privacy controller 250 can generate access instructions that can be sent to the relevant frame unit 119. For example, access instructions may be sent to the hardware frame unit, UI frame unit, and application frame unit. Access instructions may include blocking directional callbacks and restricting the use of rotation vector sensors. Therefore, system 120 can prevent / block access to the content the user is viewing by rotating the electronic device 110.

[0080] Figure 8This is a flowchart illustrating an example method 800 for accessing content by a control frame unit 119 on an electronic device 110 according to various embodiments. In embodiments, the operation of method 800 may be performed by system 120, for example, by processor 202 of system 120 in conjunction with multiple modules 201 and memory 204, which may be integrated within electronic device 110 or provided separately and operatively coupled.

[0081] At S802, method 800 includes capturing the best interaction (e.g., interaction) that indicates the user's interaction with each type of content.

[0082] At S804, method 800 includes extracting a set of attribute values ​​based on content and optimal interaction, wherein the set of attribute values ​​indicates user behavior patterns for each content type.

[0083] At S806, method 800 includes classifying the content based on an extracted set of attribute values, wherein the classification determines the sensitivity of the content as either interesting or uninteresting.

[0084] At S808, method 800 includes controlling access to content for one or more applications associated with an electronic device based on whether the content is of interest or not to the user according to the classification.

[0085] Figure 9 This is a flowchart illustrating an example method 900 for dynamically restricting access to content by an application 111 on an electronic device 110 according to various embodiments. In embodiments, the operation of method 900 may be performed by system 120, for example, by processor 202 of system 120 in conjunction with multiple modules 201 and memory 204, which may be integrated within electronic device 110 or provided separately and operatively coupled.

[0086] At S902, method 900 includes capturing user interactions with content types, the captured user interactions forming a user behavior pattern about attributes that indicate the user's response to each content type.

[0087] At S904, method 900 includes training a machine learning model on the device based on content type and captured interactions to extract user-specific attributes.

[0088] At S906, method 900 includes classifying the content based on the extracted attributes, wherein the classification determines the sensitivity of the content as either interesting or uninteresting.

[0089] At S908, method 900 includes capturing user behavior when interacting with new content on an electronic device.

[0090] In S910, method 900 includes using a trained on-device machine learning model to predict whether interaction with new content will lead to the user being interested in or uninterested in the content.

[0091] At S912, method 900 includes controlling an application on an electronic device to access content.

[0092] Although shown and described in a given order Figure 8 and Figure 9 The steps described above are the same, but according to various embodiments, these steps can occur in different orders. Furthermore, with... Figure 8 and Figure 9 Detailed descriptions of each step are already available in the relevant section. Figure 1 , Figure 2 , Figure 3 Figure 4 Figure 5 , Figure 6 and Figure 7 This is covered in the relevant descriptions and will not be repeated here for the sake of brevity.

[0093] Figure 10 This is a diagram illustrating example use case scenarios of a system 120 for controlling access to content according to various embodiments. Figure 10 An electronic device 1000 is shown, which a user uses to view social media content on an application. The user can scroll through various social media content 1002 on the user interface of the electronic device 1000. Furthermore, the user can perform other actions, such as capturing screenshots, long-viewing content, and long-pressing the screen. According to conventional techniques, user interaction with the social media content 1002 can be tracked, and relevant advertisements can be displayed to the user based on the content being consumed. However, the system and method disclosed herein prevent access to social media content, and third-party applications cannot track user interactions and content. Therefore, user privacy is protected.

[0094] Figure 11 This is a diagram illustrating example use case scenarios of a system 120 for controlling access to content according to various embodiments. Figure 11 The illustration shows user 1102 conversing with other users 1104 and 1106 on a specific topic. Topics could include, for example, travel plans, shopping-related discussions, or entertainment-related discussions. Even if the user does not intend to provide audio input to electronic device 1108, user 1102's electronic device 1108 may still listen to user 1102. According to conventional techniques, applications on electronic device 1108 track audio of users outside of their intended conversation and present advertisements on that topic to the user. However, the system and method disclosed herein do not track audio of users outside of their intended conversation and prevent or reduce privacy breaches.

[0095] Figure 12This is a diagram illustrating example use case scenarios of a system 120 for controlling access to content according to various embodiments. Figure 12 An electronic device 1200 is shown, which a user uses to view content on application 1204. In the example shown, the user may be viewing content to book a taxi to a destination via one of several travel applications within the electronic device 1200, application 1204. According to conventional techniques, user interactions with travel applications can be tracked, and another travel application could alternatively trigger a notification that the user has booked a taxi via another travel application. However, using the system and method of this disclosure, access to the user's interest data (used for booking actions to travel to a destination) is blocked for other travel applications.

[0096] Figure 13 This is a diagram illustrating example use case scenarios of a system 120 for controlling access to content according to various embodiments. Figure 13 Example scenarios 13A, 13B, and 13C are illustrated, in which user 1302 accesses electronic device 1304 (such as the user's mobile phone) to view content, as shown in 13A. For example, user 1302 can view ticket prices for an entertainment event by a specific artist. User 1302 can access different websites via electronic device 1304 to view ticket prices. As shown in 13B, user 1302 can show details to friend 1306 on electronic device 1304. For example, user 1302 can change the orientation of electronic device 1304 to face friend 1306 standing next to user 1302. Cloud service 1310 can track user 1302's interests by analyzing user interactions associated with electronic device 1304. User interactions can include changes in the orientation of electronic device 1304 when showing content to friend 1306. According to conventional techniques, user 1302's interactions with electronic device 1304 can be tracked, and targeted advertisements related to an entertainment event by a specific artist can also be displayed to friend 1306 on friend 1306's device 1308. However, by utilizing the systems and methods of this disclosure, as shown in 13C, user interactions are not tracked and targeted advertisements are not displayed, thereby preventing and / or reducing privacy breaches.

[0097] In various examples, the methods and systems disclosed herein prevent / block the tracking of a user's geographic location. Furthermore, they prevent surveillance of shared interests among two or more users. Additionally, they can prevent or block the detection of nearby devices by blocking access to location and wireless signals (Bluetooth, Wi-Fi, etc.).

[0098] This disclosure provides various technological advancements based on the key features discussed above. The methods and systems currently disclosed enable dynamic control of access to content and user interests. If content of interest to the user is detected, application access to that content is restricted. Furthermore, access control can be user-controlled or automatic. Therefore, user privacy is maintained while the user interacts with content via the application.

[0099] Furthermore, user interest data is protected based on real-time user interaction. In other words, this method and system can protect user interest data and / or content from third-party application interactions / operations based on the concept of user interest leakage, thereby preventing and / or reducing privacy breaches. User interest data and / or content is generated based on real-time interactions between the user and content of defined content types within defined application categories. Therefore, user privacy is not compromised when a third-party application attempts to access the real-time generated interest data and / or content to infer user interests.

[0100] While this disclosure has been shown and described with reference to various exemplary embodiments, it should be understood that these exemplary embodiments are intended to be illustrative and not restrictive. Those skilled in the art will further understand that various changes in form and detail may be made without departing from the true spirit and full scope of this disclosure (including the appended claims and their equivalents). It should also be understood that any embodiment described herein may be used in conjunction with any other embodiment described herein.

Claims

1. A method for controlling a frame unit for accessing content on an electronic device, the method comprising: Capture interactions that indicate the user's interactions with each type of content; Extract a set of attribute values ​​based on content and optimal interaction, where the set of attribute values ​​indicates user behavior patterns for each content type; Content is categorized based on a set of extracted attribute values, where the categorization determines the content's sensitivity as either "interested" or "uninterested"; and Category-based content is either of interest or disinterest to the user, and access to the content is controlled by one or more applications associated with the electronic device.

2. The method according to claim 1, wherein, The captured interactions include: Receives an interactive stream associated with an electronic device from a set of frame units associated with the electronic device; Generate composite interactions based on the received interaction stream and content; Analyzing complex interactions based on an interaction classifier model; and Interactions are selected based on the analysis of complex interactions.

3. The method according to claim 1, wherein, Extracting a set of attribute values ​​includes: Filter content and interactions to generate filtered data, where filtering content and interactions includes removing outliers, abnormal and corrupted data; Convert the filtered data to a specified format; and Based on one or more pattern recognition technologies, a set of attribute values ​​are extracted from filtered data in a specified format.

4. The method according to claim 1, wherein, Content categorization includes: Determine the data type associated with a set of attribute values; Select an estimation model from multiple estimation models based on a given data type; Based on the selected estimation model, a first probability value associated with sensitivity to content of interest and a second probability value associated with sensitivity to content of uninterested interest are determined; and The content is classified based on the first probability value and the second probability value.

5. The method according to claim 4, wherein, At least one of the plurality of estimation models includes a mathematical model trained on an electronic device, and wherein the at least one estimation model is configured to be trained based on a global estimation model stored on a cloud-based server.

6. The method according to claim 1, wherein, Controlling access to content includes: Based on content and interaction, determine a variety of possible tracking methods to be used by the one or more applications to track content; Select one or more feasible tracking methods from the various possible tracking methods; Based on the content being determined to be of interest to the user, an access instruction is generated to block access to the content; and The generated access command is sent to the corresponding frame unit in a set of frame units to control the access of the one or more applications to the content.

7. The method according to claim 1, wherein, Content types are associated with one or more of the following: images, music, videos, text, links, audio, documents, and visual motion elements.

8. The method according to claim 1, wherein, Interactions are associated with one or more of the following: touch, physical buttons, gestures, voice commands, text input, navigation, app interaction, media control, camera and multimedia, communication, notifications, security, settings, customization, file management, payments, accessibility, smart device control, games, screen recording, search, battery, power management, and emergency services.

9. A system for controlling a frame unit for accessing content on an electronic device, the system comprising: The memory is configured to store multiple modules in the form of programmable instructions; At least one processor, including processing circuitry communicatively coupled to memory, the at least one processor being individually and / or collectively configured to execute programmable instructions associated with the plurality of modules, the plurality of modules comprising: An interaction estimator is configured to capture interactions that indicate user interactions with each type of content. An attribute extractor is configured to extract a set of attribute values ​​based on content and interaction, wherein the set of attribute values ​​indicates user behavior patterns for each content type; An interest estimator is configured to classify content based on an extracted set of attribute values, where the classification determines the content's sensitivity as either interesting or uninteresting; and A privacy controller is configured to control access to content for one or more applications associated with an electronic device based on whether the content is categorized as either of interest or disinterest to the user.

10. The system according to claim 9, wherein, To capture the best interactions, the interaction estimator is configured as follows: Receives an interactive stream associated with an electronic device from a set of frame units associated with the electronic device; Generate composite interactions based on the received interaction stream and content; Analyzing complex interactions based on an interaction classifier model; as well as Interactions are selected based on the analysis of complex interactions.

11. The system according to claim 9, wherein, To extract a set of attribute values, the attribute extractor is configured as follows: Filter content and interactions to generate filtered data, where filtering content and interactions includes removing outliers, abnormal and corrupted data; Convert the filtered data to a specified format; and Based on one or more pattern recognition technologies, a set of attribute values ​​are extracted from filtered data in a specified format.

12. The system according to claim 9, wherein, To categorize the content, the interest estimator is configured as follows: Determine the data type associated with a set of attribute values; Select an estimation model from multiple estimation models based on a given data type; Based on the selected estimation model, a first probability value associated with sensitivity to content of interest and a second probability value associated with sensitivity to content of no interest are determined. as well as The content is classified based on the first probability value and the second probability value.

13. A method for dynamically restricting application access to content in an electronic device, the method comprising: Capture user interactions with content types; the captured user interactions form user behavior patterns about attributes that indicate user responses to each content type. Train on-device machine learning models based on content type and captured interactions to extract user-specific attributes; The content is categorized based on the extracted attributes, where the categorization determines the sensitivity of the content as either interesting or uninteresting. Capture user behavior as they interact with new content on electronic devices; Using machine learning models trained on the device, predict whether interaction with new content will lead to user interest or disinterest; and Controls how applications on electronic devices access content.

14. The method according to claim 13, wherein, The capture is performed by a first monitoring service and a second monitoring service, which are respectively run by the application and the electronic device.

15. The method according to claim 13, wherein, Based on user interactions with content on the application, perform one of the following: monitor user interactions in a first monitoring service running on the application or monitor user interactions in a second monitoring service running on the electronic device.

Images