Updating a machine learning model based on unlearning a dataset or features

Abstract

An electronic device and method are disclosed for performing an update of a machine learning model based on unlearning a dataset or features. The electronic device receives a data subset of a first dataset associated with a user. A first machine learning model is trained based on the first dataset. The electronic device trains a second machine learning model based on the received data subset. The electronic device applies a transformation function to the trained first machine learning model based on the trained second machine learning model. The electronic device updates the trained first machine learning model based on the application of the transformation function. The update of the trained first machine learning model corresponds to unlearning at least one of the received data subset or a feature set associated with the second machine learning model.

Description

[Technical field]

[0001] CROSS REFERENCE TO RELATED APPLICATIONS / INCORPORATION BY REFERENCE This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 362,139, filed on March 30, 2022, which is a continuation of U.S. Provisional Patent Application No. 18 / 159,902, filed in the U.S. Patent and Trademark Office on January 26, 2023, which claims priority to U.S. Provisional Patent Application Serial No. 63 / 362,139, filed on March 30, 2022, the contents of which are incorporated herein by reference in their entirety.

[0002] Various embodiments of the present disclosure relate to machine learning models, and more particularly, to electronic devices and methods for updating machine learning models based on unlearning of datasets or features. [Background technology]

[0003] Advances in software technology have led to the development and use of various types of machine learning (ML) models. ML models can be employed in a variety of applications, such as making predictions, recommendations, and classifications. Typically, ML models can be trained for any application field. Training can be done using a training dataset that can be provided to the ML model. The training dataset can include data points across various features and a predefined label associated with each data point. The ML model can be trained based on an analysis of correlations or associations between the various features and the corresponding predefined labels. However, in some cases, the training dataset may include erroneous data. Training an ML model based on such erroneous data can cause the ML model to produce misleading or erroneous outputs, such as inaccurate predictions or incorrect classes. Summary of the Invention [Problem to be solved by the invention]

[0004] The limitations and disadvantages of the conventional approach will become apparent to one skilled in the art by comparing the described system with certain aspects of the present disclosure illustrated in the remaining portions of this application and with reference to the drawings. [Means for solving the problem]

[0005] Provided are electronic devices and methods for updating a machine learning model based on unlearning a dataset or features substantially as illustrated and / or described in connection with at least one of the figures and more fully as set forth in the claims.

[0006] These and other features and advantages of the present disclosure will become apparent from a consideration of the following detailed description of the disclosure when taken in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram illustrating an example network environment for updating a machine learning model based on unlearning a dataset or features, in accordance with an embodiment of the present disclosure. [Diagram 2] 2 is a block diagram illustrating the example electronic device of FIG. 1 in accordance with an embodiment of the present disclosure. [Diagram 3] FIG. 1 illustrates an example scenario for performing an update of a machine learning model based on unlearning a dataset or features, according to an embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates an example scenario for performing an update of a machine learning model based on unlearning a dataset or features, according to an embodiment of the present disclosure. [Diagram 5] FIG. 1 illustrates an exemplary scenario for training a first machine learning model, according to an embodiment of the present disclosure. [Figure 6] FIG. 1 illustrates an example scenario of updating a machine learning model based on unlearning a dataset or features, according to an embodiment of the present disclosure. [Figure 7]FIG. 2 illustrates an exemplary scenario of label determination according to an embodiment of the present disclosure. [Figure 8] FIG. 1 illustrates an example scenario for updating a first ML model based on application of a transformation function, according to an embodiment of the present disclosure. [Figure 9] FIG. 1 illustrates an exemplary processing pipeline for performing an update of a machine learning model based on unlearning a dataset or features, according to an embodiment of the present disclosure. [Figure 10] FIG. 1 illustrates an example scenario of updating a machine learning model for content-based recommendation, according to an embodiment of the present disclosure. [Figure 11] 4 is a flowchart illustrating the operation of an exemplary method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Electronic devices and methods for updating a machine learning model based on unlearning a dataset or features can find embodiments described below. An exemplary aspect of the present disclosure can provide an electronic device that can receive a data subset of a first dataset associated with a user. A first machine learning model can be trained based on the received first dataset associated with the user. In an example, a data point in the dataset can be generated at a particular time point, and the data point can have an associated timestamp. The electronic device can enable a user to provide a user input indicating a time interval associated with the data (i.e., the data subset) that was improperly captured. The electronic device can train a second machine learning model based on the received data subset. The electronic device can further apply a transformation function to the trained first machine learning model based on the trained second machine learning model. The electronic device can further update the trained first machine learning model based on the application of the transformation function to the trained first machine learning model. The update of the trained first machine learning model can correspond to unlearning at least one of the data subset or feature set associated with the second machine learning model.

[0009] Typically, a machine learning (ML) model can be trained to make predictions, recommendations, classifications, and the like by training it based on a training dataset. Based on the training, the ML model can provide an output (e.g., a prediction) related to a given input or feature set. However, in some cases, the training dataset may include erroneous data. In such cases, the trained ML model may provide an erroneous output. The electronic device of the present invention can train a first machine learning model based on a first dataset related to a user. The electronic device can extract the erroneous data from the first dataset as a data subset. For example, the electronic device can determine or receive a potentially erroneous data subset based on a user input indicating a time interval corresponding to a particular dataset. The electronic device can further train a second machine learning model based on the received data subset. Thereafter, the electronic device can apply a transformation function to the trained first machine learning model based on the trained second machine learning model, and update the trained first machine learning model based on the application of the transformation function. The update of the trained first machine learning model can correspond to unlearning at least one of the data subset or the feature set related to the second machine learning model. Thus, the electronic device can unlearn the incorrectly introduced / erroneous data set or undesired feature set in the existing model to obtain an updated model that achieves the desired output performance. Since the trained first ML model can unlearn at least one of the data subset or feature set associated with the second machine learning model, the updated ML model is optimal and can not make incomplete recommendations.

[0010] FIG. 1 is a block diagram illustrating an example network environment for updating a machine learning model based on unlearning a dataset or features, according to an embodiment of the present disclosure. FIG. 1 illustrates a network environment 100. The network environment 100 can include an electronic device 102, a server 104, a database 106, and a communication network 108. The electronic device 102 can include a first machine learning model 110 and a second machine learning model 112. FIG. 1 also illustrates a first dataset 114 that includes a plurality of datasets 116, such as data subset 116A. Also illustrated is a user 118 associated with or operating the electronic device 102.

[0011] The electronic device 102 may include suitable logic, circuitry, interfaces, and / or code that may be configured to receive a data subset (e.g., data subset 116A) of a first dataset (e.g., first dataset 114) associated with a user (e.g., user 118). A first machine learning model (e.g., first machine learning model 110) may be trained based on the received first dataset (e.g., first dataset 114) associated with the user, the first dataset including the received data subset. The electronic device 102 may train a second machine learning model (e.g., second machine learning model 112) based on the received data subset 116A. Examples of the electronic device 102 may include, but are not limited to, a computing device, a smart phone, a cellular phone, a mobile phone, a gaming device, a mainframe machine, a server, a computer workstation, a machine learning device (e.g., enabled with or hosting computing, memory, and network resources), and / or a consumer electronics (CE) device.

[0012] The server 104 may include suitable logic, circuitry, interfaces, and / or code that may be configured to apply a transformation function to the trained first machine learning model 110 and the trained second machine learning model 112. The server 104 may be configured to update the trained first machine learning model 110 based on the application of the transformation function to the trained first machine learning model 110. The update of the trained first machine learning model 1110 may correspond to unlearning at least one of the data subset 116A or the feature set associated with the second machine learning model 112. The server 104 may be implemented as a cloud server and may perform operations through web applications, cloud applications, HTTP requests, repository operations, file transfers, and the like. Other implementations of the server 104 may include, but are not limited to, a database server, a file server, a web server, a media server, an application server, a mainframe server, a machine learning server (e.g., enabled or hosting computing resources, memory resources, network resources), or a cloud computing server, and the like.

[0013] In at least one embodiment, the server 104 can be implemented as a number of distributed cloud-based resources using a number of techniques known to those skilled in the art. Those skilled in the art will appreciate that the scope of the present disclosure may not be limited to the implementation of the server 104 and the electronic device 102 as two independent entities. In some embodiments, the functionality of the server 104 can be incorporated in whole or at least in part into the electronic device 102 without departing from the scope of the present disclosure. In some embodiments, the server 104 can host the database 106. Alternatively, the server 104 can be separate from the database 106 and communicatively coupled to the database 106.

[0014] The database 106 may include suitable logic, interfaces, and / or code that may be configured to store the first data set 114. The first data set 114 may include multiple data sets 116, which may include data sets such as data subset 116A. The database 106 may be derived from data from a relational or non-relational database, or from a comma-separated values ​​(csv) file set in traditional storage or big data storage. The database 106 may be stored or cached on a device, such as a server (e.g., server 104) or electronic device 102. The device that stores the database 106 may be configured to receive a query for the first data set 114 from the electronic device 102 or server 104. In response, the device of the database 106 may be configured to retrieve and provide the queried first data set 114 to the electronic device 102 or server 104 based on the received query.

[0015] In some embodiments, database 106 may be hosted on multiple servers stored in the same or different locations. Operations of database 106 may be performed using hardware, including a processor, a microprocessor (e.g., performing or controlling the execution of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In some other cases, database 106 may be implemented using software.

[0016] The communication network 108 may include a communication medium that allows the electronic device 102 and the server 104 to communicate with each other. The communication network 108 may be either a wired connection or a wireless connection. Examples of the communication network 108 may include, but are not limited to, the Internet, a cloud network, a cellular or wireless mobile network (such as Long Term Evolution and Fifth Generation (5G) New Radio (NR)), a satellite communication system (e.g., using low-earth orbit satellites), a Wireless Fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). Various devices in the network environment 100 may be configured to connect to the communication network 108 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols include, but are not limited to, at least one of Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE802.11, Light Fidelity (Li-Fi), 802.16, IEEE802.11s, IEEE802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocols, and Bluetooth (BT) communication protocols.

[0017] Each of the first machine learning (ML) model 110 and the second ML model 112 can be a recommendation model that can be trained to identify relationships between inputs, such as features in a training dataset, and output labels, such as recommended video files. Each of the first ML model 110 and the second ML model 112 can be defined by hyperparameters, such as the number of weights, a cost function, an input size, and a number of layers. The parameters of the ML model can be updated to move toward a global minimum of the ML model's cost function, and the weights can also be updated in this manner. The ML model can be trained to output prediction / classification results for the input dataset after training for several epochs on feature information in the training dataset. The prediction results can indicate a class label (e.g., input features extracted from new / undiscovered instances) for each input of the input dataset.

[0018] The ML model may include electronic data that may be implemented, for example, as a software component of an application executable on the electronic device 102. The ML model may rely on libraries, external scripts, or other logic / instructions for execution by a processing device, such as the server 104 or the electronic device 102. The ML model may include code and routines configured to enable a computing device, such as the server 104 or the electronic device 102, to perform one or more operations, such as making a recommendation to the user 118. Additionally or alternatively, the ML model may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling the execution of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). Alternatively, in some embodiments, the ML model may be implemented using a combination of hardware and software. Examples of ML models may include a linear regression model, a logistic regression model, a decision tree model, a support vector machine (SVM) model, a naive Bayes model, a k-nearest neighbor (kNN) model, a K-means clustering model, a random forest model, a dimensionality reduction model (e.g., a principal component analysis (PCA) model), or a gradient boosting model.

[0019] The first dataset 114 may include data related to the user 118 that may be used to train the first machine learning model 110. For example, if the first machine learning model 110 is a content-based recommendation model, the first dataset 114 may include one or more media contents that the user 118 has interacted with (e.g., viewed or rated). For example, the first dataset 114 may include video files that the user 118 may have viewed. The first machine learning model 110 may make recommendations to the user 118 based on such video files. The first dataset 114 may include multiple datasets 116, such as data subset 116A. Each dataset of the multiple datasets 116 may be stored as a data shard. In an embodiment, the first dataset 114 may be stored in the database 106. In another embodiment, the first dataset 114 may be stored in the electronic device 102.

[0020] The data subset 116A may include data associated with the user 118 that may be used to train the second machine learning model 112. The data subset 116A may be undesirable, unlearnable data (and / or erroneous / incorrectly introduced data) associated with the user 118. The data subset 116A may be extracted from the first data set 114 to train the second machine learning model 112. In some embodiments, the data subset 116A may be extracted based on a first user input indicating a time period associated with the data subset 116A. The first machine learning model 110 may be updated based on this extracted first data set 114. Recommendations made by the updated first machine learning model 110 may be independent of the data subset 116A. Thus, the first machine learning model 110 may "unlearn" features associated with the data subset 116A.

[0021] In operation, the electronic device 102 may be configured to receive a data subset 116A of a first dataset 114 associated with a user 118. The first machine learning model 110 may be trained based on the first dataset 114 associated with the user 118. The first dataset 114 associated with the user 118 may be behavioral data associated with the user 118. Behavioral data over a period of time associated with the user 118 may be collected and stored as the first dataset 114. The first machine learning model 110 may be trained based on the first dataset 114 associated with the user 118. The first dataset 114 may include multiple datasets 116. However, in some cases, one or more datasets of the multiple datasets 116 may not be associated with the behavior of the user 118. Such a dataset that may not be associated with the user 118 or may be a misrepresentation of the behavior of the user 118 may be referred to as a data subset 116A. More details regarding the first ML model 110 are further described, for example, in FIG. 3.

[0022] The electronic device 102 may be further configured to train a second machine learning model 112 based on the received data subset 116A. The second ML model 112 may be trained to make recommendations based on the received data subset 116A. Details regarding training the second ML model 112 are further described, for example, in FIG.

[0023] The electronic device 102 may be further configured to apply a transformation function to the trained first machine learning model 110 based on the trained second machine learning model 112. The transformation function may help transform the trained first machine learning model 110 from an incomplete state to an updated (or accurate) state so that the updated first ML model can make optimal recommendations. Details related to the transformation function are further shown, for example, in FIG. 8.

[0024] The electronic device 102 may further be configured to update the trained first ML model 110 based on application of the transformation function to the trained first ML model 110. The update of the trained first ML model 110 may correspond to unlearning at least one of the feature set associated with the data subset 116A or the second machine learning model 112. By updating the trained first ML model 110, recommendations of the updated first ML model 110 may be independent of the data subset 116A or the feature set corresponding to the data subset 116A. Thus, based on the update of the first ML model 110, incomplete output of the first ML model 110 that may be associated with the data subset 116A may be prevented. More details regarding the update of the trained first ML model are further described, for example, in FIG. 8.

[0025] Figure 2 is a block diagram illustrating the example electronic device of Figure 1 in accordance with an embodiment of the present invention. The description of Figure 2 is provided with reference to the elements of Figure 1. Figure 2 illustrates an example electronic device 102. The electronic device 102 may include circuitry 202, memory 204, input / output (I / O) devices 206, and a network interface 208. The memory 204 may store a first data set 114 that includes a plurality of data sets 116, such as data subset 116A. The input / output (I / O) devices 206 may include a display device 210.

[0026] The circuitry 202 may include suitable logic, circuits, and / or interfaces that may be configured to execute program instructions associated with different operations performed by the electronic device 102. These operations may include receiving the data subset, training the second ML model, applying a transformation function, and updating the first ML model. The circuitry 202 may include one or more processing units that may be implemented as independent processors. In some embodiments, the one or more processing units may be implemented as an integrated processor or processors that collectively perform the functions of the one or more processing units. The circuitry 202 may be implemented based on multiple processor technologies known in the art. Example implementations of the circuitry 202 may be an X86-based processor, a graphic processing unit (GPU), a reduced instruction set computing (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computing (CISC) processor, a microcontroller, a central processing unit (CPU), and / or other control circuitry.

[0027] The memory 204 may include suitable logic, circuitry, interfaces, and / or code that may be configured to store one or more instructions executed by the circuit 202. The one or more instructions stored in the memory 204 may be configured to perform different operations of the circuit 202 (and / or the electronic device 102). The memory 204 may be configured to store a first data set 114 that includes a plurality of data sets 116, such as data subset 116A. Example implementations of the memory 204 may include, but are not limited to, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid-state drive (SSD), a CPU cache, and / or a secure digital (SD) card.

[0028] The I / O device 206 may include suitable logic, circuitry, interfaces, and / or code that may be configured to receive input and provide output based on the received input. For example, the I / O device 206 may receive a first user input indicating a selection of the first dataset 114 and / or the data subset 116A. The I / O device 206 may be further configured to display or render a recommendation output of the trained / updated first ML model 110 or the trained second ML model 112, the first dataset 114, and / or the data subset 116A. The I / O device 206 may include a display device 210. Examples of the I / O device 206 may include, but are not limited to, a display (e.g., a touch screen), a keyboard, a mouse, a joystick, a microphone, or a speaker, etc. Examples of the I / O device 206 may further include a Braille I / O device, such as a Braille keyboard and a Braille reader.

[0029] The network interface 208 may include suitable logic, circuitry, interfaces, and / or code that may be configured to facilitate communications between the electronic device 102 and the server 104 over the communications network 108. The network interface 208 may be implemented to support wired or wireless communications between the electronic device 102 and the communications network 108 using various known technologies. The network interface 208 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, or a local buffer circuit.

[0030] The network interface 208 may be configured to communicate via wireless communications with a network, such as the Internet, an intranet, a wireless network, a cellular telephone network, a wireless local area network (LAN) or a metropolitan area network (MAN). The wireless communication may be configured to use one or more of a number of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Fifth Generation (5G) New Radio (NR), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (WiFi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Worldwide Interoperability for Microwave Access (Wi-MAX), protocols for email, instant messaging, and short message service (SMS).

[0031] The display device 210 may include suitable logic, circuits, and interfaces that may be configured to display or render the recommended output of the trained / updated first ML model 110 or the trained second ML model 112, the first dataset 114, and / or the data subset 116A. The display device 210 may be a touch screen that allows a user (e.g., the user 118) to provide user input via the display device 210. The touch screen may be at least one of a resistive touch screen, a capacitive touch screen, or a thermal touch screen. The display device 210 may be realized through a number of known technologies, such as, but not limited to, at least one of a liquid crystal display (LCD) display, a light emitting diode (LED) display, a plasma display, or an organic LED (OLED) display technology, or other display devices. According to an embodiment, the display device 210 may refer to a display screen of a head mounted display (HMD), a smart glasses, a see-through display, a projection-based display, an electrochromic display, or a transparent display. Various operations of the circuit 202 for performing updates of a machine learning model based on unlearning of a dataset or features are further described in, for example, Figures 3, 4, 6, 7, 8 and 9.

[0032] Figure 3 illustrates an exemplary scenario for performing an update of a machine learning model based on unlearning a dataset or features, according to an embodiment of the present disclosure. The description of Figure 3 is provided with reference to elements of Figures 1 and 2. Figure 3 illustrates an exemplary scenario 300. The scenario 300 can include a first dataset 302 including a dataset 302A, a Softmax model 304, an existing ML model 306, a first ML model 110, a first dataset 308 including a data subset 308A, an updated first ML model 310, and a transformation function 312. A series of operations associated with the scenario 300 can now be described.

[0033] In the scenario 300 of FIG. 3, at time T1, a first dataset 302 associated with a user 118 may be received. As shown, the first dataset 302 may include multiple datasets, such as dataset 302A. The first dataset 302 may be received based on a cloud of data points related to behavioral data associated with a user (such as the user 118 of FIG. 1). For example, the behavioral data may correspond to a viewing history or rating score of media content consumed by the user 118 on a media streaming platform. In another example, the behavioral data may be a purchase history or rating score of products purchased by the user 118 from an e-commerce portal. The first dataset 302 and the existing ML model 306 may be provided to a Softmax model 304. The Softmax model 304 may be understood to be capable of determining a probability of each class of a set of predefined classes into which the data points of the first dataset 302 may be classified. The Softmax model 304 may include a layer of neurons, referred to as a Softmax layer, at an output layer of the existing ML model 306. The first ML model 110 can be trained based on the first dataset 302 and the application of the Softmax model.

[0034] It should be noted that in the process of collecting behavioral data of the user 118, erroneous or undesirable data may be captured as the first data set 302 in various cases. For example, the user 118 may feel depressed on one day and watch a tragic movie, but generally prefer comedy movies. Thus, the sad mood of the user 118 may be exceptional for the user 118, and the tragic movie that the user 118 watched when he / she is sad may not represent the normal behavior of the user 118. Thus, in the first data set 302, information about the tragic movie may be undesirable data. In another example, the user 118 may lend the electronic device 102 to another person (e.g., a friend or family member of the user 118). The other person may have different interests than the user 118. Thus, the other person who has borrowed the electronic device 102 for a certain period of time may watch movies in a genre other than comedy (e.g., thriller). In this case, again, information about such movies (e.g., thriller movies) may not represent the normal behavior of the user 118, and may need to be deleted as undesirable data.

[0035] The presence of such erroneous data in the first dataset 302 may cause the first ML model 110 to be incomplete. At time “T2”, the data subset 308A of the first dataset 308 may be determined to be an incomplete or erroneous dataset. In other words, it may be identified that the behavioral data, such as the previously used first dataset 308, may be erroneous causing the first ML model 110 to be incomplete. Since the training of the first ML model 110 is based on the erroneous first dataset 308 (which may be included in the data subset 308A), the first ML model 110 may be an incomplete model. Thus, the recommendations made by the first ML model 110 may not be optimal. To improve the prediction accuracy of the first ML model 110, it may be necessary to transition the first ML model 110 from an incomplete state to a non-incomplete state.

[0036] At time "T3," the first ML model 110 may be updated based on application of the transformation function 312 to obtain an updated first ML model 310. The updated first ML model 310 may unlearn certain features (e.g., undesirable features) associated with the data subset 308A. The updated first ML model 310 may provide optimal recommendations compared to the first ML model 110. More details regarding the transformation function 312 are described further in, for example, FIG. 8.

[0037] In an embodiment, the trained first machine learning model 110 can correspond to a recommendation model, and the updated first machine learning model 310 can be configured to output personalized recommendations based on the received first user input. Before the update of the first machine learning model 110, the recommendation model can output recommendations based on training the first ML model 110 on the received first dataset 302. For example, the recommendation model can be a content-based recommendation model. It can be understood that each user associated with the content-based recommendation model can have a certain personality trait. For example, there may be users who like to watch comedy videos, other users who like to watch action movies, and there may be another group of users who like to watch documentaries or news. A certain user may watch more videos of a certain type or genre than other types or genres of videos based on the personality traits of the user. Thus, the electronic device 102 can receive a first dataset 302 at time “T1” that can be used to train the first ML model 110 based on the user's behavior. The trained first ML model 110 can recommend videos to the user that are believed to match the user's tastes or preferences. For example, if the user prefers comedy videos over videos of other genres, the trained first ML model 110 can recommend a set of comedy videos in the recommendation list. As a result, the user can select the videos he or she wants to watch from the recommendation list. In some cases, the first dataset 302 may include one or more incomplete datasets. For example, a user who likes watching comedy videos may lend the electronic device 102 to another person for a period of time during which the other person watches action movies. Thus, after a certain period of time, the first dataset 302 may include behavioral data related to action movies. Because the first ML model 110 is trained based on the first dataset 302, the first ML model 110 may be incomplete and may recommend action videos to the user even though the user still prefers comedy videos.At time "T2," the data subset 308A may be determined to be incomplete based on the personalized recommendations by the trained first ML model 110. At time "T3," the first ML model 110 may be updated based on application of the transformation function 312 to obtain an updated first ML model 310. At this point, the updated first ML model 310 may recommend comedy videos over action videos as recommendations to the user.

[0038] In an embodiment, the circuit 202 can be further configured to receive a first user input indicating a time period associated with the data subset 308A, and can receive the data subset 308A based on the received first user input. In an example, the user 118 can receive the recommendation and realize that the personalized recommendation made by the trained first ML model 110 is incomplete. The trained first ML model 110 may recommend action videos (over comedy videos) based on the viewing history of another person, not the user 118. The user 118 can provide the first user input indicating the time period associated with the data subset 308A. For example, the user 118 can communicate that videos viewed from a particular date / time to another date / time are not desirable, and information related to such videos may correspond to the data subset 308A. As a result, the electronic device 102 can extract videos viewed during the aforementioned time period and update the first machine learning model 110 based on the received data subset 308A to obtain an updated first ML model 310. Here, the first machine learning model 110 can unlearn features associated with the received data subset 308A. For example, the features that can be unlearned associated with the received data subset 308A can include first features such as the number of videos of a particular genre watched by the user 118, the length of the videos watched by the user 118, the types of genres watched by the user 118, and the ratings or reviews associated with the videos watched by the user 118. The updated first machine learning model 310 can then be configured to output personalized recommendations based on the received first user input. For example, the updated trained first machine learning model 310 can unlearn features associated with the received data subset 308A based on the received first user input, so that the updated first machine learning model 310 can now recommend comedy videos to the user.

[0039] It should be noted that scenario 300 in FIG. 3 is for illustrative purposes and should not be construed as limiting the scope of the present disclosure.

[0040] Figure 4 illustrates an example scenario for performing an update of a machine learning model based on unlearning a dataset or features, according to an embodiment of the present disclosure. Figure 4 is described with reference to elements of Figures 1, 2, and 3. Figure 4 illustrates an example scenario 400. The scenario 400 can include a first dataset 402, a data subset 404, a first ML model 110, a transformation function 312, and an updated first ML model 310. A series of operations associated with the scenario 400 can now be described.

[0041] In the scenario 400 of FIG. 4, a first data set 402 associated with a user (such as user 118 of FIG. 1) may be received. For example, the first data set 402 may include a first data point (such as "User_1|25|M|20|5|s1,s2,s3..."), a second data point (such as "User_2|23|M|40|2|s1,s2,s3..."), ..., and a kth data point (such as "User_k|45|F|23|50|s1,s2,s3..."). The first data set 402 may include features that may train the first ML model 110. For example, the first data point "User_1|25|M|20|5|s1,s2,s3..." may indicate that a male user with an identity of "user_1" and age "25" may have interacted with items "s1, s2, and s3, etc." Based on the received first data set 402, the first ML model 110 may be trained. Further, a data set including the first data point (i.e., "User_1|25|M|20|5|s1,s2,s3...") and the second data point (i.e., "User_2|23|M|40|2|s1,s2,s3...") may be received as a data subset 404. Features associated with the data subset 404 may require unlearning. The first ML model 110 and the data subset 404 may be applied to a transformation function 312 to obtain an updated first ML model 310. The updated first ML model 310 may be obtained based on the first ML model 110 unlearning features associated with the data subset 404. Details of the transformation function 312 may be further described, for example, in FIG. 8.

[0042] It should be noted that scenario 400 in FIG. 4 is for illustrative purposes and should not be construed as limiting the scope of the present disclosure.

[0043] Figure 5 illustrates an exemplary scenario for training a first machine learning model, according to an embodiment of the present invention. Figure 5 is described with reference to elements of Figures 1, 2, 3, and 4. Figure 5 illustrates an exemplary scenario 500. Scenario 500 may include a first dataset 502, a first ML model 110, and an output 504. Output 504 may include N items, such as item-1 504A, ..., and item-N 504N. A series of operations associated with scenario 500 will now be described.

[0044] The N inputs and items shown in Figure 5 are presented as an example only. Output 504 may include only one input and item, or more than N inputs and items without departing from the scope of this disclosure. For simplicity, only N inputs and items are shown in Figure 5. However, in some embodiments, there may be more than N inputs and items without limiting the scope of the disclosure.

[0045] In the scenario 500 of FIG. 5, a first dataset 502 may be received. The first dataset 502 may include one or more features, such as a first feature including an identity of a user (e.g., user 118), a second feature indicative of the user's age, a third feature indicative of the user's gender, a fourth feature indicative of the user's viewing time, a fifth feature indicative of inactive time since last viewing, and a sixth feature indicative of the user's viewing history. Each feature may be provided as an input to an input layer of the first ML model 110. The first ML model 110 may be trained based on the received first dataset 502 and may make recommendations including N items, such as item-1 504A, ..., item-N 504N. In one example, the trained first ML model 110 may output a first item as an identification of the recommended video, a second item as a title of the recommended video, a third item as a video length of the recommended video, a fourth item as a genre of the recommended video, a fifth item as a language of the recommended video, and a sixth item as a category, such as a show or movie, of the recommended video.

[0046] It should be noted that scenario 500 in FIG. 5 is for illustrative purposes and should not be construed as limiting the scope of the present disclosure.

[0047] Figure 6 illustrates an example scenario for updating a machine learning model based on unlearning a dataset or features, according to an embodiment of the present disclosure. The description of Figure 6 is provided with reference to elements of Figures 1, 2, 3, 4, and 5. Figure 6 illustrates an example scenario 600. The scenario 600 may include a first dataset 602, a first label set 604, a data subset 606, and a second label set 608. The scenario 600 further illustrates a user 610. A series of actions associated with the scenario 600 will now be described.

[0048] In an embodiment, the circuit 202 can be further configured to extract a second label set 608 associated with the received data subset 606. The circuit 202 can be further configured to extract a first label set 604 associated with the received first data set 602. The circuit 202 can be further configured to remove the extracted second label set 608 associated with the data subset 606 from the extracted first label set 604 associated with the received first data set 602. The circuit 202 can be further configured to determine a third label set based on removing the extracted second label set 608 associated with the data subset 606 from the extracted first label set 604 associated with the received first data set 602.

[0049] Labels can be understood to provide meaningful context for a given dataset such that a given machine learning model can learn from the labels. For example, if a given dataset includes images, the labels of the images can indicate an animal, such as a cat, present in the image. In other words, the labels can correspond to objects (or classes) that a given machine learning model can be configured to identify (or classify) for a given input data point. For example, if an input data point such as "User_1|25|M|20|5|s1,s2,s3..." is provided to the first ML model 110, the first ML model 110 can output a prediction such as a video file that includes an item set as "Vid_17,ABC,large,drama,English,and show". In such a case, "Vid_17" can be an identifier of the video, "ABC" can be the title of the video, "large" can be the length of the video, "drama" can be the genre of the video, "English" can be the language of the video, and "show" can be the category of the video file predicted by the first ML model 110 based on the provided input data. Each item in the item set of the video file to be predicted can correspond to a label. In the above example, "Vid_17" can correspond to the first label, "ABC" can correspond to the second label, "large" can correspond to the third label, "drama" can correspond to the fourth label, "English" can correspond to the fifth label, and "show" can correspond to the sixth label. The training dataset and the respective labels can be used to train a given machine learning model.

[0050] In the scenario 600 of FIG. 6, behavioral data associated with a user 610 can be received as a first dataset 602. The acquisition of the behavioral data can lead to the development of a recommendation model through training of a first machine learning model 110. In the scenario 600, the first dataset 602 can include data points such as a first data point (e.g., “User_1|25|M|20|5|s1,s2,s3...”), a second data point (e.g., “User_2|23|M|40|2|s1,s2,s3...”), ..., a k-th data point (e.g., “User_k|45|F|23|50|s1,s2,s3...”). The circuit 202 can extract a first label set 604 associated with the received first dataset 602. In the scenario 600, a first extracted label set 604 associated with a received first data set 602 includes an aggregate label “LA1” as “Label1, Label2, … Labelk” corresponding to a first data point, an aggregate label “LA2” as “Label1, Label2, … Labelk” corresponding to a second data point, and an aggregate label “LA3” as “Label1, Label2, … Labelk” corresponding to a k data point. n ". The electronic device 102 can use the extracted first label set 604 and the received first dataset 602 to train a first ML model 110 (not shown in FIG. 6).

[0051] The user 610 may realize that a particular recommendation made by the first ML model 110 may not be optimal. This is because the first dataset 602 includes a data set (including one or more data points) that may not correspond to the user 610, which may cause the first ML model 110 to be incompletely trained. For example, the user 610 may lend the electronic device 102 to a friend, who may watch certain videos, such as news and sports, that are not relevant to the user 610. However, as the electronic device 102 continuously receives behavioral data, the first ML model 110 may also be retrained based on behavioral data related to the user 610's friends. Thus, the retrained first ML model 110 may recommend videos related to topics, such as news and sports, that may not be relevant to the user 610. In another example, the user 610 may watch certain videos, such as pessimistic videos, during a period of time due to an unexpected event in his / her life. However, the user 610 may not prefer to watch such videos (such as pessimistic videos) again. The first ML model 110 may recommend videos similar to the pessimistic video based on the updated viewing history of the user 610. To mitigate the above-mentioned problems, the user 610 may trigger a request to allow the first ML model 110 to unlearn data acquired between two specified timestamps. In such a case, the user 610 may provide a first user input indicating a period during which the data subset (e.g., the data subset 606) may have been received. For example, the user 610 may provide an input indicating that he or she watched a pessimistic video from 6:00 AM to 12:00 PM on a certain day, but does not wish to watch such videos again. The first ML model 110 may need to unlearn the corresponding user behavior data (e.g., the viewing history of the user 610 associated with a particular period).For example, the electronic device 102 may provide user interface elements, such as dialog boxes, drop-down menus, and date-time pickers, through the display device 210 of the electronic device 102 to receive a first user input. Based on the received first user input, a data subset 606 may be received. In the scenario 600, the data subset 606 may be received that includes data points, such as a second data point (e.g., "User_2|23|M|40|2|s1,s2,s3...") and a first data point (e.g., "User_1|25|M|20|5|s1,s2,s3..."). A second set of labels 608 associated with the received data subset 606 may be extracted. In scenario 600, a second label set 608 associated with the received data subset 606 may include "Label_u1, Label_u2, ... Label_uk" associated with the second data point and "Label_u1, Label_u2, ... Label_uk" associated with the first data point. The second label set 606 and the received data subset 606 may be used to train a second ML model 112 that may update the first ML model 110 based on application of a transformation function (such as transformation function 312 of FIG. 3). Determining the labels for updating the first ML model 110 is further described, for example, in FIG. 7.

[0052] It should be noted that scenario 600 in FIG. 6 is for illustrative purposes and should not be construed as limiting the scope of the present disclosure.

[0053] FIG. 7 illustrates an exemplary scenario for determining labels, according to an embodiment of the present disclosure. FIG. 7 is described with reference to elements of FIG. 1, FIG. 2, FIG. 3, FIG. 4, FIG. 5, and FIG. 6. FIG. 7 illustrates an exemplary scenario 700. The scenario 700 can include a first label set 702, a second label set 704, a fourth label 708, a fifth label 710, a new label 712, a database 106, and a second machine learning model 112. The database 106 can include the new label 712 and a data subset 714. The scenario 700 further illustrates an exemplary operation 706. A series of operations related to the scenario 700 will now be described.

[0054] In the scenario 700 of FIG. 7, the circuit 202 is configured to receive a second set of labels (Y un ) 704. As described above, the second label set 704 can provide meaningful context to a received data subset (not shown) such that a second ML model (such as the second ML model 112 of FIG. 1 ) can learn from the second label set 704. Note that the first ML model 110 can use the extracted second label set (Y un Such labels may be referred to as an "unlearned" label set, since it may be necessary to unlearn the first set of labels 704. Further details regarding the extraction of the second set of labels 704 are described further in, for example, FIG.

[0055] The circuit 202 generates a first set of labels (Y Tr ) 702. The first label set 702 can provide a meaningful context for a received first dataset (not shown) so that the first ML model 110 can learn from the first label set 702. Note that the extracted first label set (Y Tr) 702 may also be referred to as a training label set, since it may be used to train the first ML model 110. Details regarding the extraction of the first label set 702 are further described in, for example, FIG.

[0056] The circuit 202 may be configured to remove the extracted second label set 704 (associated with the data subset 606) from the extracted first label set 702 (associated with the received first data set 602). The circuit 202 may be further configured to determine a third label set based on removing the extracted second label set 704 (associated with the data subset 606) from the extracted second label set 704 (associated with the received first data set 602). The extracted second label set 704 may be removed based on the following equation (1): Y'' Un =Y Tr -Y Un (1) Here, "Y'' Un " may represent a third set of labels, "Y Tr " may represent the first label set 702 extracted, "Y Un " may represent the extracted second label set 704.

[0057] Here, the extracted second label set 704 can be removed from the extracted first label set 702 such that the determined third label set does not include the extracted second label set 704.

[0058] In an embodiment, the circuit 202 may be further configured to determine whether each label of the determined third label set corresponds to a category label, e.g., as further described at 706. The circuit 202 may be further configured to determine a total number of determined third label sets based on the determination that each label of the determined third label set corresponds to a category label. The circuit 202 may be further configured to determine a fourth label 708 based on the determined total number of determined third label sets. The fourth label 708 may correspond to a maximum number in the determined third label set, and the second machine learning model 112 (not shown) may be further trained based on the determined fourth label 708.

[0059] For example, in 706 of the scenario 700 of FIG. 7, the circuit 202 may determine a category of the determined third label set. In one embodiment, the circuit 202 may determine a category of the determined third label set (Y″ Un ) may be determined to be categorical or not. If the determined third label set is categorical, it may be understood that the value of the determined third label set may be a discrete value such as "yes" or "no" (or "0" or "1").

[0060] Each of the extracted second label set 704 and the extracted first label set 702, or the determined third label set if the determined third label set is categorical, may be determined to have a total number of the determined third label set. The most numerous label in the determined third label set may be determined as the fourth label 708.

[0061] The circuit 202 may determine a fourth label 708 based on the determined total number of the determined third label set. The fourth label 708 may correspond to a maximum number in the determined third label set, and may further train the second machine learning model 112 (not shown) based on the determined fourth label 708. The fourth label 708 may be determined according to the following equation (2): Y' Un =Max(Y Tr -Y Un ) (2) Here, "Y' Un " may represent the determined fourth label 708, "Y Tr " may represent the first label set 702, "Y Un " may represent the second label set 704.

[0062] As is evident from equation (1), the first set of labels (Y Tr ) 702 and the second label set (Y Un ) 704 may be determined as a third label set. Thus, based on equations (1) and (2), the most numerous label in the third label set may be determined as a fourth label 708. If the determined category of the determined third label set is categorical, then the determined fourth label 708 may be considered as a new label 712. However, in some cases, the determined category of the determined third label set may be numerical.

[0063] In an embodiment, the circuit 202 may be further configured to determine whether each label in the determined third label set corresponds to a numerical label. The circuit 202 may be further configured to determine an average of the determined third label set based on a determination that each of the determined third label set corresponds to a numerical label. The circuit 202 may be further configured to determine a fifth label 710 based on the average of the determined third label set. The second machine learning model 112 may be further trained based on the determined fifth label 710.

[0064] For example, the circuit 202 may determine the category of the determined third label set Y″ at 706 of the scenario 700 of FIG. Un It can be determined whether Y″ is a numeric value. If the determined third label set is non-categorical or numeric, it can be understood that the output of the determined third label set can be a non-discrete or continuous value, such as 0.1, 0.11, 0.123, and 5.893. Un If , then, the average of the determined third set of labels may be determined to determine a fifth label 710 that may be assigned as a new label 712. The fifth label 710 may be determined according to the following equation (3): TIFF2025513761000002.tif20150(3) Here, "Y' Un " may represent the determined fifth label 710, "Y Tr " may represent the first label set 702 extracted, "Y Un " may represent the extracted second label set 704, "n" may represent the total number of labels present in the third label set.

[0065] If the determined category of the determined third label set is numeric, the determined fifth label 710 may be considered as a new label 712. The determined new label 712 and the received data subset 714 may be used to train the second ML model 112. The second ML model 112 may further be used to update the first ML model 110 based on a transformation function (such as the transformation function 312 of FIG. 3). Updating the first ML model 110 enables the updated first ML model 110 to output the new label 712 for the received data subset 714. More details regarding the transformation function are further described, for example, in FIG. 8.

[0066] It should be noted that scenario 700 in FIG. 7 is for illustrative purposes and should not be construed as limiting the scope of the present disclosure.

[0067] FIG. 8 illustrates an exemplary scenario for updating a first ML model based on application of a transformation function, according to an embodiment of the present disclosure. FIG. 8 is described with reference to elements of FIG. 1, FIG. 2, FIG. 3, FIG. 4, FIG. 5, FIG. 6, and FIG. 7. FIG. 8 illustrates an exemplary scenario 800. The scenario 800 may include a first dataset 402, a data subset 404, a first ML model 110, a second ML model 112, a stack layer 802, a convolutional neural network (CNN) layer 804, a fully connected (FC) layer 806, an output layer 808, and an updated first ML model 810. The scenario 800 further illustrates a matrix 812 including a first row 812A, a second row 812B, and a third row 812C. A series of operations associated with the scenario 800 will now be described.

[0068] In the scenario 800 of Figure 8, a first dataset 402 associated with a user, such as user 118, may be received. Details of the first dataset 402 are further described in, e.g., Figure 4. A first ML model 110 may be trained based on the received first dataset 402. Additionally, a data subset 404 may be received. A second ML model 112 may be trained based on the data subset 404. Details of the data subset 404 are further described in, e.g., Figure 4. A transformation function may be applied to the trained first machine learning model 110 and the trained second machine learning model 112.

[0069] In an embodiment, the circuit 202 may be further configured to build a stack layer 802 associated with the transformation function. The stack layer 802 may be configured to stack the trained first machine learning model 110 and the trained second machine learning model 112 to update the trained first machine learning model 110. In other words, the circuit 202 may stack the trained first machine learning model 110 with the trained second machine learning model 112 using the stack layer 802 as an intermediate layer between the trained first machine learning model 110 and the trained second machine learning model 112. In an embodiment, the transformation function may include the built stack layer 802 and a series of deep neural network (DNN) layers.

[0070] In the scenario 800 of FIG. 8, the trained first machine learning model 110 may be stacked with the trained second machine learning model 112 using the stack layer 802, such that the trained first machine learning model 110 forms the first row 812A of the matrix 812. The trained second machine learning model 112 may form the third row 812C of the matrix 812. The stack layer 802 may be the second row 812B of the matrix 812 and may be located between the first row 812A and the third row 812C. In an embodiment, weights associated with the trained first machine learning model 110 may be stored in the first row 812A, and weights associated with the trained second machine learning model 112 may be stored in the third row 812C. Based on the application of the stack layer 802, the trained second machine learning model 112 may be used to apply a transformation function to the trained first machine learning model 110.

[0071] In an embodiment, the transformation function may correspond to a dot product of a first output of the trained first machine learning model 110 and a second output of the second machine learning model 112. The stack layer 802 may determine an element-wise product interaction between the first output of the trained first machine learning model 110 and the second output of the second machine learning model 112 to update the trained first machine learning model 110.

[0072] In one embodiment, the transformation function may further correspond to a normalization of the dot product based on the first output, i.e., the determined element-wise product interaction may be divided by the modulus of the first output of the trained first machine learning model 110. In other words, the transformation function may be determined according to the following equation (4): TIFF2025513761000003.tif20150(4) Here, "O dot " can represent a transformation function, “Φ(x)” may represent the first output of the trained first machine learning model 110, “Ψ(x)” may represent the second output of the trained second machine learning model 112, “ε” can be a parameter to avoid erroneous results when the absolute value of the transformation function of the trained first machine learning model 110 is “0” or a very small value close to “0”.

[0073] In the scenario 800 of FIG. 8, a CNN layer 804 and an FC layer 806 may be applied after the application of the stack layer 802. It may be understood that the CNN layer 804 may include an input layer, one or more hidden layers, and an output layer. Thus, for feature extraction from the trained first ML model 110 and the trained second ML model 112, stacked outputs of both models may be fed to the CNN layer 804. Thus, attributes or features that may cause over-generalization of the performance of the trained first ML model 110 may be unlearned from the trained first ML model 110. However, in contrast to a typical neural network, each neuron in the input layer (of the CNN layer 804) may not be connected to neurons in one or more hidden layers (of the CNN layer 804). Only neurons in the local receptive field may be connected to the hidden layers. Based on the convolution operation, a feature mapping may be created. The output of the CNN layer 804 may be fed to the input of the FC layer 806. The FC layer 806 can be understood to be a simple feed-forward neural network. The CNN layer 804 can be used as a feature of the spatial data that can be efficiently extracted. The output of the CNN layer 804 can be connected to the output layer 808 (through the FC layer 806) to obtain an updated first ML model 810.

[0074] The updated first ML model 810 can be an unlearned version of at least one of the feature set associated with the data subset 404 or the second machine learning model 112. Thus, the updated first ML model 810 can make recommendations more accurately than the first ML model 110 before the update. For example, a first user (such as the user 118 of FIG. 1 ) may lend the electronic device 102 or video streaming account to a second user so that the second user can watch videos for a period of time. The second user may be able to watch sports videos, but the first user may not be interested in such sports videos. However, the first ML model 110 may continue to learn the behavioral data and begin to make recommendations related to sports videos to the first user. Eventually, the first user may lose interest in these recommendations and provide information as a first user input indicating a period of time during which the second user is believed to have watched sports videos. The data subset 404 may be received based on the received first user input to train the second ML model 112. Further, a transformation function may be applied to the trained first machine learning model 110 and the second machine learning model 112 to obtain an updated first machine learning model 810. Based on the update of the first ML model 110, the first ML model 110 may unlearn the data subset 404 related to sports videos and / or unlearn a feature set related to sports videos (e.g., features related to a video type as sports / game videos). At this point, the updated first machine learning model 810 may make recommendations that are deemed to be relevant to the first user. Thus, the attributes of the trained second machine learning model 112, which may be configured for over-generalized performance (which may also be referred to as an "unlearnable model"), may be useful in unlearning undesirable data from the trained first machine learning model 110.

[0075] It should be noted that scenario 800 in FIG. 8 is for illustrative purposes and should not be construed as limiting the scope of the present disclosure.

[0076] FIG. 9 illustrates an exemplary processing pipeline for performing an update of a machine learning model based on unlearning a dataset or features, according to an embodiment of the present disclosure. FIG. 9 is described with reference to elements of FIGS. 1, 2, 3, 4, 5, 6, 7, and 8. FIG. 9 illustrates an exemplary processing pipeline 900 illustrating exemplary operations 902-908 for performing an update of a machine learning model based on unlearning a dataset or features. The exemplary operations 902-908 can be performed by any computer system, such as, for example, the electronic device 102 of FIG. 1 or the circuit 202 of FIG. 2. The exemplary processing pipeline 900 further illustrates the first ML model 110, the second ML model 112, the first dataset 114, the data subset 116A, the updated first ML model 310, and the transformation function 312.

[0077] At 902, a data subset receiving operation can be performed. The circuit 202 can be configured to receive a data subset 116A of a first dataset 114 associated with a user (such as the user 118 of FIG. 1 ), and can train a first machine learning model 110 based on the first dataset 114 associated with the user. The first dataset 114 associated with the user 118 can be behavioral data associated with the user 118. Behavioral data over a period of time associated with the user 118 can be collected and received as the first dataset 114. The first machine learning model 110 can be trained based on the first dataset 114 associated with the user 118. The first dataset 114 can include multiple datasets 116. However, in some cases, one or more datasets of the multiple datasets 116 may not be associated with the user 118. Such a dataset that may not be associated with the user 118 can be referred to as a data subset 116A. The first machine learning model 110 can be trained based on the first dataset 114, and therefore can also make recommendations based on the data subset 116A. In one embodiment, the first machine learning model 110 may be a recommendation model. Details regarding the first machine learning model 110 are further described in, for example, FIG.

[0078] In an embodiment, the circuit 202 can be further configured to receive a first user input indicating a time period, and the data subset 116A is received based on the received first user input. For example, the time period can relate to a date or a time interval during which the electronic device 102 can collect the data subset 116A, etc. For example, when the user 118 realizes that some data may need to be unlearned, the user 118 can specify a specific date or time interval to select the data to be unlearned. For example, a user master file can be invoked from the server 104 to fetch the specified data to be unlearned (hereinafter also referred to as "unlearned data"). The circuit 202 can extract the specified "unlearned data" for the above-mentioned time interval as the data subset 116A. Details regarding the first user input are further described, for example, in FIG. 3.

[0079] In an embodiment, the circuit 202 may be further configured to compare the output of the trained first machine learning model 110 to a threshold. The circuit 202 may be further configured to determine the output to be incomplete based on a comparison of the output of the trained first machine learning model 110 to the threshold. The circuit 202 may be further configured to send a notification based on the determination that the output is incomplete. The circuit 202 may be further configured to receive a second user input based on the sent notification indicating the incomplete output, and may receive the data subset 116A based on the second user input. The threshold here may be a minimum value of a parameter used to evaluate the performance (e.g., precision, accuracy, recall, or F-score) of the first ML model 110.

[0080] The circuit 202 may be further configured to determine the output of the trained first machine learning model 110 as incomplete based on a comparison of the output of the trained first machine learning model 110 to a threshold. Examples of parameters that may be used to evaluate the performance of the first ML model 110 may include, but are not limited to, a confusion matrix, an F1 score, precision, accuracy, sensitivity, and specificity. If the measured parameters of the first ML model 110 are less than the threshold, the first ML model 110 may be considered suitable for recommendation.

[0081] The circuit 202 may be further configured to send an alert notification to the user 118 based on the determination that the output is incomplete. If the output of the trained first machine learning model 110 is below a threshold, the output of the first ML model 110 may be determined to be incomplete. In such a case, the output, such as a recommendation made by the first ML model 110, may not be optimal. The user 118 may be notified that the output of the first ML model 110 is incomplete. The notification may be a pop-up notification displayed on a display (such as the display device 210 of FIG. 2) of the electronic device 102. As an example, the notification may be provided to the user 118 based on haptic feedback. For example, the electronic device 102 may vibrate to indicate that the output of the first ML model 110 is incomplete.

[0082] The circuit 202 may be further configured to receive a second user input based on the notification indicating the transmitted incomplete output, and may receive the data subset 116A based on the second user input. For example, if the output of the first ML model 110 is determined to be incomplete, the user 118 may be prompted to provide an input indicating whether or not he or she wishes to perform unlearning of the particular data. In one example, a dialog box may be displayed on the display of the electronic device 102 displaying a question such as "The model is incomplete. Do you want to perform unlearning?". Further options such as "yes" or "no" may also be provided to the user 118. The second user input may be received based on a user selection of one of the options displayed on the display of the electronic device 102. For example, if the user 118 selects "yes," the electronic device 102 may receive the data subset 116A associated with the user 118 based on the second user input. In one embodiment, the circuit 202 may also display a time period associated with the data subset 116A. Additionally, the user 118 may be prompted to indicate whether they want the first ML model 110 to unlearn features associated with the data subset 116A collected within that time period. In another embodiment, the user 118 may simply be informed that the output of the data subset 116A is incomplete and the user 118 may be requested to provide the time period associated with the data subset 116A.

[0083] At 904, a training operation of the second machine learning model may be performed. The circuit 202 may be further configured to train the second machine learning model 112 based on the received data subset 116A. In an embodiment, the training of the second machine learning model 112 may be supervised. In an embodiment, the second label set (such as the second label set 608 of FIG. 6) and the received data subset 116A may be provided as input to the second ML model 112. In another embodiment, the training of the second machine learning model 112 may be unsupervised. Here, only the data subset 116A may be provided as input to the second ML model 112. The second ML model 112 may learn with experience. Details regarding the training of the second ML model 112 are further described, for example, in FIG. 8.

[0084] At 906, a conversion function application operation may be performed. The circuit 202 may be further configured to apply a conversion function 312 to the trained first machine learning model 110 based on the trained second machine learning model 112. The conversion function 312 may be applied based on stacking the trained first ML model 110 and the trained second ML model 112 using an intermediate layer (such as the stack layer 802 in FIG. 8 ) between the trained first ML model 110 and the trained second ML model 112. Furthermore, to obtain the conversion function 312, a dot product of the trained first ML model 110 and the trained second ML model 112 may be divided by the absolute value of the trained first ML model 110. More details regarding the conversion function are further described in, for example, FIG. 8 .

[0085] At 908, an update operation of the trained first ML model may be performed. The circuit 202 may be further configured to update the trained first ML model 110 based on the application of the transformation function, and the update of the trained first ML model 110 may correspond to unlearning at least one of the feature set associated with the data subset 116A or the second machine learning model 112. Updating the trained first ML model 110 may prevent the updated first ML model 310 from making recommendations based on the data subset 116A. Since the data subset 116A may not be relevant to the user 118 of the electronic device 102, unlearning at least one of the feature set associated with the data subset 116A or the second machine learning model 112 by the trained first ML model 110 may enable the updated first ML model 310 to make optimal recommendations according to the behavior of the user 118. Thus, incomplete output may be prevented. Thus, the electronic device 102 can unlearn the erroneously captured / erroneous data set or undesired feature set in the existing model to obtain an updated model that achieves the desired output performance. Since the trained first ML model can unlearn at least one of the data subset or feature set associated with the second machine learning model, the updated ML model can be optimal and not make erroneous recommendations. More details regarding updating the trained first ML model are further described in, for example, FIG. 8.

[0086] FIG. 10 illustrates an exemplary scenario of updating a machine learning model for content-based recommendation, according to an embodiment of the present invention. FIG. 10 is described with reference to elements of FIG. 1, FIG. 2, FIG. 3, FIG. 4, FIG. 5, FIG. 6, FIG. 7, FIG. 8, and FIG. 9. FIG. 10 illustrates an exemplary scenario 1000. The scenario 1000 may include an electronic device 102, a display device 210, an action video-1 1002, a recommended video 1004, a comedy video-1 1006, and a recommended video 1008. The recommended video 1004 may include an action video-2 1004A, an action video-3 1004B, and an action video-4 1004C. The recommended video 1008 may include a comedy video-2 1008A, a comedy video-3 1008B, and a comedy video-4 1008C. A series of operations related to the scenario 1000 will now be described.

[0087] In the scenario 1000 of FIG. 10, a user such as the user 118 of FIG. 1 may use the electronic device 102. The user 118 may loan the electronic device 102 to person A for a period of time. The person A may watch action video-1 1002 on the media streaming platform of the electronic device 102. However, the user 118 may not like action videos and may prefer to watch comedy videos. Alternatively, the user 118 may have been watching the action video-1 1002 to relax. However, the user 118 may not like such action videos and may prefer to watch comedy videos. In either case, since the action video-1 1002 may have been watched, the first ML model 110 of FIG. 1 may be trained on such videos and may recommend recommended videos 1004 to the user 118, including other action videos (e.g., action video-2 1004A, action video-3 1004B, action video-4 1004C, etc.). Since the user 118 may not like watching the recommended video 1004, the first user input and the second user input may be provided to update the trained first machine learning model 110 so that the trained first machine learning model 110 can unlearn information related to the action video-1 1002. For example, as shown in FIG. 3, FIG. 4, FIG. 5, FIG. 6, FIG. 7, FIG. 8, and FIG. 9, the trained first ML model may be updated based on the application of a transformation function to the trained first machine learning model 110 with the trained second machine learning model 112. The updated first ML model 110 may make optimal recommendations, such as recommended videos 1008 including comedy video-2 1008A, comedy video-3 1008B, and comedy video-4 1008C, based on the comedy video-1 1006 watched by the user. Thus, based on unlearning by the trained first ML model 110, videos can be recommended to the user 118 based on the normal behavior of the user 118 and not on the behavior of person A (or the abnormal behavior of the user 118).

[0088] It should be noted that the electronic device 102 of the present invention can be used in multiple applications. In one example, the electronic device 102 can be used in an artificial intelligence (AI)-based model that aims to provide personalized recommendations to a user. The personalization process may require continuous acquisition of user behavior data. If the newly received data does not make sense, the existing AI-based model may be corrupted, as such data may cause the AI-based model to be incomplete. In a few cases, when the integrity of the data is in doubt after the development of the AI-based model, unlearning as described herein may be one of the preferred options, if not the only option.

[0089] In a second application, privacy legislation and laws of various countries and jurisdictions may include provisions requiring a right to be forgotten. Thus, the recommendation model may address the request to be forgotten based on unlearning a set of features associated with undesired data or a machine learning model trained on such undesired data. For example, a user 118 may have watched some videos related to news. However, the user 118 may want the trained first ML model (such as the trained first ML model 110 of FIG. 1) to forget the videos the user 118 watched. In such a case, the trained first machine learning model (e.g., the trained first machine learning model 110 of FIG. 1) may be updated to unlearn a set of features associated with at least one dataset (related to news videos) or a second machine learning model (such as the trained second machine learning model 112 of FIG. 1).

[0090] In a third application, unexpected events may occur that may affect a user in a way that deviates from the user's normal behavior or personality traits over a period of time. Detecting such events and unlearning the behavioral data may be necessary to achieve reasonable model performance. For example, a user may have been depressed by unexpected events in their life. A user may have watched a sad video that is generally considered to be unpopular. In such a case, a trained first machine learning model (such as the trained first machine learning model 110 of FIG. 1) may be updated to unlearn at least a dataset or features related to sad videos.

[0091] It should be noted that the scenario 1000 of FIG. 10 is for illustrative purposes and should not be construed as limiting the scope of the present disclosure.

[0092] Figure 11 is a flowchart illustrating the operation of an exemplary method according to an embodiment of the present invention. The description of Figure 11 will be made with reference to elements of Figures 1, 2, 3, 4, 6, 7, 8, 9, and 10. Figure 11 shows a flowchart 1100. The flowchart 1100 may include operations 1102-1110 and may be performed by the electronic device 102 of Figure 1 or the circuit 202 of Figure 2. The flowchart 1100 may start at 1102 and proceed to 1104.

[0093] At 1104, a data subset 116A of a first dataset 114 associated with a user (such as the user 118 of FIG. 1 ) can be received, and a first machine learning model (such as the first machine learning model 110 of FIG. 1 ) can be trained based on the first dataset (such as the first dataset 114 of FIG. 1 ) associated with the user (such as the user 118 of FIG. 1 ). The circuit 202 can be configured to receive the data subset 116A associated with the user 118, and can train the first machine learning model 110 associated with the user 118 based on the first dataset 114 associated with the user 118. The first dataset 114 can be behavioral data associated with the user 118. The first dataset 114 can include multiple datasets (such as the multiple datasets 116 of FIG. 1 ). However, in some cases, one or more of the multiple datasets may not be associated with the user 118 and may be an incorrect dataset for the user 118. Such a dataset that may not be associated with the user 118 can be referred to as a data subset 116A. Details of the first ML model 110 are further described in, for example, FIG.

[0094] At 1106, a second machine learning model (such as the second machine learning model 112 of FIG. 1) may be trained based on the received data subset (such as the received data subset 116A of FIG. 1). The circuit 202 may be further configured to train the second machine learning model 112 based on the received data subset 116A. Details regarding training the second ML model are further described in, for example, FIG.

[0095] At 1108, a transformation function (such as transformation function 312 of FIG. 3) may be applied to the trained first machine learning model (such as trained first machine learning model 110 of FIG. 1) based on the trained second machine learning model (such as trained second machine learning model 112 of FIG. 1). The circuit 202 may be further configured to apply a transformation function to the trained first machine learning model 110 based on the trained second machine learning model 112. The transformation function may be applied based on stacking the trained first ML model 110 with the trained second ML model 112 using an intermediate layer (such as stack layer 802 of FIG. 8) between the trained first ML model 110 and the trained second ML model 112. More details regarding the transformation function are further described in, for example, FIG. 8.

[0096] At 1110, the trained first ML model 110 (such as the trained first ML model 110 of FIG. 1 ) may be updated based on application of a transformation function (such as the transformation function 312 of FIG. 3 ) to the trained first ML model 110, where the updating of the trained first ML model 110 may correspond to unlearning at least one of a feature set associated with the data subset (such as the data subset 116A of FIG. 1 ) or the second machine learning model 112. The circuit 202 may be further configured to update the trained first ML model 110 based on application of the transformation function to the trained first ML model 110. The updating of the trained first ML model 110 may correspond to unlearning at least one of a feature set associated with the data subset 116A or the second machine learning model 112. Updating the trained first ML model 110 may prevent the updated first ML model 110 from making recommendations based on the data subset 116A. Details regarding updating the trained first ML model 110 are further described in, e.g., Figure 8. Control may proceed to END.

[0097] Although flowchart 1100 is depicted as discrete operations, such as 1104, 1106, 1108, and 1110, the disclosure is not so limited. Thus, in some embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the implementation, without departing from the essence of the disclosed embodiments.

[0098] Various embodiments of the present disclosure may provide a non-transitory computer-readable medium and / or storage medium having computer-executable instructions executable by a machine and / or computer to operate an electronic device (such as, for example, electronic device 102 of FIG. 1). Such instructions may cause electronic device 102 to perform operations that may include receiving a data subset (such as data subset 116A of FIG. 1) of a first dataset (such as first dataset 114 of FIG. 1) associated with a user (such as user 118 of FIG. 1). A first machine learning model (such as first machine learning model 110 of FIG. 1) may be trained based on the received first dataset (such as first dataset 114 of FIG. 1) associated with user 118. The operations may further include training a second machine learning model (such as second machine learning model 112 of FIG. 1) based on the received data subset 116A. The operations may further include applying a transformation function (such as transformation function 312 of FIG. 3 ) to the trained first machine learning model 110 based on the trained second machine learning model 112. The operations may further include updating the trained first machine learning model 110 based on application of the transformation function 312 to the trained first machine learning model 110. The updating of the trained first machine learning model 110 may correspond to unlearning at least one of the data subset 116A or a feature set associated with the second machine learning model 112.

[0099] An exemplary aspect of the disclosure may provide an electronic device (such as electronic device 102 of FIG. 1 ) including a circuit (such as circuit 202 ). Circuit 202 may be configured to receive a data subset (such as data subset 116A of FIG. 1 ) of a first dataset (such as first dataset 114 of FIG. 1 ) associated with a user (such as user 118 of FIG. 1 ). A first machine learning model (such as first machine learning model 110 of FIG. 1 ) may be trained based on the received first dataset (such as first dataset 114 of FIG. 1 ) associated with the user. Circuit 202 may be configured to train a second machine learning model (such as second machine learning model 112 of FIG. 1 ) based on the received data subset 116A. Circuit 202 may be configured to apply a transformation function (such as transformation function 312 of FIG. 3 ) to trained first machine learning model 110 based on trained second machine learning model 112. The circuitry 202 may be configured to update the trained first machine learning model 110 based on application of the transformation function 312 to the trained first machine learning model 110. The update of the trained first machine learning model 110 may correspond to unlearning at least one of the data subset 116A or a feature set associated with the second machine learning model 112.

[0100] In an embodiment, the circuit 202 can be further configured to receive a first user input indicating a time period and can receive the data subset 116A based on the received first user input.

[0101] In an embodiment, the trained first machine learning model 110 may correspond to a recommendation model, and the updated first machine learning model (e.g., the updated first machine learning model 810) may be configured to output personalized recommendations based on the received first user input.

[0102] In an embodiment, the circuit 202 can be further configured to compare the output of the trained first machine learning model 110 to a threshold. The circuit 202 can be further configured to determine the output to be incomplete based on the comparison of the output of the trained first machine learning model 110 to the threshold. The circuit 202 can be further configured to send a notification based on the output being determined to be incomplete. The circuit 202 can be further configured to receive a second user input based on the sent notification indicating the incomplete output, and can receive the data subset 116A based on the second user input.

[0103] In an embodiment, the circuit 202 can be further configured to extract a first label set (e.g., first label set 702) associated with the received first data set 114. The circuit 202 can be further configured to extract a second label set (e.g., second label set 704) associated with the received data subset 116A. The circuit 202 can be further configured to remove the extracted second label set 704 associated with the data subset 116A from the extracted first label set 702 associated with the received first data set 114. The circuit 202 can be further configured to determine a third label set based on removing the extracted second label set 704 associated with the data subset 116A from the extracted first label set 702 associated with the received first data set 114.

[0104] In an embodiment, the circuit 202 can be further configured to determine whether each label of the determined third label set corresponds to a category label. The circuit 202 can be further configured to determine a total number of the determined third label sets based on a determination that the determined category of the determined third label set is a category label. The circuit 202 can be further configured to determine a fourth label (e.g., a fourth label 708) based on the determined total number of the determined third label sets. The fourth label 708 can correspond to a maximum number in the determined third label set. The second machine learning model 112 can be further trained based on the determined fourth label 708.

[0105] In an embodiment, the circuit 202 can be further configured to determine whether each label in the determined third label set corresponds to a numerical label. The circuit 202 can be further configured to determine an average of the determined third label set based on the determination that each label in the determined third label set corresponds to a numerical label. The circuit 202 can be further configured to determine a fifth label (e.g., the fifth label 710) based on the determined average of each label in the determined third label set. The second machine learning model 112 can be further trained based on the determined fifth label 710.

[0106] In an embodiment, the circuit 202 may be further configured to build a stack layer (e.g., stack layer 802) associated with the transformation function 312, and the stack layer 802 may be configured to stack the trained first machine learning model 110 and the trained second machine learning model 112 to update the trained first machine learning model 110.

[0107] In one embodiment, the transform function 312 may include constructed stack layers 802 and a series of deep neural network (DNN) layers (such as CNN layers 804).

[0108] In one embodiment, the transformation function 312 may correspond to a dot product of a first output of the trained first machine learning model 110 and a second output of the second machine learning model 112.

[0109] In an embodiment, the transformation function may further accommodate a normalization of the dot product based on the first output.

[0110] The present disclosure may also be situated in a computer program product that includes all features enabling the implementation of the methods described herein and that is capable of executing these methods when loaded into a computer system. A computer program in this context means any expression, in any language, code or notation, of a set of instructions intended to cause a system capable of processing information to execute a particular function, either directly, or after a) conversion into another language, code or notation, b) reproduction in a different content form, or both.

[0111] While the present disclosure has been described with reference to several embodiments, those skilled in the art will recognize that various modifications can be made and equivalents substituted without departing from the scope of the present disclosure. Many modifications can also be made to adapt a particular situation or material to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, it is not intended that the present disclosure be limited to the particular embodiments disclosed, but rather, it is intended to include all embodiments falling within the scope of the appended claims. [Explanation of symbols]

[0112] 110 First Machine Learning Model 112 Second Machine Learning Model 402 First Data Set 404 Data Subset 800 Scenarios 802 stack layers 804 Convolutional Neural Network Layers 806 fully connected layer 808 Output layer 810 Updated 1st ML Model 812A 1st line 812B 2nd line 812C 3rd line

Claims

1. 1. An electronic device comprising: receiving a data subset of the first data set associated with the user; A first machine learning model is trained based on the first dataset associated with the user; training a second machine learning model based on the received data subset; applying a transformation function to the trained first machine learning model based on the trained second machine learning model; updating the trained first machine learning model based on application of the transformation function to the trained first machine learning model. A circuit configured as follows: the update of the trained first machine learning model corresponds to unlearning at least one of the received data subset or a feature set associated with the second machine learning model.

1. An electronic device comprising:

2. the circuitry is further configured to receive a first user input indicating a time period associated with the data subset, the data subset being received based on the received first user input.

2. The electronic device of claim 1.

3. the trained first machine learning model corresponds to a recommendation model; the updated first machine learning model is configured to output personalized recommendations based on the received first user input.

3. The electronic device of claim 2.

4. The circuit comprises: comparing an output of the trained first machine learning model to a threshold; determining that the output of the trained first machine learning model is incomplete based on a comparison of the output of the trained first machine learning model to the threshold; sending a notification based on said determining that said output is incomplete; receiving a second user input based on the transmitted notification indicating the incomplete output; The method further comprises: the data subset is received based on the second user input.

2. The electronic device of claim 1.

5. The circuit comprises: Extracting a first set of labels associated with the first dataset; Extracting a second set of labels associated with the received data subset; removing the extracted second set of labels associated with the received data subset from the extracted first set of labels associated with the first data set; determining a third set of labels based on removing the extracted second set of labels associated with the data subset from the extracted first set of labels associated with the first data set; The electronic device of claim 1 , further configured to:

6. The circuit comprises: determining whether each label in the determined third label set corresponds to a category label; determining a total number of the determined third label sets based on the determination that each of the determined third label sets corresponds to the category label; determining a fourth label based on the determined total number of the determined third label set; The method further comprises: the fourth label corresponds to a maximum number in the determined third label set; the second machine learning model is further trained based on the determined fourth label.

6. The electronic device of claim 5.

7. The circuit comprises: determining whether each label in the determined third label set corresponds to a numeric label; determining an average of the determined third set of labels based on the determination that each of the determined third set of labels corresponds to the numerical label; determining a fifth label based on the determined average of the determined third set of labels; The method further comprises: the second machine learning model is further trained based on the determined fifth label.

6. The electronic device of claim 5.

8. The circuit comprises: further configured to build a stack layer associated with the transformation function; The stack layer is configured to stack the trained first machine learning model and the trained second machine learning model to update the trained first machine learning model.

2. The electronic device of claim 1.

9. The transformation function includes the constructed stack layer and a series of deep neural network (DNN) layers.

9. The electronic device of claim 8.

10. the transformation function corresponds to a dot product of a first output of the trained first machine learning model and a second output of the trained second machine learning model.

9. The electronic device of claim 8.

11. The transformation function further corresponds to a normalization of the dot product based on the first output.

12. In an electronic device, receiving a data subset of a first data set associated with a user, A first machine learning model is trained based on the first dataset associated with the user; and training a second machine learning model based on the received data subset; applying a transformation function to the trained first machine learning model based on the trained second machine learning model; updating the trained first machine learning model based on application of the transformation function to the trained first machine learning model; Including, the update of the trained first machine learning model corresponds to unlearning at least one of the received data subset or a feature set associated with the second machine learning model. A method comprising:

13. and receiving a first user input indicating a time period associated with the data subset, the data subset being received based on the received first user input. The method of claim 12.

14. the trained first machine learning model corresponds to a recommendation model; the updated first machine learning model is configured to output personalized recommendations based on the received first user input. The method of claim 13.

15. extracting a first set of labels associated with the first dataset; extracting a second set of labels associated with the received data subset; removing the extracted second set of labels associated with the received data subset from the extracted first set of labels associated with the first data set; determining a third label set based on removing the extracted second label set associated with the data subset from the extracted first label set associated with the first data set; The method of claim 12 further comprising:

16. determining whether each label in the determined third label set corresponds to a category label; determining a total number of the determined third label sets based on the determination that each of the determined third label sets corresponds to the category label; and determining a fourth label based on the determined total number of the determined third label set; and Further comprising: the fourth label corresponds to a maximum number in the determined third label set; the second machine learning model is further trained based on the determined fourth label. The method of claim 15.

17. determining whether each label in the determined third label set corresponds to a numerical label; determining an average of the determined third set of labels based on the determination that each of the determined third set of labels corresponds to the numerical label; and determining a fifth label based on the determined average of the determined third set of labels; and Further comprising: the second machine learning model is further trained based on the determined fifth label. The method of claim 15.

18. building a stack layer associated with the transformation function; The stack layer is configured to stack the trained first machine learning model and the trained second machine learning model to update the trained first machine learning model. The method of claim 12.

19. the transformation function corresponds to a dot product of a first output of the trained first machine learning model and a second output of the trained second machine learning model.

20. The method of claim 18.

20. A non-transitory computer-readable medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by an electronic device, receiving a data subset of a first data set associated with a user, A first machine learning model is trained based on the first dataset associated with the user; and training a second machine learning model based on the received data subset; applying a transformation function to the trained first machine learning model based on the trained second machine learning model; updating the trained first machine learning model based on application of the transformation function to the trained first machine learning model; causing the electronic device to perform operations including the update of the trained first machine learning model corresponds to unlearning at least one of the received data subset or a feature set associated with the second machine learning model. A non-transitory computer-readable medium comprising:

Images