Electronic device for applying a personalized artificial intelligence model to another model

Abstract

An electronic device, comprising: at least one memory configured to store at least one instruction and a first neural network model; a communicator comprising communication circuitry; and at least one processor configured to execute the at least one instruction to: receive, from an external electronic device through the communicator, information about a second neural network model stored in the external electronic device; compare the first neural network model with the second neural network model based on the information about the second neural network model; and based on the comparison between the second neural network model and the first neural network model, control the communicator to transmit, to the external electronic device, information about weights between nodes of the first neural network model.

Description

Technical Field

[0001] This disclosure relates to an electronic device for applying a personalized or customized neural network model to another neural network model. More specifically, this disclosure relates to an electronic device for efficiently reflecting the training results of a personalized or customized neural network model onto another electronic device or another neural network model. Background Technology

[0002] Artificial neural network models can be designed and trained to perform a wide variety of functions, and their applications include image processing, speech recognition, inference / prediction, knowledge representation, and motion control.

[0003] Neural network models can be stored on a user's electronic device and can be trained and personalized based on the user's individual input or personal information on the electronic device.

[0004] Servers (such as application providers) can improve the performance of previously distributed older versions of neural network models, thereby distributing updated versions of neural network models or more advanced types of deep neural network models.

[0005] If a new version of the neural network model is received, the personalized artificial intelligence (AI) model will no longer be used by using the old version of the neural network model in the device. Therefore, there is an inconvenience in training the new version of the neural network model for personalization again.

[0006] If you purchase a new device, you need to bring and use the personalized AI model that was already present on your previous device. Summary of the Invention

[0007] Technical issues

[0008] Embodiments of this disclosure provide one or more electronic devices for effectively applying a personalized neural network model to another non-personalized neural network model.

[0009] Furthermore, embodiments of this disclosure provide an electronic device that, upon receiving a new neural network model for replacing a personalized neural network model, effectively applies the personalized neural network model to the new neural network model.

[0010] Solution to the problem

[0011] According to one aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one memory configured to store at least one instruction and a first neural network model; a communicator including communication circuitry; and at least one processor configured to execute at least one instruction to: receive information about a second neural network model stored in the external electronic device via the communicator; compare the first neural network model with the second neural network model based on the information about the second neural network model; and control the communicator to transmit information about the weights between nodes of the first neural network model to the external electronic device based on the comparison between the second neural network model and the first neural network model.

[0012] The at least one processor may also be configured to execute at least one instruction to: identify, based on received information about the second neural network model, whether the version of the first neural network model is the same as the version of the second neural network model; and, based on the fact that the version of the first neural network model is the same as the version of the second neural network model, control the communicator to send information about the weights between the nodes of the first neural network model to an external electronic device.

[0013] The at least one processor may also be configured to execute at least one instruction to: customize the first neural network model in the electronic device based on the nodes being added to the first neural network model, and control the communicator to send information about the added nodes and information about the weights between the nodes of the first neural network model to an external electronic device.

[0014] The at least one processor may also be configured to execute at least one instruction to: control the communicator to send one of the first neural network model and training data for customizing the first neural network model, based on the fact that the version of the first neural network model is different from the version of the second neural network model.

[0015] The at least one processor may also be configured to execute at least one instruction to: identify whether training data used to train the first neural network model is stored in at least one memory, based on the fact that the version of the first neural network model is different from the version of the second neural network model; and control a communicator to send the training data to an external electronic device based on the fact that the training data is stored in at least one memory.

[0016] The at least one processor may also be configured to execute at least one instruction to: control the communicator to send the first neural network model to an external electronic device, based on the fact that the training data is not stored in at least one memory.

[0017] According to one aspect of another embodiment of the present disclosure, an electronic device is provided, comprising: at least one memory configured to store at least one instruction and a second neural network model; a communicator including circuitry; and at least one processor configured to execute at least one instruction to: receive information about a first neural network model stored in an external electronic device via the communicator; identify a difference between the first neural network model and the second neural network model based on the received information about the first neural network model; and update the second neural network model based on the difference between the first neural network model and the second neural network model.

[0018] The at least one processor may also be configured to execute at least one instruction to update a second neural network model based on the weights between nodes, based on information received from an external electronic device via a communicator regarding the weights between nodes of the first neural network model.

[0019] The at least one processor may also be configured to execute at least one instruction to update a second neural network model based on information received from an external device via a communicator regarding nodes added to the first neural network model and information regarding the weights between nodes in the first neural network model.

[0020] The at least one processor may also be configured to execute at least one instruction to: train a second neural network model based on training data received from an external device via a communicator for customizing a first neural network model.

[0021] The at least one processor may also be configured to execute at least one instruction to: generate training data using the first neural network model received via a communicator; and train a second neural network model based on the generated training data.

[0022] The at least one processor may also be configured to execute at least one instruction to: obtain one or more input values ​​that give the output data of the first neural network model a specific output value based on a first neural network model received via a communicator, and train a second neural network model based on the output value and one or more input values.

[0023] The at least one processor may also be configured to execute at least one instruction to: obtain first output data by inputting first input data into a first neural network model, and obtain second output data by inputting the first input data into a second neural network model; store the first input data and the first output data in at least one memory based on the fact that the first output data is different from the second output data; and train the second neural network model based on the first input data and the first output data stored in the at least one memory.

[0024] The at least one processor may also be configured to execute at least one instruction to train a second neural network model based on generated training data when no input data for using the second neural network model is input.

[0025] According to one aspect of another embodiment of the present disclosure, an electronic device is provided, comprising: at least one memory configured to store at least one instruction and a first neural network model; a communicator including communication circuitry; and at least one processor configured to execute at least one instruction to: receive a second neural network model from a server device via the communicator, the second neural network model being an updated version of the first neural network model, based on a user command received to download a second neural network model; and train the received second neural network model based on information about the first neural network model and information about the second neural network model.

[0026] The at least one processor may also be configured to execute at least one instruction to: store training data based on the first neural network model in at least one memory, and train the received second neural network model based on the stored training data.

[0027] The at least one processor may also be configured to execute at least one instruction to: generate training data based on the fact that training data for the first neural network model is not stored in the at least one memory, use the first neural network model to generate training data, and train the received second neural network model based on the generated training data.

[0028] The at least one processor may also be configured to execute at least one instruction to: obtain one or more input values ​​that cause the first neural network model to output a specific output value, and train the received second neural network model based on the one or more input values.

[0029] The at least one processor may also be configured to execute at least one instruction to: obtain first output data by inputting first input data into the first neural network model, and obtain second output data by inputting the first input data into the second neural network model; store the first input data and the first output data in at least one memory based on the difference between the first output data and the second output data; and train the second neural network model based on the first input data and the first output data stored in the at least one memory.

[0030] The at least one processor is also configured to execute at least one instruction to train a second neural network model based on generated training data when no input data for using the second neural network model is input.

[0031] Beneficial effects of the present invention

[0032] According to this disclosure, an electronic device including a personalized model can have the effect that the personalized model can transfer (apply) an experience gained through training or updating on the electronic device to another model or another electronic device included in the electronic device.

[0033] According to this disclosure, an electronic device including a personalized model can have the effect of minimizing the amount of data transfer between devices used to apply the personalized model to the external electronic device by selectively sending information about the personalized model, at least some information about the personalized model, training data for personalization, etc., to the external electronic device based on the difference between the model stored in the external electronic device and the personalized model.

[0034] An electronic device according to this disclosure, including an unpersonalized neural network model, can have the effect of updating a pre-stored model when training data is received from an external electronic device and when at least a portion of a personalized model is received from an external electronic device.

[0035] According to this disclosure, an electronic device that applies a pre-stored personalized model to a newly received model can obtain or generate training data based on the personalized model and use the training data to effectively personalize the received model. Attached Figure Description

[0036] The above and other aspects, features, and advantages of certain embodiments of the present disclosure will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0037] Figure 1 This is a diagram that briefly illustrates the process of applying a personalized neural network model (personalized model) stored in an electronic device to other electronic devices according to an embodiment;

[0038] Figure 2A This is a diagram illustrating an embodiment of a device that applies a personalized model when the version of the model stored in the electronic device is the same as the version of the model stored in other electronic devices.

[0039] Figure 2B This is a diagram illustrating an embodiment of a device that applies a personalized model when the version of the model stored in the electronic device is the same as the version of the model stored in other electronic devices.

[0040] Figure 2C This is a diagram illustrating an embodiment of a device associated with updating a neural network model based on node connection data that indicates how nodes in another neural network model are connected to each other;

[0041] Figure 3 This is a diagram illustrating an embodiment of an apparatus for applying a personalized model when pre-stored training data is available;

[0042] Figure 4 This is a diagram illustrating an embodiment of an apparatus for applying a personalized model when there is no pre-stored training data;

[0043] Figure 5 This is a diagram illustrating an embodiment of an apparatus for applying a personalized model when there is no pre-stored training data;

[0044] Figure 6 This is a sequence diagram illustrating an example of the device's operation before the application of the personalized model;

[0045] Figure 7A It is shown in Figure 6 During the process, the user interface (UI) provided by the device will offer the user sample illustrations.

[0046] Figure 7B It is shown in Figure 6 During the process, the user interface (UI) provided by the device will offer the user sample illustrations.

[0047] Figure 7C It is shown in Figure 6 During the process, the user interface (UI) provided by the device will offer the user sample illustrations.

[0048] Figure 7D It is shown in Figure 6 During the process, the user interface (UI) provided by the device will offer the user sample illustrations.

[0049] Figure 8This is a sequence diagram illustrating another example of the device's operation before the application of the personalized model;

[0050] Figure 9A This is a block diagram illustrating the configuration of an electronic device according to an embodiment for applying a personalized neural network model to an external electronic device;

[0051] Figure 9B It shows Figure 9A A block diagram detailing the configuration of the electronic device;

[0052] Figure 10A It is a block diagram illustrating the configuration in which a personalized neural network model from an external electronic device is applied to an electronic device;

[0053] Figure 10B It shows Figure 10A A block diagram illustrating a more detailed configuration of the electronic device;

[0054] Figure 11 This is a diagram that briefly illustrates an example of an electronic device, according to an embodiment, for applying a personalized model to a newly received model;

[0055] Figure 12 It is a sequence diagram illustrating the process by which an electronic device receives an updated model and trains the model according to an embodiment;

[0056] Figure 13A This is a block diagram illustrating the configuration of an electronic device that applies a personalized model to a newly received model according to an embodiment;

[0057] Figure 13B It shows Figure 13A A block diagram illustrating a detailed configuration of an electronic device;

[0058] Figure 14 This is a flowchart illustrating a method for applying a personalized model of an electronic device based on the transmitting and receiving devices, respectively;

[0059] Figure 15 This is a flowchart illustrating a method for applying a personalized model of an electronic device based on the transmitting and receiving devices, respectively; and

[0060] Figure 16 This is a flowchart illustrating a method for applying a personalized model according to an embodiment. Detailed Implementation

[0061] Before describing this disclosure in detail, an overview for understanding this disclosure and the accompanying drawings will be provided.

[0062] The terminology used in this disclosure and claims are general terms identified in consideration of the functionality of the various exemplary embodiments of this disclosure. However, these terms may vary depending on the intent of those skilled in the art, legal or technical interpretations, the emergence of new technologies, etc. Furthermore, some arbitrary terms may be used. Unless a specific definition of a term exists, it may be understood based on the overall content and the technical common sense of those skilled in the art.

[0063] Furthermore, throughout this disclosure, the same reference numerals indicate the same components that perform substantially the same function. For ease of description and understanding, the same reference numerals or symbols are used and described in different exemplary embodiments. In other words, although all elements with the same reference numerals are shown in multiple figures, the multiple figures do not refer to a single embodiment.

[0064] Terms such as "first" and "second" can be used to describe various elements, but these elements should not be limited by these terms. These terms are used for the purpose of distinguishing one element from another. For example, elements associated with ordinal numbers should not be limited by sequence or the order in which numbers are used. Ordinal numbers can be interchanged if necessary.

[0065] Unless otherwise stated, singular expressions include plural expressions. It should be understood that terms such as "comprising" or "consisting of" can be used, for example, to specify the presence of a characteristic, quantity, step, operation, element, component, or combination thereof, and do not preclude the presence or possibility of adding one or more of other characteristics, quantities, steps, operations, elements, components, or combinations thereof.

[0066] Terms such as “module,” “unit,” and “component” can refer to an element that performs, for example, at least one function or operation, and such an element can be implemented as hardware or software, or a combination of hardware and software. Furthermore, except where each of the multiple “modules,” “units,” “components,” etc., needs to be implemented in separate hardware, a component can be integrated in at least one module or chip and implemented in at least one processor.

[0067] When any component is connected to another component, this includes both direct connection and indirect connection via another medium. Furthermore, when a component includes an element, unless otherwise stated, additional elements may be included, rather than excluded.

[0068] When a statement such as "at least one of..." precedes a list of elements, it modifies the entire list of elements and does not modify any individual elements in the list. For example, the statement "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or any variation of the above examples.

[0069] In this disclosure, applying a personalized or customized neural network model to another neural network model means that the performance of training or updates performed during the final personalization process of the personalized model can also be reflected in the other model. Various schemes exist for applying personalized models, and these schemes will be described with reference to various embodiments.

[0070] Figure 1 This is a diagram that briefly illustrates the process of applying a personalized or customized neural network model (personalized model or customized model) stored in an electronic device to other electronic devices according to an embodiment.

[0071] A neural network model may include one or more nodes or one or more layers that define the relationship between input and output values. During the training of a neural network model, the relationships between nodes (e.g., weights) or the relationships between layers can change.

[0072] exist Figure 1 In the first electronic device 100, a personalized artificial intelligence (AI) model 11 is stored in the first electronic device 100, and a non-personalized AI model 12 is stored in the second electronic device 200.

[0073] Each of the first electronic device 100 and the second electronic device 200 can be the same type or different types of devices. Each of the first electronic device 100 and the second electronic device 200 can correspond to various user devices, such as smartphones, televisions (TVs), tablet PCs, desktop PCs, laptop PCs, etc., but is not limited thereto.

[0074] The first electronic device 100 and the second electronic device 200 can be devices belonging to the same user or devices belonging to different users.

[0075] The personalized model 11 can be a user-specific model based on user input to the first electronic device 100 or user personal information stored in the first electronic device 100, etc. Specifically, if the model 11 is a neural network model, the first electronic device 100 can personalize the model 11 by changing the weights between nodes of the model 11, the node configuration of the model 11, the activation function of each node, etc., based on user input or user personal information.

[0076] The first electronic device 100 and the second electronic device 200 can apply model 11 stored in the first electronic device 100 to model 12 stored in the second electronic device 200 based on user input received in at least one of the first electronic device 100 and the second electronic device 200. (Refer to...) Figure 1 This will be described.

[0077] refer to Figure 1 In operation S51, the first electronic device 100 can compare the structures of model 11 and model 12. If model 11 and model 12 are neural network models, then the structure can refer to, for example, a layer structure or internal node configuration, etc. Figure 1 Assume that Model 11 and Model 12 are neural network models.

[0078] The first electronic device 100 can receive information about model 12 from the second electronic device 200, and compare the structures of model 11 and model 12 based on the received information about model 12. The information about model 12 may include information about the layer structure, internal node configuration, etc. of model 12.

[0079] Information about Model 12 can include information about its version. If the versions of the models are consistent with each other, then the layer structure of the models can be the same.

[0080] The steps of S51 can be performed by the second electronic device 200 instead of the first electronic device 100. Figure 8 Further description of relevant embodiments follows.

[0081] In operation S52, if the structures of the two models 11 and 12 are identical, then in operation S53, the first electronic device 100 can send information about the weights of the personalized model 11. Alternatively, if the versions of the two models 11 and 12 are identical, then the first electronic device 100 can send information about the weights of the personalized model 11.

[0082] In operation S54, the second electronic device 200 can update model 12 by changing the inter-node weights of model 12 based on the received information about the weights. This corresponds to the first scheme of applying (or converting) the personalized model 11 to model 12. Reference will be made below. Figure 2A and Figure 2B The first option will be described in more detail.

[0083] In operation S52-No, if the structures of the two models 11 and 12 are inconsistent, then in operation S55, the first electronic device 100 can identify whether training data has been stored in the first electronic device 100. Here, the training data may be data used to train model 11 during a previous personalization process of personalized model 11. The training data can be defined according to the types of input and output data of neural network model 11 (and neural network model 12), and may include user input received at the first electronic device 100, user personal information, sensing data received from various sensors, etc.

[0084] In operation S55, if the training data is pre-stored, then in operation S56, the first electronic device 100 can send the training data to the second electronic device 200. As a result, in operation S57, the second electronic device 200 can train model 12 using the received training data. This is a second scheme for applying personalized model 11 to model 12. (See reference...) Figure 3 The second option will be described further.

[0085] In operation S55-No, when the training data is not pre-stored, in operation S58, the first electronic device 100 can send the personalized model 11 itself to the second electronic device 200.

[0086] In operation S59, the second electronic device 200 can use the received model 11 to generate training data in reverse. The second electronic device 200 can then use the generated training data to train model 12. This is a third approach to applying the personalized model 11 to model 12. (See reference...) Figure 4 The third option will be described further.

[0087] The second electronic device 200 can obtain output data from the received models 11 and 12 respectively by inputting input data to the received models 11 and 12 respectively. In operation S60, the second electronic device 200 can identify whether there is an output difference between the models 11 and 12.

[0088] If an output difference exists, the second electronic device 200 can identify the output data of model 11, which differs from the output data of model 12 relative to the same input data, as training data, and in operations S59 and S57, train model 12 based on the training data. This corresponds to the fourth scheme of applying personalized model 11 to model 12. Reference will be made below. Figure 5 The fourth option will be described in more detail.

[0089] It is not necessary to use only one of the four options. Specifically, the second option (S56-S57), the third option (S58-S59-S57), and the fourth option (S58-S60-S59-S57) are not contradictory to each other, therefore, unlike... Figure 1 At least two schemes can be executed together.

[0090] The operation of the first electronic device 100 and the second electronic device 200 will be further described with reference to the accompanying drawings.

[0091] exist Figure 2A , Figure 2B , Figure 3 , Figure 4 and Figure 5 In the first electronic device 100, the personalized model 115 is stored in the first electronic device 100 and the non-personalized model 215 is stored in the second electronic device 200.

[0092] Figure 2A and Figure 2B The first scheme for applying the personalized model described above is further illustrated, in which the model stored in one electronic device and the model stored in another electronic device are the same version, that is, it is assumed that the layer structure of model 115 and the layer structure of model 215 are consistent with each other.

[0093] refer to Figure 2A The first electronic device 100 can send information about the weights between nodes of the personalized model 115 to the second electronic device 200.

[0094] In this example, the first electronic device 100 can send all the information about the weights forming the model 115 or some of the information about the weights.

[0095] The first electronic device 100 can send only information about the weights in the model 115 that have changed during the personalization process to the second electronic device 200.

[0096] For example, if model 115 is configured with convolutional layers and one or more independent fully connected layers as a classifier, and only the weights of the fully connected layers are changed during the personalization process performed by the first electronic device 100, then the first electronic device 100 may send only information about the weights of the fully connected layers to the second electronic device 200.

[0097] Information about the weights of the fully connected layer that have changed during the personalization process can be sent to the second electronic device 200.

[0098] The electronic device can identify the difference between the weights of model 115 before the personalization process and the weights of model 115 after the personalization process, and send information about the identified differences to the second electronic device 200.

[0099] The second electronic device 200 can update the weights between nodes in model 215 based on the weight information received from the first electronic device 100.

[0100] Regarding the layer structure, the second electronic device 200 can change the weights at positions corresponding to the weights of model 115 among the weights between nodes of model 215 to adapt to the weights of model 115. Therefore, the second electronic device 200 can obtain a personalized model 215'.

[0101] For example, model 215 may include a first layer with nodes n11-n13, a second layer with nodes n21-n23, and a third layer with nodes n31-n32. Each of nodes n11-n13, n21-n23, and n31-n32 has its own activation functions f11-f13, f21-f23, and f31-f32. Node n21 in the second layer may be connected to nodes n11, n12, and n13 in the first layer. Node n21 may receive inputs X1, X2, and X3 from nodes n11, n12, and n13, and may use activation function f21 to obtain output Y1. Specifically, node n21 may apply weights w1, w2, and w3 to the values ​​X1, X2, and X3 provided by nodes n11, n12, and n13, respectively, to obtain output Y1. When a weight-based update is performed, the weights w1, w2, and w3 can be updated to weights w1', w2', and w3' as shown in model 215'. Furthermore, the bias b of the activation function f21 can be updated to b'. Here, the updated weights w1', w2', and w3' and the updated bias b' can have the same or different values ​​as the weights w1, w2, and w3 and the bias b.

[0102] Figure 2B Suppose that the node configuration included in model 115 has changed, and the weights between the nodes of model 115 have been changed during the personalization process. Specifically, during the personalization process, a new node 115-1 can be added to the output of model 115.

[0103] For example, model 115 could be a classifier that can identify only people and animals from an image before personalization, and during the personalization process, it could add categories for identifying plants other than people and animals based on user requests, feedback, etc. (which could be implemented as output nodes).

[0104] In this example, the first electronic device 100 can send information about the added node 115-1 and the weights between the nodes to the second electronic device 200. The information about the added node 115-1 may be about the position or order of the node 115-1 in the layer structure of model 115.

[0105] In this example, information about the weights between nodes could include information about the weights associated with the added node 115-1.

[0106] The second electronic device 200 can add node 215'-1 corresponding to node 115-1 to model 215 based on information received from the first electronic device 100 about the added node 115-1.

[0107] The second electronic device 200 can update all the weights of the model 215 with added nodes 215'-1 based on the received information about the weights, so as to obtain a personalized model 215'.

[0108] Furthermore, model 215 with a fully-connected neural network (FCNN) can be updated to model 215' with a partially-connected neural network (PCNN). PCNN may contain only a subset of the entire set of possible connections of the network model.

[0109] Figure 2C This is a diagram illustrating an embodiment of a device associated with updating a neural network model based on node connection data that indicates how nodes in another neural network model are connected to each other.

[0110] like Figure 2C As shown, the second electronic device 200 can receive node connection data from the first electronic device 100 and can update model 215 to model 215' based on the node connection data, such that the updated model 215' has a dense convolutional network in which each layer is connected to every other layer in a feedforward manner. Unlike the neural network of model 215, which has connections between each layer and its subsequent layers, each layer of the neural network of model 215' can use inputs from all previous layers. Specifically, for each layer, the feature maps of all previous layers of model 215' are used as inputs, and its own feature map is used as input to all subsequent layers.

[0111] Figure 3This is a diagram illustrating an embodiment of an apparatus for applying a personalized model when pre-stored training data is present. The presence of pre-stored training data can refer to, for example, the simple presence of pre-stored training data, although it is not much, or it can refer to, for example, the presence of an amount of pre-stored training data that contributes to the personalization of model 215 (i.e., above a predetermined threshold).

[0112] Figure 3 This is a more detailed illustration of the second approach described above for applying the personalized model. Figure 1 In this context, it is assumed that the versions of the models are different, but even if the versions of the models are consistent with each other, this method can still be used without using the first approach.

[0113] refer to Figure 3 The first electronic device 100 can store training data 116 used in the personalization process of model 115. In this example, the first electronic device 100 can send the training data 116 to the second electronic device 200.

[0114] If the training data 116 includes personal information about the user's identity or body, the first electronic device 100 may send the training data 116 to the second electronic device 200 only if it receives user input that accepts the transmission of personal information via user input or the like.

[0115] If training data 116 is sent, the second electronic device 200 can obtain a personalized model 215' by training the model 215 based on the received training data 116.

[0116] Figure 4 and Figure 5 This is a diagram illustrating an embodiment of an apparatus for applying a personalized model when there is no pre-stored training data.

[0117] refer to Figure 4 The first electronic device 100 can send the model 115 to the second electronic device 200.

[0118] In this example, the second electronic device 200 can generate training data from the model 115 in reverse. For example, the second electronic device 200 can obtain one or more input values ​​that cause the personalized model 115 to output a specific output value.

[0119] Based on the specific output value of the personalized model 115, an objective function can be obtained, and one or more input values ​​can be obtained based on the objective function.

[0120] A specific output value can be any of the output values ​​that can be included in the output data output by model 115.

[0121] An objective function can be defined as a function that finds input values ​​that cause model 115 to output a specific output value. The objective function can be a function that includes the output values ​​as input to model 115 and the specific output value. The objective function can indicate the relationship between the output values ​​and the results of inputting (arbitrary) input values ​​into the neural network model.

[0122] Predefined conditions may include conditions under which the value of the objective function is minimum or maximum. This can be used to obtain the input values ​​at points where the value of the objective function is minimum / maximum (this is the result of changing the input values ​​and applying these values ​​to the objective function in turn).

[0123] For example, the objective function could be "L(x) = |M(x) - y|", and the precondition could be that L(x) is the minimum value. In this example, x is the input value, M(x) is the output value obtained by inputting the input value into model 115, and y is a specific output value.

[0124] In this example, the second electronic device 200 can compare sequential values ​​(L(x1), L(x2), L(x3), ...) of the gradually changing input values ​​(x1, x2, x3, ...) to identify the input value where the value of the objective function becomes minimum. Alternatively, the second electronic device 200 can use a first-order or higher-order difference of the value of the objective function for the gradually changing input values ​​to obtain the input value where the value of the objective function (L(x)) becomes minimum.

[0125] The second electronic device 200 can obtain multiple input values ​​that satisfy predetermined conditions by changing the input values ​​until the objective function of a specific output value satisfies predetermined conditions for each of the multiple random input values.

[0126] For each of the multiple specific output values ​​of model 115, the second electronic device 200 can obtain multiple objective functions and multiple input values ​​that satisfy predetermined conditions for each of the multiple objective functions.

[0127] For each of a plurality of (e.g., N) specific output values, the second electronic device 200 can obtain a plurality of (e.g., M) input values. In this example, N*M input values ​​can be obtained.

[0128] The second electronic device 200 can generate training data that includes the specific output value and one or more obtained input values. The generated training data may include one or more training data pairs. Each of the one or more training data pairs can be configured with an obtained input value and a specific output value for the corresponding input value.

[0129] The second electronic device 200 can train the model 215 based on the generated training data.

[0130] exist Figure 5 In this process, the first electronic device 100 can send the model 115 itself to the second electronic device 200.

[0131] In this example, if model 215 is used during the operation of the second electronic device 200, that is, input data is input to model 215, the second electronic device 200 can also input input data to model 115.

[0132] The output data of model 115 and the output data of model 215 can be compared with each other.

[0133] If there is a difference between the output data of model 115 and the output data of model 215, the second electronic device 200 can store the output data and input data of model 115 with the difference.

[0134] Model 215 can be trained based on training data, including the stored input and output data. As a result, a personalized model 215' can be obtained.

[0135] If the stored input and training data exceed the predetermined amount, the stored input and training data can be used to train model 215. As mentioned above, by performing training after a sufficient amount of training data has been collected, the time efficiency and accuracy of the training can be ensured.

[0136] As a specific example, suppose the first electronic device 100 is an older version of the cleaning robot and the second electronic device 200 is a newer version of the cleaning robot, and each of models 115 and 215 is an image-based object recognition model. In this example, model 215 is a model with updated overall performance compared to model 115, but it is not personalized like model 115.

[0137] For example, it can be assumed that when model 115 is personalized via the first electronic device 100, new categories are added to model 115 (e.g., suppose the new type objects are "dog" and "cat"). In this example, the new categories (i.e., "dog" or "cat") cannot be classified (recognized) by the non-personalized model 215.

[0138] According to the fourth transmission scheme described above, the personalized model 115 can be transmitted from the first electronic device 100 to the second electronic device 200. The second electronic device 200 can input images that are captured in real time by a camera during driving and cleaning to the received model 115 and model 215.

[0139] If model 115 outputs a recognition result for "dog" or "cat" that model 215 cannot recognize, the second electronic device 200 can obtain input data (e.g., images) and output data (e.g., "dog" or "cat", or the probability value of "dog / cat" existence, etc.) as training data and store the training data in memory.

[0140] The second electronic device 200 can add nodes associated with new categories (“dog”, “cat”) identified by model 115 to model 215. Furthermore, if the stored training data is sufficiently accumulated (e.g., exceeding a predetermined amount of data) for the personalization of model 215, model 215 can be trained based on the stored training data.

[0141] The second electronic device 200 can train model 215 based on received / generated / stored training data, without inputting data for using model 215, according to the above embodiments. As a specific example, if the second electronic device 200 is a cleaning robot, model 215 can be trained in a rest mode when cleaning is not performed.

[0142] The second electronic device 200 can train the model 215 while the second electronic device 200 is being charged. In this example, insufficient computation and power can be prevented during training.

[0143] While training model 215 is being performed, the second electronic device 200 can display a graphical user interface (UI) indicating that "training (learning) is in progress." If the user begins to manipulate the second electronic device 200, training can be stopped while information about the most accurate weights at the corresponding time points is stored. In this example, if an application including model 215 is executed, the existing untrained model 215 can be used to perform the actions based on user input. If no further user manipulation is input (e.g., for a threshold time or longer), training can be resumed by reloading the information about the stored weights. In this example, the graphical user interface (UI) notifying that "training is in progress" can be displayed again.

[0144] If the training (personalization) of model 215 is terminated, the second electronic device 200 can notify the user that the personalization of model 215 based on model 115 is complete.

[0145] Figure 6 This is a sequence diagram illustrating an example of the device's operation before the application of a personalized model.

[0146] refer to Figure 6In operation S610, the first electronic device 100 can initiate a transmission mode. In this example, the first electronic device 100 can initiate the transmission mode based on user input received in the first electronic device 100. After executing the application for sending and receiving AI models based on user input, the first electronic device 100 can be designated as a transmission device based on user input, so that the operation mode of the first electronic device 100 can be switched to transmission mode.

[0147] In operation S620, the second electronic device 200 can activate a receiving mode. The second electronic device 200 can activate the receiving mode based on user input received in the second electronic device 200. In this example, after executing an application for sending and receiving AI models based on user input, the second electronic device 200 can be designated as a receiving device based on user input, allowing its operating mode to be switched to receiving mode.

[0148] In operation S630, the first electronic device 100 and the second electronic device 200 can connect to each other using each communicator. Electronic devices 100 and 200, each activating transmit and receive modes, can search for each other on wired and wireless communication networks based on user input.

[0149] According to an embodiment, the first electronic device 100 and the second electronic device 200 may additionally perform device authentication based on email or account (personal, family) to receive and send information related to the personalized AI model.

[0150] In operation S630, the first electronic device 100 and the second electronic device 200 can be connected directly via a wired connection or via a wireless communication method such as direct WiFi or Bluetooth.

[0151] When connected wirelessly, personalized models can be sent and received without going through a server by using communication methods such as direct WiFi or Bluetooth connections. As a result, situations such as leaking personalized models to external devices against the user's will can be prevented.

[0152] After the first electronic device 100 and the second electronic device 200 are connected to each other, in operation S640, the second electronic device 200 can send various information to the first electronic device 100. In this example, if the first electronic device 100 successfully authenticates the user on the second electronic device 200, the second electronic device 200 can send various information to the first electronic device 100.

[0153] The second electronic device 200 can send information about one or more models stored in the second electronic device 200, as well as information about the second electronic device 200, to the first electronic device 100.

[0154] Information about a model can include information about its functionality, its input and output data, its version, its layer structure, and its weights.

[0155] Information about the second electronic device 200 may include information about the memory capacity of the second electronic device 200, information about the remaining memory capacity of the second electronic device 200, information about the performance of the NPU or CPU of the second electronic device 200, information about the types of sensors included in the second electronic device 200, etc. If the personalized model being sent and received is an image-based object recognition model, the information about the second electronic device 200 may also include information about the performance of the camera provided in the second electronic device 200.

[0156] In operation S650, the first electronic device 100 can select a model to be sent from one or more models stored in the first electronic device 100. The model to be sent can be selected based on user input received in the first electronic device 100.

[0157] In operation S660, a transmission scheme for the selected model can be selected based on the selected model and various information received from the second electronic device 200, and in operation S670, transmission can be performed according to the selected transmission scheme. In this example, if the second electronic device 200 successfully authenticates the user on the first electronic device 100, the first electronic device 100 can perform transmission in step S670.

[0158] The first electronic device 100 may send information reflecting the personalized characteristics of the personalized model to the second electronic device 200 according to a selected transmission scheme. For example, the first electronic device 100 may send to the second electronic device 200 at least one of the following: information about the selected model (e.g., information about the weights), training data for personalization of the selected model, and the selected model itself.

[0159] For example, if a model belonging to the same application / function as the selected model is not stored in the second electronic device 200, the first electronic device 100 can send the selected model to the second electronic device 200.

[0160] If a model belonging to the same application / function as the selected model is stored in the second electronic device 200, and if the selected model is a personalized model on the first electronic device 100, the first electronic device 100 may send information about the weights of the personalized model, training data, and at least one of the personalized model to the second electronic device 200.

[0161] The first electronic device 100 may consider whether the version (or layer structure) of the selected model is consistent with the model stored in the second electronic device 200, and whether personalized training data for the selected model is stored in the first electronic device 100, etc. (References already provided) Figures 1 to 5 This was described.

[0162] Even if a model belonging to the same application / function as the selected model is stored in the second electronic device 200, if the selected model is a non-personalized model, the first electronic device 100 may send the selected model to the second electronic device 200 only if the selected model is a higher version than the model stored in the second electronic device 200.

[0163] The first electronic device 100 can select a transmission scheme based on user input. Specifically, the first electronic device 100 can provide the user with information about possible transmission schemes among the first to fourth schemes, and apply the model to the second electronic device 200 according to the transmission scheme selected by the user input.

[0164] According to the selected transmission scheme described above, the first electronic device 100 can transmit the weight of the selected model or at least a portion of the selected model to the second electronic device 200 and also transmit information about the selected model to the second electronic device 200.

[0165] In this example, information about the model may include information about the model's functionality, information about the model's input / output data, information about the model's version, information about the model's data size, information about the model's layer structure, information about the model's weights, information about the model's personalization state, information about the training data used for model personalization (e.g., whether the training data is stored in the first electronic device 100, the capacity of the training data, whether the training data includes personal information, etc.), and information about the categories added according to the model's personalization (e.g., when the model is an object classifier, the name / type of the identified object (category), its order in the output, etc., can be added by the model according to the personalization).

[0166] Figure 7A and Figure 7B It is shown in Figure 6 During the process, the user interface (UI) provided by the device will offer the user sample illustrations.

[0167] Figure 7A This diagram illustrates an example of how a transmit mode and a receive mode are activated by a first electronic device 100 and a second electronic device 200 respectively via user input. Figure 7A In this context, it is assumed that applications for sending and receiving AI models are executed in the first electronic device 100 and the second electronic device 200, respectively, and an execution screen is displayed.

[0168] refer to Figure 7A On the execution screen, the user can select "Send" 701 or "Receive" 702 by touching or other means.

[0169] refer to Figure 7A The first electronic device 100, which touches the "send" button 701, can activate the sending mode, such as... Figure 6 The operation S610, wherein the second electronic device 200, which touches the "receive" 702, can activate the receiving mode, such as Figure 6 Operation of S620.

[0170] Figure 7B It shows the relationship with Figure 6 The step S630 is related to an example view of the user interface (UI) provided by electronic device 100 and electronic device 200 for selecting a connection method.

[0171] refer to Figure 7B Each of the first electronic device 100 and the second electronic device 200 can display each item in order to select one of a “cable” 711 for transmitting and receiving AI models via wired means and a “wireless connection” 712 for transmitting and receiving AI models wirelessly.

[0172] If “Cable” 711 is selected, and the cable or the like connecting the first electronic device 100 and the second electronic device 200 is directly attached by the user, then the first electronic device 100 and the second electronic device 200 can be connected to each other.

[0173] If “Wireless Connection” 712 is selected, the first electronic device 100 and the second electronic device 200 can connect to each other by searching for each other via direct WiFi, Bluetooth communication, etc.

[0174] Figure 7C It shows about Figure 6 The S650 is a view of an example of a user interface (UI) provided by the first electronic device 100 for receiving a selection of a model to be sent to the second electronic device 200.

[0175] refer to Figure 7CThe first electronic device 100 may display a menu 721 representing models (or applications including models) stored in the first electronic device 100 (such as "DJ" 721-1, "Refrigerator Management" 721-2, "Camera Object Recognition" 721-3, "Voice Assistant" 721-4, etc.). If "Send" 722 is selected after at least one of the models is selected, the first electronic device 100 may send the selected model, information about the weights of the selected model, or personalized training data for the selected model to the second electronic device 200.

[0176] refer to Figure 7D If the selected model is a personalized model, the first electronic device 100 may request the user's confirmation before sending the personalized model.

[0177] refer to Figure 7D The first electronic device 100 can notify the selected model "DJ" that it is in a personalized state according to the user's (e.g., KIM) taste and display a UI 735 for selecting whether to send.

[0178] In this example, if the user selects "Yes" 735-1, the personalized model "DJ" by the first electronic device 100 can be applied to the second electronic device 200. That is, the first electronic device 100 can send at least a portion of the personalized model "DJ" or training data to the second electronic device 200.

[0179] In this example, the first electronic device 100 can use information received from the second electronic device 200 to identify a transmission scheme and select a transmission scheme based on user input.

[0180] If the user selects "No" 735-2, then "DJ", as a model in a non-personalized state, can be applied to the second electronic device 200.

[0181] Unlike Figure 6 The transmission scheme can be identified by the second electronic device 200.

[0182] Figure 8 This is a sequence diagram illustrating another example of the device's operation before the application of the personalized model.

[0183] refer to Figure 8 The first electronic device 100 and the second electronic device 200 can respectively activate the transmit mode and receive mode in operation S810 and S820, and can be connected to each other in operation S830, and... Figure 6 Unlike other devices, in operation S840, the first electronic device 100 can send various information to the second electronic device 200.

[0184] The first electronic device 100 can send information about one or more models stored in the first electronic device 100, as well as information about the first electronic device 100, to the second electronic device 200.

[0185] In this example, information about the model may include information about the model's functionality, information about the model's input / output data, information about the model's version, information about the model's data size, information about the model's layer structure, information about the model's weights, information about the model's personalization state, information about the training data used for model personalization (e.g., whether the training data is stored in the first electronic device 100, the capacity of the training data, whether the training data includes personal information, etc.), and information about the categories added according to the model's personalization (e.g., when the model is an object classifier, the name / type of the object (category) that can be identified by the model according to personalization, the order of the categories added from the categories that can be classified by the model, etc.).

[0186] Information about the first electronic device 100 may include information about the performance of the NPU or CPU of the first electronic device 100, information about the types of sensors included in the first electronic device 100, etc. If the personalized model being sent and received is an image-based object recognition model, the information about the first electronic device 100 may also include information about the performance of the camera provided in the first electronic device 100.

[0187] In operation S850, the second electronic device 200 can select a model (or application) to be received from the first electronic device 100 based on information about one or more models stored in the first electronic device 100. In this example, the second electronic device 200 can select the model to be received based on user input received in the second electronic device 200.

[0188] In operation S860, the second electronic device 200 can identify a transmission scheme based on information about the selected model, information about the model stored in the second electronic device 200, etc., for use with the model selected from the first electronic device 100. In operation S870, the second electronic device 200 can send information about the identified transmission scheme to the first electronic device 100.

[0189] As a result, in operation S880, the second electronic device 200 can receive information reflected by the personalized features of the personalized model from the first electronic device 100 based on the identified transmission scheme. That is, the second electronic device 200 can receive from the first electronic device 100 at least one of the following: information about the selected model (e.g., information about the weights), personalized training data for the selected model, and the selected model itself.

[0190] If a model belonging to the same application / function as the selected model is not stored in the second electronic device 200, the second electronic device 200 may send a request to send the selected model to the first electronic device 100.

[0191] If a model belonging to the same application / function as the selected model is stored in the second electronic device 200, and if the selected model is a personalized model on the first electronic device 100, the second electronic device 200 may send a request to the first electronic device 100 to send information about the weights of the personalized model, training data, and at least one of the personalized model.

[0192] The second electronic device 200 may consider whether the selected model matches the version (or layer structure) of the model stored in the second electronic device 200, and whether personalized training data for the selected model is stored in the first electronic device 100, etc. (Reference) Figures 1 to 5 This was described.

[0193] Even if a model belonging to the same application / function as the selected model is stored in the second electronic device 200, if the selected model is not a personalized model, the second electronic device 200 may send a request to the first electronic device 100 to send the selected model to the second electronic device 200 only if the selected model has a higher version than the model stored in the second electronic device 200.

[0194] The second electronic device 200 can identify the transmission scheme based on user input.

[0195] Figure 9A This is a block diagram illustrating the basic configuration of a first electronic device 100 according to the various embodiments described above. In the following, the operation of the first electronic device 100 will be described based on its components and according to the various embodiments described above.

[0196] The first electronic device 100 may include a memory 110, a communicator (e.g., a communication interface) 120, and a processor 130.

[0197] The memory 110 is configured to store various data related to the operating system (OS) and components of the first electronic device 100 for controlling the overall operation of the components of the first electronic device 100. The memory 110 may include at least one instruction associated with one or more components of the first electronic device 100.

[0198] The memory 110 can be implemented as a non-volatile memory (e.g., hard disk, solid-state drive (SSD), flash memory), volatile memory, etc.

[0199] The memory 110 can store the neural network model 115.

[0200] The neural network model 115 may include multiple neural network layers. Each layer has multiple weight values ​​and performs layer operations based on the results of computation by previous layers and operations on the multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), and deep Q-networks, and the neural networks in this disclosure are not limited to the examples described above unless otherwise indicated.

[0201] The neural network model 115 can be a personalized model based on user personal information or user input received in the first electronic device 100, which is trained and updated.

[0202] The communicator 120 is configured to perform communication with at least one external device to transmit and receive signals / data by the first electronic device 100. For this purpose, the communicator 120 may include circuitry.

[0203] The communicator 120 may include a wireless communication module, a wired communication module, etc.

[0204] The wireless communication module may include at least one of the following: a Wi-Fi communication module for receiving content from an external server or external device, a Direct Wi-Fi communication module, a Bluetooth module, an Infrared Data Association (IrDA) module, a third-generation (3G) mobile communication module, a fourth-generation (4G) mobile communication module, and a fourth-generation Long Term Evolution (LTE) communication module.

[0205] Wired communication modules can be implemented as wired ports, such as Thunderbolt ports, Universal Serial Bus (USB) ports, etc.

[0206] When the communicator 120 is connected to the second electronic device 200 via wireless communication, the communicator 120 can use a direct-connect WiFi communication module, Bluetooth module, infrared communication module, etc.

[0207] The processor 130 controls the overall operation of the first electronic device 100. The processor 130 can be connected to the memory 110 and the communicator 120 to control the first electronic device 100.

[0208] For this purpose, the processor 130 may include a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), etc., and may perform operations or data processing on control of other components included in the first electronic device 100.

[0209] The processor 130 may be implemented as a microprocessor unit (MPU), or may correspond to a computer in which random access memory (RAM), read-only memory (ROM), etc. are connected to the CPU via a system bus.

[0210] The processor 130 can control one or more software modules included in the first electronic device 100 and hardware components included in the first electronic device 100, and the results of the processor 130 controlling the software modules can be derived as operations of hardware configuration.

[0211] The processor 130 may be configured with one or more processors. The one or more processors may then be general-purpose processors (such as CPUs, APs, etc.), pure graphics processors (such as GPUs, VPUs), or pure AI processors (such as NPUs).

[0212] One or more processors control the processing of input data based on predefined operating rules or AI models stored in memory. The predefined operating rules or AI models are generated through learning.

[0213] In this paper, referentiality is generated through learning by applying a learning algorithm to multiple learning datasets to produce predetermined operational rules or AI models with desired characteristics. Learning can be performed within the device itself that performs the AI ​​according to this disclosure, or it can be implemented via a separate server / system.

[0214] A learning algorithm is a method of training a predetermined target device (e.g., a robot) using multiple training data sets so that the predetermined target device can make determinations or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and unless otherwise specified, the learning algorithms in this disclosure are not limited to the examples above.

[0215] The processor 130 can control the operation of the first electronic device 100 described in the various embodiments above.

[0216] The processor 130 can receive information about a neural network model stored in the second electronic device 200 via the communicator 120. The processor 130 can control the communicator 120 to send information about the weights between nodes of the personalized neural network model 115 to the second electronic device 200 based on the received information about the neural network model and information about the personalized neural network model 115 stored in the memory 110.

[0217] The processor 130 can identify, based on received information about the neural network model, whether the version of the neural network model 115 is the same as the version of the neural network model stored in the second electronic device 200. When the version of the neural network model 115 is the same as the version of the neural network model stored in the second electronic device 200, the processor 130 can control the communicator 120 to send information about the weights between the nodes of the personalized neural network model 115 to the second electronic device 200.

[0218] If the node added to the first electronic device 100 is included in the personalized neural network model 115, the processor 130 can control the communicator 120 to send information about the added node and the weights between the nodes in the personalized neural network model 115 to the second electronic device 200.

[0219] If the version of neural network model 115 is different from the version of neural network model stored in the second electronic device 200, the processor 130 can control the communicator 120 to send one of the neural network model 115 and the training data used to personalize the neural network model 115 to the second electronic device 200.

[0220] In this example, if the version of neural network model 115 is different from the version of neural network model stored in the second electronic device 200, the processor 130 can identify whether the training data used to train neural network model 115 is stored in memory 110.

[0221] If the training data is stored in memory 110, the processor 130 can control the communicator 120 to send the training data to the second electronic device 200. If the training data is not stored in memory 110, the processor 130 can control the communicator 120 to send the personalized neural network model 115 to the second electronic device 200.

[0222] Figure 9B It shows Figure 9A A block diagram showing the detailed configuration of the electronic device.

[0223] refer to Figure 9B In addition to the memory 110, the communicator 120 and the processor 130, the first electronic device 100 may also include a user input device (e.g., a user input interface) 140 and an output device (e.g., an output interface) 150.

[0224] refer to Figure 9B In addition to the neural network model 115, the memory 110 may also include training data 116. The training data 116 is the data used to train the neural network model 115 during the personalization process of the neural network model 115 on the first electronic device 100.

[0225] Training data 116 can be defined according to the type of input / output data of neural network model 115, and can be generated based on user input received via user input device 140 or sensing data received by various sensors in the first electronic device 100.

[0226] refer to Figure 9B The processor 130 can control the personalization module 131 and the personalization conversion module 132 stored in the memory 110. Each of the current modules can, as Figure 9B It is stored in memory 110 as software and can be selectively executed by processor 130, or it can be implemented as hardware, different from FIG. 9, and controlled by processor 130. Alternatively, each of the current modules can be in the form of a combination of software and hardware.

[0227] Personalization module 131 is a module used to personalize neural network model 115. Personalization module 131 can be trained or updated based on user input for the first electronic device 100 or different input data.

[0228] The personalization conversion module 132 is a module for sending at least a portion of the personalized neural network model 115 or training data 116 to an external electronic device (e.g., a second electronic device 200).

[0229] The personalization conversion module 132 can send at least a portion of the neural network model 115 and / or training data 116 to the external electronic device based on the layer structure or version differences between the neural network model 115 and the neural network model stored in the external electronic device.

[0230] User input device 140 is configured to receive various user inputs (such as user commands, (user) input data, etc.).

[0231] Based on user commands received via user input device 140, processor 130 can operate neural network model 115. In this example, processor 130 can input input data received via user input device 140 into neural network model 115 and obtain the output of neural network model 115.

[0232] User input device 140 may include one or more buttons, keyboards, mice, etc. User input device 140 may also include a touch panel implemented with a display (not shown) or a separate touchpad (not shown).

[0233] User input device 140 may include a microphone for receiving user commands or input data as speech, or may include a camera for receiving user commands or input data as images or motion.

[0234] The output device 150 is configured to provide various information included in the first electronic device 100 to the user.

[0235] The processor 130 can provide the user with output data from the neural network model 115 and information related to the neural network model 115 through the output device 150.

[0236] The processor 130 can also control the output device 150 to provide a visual / auditory UI to receive user commands related to operations such as the use of the neural network model 115 and the application of the neural network model 115. Figures 7A to 7D Corresponding to its example.

[0237] Output device 150 may include an audio output device, a display, etc. The audio output device may be implemented as a speaker and / or an audio / headphone terminal, etc.

[0238] The first electronic device 100 may include various types of sensors to receive various information about the user's living environment. Using the sensing data obtained through the sensors, the processor 130 can personalize the neural network model 115 and use the personalized neural network model 115 to provide various services.

[0239] Figure 10A This is a block diagram illustrating a configuration in which a personalized neural network model from an external electronic device is applied to an electronic device. In the following, based on the components of the second electronic device 200, the operation of the second electronic device 200 according to the various embodiments described above will be described.

[0240] refer to Figure 10A The second electronic device 200 may include a memory 210, a communicator (e.g., a communication interface) 220, and a processor 230. The memory 210 may store a neural network model 215.

[0241] The processor 230 can be connected to the memory 210 and the communicator 220, and control the second electronic device 200.

[0242] The processor 230 can receive information about the personalized neural network model 115 stored in the first electronic device 100 via the communicator 220. The processor 230 can update the neural network model 215 based on the differences between the personalized neural network model 115 and the neural network model 215.

[0243] If information about the weights between nodes is received from the first electronic device 100 via the communicator 220, the processor 230 can update the neural network model 215 based on the received weights between nodes.

[0244] If the processor 230 receives information about the nodes added to the personalized neural network model 115 and the weights between the nodes in the personalized neural network model 115 via the communicator 220, the processor 230 can update the neural network model 215 based on the received information about the nodes and the weights between the nodes.

[0245] If the processor 230 receives training data 116 for personalizing the personalized neural network model 115 via the communicator 220, it can train the neural network model 215 based on the received training data.

[0246] If the personalized neural network model 115 is received via communicator 220, processor 230 can use the personalized neural network model 115 to generate training data and train neural network model 215 based on the generated training data. Processor 230 can train neural network model 215 based on the generated training data without input data for using neural network model 215.

[0247] The processor 230 can obtain one or more input values ​​that cause the neural network model 115 to output a specific output value, and train the neural network model 215 based on the obtained one or more input values.

[0248] When the processor 230 receives the personalized neural network model 115 via the communicator 220, it can input the first input data into the neural network model 115 to obtain the first output data, and input the first input data into the neural network model 215 to obtain the second output data.

[0249] If the first output data is different from the second output data, the first input data and the second output data can be stored in the memory 210, and the neural network model 215 can be trained based on the first input data and the first output data stored in the memory 210.

[0250] Figure 10B It shows Figure 10A A block diagram illustrating a more detailed configuration of the electronic device. (See reference...) Figure 10B In addition to the memory 210, the communicator 220 and the processor 230, the second electronic device 200 may also include a user input device (e.g., a user input interface) 240, an output device (e.g., an output interface) 250, etc.

[0251] refer to Figure 10B The processor 230 can control the update module 231, training module 232, training data generation module 233, personalization module 234, etc., stored in the memory 210. Each of these modules can be stored in the memory 210 in software format and can be selectively executed by the processor 230, or executed in conjunction with... Figure 10B Different hardware formats are implemented and controlled by processor 230. Alternatively, each of the current modules can be a combination of software and hardware.

[0252] The update module 231 is a module that updates the neural network model 215 based on information received from an external electronic device regarding weights or information regarding added nodes.

[0253] Training module 231 is a module that trains neural network model 215 based on training data received from external electronic devices or training data generated in the second electronic device 200.

[0254] The training data generation module 233 is a module used to train the neural network model 215. Specifically, the training data generation module 233 can generate training data that includes input data and output data from a model (e.g., a personalized neural network model 115) received from an external electronic device. Alternatively, training data can be generated based on the output differences between the received model 115 and the neural network model 215 for the same input data.

[0255] The personalization module 234 is a module used to personalize the neural network model 215. Specifically, the personalization module 234 can be updated or trained based on user input or various input data for the first electronic device 100.

[0256] refer to Figure 9A , Figure 9B , Figure 10A , Figure 10B The above configuration is based on the assumption that the first electronic device 100 is a device for applying a personalized neural network model to another device and the second electronic device 200 is a device for applying a personalized neural network model stored in another device.

[0257] The electronic device according to this disclosure can apply a personalized model stored in the electronic device to another device, or a personalized model stored in another electronic device can be applied. Therefore, the configuration of each of the first electronic device 100 and the second electronic device 200 can be entirely included in a single electronic device. In this example, memory 110 and memory 210 do not need to be implemented as separate memories, but can be implemented as a single memory. This also applies to other configurations.

[0258] The embodiments described above pertain to applying a personalized model stored in one electronic device to a model stored in another electronic device, but it is also possible to apply a personalized model to another model within one electronic device. (Refer to...) Figure 11 , Figure 12 , Figure 13A , Figure 13B This will be described in detail.

[0259] Figure 11 This is a diagram that briefly illustrates an example of an electronic device, according to an embodiment, for applying a personalized model to a newly received model.

[0260] refer to Figure 11 While the personalized model 13 is stored in the electronic device 300, a new model 14 can be received from the outside during operation S1101.

[0261] In this example, during operation S1102, the electronic device 300 can identify whether the training data used in the personalization of the personalization model 13 is stored in the electronic device 300.

[0262] If training data is stored, in operation S1103, electronic device 300 can use the training data to train a new model 14.

[0263] If no training data is stored, in operation S1104, the electronic device 300 can use the personalized model 13 to generate training data.

[0264] Personalized model 13 can obtain one or more input values ​​that cause personalized model 13 to output a specific output value, and generate training data including training data pairs of each of the obtained input values ​​configured to be obtained.

[0265] The electronic device 300 can use the generated training data to train a new model 14.

[0266] If no training data is stored, in operation S1105, the electronic device 300 can identify the output difference between the personalized model 13 and the new model 14, and in operation S1104, generate training data based on the difference.

[0267] In this example, electronic device 300 can use the generated training data to train a new model.

[0268] Operation of electronic device 300 Figure 11 The embodiments correspond to Figure 1 The fourth scheme described herein is consistent with the operation of the second electronic device 200 after receiving training data or a personalized model.

[0269] Figure 12 This is a sequence diagram illustrating the process by which an electronic device receives an updated model and trains the model according to an embodiment.

[0270] refer to Figure 12 The electronic device 300 can connect to the server device 400 via various communication methods. If an updated model exists compared to the personalized model stored in the electronic device 300, the server device 400 can send a notification to the electronic device 300 that an updated model exists during operation S1210.

[0271] In this example, electronic device 300 can notify the user that an updated model exists. Therefore, if a user command to download the updated model is received in operation S1220, electronic device 300 can request the updated model from server device 400 in operation S1230.

[0272] Therefore, in operation S1240, electronic device 300 can receive the updated model from server device 400.

[0273] In operation S1250, electronic device 300 can, as Figure 11 The received model is trained in the same way.

[0274] If training (personalization) on the updated model is terminated, the electronic device 300 can delete the pre-stored personalized model from the memory.

[0275] Figure 13A This is a block diagram illustrating the configuration of an electronic device 300 that applies a personalized model to a newly received model according to an embodiment. In the following, it will be based on... Figure 13A Further description of the configuration Figure 11 and Figure 12Examples of implementations.

[0276] refer to Figure 13A The electronic device 300 may include a memory 310, a communicator 320, a processor 330, etc. The electronic device 300 may correspond to various user devices, such as smartphones, TVs, tablet PCs, desktop PCs, laptop PCs, etc., but is not limited to these.

[0277] The personalized neural network model 315 in the electronic device 300 can be stored in the memory 310.

[0278] When a user command is received to download a neural network model (which is an updated version of neural network model 315), processor 330 can receive the updated neural network model from server device 400 via communicator 320.

[0279] The processor 330 can train (personalize) the updated neural network model based on information about the neural network model 315 and information about the updated neural network model.

[0280] If the training data of the personalized neural network model 315 is stored in the memory 310, the processor 330 can train the updated neural network model based on the stored training data.

[0281] If the training data for the personalized neural network model 315 is not stored in the memory 310, the processor 330 can use the personalized neural network model 315 to generate training data. The processor 330 can then train the updated neural network model based on the generated training data. In this example, the processor 330 can train the updated neural network model based on the generated training data without requiring input data for using the updated neural network model.

[0282] In this example, processor 330 can obtain one or more input values ​​that cause neural network model 315 to output a specific output value, and train an updated neural network model based on the corresponding output value and the obtained one or more input values.

[0283] Alternatively, processor 330 can input the first input data into neural network model 315 to obtain first output data, input the first input data into an updated neural network model to obtain second output data, and if the first output data differs from the second output data, processor 330 can store the first input data and the first output data in memory 310. Processor 330 can train the updated neural network model based on training data including the first input data and the first output data stored in memory 310.

[0284] Figure 13B It shows Figure 13A A block diagram illustrating a more detailed configuration of the electronic device 300. (See reference...) Figure 13B In addition to the memory 310, the communicator 320 and the processor 330, the electronic device 300 may also include a user input device 340, an output device 350, etc.

[0285] refer to Figure 13B The memory 310 can store training data 316 used in the personalization process of the neural network model 315.

[0286] refer to Figure 13B The processor 330 can control the training module 331, training data generation module 332, personalization module 333, etc., stored in the memory 310. For example... Figure 13B As shown, each of the modules can be stored in software form on memory 310, and can be selectively executed by processor 330, or integrated with... Figure 13B Unlike other components, this is implemented in hardware and controlled by processor 330. Alternatively, each module can be a combination of software and hardware.

[0287] Training module 331 is a module used to train neural network model 215 or an updated neural network model based on training data.

[0288] The training data generation module 332 is used to generate training data for training the updated neural network model. The training data generation module 332 can generate training data that includes input data and output data from a pre-stored personalized neural network model 315. Alternatively, the training data generation module 332 can generate training data based on the output differences between the neural network model 315 and the updated neural network model for the same input data.

[0289] The personalization module 333 is used to personalize the neural network model 315. The personalization module 333 can train or update the personalization module 315 based on user input or various input data from the electronic device 300.

[0290] The personalization module 333 can also personalize newly received updated neural network models in the same way.

[0291] pass Figure 13A and Figure 13B The electronic device 300 shown and described can also operate as the first electronic device 100 and / or the second electronic device 200 described above, and may also include, in addition to, Figure 13B Modules other than those shown and described in the document.

[0292] Figure 14 This is a flowchart illustrating a method for creating a personalized model of an applied electronic device according to an embodiment.

[0293] refer to Figure 14 In operation S1410, the electronic device can receive information about a model stored on an external electronic device. In this example, the information about the model may include information about the model's version, information about the model's functionality and the applications that use the model, information about the model's layer structure, etc.

[0294] In operation S1420, the electronic device can send information about the weights between nodes of the personalized model to an external electronic device based on the received information and information about the personalized model stored in the electronic device. This is based on the assumption that the model stored in the external electronic device and the personalized model have the same functionality.

[0295] If the model stored in the external electronic device and the personalized model are the same version, information about the weights between nodes in the personalized model can be sent to the external electronic device. This is based on the assumption that the same version of the model has the same layer structure.

[0296] However, if the versions are different, the electronic device can send training data used in at least part of the personalized model or in the personalization process of the personalized model to an external electronic device.

[0297] Figure 15 This is a flowchart illustrating a method for creating a personalized model of an applied electronic device according to an embodiment.

[0298] refer to Figure 15 In operation S1510, the electronic device can receive information about a personalized model stored in an external electronic device. In this example, the electronic device can identify, based on the received information, whether the personalized model and the model stored in the electronic device have the same functionality.

[0299] Electronic devices can compare personalized models with versions and / or layer structures of models stored in the electronic device based on received information.

[0300] In operation S1520, the electronic device can update the pre-stored model based on the difference between the model pre-stored in the electronic device and the personalized model.

[0301] If the personalized model and the model stored in the electronic device have the same functionality and the same version, then when information about the personalized model's nodes is received from an external electronic device, the model stored in the electronic device can be updated based on the information about the received nodes.

[0302] If the personalized model and the model stored in the electronic device have the same functionality but are different versions, the electronic device can train the model stored in the electronic device based on training data received from an external electronic device and / or use the personalized model received from an external electronic device to generate training data, and train the model stored on the electronic device based on the training data.

[0303] Figure 16 This is a flowchart illustrating a method for applying a personalized model according to an embodiment.

[0304] According to an embodiment, in the personalized model application method based on an electronic device in which a personalized model is stored, during operation S1610, the electronic device can receive an updated model from the outside that has the same functions as the personalized model.

[0305] In operation S1620, the electronic device can train the updated model based on the difference between the pre-stored personalized model and the updated model.

[0306] For example, the training data used in the personalization process of a personalized model can be used to train the updated model.

[0307] If no training data is stored, a personalized model can be used to generate new training data, and the updated model can be trained based on the generated training data. Training data can also be generated based on the output difference between the personalized model and the updated model, and the updated model can be trained based on the generated training data.

[0308] The various example embodiments described above can be implemented in a recordable medium that can be read by a computer or a computer-like device using software, hardware, or a combination of software and hardware.

[0309] Through hardware implementation, embodiments of this disclosure may be implemented using, but are not limited to, at least one of, such as, application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electrical units for performing other functions, etc.

[0310] In some cases, the embodiments described herein can be implemented by the processor 120 itself. Depending on the software implementation, embodiments (such as the processes and functions described herein) can be implemented using separate software modules. Each of the aforementioned software modules can perform one or more of the functions and operations described herein.

[0311] Computer instructions for performing processing operations of the first electronic device 100 according to the various embodiments described above may be stored in a non-transitory computer-readable medium. When executed by the device's processor, the computer instructions stored in the non-transitory computer-readable medium can cause the specific device to perform the processing operations in the first electronic device 100 according to the various example embodiments described above.

[0312] For example, non-volatile computer-readable media can refer to media that stores data semi-permanently and can be read by a device. For example, the various applications or programs mentioned above can be stored in non-transitory computer-readable media (e.g., CD, DVD, hard disk, Blu-ray disc, Universal Serial Bus (USB), memory card, read-only memory (ROM), etc.).

[0313] The foregoing exemplary embodiments and advantages are merely illustrative and should not be construed as limiting this disclosure. This disclosure can be readily applied to other types of devices. The description of embodiments of this disclosure is intended to be illustrative and not to limit the scope of the claims, and various substitutions, modifications, and variations will be apparent to those skilled in the art.

Claims

1.An electronic device comprising: a display; at least one memory configured to store at least one instruction and a personalized artificial intelligence (AI) model; a communicator including communication circuitry; and at least one processor configured to execute the at least one instruction to: control the display to provide a user interface that enables a user to select an AI model communication mode between a transmission mode and a reception mode, based on the AI model communication mode of the electronic device being set to the transmission mode, receive, through the communicator, information about a non-personalized AI model stored in an external electronic device that is set to the reception mode, compare a layer structure of the personalized AI model with a layer structure of the non-personalized AI model based on the information about the non-personalized AI model, and based on a comparison between the layer structure of the non-personalized AI model and the layer structure of the personalized AI model, control the communicator to transmit information about weights between nodes of the personalized AI model to the external electronic device, based on the layer structure of the personalized AI model being identical to the layer structure of the non-personalized AI model, control the communicator to transmit the information about the weights between the nodes of the personalized AI model to the external electronic device without transmitting information of the entire personalized AI model, wherein the at least one processor is further configured to execute the at least one instruction to, based on the layer structure of the personalized AI model being different from the layer structure of the non-personalized AI model, control the communicator to transmit training data for customizing the personalized AI model to the external electronic device, wherein the weights between the nodes of the personalized AI model are changed in a personalization process. the at least one processor is further configured to execute the at least one instruction to: 2.The electronic device of claim 1, wherein, based on a node being added to the personalized AI model during a personalization process performed by the electronic device to customize the personalized AI model in the electronic device, control the communicator to transmit information about the added node and information about the weights between the nodes of the personalized AI model to the external electronic device. the at least one processor is further configured to execute the at least one instruction to: 3.The electronic device of claim 1, wherein based on the layer structure of the personalized AI model being different from the layer structure of the non-personalized AI model, identify whether training data for training the personalized AI model is stored in the at least one memory, and based on the training data being stored in the at least one memory, control the communicator to transmit the training data to the external electronic device. the at least one processor is further configured to execute the at least one instruction to: 4.The electronic device of claim 1, wherein based on receiving a user command to download the non-personalized AI model, receive, through the communicator, the non-personalized AI model from a server device, and train the received non-personalized AI model based on the information about the personalized AI model and the information about the non-personalized AI model. ​

Images