Embedding normalization method and electronic device utilizing the same

Abstract

To allow an embedding normalization method and an electronic device using the same to easily predict a click through rate.SOLUTION: The present invention is directed to a method of training a neural network model for predicting a click through rate of a user in an electronic device according to various embodiments. The method has the steps of normalizing an embedding vector based on a feature-wise linear transformation parameter and inputting the normalized embedding vector to a neural network layer. The feature-wise linear transformation parameter is defined so that the same value should be applied to all components of the embedding vector.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] Various embodiments of this disclosure relate to embedding normalization methods and electronic devices utilizing the same. More specifically, they relate to a method for training a neural network model while preserving the importance of feature vectors. [Background technology]

[0002] Artificial intelligence (AI) is being used in a variety of industrial fields. Operating in a manner similar to human thought, AI can be used to extract features from objects that a sample is approaching.

[0003] Recently, research has been progressing on identifying primary concerns for specific objects through neural network models. If concerns for a particular object are identified, the neural network model can be easily guided to the desired destination through those concerns. The analysis of concerns can be improved by controlling various stages during the training of the neural network model, and methods can be used to formulate and extract primary concerns with high importance. [Overview of the project] [Problems that the invention aims to solve]

[0004] Learning interactions between feature vectors for a specific target can be a fundamental challenge in click-through rate prediction. FM, which simultaneously considers first-order and second-order feature interactions, is perhaps the most representative model for performing click-through rate prediction. First-order construct interactions can refer to interactions within individual constructs themselves, while second-order construct interactions can refer to pair-wise interactions between constructs. For example, AFM can utilize an attention mechanism to automatically capture the importance of construct interactions. Recently, NFM, Wide&Deep, DeepFM, xDeepFM, PNN, Autolnt, and AFN have been used to model high-dimensional construct interactions through deep neural networks.

[0005] Recently, there have been various attempts to apply normalization methods to perform click-through rate prediction. NFM, AFN, and AutoFIS can reliably train the components of deep neural networks using batch normalization (BN). PNN and MINA, on the other hand, utilize layer normalization (LN) to train their click-through rate prediction models.

[0006] Click-through rate prediction models that apply normalization methods do not preserve the importance of feature embeddings, but only consider the training stability of the deep neural network components. In other words, batch normalization and layer normalization cannot reflect weighted values ​​for importance by utilizing constant parameters in the same dimension when performing normalization on individual components. If the resulting values ​​through normalization do not reflect importance, there is a problem in that the accuracy of click-through rate predictions may be low. [Means for solving the problem]

[0007] A method for training a neural network model to predict the user's click-through rate in an electronic device according to various embodiments of the present disclosure includes the steps of: normalizing the embedding vector based on a feature-wise linear transformation parameter; and inputting the normalized embedding vector into a neural network layer, wherein the feature-wise linear transformation parameter is defined such that the same value is applied to all components of the embedding vector.

[0008] The normalization steps may include: calculating the mean of the components of the embedding vector; calculating the variance of the components of the embedding vector; and normalizing the embedding vector based on the mean, the variance, and the feature-specific linear transformation parameters.

[0009] The scale parameter and the shift parameter are each vectors of the same dimension as the embedding vector, and all components may have the same value.

[0010] The scale parameter and the shift parameter may be scalar values.

[0011] A neural network system for predicting a user's click through rate implemented by at least one electronic device in various embodiments of the present disclosure includes an embedding layer, a normalization layer, and a neural network layer. The embedding layer maps features included in a feature vector to an embedding vector. The normalization layer normalizes the embedding vector based on feature-wise linear transformation parameters. The neural network layer performs neural network operations based on the normalized embedding vector. The feature-wise linear transformation parameters are defined such that the same value is applied to all components of the embedding vector.

[0012] The normalization layer can calculate the mean of the components of the embedding vector, calculate the variance of the components of the embedding vector, and normalize the embedding vector based on the mean, the variance, and the feature-wise linear transformation parameters.

[0013] The feature-wise linear transformation parameters can include a scale parameter and a shift parameter.

[0014] The scale parameter and the shift parameter are each vectors of the same dimension as the embedding vector, and all components can have the same value.

[0015] The scale parameter and the shift parameter can be scalar values.

Advantages of the Invention

[0016] Unlike variance-only layer normalization (VO-LN), which is being studied to overcome such limitations, the embedding normalization methods according to the various embodiments of this disclosure can be calculated based on parameters that can reflect weights for feature vectors. Based on the parameters calculated by embedding normalization, the normalized values ​​can be widely used in various neural network model classes (e.g., deep neural networks, shallow neural networks) and can reflect weights for importance.

[0017] According to the embedding normalization methods of various embodiments of this disclosure, electronic devices can improve the performance of click-through rate prediction models by preserving the importance of feature vectors. While methods such as batch normalization or layer normalization may excessively equalize the norm of feature embeddings and potentially impair model performance, the embedding normalization methods of this disclosure can improve the accuracy of click-through rates by explicitly modeling the norm of individual feature embeddings, thereby quickly performing the distribution and convergence of importance for individual configurations. [Brief explanation of the drawing]

[0018] [Figure 1] This is a schematic block diagram showing the configuration of an electronic device according to various embodiments of this disclosure. [Figure 2] This is a schematic flowchart of the embedding normalization method according to various embodiments of the present disclosure. [Figure 3] This diagram schematically illustrates the structure of a neural network training model including an embedded normalization layer according to various embodiments of this disclosure. [Figure 4]This is an illustrative diagram relating to performing embedding normalization on feature vectors according to various embodiments of the present disclosure. [Figure 5] This diagram schematically illustrates the structure for performing embedding normalization of feature vectors according to various embodiments of this disclosure. [Figure 6] Figure 4 is a specific example diagram illustrating the normalized values ​​obtained by performing the embedding normalization process. [Modes for carrying out the invention]

[0019] The terminology used in the embodiments has been selected as widely used and general terms as possible, taking into account the functions described herein, although this may change depending on the intent of engineers in the art, case law, the emergence of new technologies, etc. In some cases, the applicant has arbitrarily selected terms, in which case their meaning will be described in detail in the relevant sections of the description. Therefore, the terminology used in this disclosure is not simply a set of names, but must be defined based on the meaning of the term and the overall content of this disclosure.

[0020] When a specification states that a part "includes" a certain component, unless otherwise specified, this does not mean that other components are excluded, but rather that other components may be included. Furthermore, terms such as "...part" and "...module" used in the specification refer to a unit that processes at least one function or operation, which may be embodied in hardware or software, or in a combination of hardware and software.

[0021] The expression “at least one of a, b, and c” as described throughout the specification may encompass “a alone,” “b alone,” “c alone,” “a and b,” “a and c,” “b and c,” or “all of a, b, and c.”

[0022] The term "terminal" as used below can refer to a computer or portable terminal that can connect to a server or other terminals via a network. Here, a computer includes, for example, a laptop, desktop, or laptop computer equipped with a web browser, and a portable terminal is, for example, a wireless communication device that guarantees portability and mobility, and can include all types of handheld wireless communication devices such as IMT (International Mobile Telecommunication), CDMA (Code Division Multiple Access), W-CDMA (W-Code Division Multiple Access), LTE (Long Term Evolution), smartphones, and tablet PCs.

[0023] The embodiments of this disclosure will be described below in detail with reference to the attached drawings, so as to be easily implemented by a person with ordinary skill in the art to which this disclosure pertains. However, this disclosure can be embodied in a variety of different forms and is not limited to the embodiments described herein. Embodiments of the present invention will be described in detail below with reference to the attached drawings.

[0024] In describing the embodiments, technical details that are widely known in the art to which the present invention belongs and are not directly related to the present invention will be omitted. This is to ensure that the gist of the present invention is not obscured by omitting unnecessary explanations and is communicated more clearly.

[0025] For similar reasons, some components in the attached drawings are exaggerated, omitted, or only schematically represented. Furthermore, the dimensions of each component do not fully reflect their actual size. The same reference number is assigned to identical or corresponding components in each drawing.

[0026] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and can be embodied in a variety of different forms, provided that these embodiments complete the disclosure of the present invention and fully inform a person ordinary skill in the art to which the invention pertains, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0027] At this point, it will be understood that each block of the processing flowchart and the combination of the flowchart can be performed by computer program instructions. Since these computer program instructions can be implemented in the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions performed through the processor of the computer or other programmable data processing equipment will generate means for performing the functions described in the flowchart blocks(et). Since these computer program instructions can also be stored in computer-available or computer-readable memory that can be directed to the computer or other programmable data processing equipment in order to embody the functions in a particular manner, the instructions stored in that computer-available or computer-readable memory can also produce manufactured items that contain instruction means for performing the functions described in the flowchart blocks(et). Since computer program instructions can also be implemented on a computer or other programmable data processing equipment, instructions that perform a series of operational steps on a computer or other programmable data processing equipment to generate a process executed on the computer and thus perform the computer or other programmable data processing equipment can also provide steps for performing the functions described in the flowchart blocks(e).

[0028] Furthermore, each block may represent a module, segment, or portion of code containing one or more executable instructions for performing a specified logical function(s). It should also be noted that in some alternative execution examples, the functions mentioned in a block may occur out of order. For example, two blocks illustrated consecutively may be executed virtually simultaneously, or they may be executed in reverse order depending on the function at hand.

[0029] Artificial intelligence (AI) can be a type of computer program that mimics human intelligence by operating through a set of logical algorithms that think, learn, and make decisions like humans. So-called AI can process complex calculations on a processor corresponding to the human brain through a neural network similar to the human nervous system. This specification describes machine learning, which may be included in deep learning, and the process of normalizing and modeling features through different learning methods. The terms machine learning and machine learning may be used interchangeably within this specification.

[0030] A neural network can refer to a network that models the operating principles and interconnection relationships of neurons, the fundamental units of the human nervous system. A neural network can be a data processing system in which individual nodes or processing elements are connected in the form of layers. A neural network can contain multiple layers, each of which can contain multiple neurons. Furthermore, a neural network can contain synapses, which correspond to nerve stimuli that can transmit data between neurons. In this specification, the terms layer and layer may be used interchangeably.

[0031] Specifically, a neural network can generally refer to a data processing model in which artificial neurons change the strength of synaptic connections through iterative learning, thereby possessing the ability to solve a given problem or a problem in which variables arise. Within this specification, the terms neural network and artificial neural network may be used interchangeably.

[0032] Neural networks can be trained using training data. Specifically, training can involve determining the parameters of a neural network using feature data to achieve purposes such as classifying, regression analysis, or clustering input data. More specifically, the elements that determine the parameters may be weights or biases.

[0033] Neural networks can be trained on input data to classify or crowd based on patterns, and a trained neural network can be referred to as a trained model. Specifically, training methods can be classified into supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. More specifically, supervised learning can be a method of machine learning for inferring a function from training data. Among the functions inferred through machine learning, those that output continuous result values ​​can be called regression analysis, and those that predict the class of the input data and output result values ​​can be called classification.

[0034] In supervised learning, labels may be assigned to training data, and these labels may contain meaningful result values ​​that the neural network must infer. Specifically, the result values ​​that the neural network must infer may be labeling data. More specifically, the training data and the corresponding labeling data may constitute a single training set, and the neural network can acquire input and result values ​​in the form of this training set.

[0035] Training data can contain multiple feature vectors, and a neural network can infer from this training data, label individual feature vectors, and output the labeled data as the result. The neural network can infer a function relating each data point through the training and labeled data. Furthermore, parameters for individual vectors can be optimized through feedback to the function inferred by the neural network.

[0036] Figure 1 is a schematic block diagram showing the configuration of an electronic device according to various embodiments of this disclosure.

[0037] An electronic device may include a device that incorporates a neural network. An electronic device may be capable of performing machine learning using training data, and may include a device capable of performing learning using a model composed of a neural network. For example, an electronic device may be configured to receive, classify, store, and output data used for data mining, data analysis, intelligent decision-making, and machine learning algorithms.

[0038] Electronic devices can include a variety of equipment for training neural networks. For example, an electronic device can be embodied as multiple server sets, cloud servers, or a combination thereof. Specifically, an electronic device can obtain result values ​​in data analysis or training through distributed processing.

[0039] Referring to Figure 1, the electronic device may include a processor 110, an input / output interface (I / O interface) 120, and memory 130 as components. The components of the electronic device shown in Figure 1 are not limited to these and may be added or replaced.

[0040] The processor 110 can control or predict the operation of electronic devices through data analysis and machine learning algorithms. The processor 110 can request, retrieve, receive, or utilize data and control electronic devices to perform preferred actions learned through training.

[0041] The processor 110 may be configured to derive and sense result values ​​for input values ​​based on user input or natural language input. The processor 110 may be configured to collect data for processing and storage. Data collection may include sensing data through sensors, extracting data stored in memory 130, or receiving data from external devices through input / output interface 120.

[0042] The processor 110 can digitize the operation history of the electronic device and store it in the memory 130. Based on the stored operation history data and the trained model, the processor 110 can obtain the best possible result for performing a specific operation.

[0043] When a specific operation is performed, the processor 110 can analyze the history of that operation through data analysis and machine learning algorithms. Specifically, the processor 110 can update previously trained data based on the analyzed history. In other words, the processor 110 can improve the accuracy of the data analysis and machine learning algorithms based on the updated data.

[0044] The input / output interface 120 can perform functions such as transmitting data stored in the memory 130 of the electronic device or data processed by the processor 110 to other devices, or receiving data from other devices to the electronic device.

[0045] The processor 110 can train (e.g., learn) a neural network using training data or a training dataset. For example, the processor 110 can train a neural network through data preprocessed with acquired input values. As another example, the processor 110 can train a neural network through preprocessed data stored in memory 130. Specifically, the processor 110 can determine the optimization model of the neural network and the parameters used for optimization by repeatedly training the neural network using various training methods.

[0046] Memory 130 can store models trained by processor 110 or neural networks. For example, memory 130 can store trained models or models being trained separately. Specifically, memory 130 can store models during the training process of a neural network and store trained models with training history. In addition, memory 130 can store updated trained models.

[0047] Memory 130 can store input data, training data for model training, model training history data, and other input values. The input data stored in memory 130 may be either data processed for model training or unprocessed raw data.

[0048] Depending on the embodiment, the normalization of the neural network model learned by the processor 110 may be a batch normalization (BN) method, a layer normalization (LN) method, or a method utilizing the same. Normalization may include a process of processing data using parameters, and this specification describes a process of normalizing individual feature vectors corresponding to the characteristics of individual data through variance, mean, and parameters (e.g., a first parameter or a second parameter). In particular, the normalization described herein may be referred to as embedding normalization (EN).

[0049] The embodiment allows for even faster extraction of characteristics from feature vectors, deriving result values ​​that can be used to train models that predict click-through rates (CTR). CTR can represent the probability that a user will click on a component of interest, and predicting CTR can be less accurate using existing normalization methods such as batch normalization or layer normalization. Embedding normalization not only allows for stable training of neural network models but also improves the accuracy of CTR predictions. More specifically, the model in the embodiment can increase the learning speed of the model while maintaining the importance of each component during the learning process, thereby increasing safety through iterative learning. Furthermore, by preserving the importance of components during the learning process, this disclosure can improve performance not only in deep learning models but also in click-through rate prediction models that do not have a neural component, such as FM and FFM.

[0050] Figure 2 is a schematic flowchart of the embedding normalization method according to various embodiments of this disclosure.

[0051] At S210, the electronic device can receive the first training data. The training data can include a feature vector. A feature vector can include each feature. The electronic device can acquire a feature vector as input data for training a neural network model. Features can include elements related to the user and elements related to items. For example, user-related elements may indicate the user's age, gender, platform connection time, in-platform click log, and platform usage history. The platform may be an online platform to which the user has connected. As another example, item-related elements may include the type of content within the platform, the content's structure, and the content's placement area. The content may be a notice posted within the platform. For example, a feature vector could be x={x1, x2, x3, ..., x i , ..., x n} (where n is a natural number) and each component (x) of the feature vector 1~ x n ) could be their respective features.

[0052] In this embodiment, the electronic device can map individual features to embedding vectors. The mapping may represent the execution of embedding for each feature. The mapped embedding vectors are e1, e2, e3, e4, ..., e, corresponding to each feature. n It can be mapped to such as, for example, each feature (x1~x) contained in the feature vector x of an electronic device. n ) perform an embedding lookup and the corresponding embedding vector (e1~e n) can be mapped to these. At this time, each embedding vector (e1~e n ) can be a d-dimensional (where d is a natural number) vector.

[0053] According to various embodiments, an electronic device can map the first component of a feature vector to a first embedding vector, and an electronic device can map the second component of a feature vector to a second embedding vector. That is, n features contained in an embedded feature vector can each be mapped to the first embedding vector to the nth embedding vector. The embedding vector to which each feature of the feature vector (e.g., the i-th component) is mapped can be referred to as the i-th embedding vector.

[0054] Embedding vectors can be learnable parameters. As the neural network model is trained, the embedding vectors can be learned to enable the neural network model to perform its intended purpose.

[0055] In S220, the electronic device can embed feature vectors to obtain an embedding matrix. The electronic device can also generate an input matrix by embedding feature vectors. For example, the input matrix can contain the result of embedding a number of feature vectors corresponding to the batch size. That is, if the feature vectors are n-dimensional, the embedding vectors are d-dimensional, and the batch size is b, the input matrix may be of size b*n*d.

[0056] In S230, the electronic device can normalize the embedding vectors based on the embedding matrix. Through embedding the feature vectors, the electronic device can output an embedding matrix E composed of the embedding vectors. The embedding matrix can be an input matrix.

[0057] Depending on the embodiment, normalization can be performed on the embedding vectors. The electronic device can perform normalization such that the importance (e.g., magnitude (norm)) of each embedding vector is preserved. Conventionally, normalization tends to equalize the magnitude (norm) of each embedding vector, which can lead to problems such as gradient vanishing or gradient exploding, but the electronic device of this disclosure can prevent such problems from occurring.

[0058] According to the embodiment, the electronic device can perform normalization of the embedding vector based on a linear transformation parameter. The linear transformation parameter may include a scale parameter and a shift parameter. The scale parameter and the shift parameter may each be represented by a vector with the same dimensions as the embedding vector. Furthermore, all components of the scale parameter and the shift parameter may each have the same value. That is, when the electronic device performs normalization, it can apply the same value of the scale parameter and the shift parameter to each component of the embedding vector.

[0059] According to the embodiment, embedding normalization allows the application of the same scale and shift parameters corresponding to the same dimensions to all components of individual embedding vectors. That is, the electronic device can apply the same linear transformation parameter values ​​to all components for the same index of the embedding vector. This may be a difference from layer normalization, where dimension-wise parameters are set.

[0060] Depending on the embodiment, the embedding normalization of the present disclosure can perform normalization in such a way as to preserve the importance of the embedding vector by defining linear transformation parameters such that the same scale parameter value and shift parameter value are applied to all components of a single embedding vector. That is, the electronic device can perform normalization for each embedding vector based on feature-wise linear transformation parameters. The feature-wise linear transformation parameters may include scale parameters and shift parameters. The electronic device can perform normalization based on scale parameters and shift parameters defined with the same scalar value for all components of the embedding vector.

[0061] Figure 3 is a schematic diagram illustrating the structure of a neural network model including an embedding normalization layer according to various embodiments of this disclosure. Figure 3 illustrates the process by which a feature vector 310 is input to the neural network model and a result value is output.

[0062] A neural network system can include a number of neural network layers arranged in a sequence from the lowest layer to the highest layer. A neural network system can generate a neural network output from a neural network input by processing the neural network input through each layer of the sequence.

[0063] In one embodiment, the neural network system can receive input data and generate scores for that input data. For example, if the input data for the neural network system is features corresponding to characteristics extracted from content, the neural network system can output scores for each object category. The scores for each category can indicate the selectability, including the composition of objects that the content includes in the category. As another example, if the input data for the neural network system is an image feature for an article on a specific platform, the neural network system may output scores indicating the likelihood that the image for the article on that platform will be selected.

[0064] In this embodiment, the neural network model can take the feature vector 310 as input data and output result values ​​through the embedding layer 320, the normalization layer 330, and the neural network layer 340. In this case, the feature vector 310 may be the output of a lower layer (not shown in Figure 3), and the neural network layer 340 may contain multiple neural network layers.

[0065] The embedding layer 320 can perform an embedding for the feature vector 310. The embedding layer 320 can map each feature included in the feature vector 310 to an embedding vector and generate an input matrix including the embedding vectors. The embedding operation of the embedding layer 320 can be performed in the same way as the embedding operation described in S220 of FIG. 2. The normalization layer 330 can perform normalization on the embedding vectors included in the input matrix. According to an embodiment, the normalization can be performed by the following mathematical formula (1).

[0066] Mathematical formula (1)

Number

[0067] In the mathematical formula (1), e x represents an embedding vector corresponding to the feature x included in the feature vector 310, JPEG0007871052000002.jpg6170 can be a normalized embedding corresponding to the feature x. JPEG0007871052000003.jpg6170 can mean an element-wise multiplication operation. ε is a relatively small scalar value and can be added to the variance to prevent overflow during the normalization.

[0068] In the mathematical formula (1), μ x and σ x 2 can be the mean and variance of the components of the embedding vector e x . μ x and σ x 2These can be calculated as shown in mathematical formulas (2) to (5). In mathematical formulas (2) to (5), (e x ) k is the embedding vector e x This shows the kth component, where d is the embedding vector e x This indicates the dimension (i.e., the number of components).

[0069] Mathematical formula (2)

number

[0070] Mathematical formula (3)

number

[0071] Mathematical formula (4)

number

[0072] Mathematical formula (5)

number

[0073] In mathematical formula (1), γ x and β x γ can be a linear transform parameter. x and β x These can be the scale parameter and the shift parameter, respectively. γ x and β x These can be set as shown in the following mathematical formulas (6) and (7), γ x and β x This can be a learnable parameter. That is, the learnable parameter γ x and β xEach is a vector with the same dimensions as the embedding vector, and all components are JPEG0007871052000008.jpg86 and JPEG0007871052000009.jpg96 is a possible learned parameter.

[0074] Mathematical formula (6)

number

[0075] Mathematical formula (7)

number

[0076] According to the embodiment, the electronic device can perform the normalization operation of the normalization layer 330, as described with reference to mathematical formulas (1) to (7), in a manner expressed as mathematical formula (8).

[0077] Mathematical formula (8)

number

[0078] In mathematical formula (8), e x is the embedding vector, d is the dimension of the embedding vector, and μ x σ is the average of all components of the embedding vector, x 2 is the variance of all components of the embedding vector, and (e x ) k is the embedding vector e x It is the k-th component, JPEG0007871052000013.jpg86 and JPEG0007871052000014.jpg96 may be a feature-specific linear transformation parameter.

[0079] Feature-specific linear transformation parameters JPEG0007871052000015.jpg86 and JPEG0007871052000016.jpg96 may be a learnable parameter. Feature-specific linear transformation parameters JPEG0007871052000017.jpg86 and Each element in JPEG0007871052000018.jpg96 can be a scalar value. Feature-specific linear transformation parameters can be defined so that the same value is applied to all components of the embedding vector during the normalization process. Furthermore, electronic devices can perform normalization of the embedding vector based on the mean, variance, and feature-specific linear transformation parameters of the components of the embedding vector.

[0080] According to one embodiment, the embedding vectors normalized through embedding normalization may have relatively large differences in magnitude (norm) between them. Specifically, the embedding vectors normalized by embedding normalization can be distinguished by the large difference in magnitude (norm) shown by the corresponding feature vectors. Accordingly, the embedding vectors normalized by embedding normalization can be easily identified as having different levels of importance among the feature vectors.

[0081] The electronic device according to this embodiment can utilize embedding normalization along with feature embedding without requiring additional changes to the model architecture or training process. Specifically, embedding normalization can be integrated into all types of click-through rate prediction models that use feature embedding to represent a given feature, thus possessing high applicability.

[0082] According to various embodiments, the electronic device can train a neural network model by taking the embedding normalized embedding vectors, obtained through the embedding layer 320 and the normalization layer 330, as input to the neural network layer 320. Accordingly, the electronic device can obtain the resulting values ​​for the feature vectors 310 as output.

[0083] Figure 4 is an illustrative diagram related to performing embedding normalization on feature vectors according to various embodiments of this disclosure.

[0084] In various embodiments, electronic devices can perform click-through rate prediction by leveraging features with very high cardinality and scarcity. Cardinality can refer to the number of tuples that constitute a single relation, and can refer to the number of unique values ​​in a particular data set. For example, if the data is a set related to "gender," the cardinality for "gender" can be set to 2, based on the distinction between "male" and "female." Specifically, attribute values ​​related to a given list within a data set are called tuples, and the number of such tuples can correspond to cardinality. Because the components of a feature vector can contribute differently to the click-through rate prediction of a neural network model, electronic devices may need to perform a function that clearly distinguishes the importance of feature vectors. The result of embedding normalization on a feature vector can naturally imply the importance of the corresponding feature vector.

[0085] Referring to Figure 4, the electronic device can obtain the embedding vector 410 as input, in units of the index (i) of the same component. For example, the electronic device can obtain the embedding vectors 410 (e1, e2, e3, and e4) corresponding to features x1, x2, x3, and x4, respectively, as input using individual indices: the first index, second index, third index, and fourth index. In other words, the electronic device can check the relationship with the feature vector and utilize cardinality and rarity. By using feature-wise linear transformation parameters when performing embedding normalization for each component of the individual embedding vector 410, the electronic device can clearly distinguish the importance of each embedding vector 410.

[0086] According to various embodiments, an electronic device can perform embedding normalization on an embedding vector 410 through the mean 420, variance 430, and parameter 440 of the components of the embedding vector 410. The parameter 440 is a feature-wise linear parameter that can be set so that the same value is used for all components of a single embedding vector 410 on a feature-wise basis when normalization is performed on the embedding vector 410.

[0087] According to the embodiment, parameter 440 may include a first parameter 441 and a second parameter 442. Here, the first parameter 441 may correspond to a scale parameter, and the second parameter 442 may correspond to a shift parameter. The first parameter 441 and the second parameter 442 are each vectors of the same dimension as the embedding vector 410, and all components may have the same value. Accordingly, if the embedding vector 410 has the same index, the electronic device can perform normalization by applying the same parameter value to all components of the same index.

[0088] According to one embodiment, the electronic device can perform click rate prediction through a factorization machine (FM), which is a typical recommendation system model. In particular, the prediction made by the electronic device using the embedding normalization of this disclosure can be derived as shown in the following mathematical formula (9).

[0089] Mathematical formula (9)

number

[0090] In the above mathematical formula (9), w0 represents global bias, and w xi x is the component of the i-th feature vector. i This could be a weighted value for modeling the first-order interaction. JPEG0007871052000020.jpg6170 and JPEG0007871052000021.jpg6170 is x i and x j This could be a normalized embedding.

[0091] Depending on the embodiment, we can assume a feature x' that is unrelated to the click label y and does not contain a useful signal for y. For example, x' may not be important for predicting y, so the dot product term of the normalized values ​​may be 0 for all feature vectors. That is, the dot product may correspond to 0 for low-importance feature vectors. If we do not calculate the dot product as 0 for low-importance feature vectors, noise may be introduced during the training of the neural network model of the electronic device.

[0092] According to the embodiment, the electronic device can derive an embedded matrix composed of normalized embedded vectors and normalized embedded vectors through embedding normalization of feature vectors. The electronic device may also go through a process of checking other orthogonal components to satisfy additional constraints on the embedded vectors 410 (e.g., the condition that the inner product is zero). Such constraints may reduce the effective dimension of the embedding space and harm the model capacity by utilizing only the orthogonal d-1 dimension for embedding to other configurations.

[0093] Existing normalization methods may have the side effect of equalizing the component importance of different feature vectors. The electronic device of this disclosure can perform embedding normalization by applying feature-specific linear transformation parameters to all components with the same value, while taking advantage of the benefits of batch normalization or layer normalization, and clearly preserving the meaning of individual feature vectors for clear determination of importance.

[0094] Figure 5 is a schematic diagram illustrating the structure of embedding normalization of feature vectors according to various embodiments of this disclosure.

[0095] According to various embodiments, the electronic device can train a neural network model through an embedding matrix composed of embedding vectors that embed feature vectors included in the training data.

[0096] The electronic device can perform normalization of the embedding matrix using a normalization layer 520 during training through a neural network model. The resulting values ​​obtained by normalizing the embedding matrix for the training data can be represented by normalized values. Normalized values ​​can be obtained by applying parameters 510 (e.g., parameter 440 in Figure 4) in the normalization layer 520. The parameters 510 applied by the electronic device in the normalization layer 520 to train the neural network may correspond to gamma (e.g., the first parameter 441 in Figure 4) and beta (e.g., parameter 442 in Figure 4). Furthermore, new parameters 530 generated based on parameters 510 and the normalized values ​​may correspond to new gamma and new beta values. The electronic device can perform embedding normalization by applying the new parameters 530 to other training data. According to various embodiments, the electronic device can obtain new parameters 530 through backpropagation during training with the neural network model.

[0097] According to one embodiment, the electronic device can apply the same parameters to all components of the embedding vector to reflect weighted values. For example, high weighted values ​​may be assigned to feature vectors that are of high importance and are expected to lead to clicks and selections due to their high user interest. In other words, in embedding normalization, the parameters applied to all components of the embedding vector can be determined by iterative training. The electronic device can optimize the normalization parameters (e.g., parameter 440 in Figure 4) applied to the embedding matrix by repeatedly training a neural network model.

[0098] Figure 6 is a specific example of the normalized values ​​obtained by performing the embedding normalization process shown in Figure 4. Referring to Figure 6, one can compare the results of the normalized values ​​obtained through batch normalization and layer normalization methods, along with the embedding normalization method shown in Figure 4.

[0099] According to embodiments, the embedding normalization method of the present disclosure may be used in conjunction with an embedding technique that maps features with embedding vectors in latent space. For example, given input data x, an electronic device can embed a feature vector x to generate an input matrix E composed of embedding vectors e. For example, the feature vector x may include features x1, x2, x3, and x4 as its components.

[0100] Referring to the rightmost graph in Figure 6, values ​​normalized by batch normalization (a) or layer normalization (b) cannot reflect the weights for individual components and therefore cannot show differences in importance. On the other hand, values ​​normalized by the embedding normalization method (c) in Figure 6 reflect the weights for individual components, allowing for a sufficient understanding of importance in e1 and e3. The embedding normalization method can obtain results with higher accuracy at a faster learning rate than batch normalization or layer normalization for the same input data, which is compared in terms of accuracy with other normalization methods as shown in Figure 6.

[0101] Referring to Figure 6, the electronic device can generate first normalized values ​​for all dimensional components contained in the first embedding vector using the embedding normalization layer, based on the first and second parameters determined by the first index. For example, the first embedding vector e1 is the embedding vector to which x1 is mapped, and the first embedding vector can contain 1.0, -1.3, and -0.4 as components in each dimension. Here, 1.0 may correspond to the one-dimensional component of the first embedding vector, -1.3 to the two-dimensional component, and -0.4 to the three-dimensional component.

[0102] According to the embodiment, the electronic device can calculate the mean and variance of the components of the first and second embedding vectors, respectively. Referring to Figure 6, the electronic device can calculate the mean and variance of the components of e1, which is the embedded vector to which x1 is mapped. The mean and variance of the embedding normalization are calculated for the components of the individual embedding vectors and correspond to the mean and variance for all dimensions of the individual embedding vectors. For example, in the embedding normalization of Figure 6, the mean and variance for the components of the first embedding vector are calculated to be -0.2 and 0.9.

[0103] According to the embodiment, the first and second parameters are vectors of the same dimensions as the embedding vector, and all components can have the same value. Referring to Figure 6, the value corresponding to the first index of the first parameter, gamma, is 2.5. Also, the value corresponding to the first index of the second parameter, beta, is -1.2. When performing embedding normalization, the electronic device can apply the first and second parameters, which correspond to the component indices of the individual embedding vectors (e.g., the component index of the first embedding vector is 1, and the component index of the second embedding vector is 2), to the components of all dimensions of the individual embedding vectors with the same value.

[0104] On the other hand, preferred embodiments of the present invention are disclosed herein and in the drawings, and although specific terms are used, these are merely general terms used to facilitate the explanation of the technical content of the invention and to aid in the understanding of the invention, and are not intended to limit the scope of the invention. It will be obvious to those ordinary skill in the art to which the present invention pertains that other modifications based on the technical idea of ​​the present invention are also possible in addition to the embodiments disclosed herein.

[0105] The electronic device or terminal according to the embodiment described above may include a processor, memory for storing and executing program data, permanent storage such as a disk drive, a communication port for communicating with external devices, and user interface devices such as a touch panel, keys, and buttons. The method embodied in the software module or algorithm is computer-readable code or program instructions executable on the processor and may be stored on a computer-readable recording medium. Here, computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROM, DVD (Digital Versatile Disc)). Computer-readable recording media may be distributed across computer systems connected to a network, and computer-readable code may be stored and executed in a distributed manner. The medium is computer-readable, can be stored in memory, and can be executed on the processor.

[0106] This embodiment can be represented by a functional block configuration and various processing stages. Such functional blocks can be embodied by a variety of hardware and / or software configurations that perform specific functions. For example, the embodiment may employ direct circuit configurations such as memory, processing, logic, and look-up tables, which can perform various functions under the control of one or more microprocessors or other control devices. Just as the components can be executed by software programming or software elements, this embodiment includes various algorithms embodied by combinations of data structures, processes, routines or other programming configurations, which can be embodied in programming or scripting languages ​​such as C, C++, Java, and assembler. Functional aspects can be embodied by algorithms executed by one or more processors. Furthermore, this embodiment can employ prior art for electronic environment configuration, signal processing, and / or data processing. Terms such as “mechanism,” “element,” “means,” and “configuration” can be used broadly and are not limited to mechanical and physical configurations. The terms can also include the meaning of a series of software processes (routines) in conjunction with a processor, etc.

[0107] The embodiments described above are merely illustrative examples, and other embodiments may be realized within the scope of the claims described later.

Claims

1. In a method for training a neural network model to predict the click-through rate (CTR) of an electronic device, The process involves mapping the features contained in the feature vector to the embedding vector, A step of normalizing the embedding vector based on a feature-wise linear transformation parameter, wherein the feature-wise linear transformation parameter has the same dimension as the embedding vector and all of its components have the same value; The step includes inputting the normalized embedding vector into a neural network layer, A method wherein the same value of the feature-specific linear transformation parameter is applied to all components of the embedding vector during the normalization stage.

2. The normalization step described above is: A step of calculating the mean of the components of the aforementioned embedding vector, The steps include: calculating the variance of the components of the aforementioned embedding vector; The method according to claim 1, further comprising the step of normalizing the embedding vector based on the mean, the variance, and the feature-specific linear transformation parameters.

3. The linear transformation parameters for each feature are: The method according to claim 1, comprising a scale parameter and a shift parameter.

4. The normalization step described above is: This is the stage where the calculation in mathematical formula 1 below is performed. In the mathematical formula 1 below, e x is the embedding vector, d is the dimension of the embedding vector, and μ x σ is the average of all components of the aforementioned embedding vector, x 2 (e) is the variance of all components of the embedding vector, x ) k The embedding vector e x It is the k-th component, 【number】 and 【number】 The method according to claim 1, wherein is the linear transformation parameter for each feature. [Mathematical formula 1] [Math 1]

5. A computer program stored on a computer-readable recording medium, coupled with hardware, to perform the method described in claim 1.

6. A neural network system for predicting the click-through rate (CTR) of a user embodied by at least one electronic device, wherein the neural network system includes an embedding layer, a normalization layer, and a neural network layer. The aforementioned embedding layer maps the features contained in the feature vector to the embedding vector, The normalization layer normalizes the embedding vector based on feature-wise linear transformation parameters, the feature-wise linear transformation parameters having the same dimensions as the embedding vector, and all of its components having the same values. The aforementioned neural network layer performs neural network operations based on the normalized embedding vector, A neural network system in which the same value of the feature-specific linear transformation parameter is applied to all components of the embedding vector during normalization.

Images