Neural network training method and apparatus

Abstract

To provide a neural network training method, a recording medium, and an apparatus.SOLUTION: A method includes: determining first and second pieces of backbone feature data corresponding to each input data by applying two or more sets of input data of a same scene to first and second neural network models in S110 and S120; calculating first convolution data and first view data from the first backbone feature data and determining second convolution data and second view data from the second backbone feature data in S130; and training at least one of the first neural network model and the second neural network model on the basis of a loss based on a combination of any two or more of the first convolution data, the first view data, the second convolution data, the second view data, and a convolution data clustering result in S140.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The following embodiments relate to neural network training. [Background technology]

[0002] One area of ​​research aiming to apply the efficient pattern recognition methods possessed by humans to actual computers is the development of artificial neural networks, which model the characteristics of human biological nerve cells using mathematical representations. To solve the problem of classifying input patterns into specific groups, artificial neural networks use algorithms that mimic the learning ability possessed by humans. Through these algorithms, artificial neural networks can generate mappings between input and output patterns and possess the generalization ability to generate relatively correct outputs even for input patterns that have not been used in training.

[0003] The background technology described above was acquired or obtained by the inventor in the process of deriving the content of this specification, and does not necessarily constitute prior art that was made public to the general public before this application was filed. [Overview of the project] [Problems that the invention aims to solve]

[0004] The objective of this invention is to provide neural network training. [Means for solving the problem]

[0005] A neural network training method implemented by a processor according to one embodiment includes the steps of: individually applying two or more input data for the same scene to a first neural network model to determine first backbone feature data corresponding to each input data; individually applying the two or more input data to a second neural network model to determine second backbone feature data corresponding to each input data; calculating first convolutional data based on projection and first view data based on dropout from the first backbone feature data, and determining second convolutional data based on projection and second view data based on dropout from the second backbone feature data; and training at least one of the first neural network model and the second neural network model based on a loss determined based on two or more combinations of the first convolutional data, the first view data, the second convolutional data, the second view data, and the clustering results of the convolutional data.

[0006] The training step may include determining a first partial loss based on a code value indicating the cluster to which the convolutional data of the first convolutional data belongs, extracted from any one of the two or more input data based on the first neural network model, and the convolutional data of the first convolutional data extracted from the other input data based on the first neural network model.

[0007] The step of determining the first partial loss may include determining the first partial loss based on view data of the first view data extracted from any one of the two or more input data using the first neural network model, and further based on the clustering results.

[0008] The training step may include a step of determining a second partial loss based on the convolutional data of the second convolutional data extracted from any one of the two or more input data based on the second neural network model, which indicates the cluster to which the convolutional data of the first convolutional data belongs.

[0009] The step of determining the second partial loss may include determining the second partial loss based on the view data of the second view data extracted from any one of the two or more input data based on the second neural network model, and the code value.

[0010] The training step may include determining a first partial loss for the first neural network model and a second partial loss for the second neural network model based on the cross-entropy loss between one of the first convolution data, the first view data, the second convolution data, and the second view data and the code value.

[0011] The training step may include determining a third partial loss using a gradient reversal layer (GR layer) from the first and second convolution data.

[0012] The method may further include the step of generating two or more input data from original data, which is a color image acquired by a camera sensor, based on data augmentation including one or more combinations of random noise addition, image rotation, scaling, image movement, random cropping, and color distortion.

[0013] The process may further include the step of generating two or more input data from original data, which is a lidar image acquired by a lidar sensor, based on data augmentation including one or more combinations of detection point augmentation and reflection value adjustment.

[0014] The method may further include the steps of generating one of the two or more input data by capturing a scene using one of the multiple sensors, and generating the other input data from the two or more input data by capturing the same scene using another type of sensor from the multiple sensors.

[0015] The method may include the steps of: determining first convolutional data projected from the first backbone feature data using a first projection model; determining first view data projected from the first backbone feature data using a first drop model including at least one dropout layer; determining second convolutional data projected from the second backbone feature data using a second projection model; and determining second view data projected from the second backbone feature data using a second drop model including at least one dropout layer.

[0016] The step of determining the first view data may include applying a linear layer to the first convolution data and then applying a dropout layer, and the step of determining the second view data may include applying a linear layer to the second convolution data and then applying a dropout layer.

[0017] The number of parameters in the first neural network model may be greater than the number of parameters in the second neural network model.

[0018] The method may further include the step of determining one of the following: a code value indicating the cluster to which the convolutional data output by the model having the highest recognition performance among a plurality of neural network models, including the first neural network model and the second neural network model, belongs; a code value for each convolutional data output by each model; and a code value indicating the cluster to which the convolutional data output by the model with the smallest clustering loss belongs.

[0019] The step of determining the second backbone feature data may include the steps of converting the two or more input data based on a format defined by the target hardware, and inputting the converted data into the second neural network model.

[0020] The training step may include updating the parameters of the second neural network model so that the determined loss is minimized.

[0021] The process may include the step of generating an inference result for a target task on the detection data using convolutional data extracted from the detection data based on the second neural network model that has been trained.

[0022] The method may include the step of controlling the operation of an electronic device storing the second neural network model based on the inference results.

[0023] An apparatus having a neural network according to one embodiment includes a memory for storing a first neural network and a second neural network, and a processor which individually applies two or more input data for the same scene to the first neural network model to determine first backbone feature data corresponding to each input data, individually applies the two or more input data to the second neural network model to determine second backbone feature data corresponding to each input data, determines first convolution data and first view data based on dropouts based on projection from the first backbone feature data, determines second convolution data and second view data based on dropouts based on projection from the second backbone feature data, and trains at least one of the first neural network model and the second neural network model based on a loss determined based on two or more combinations of the first convolution data, the first view data, the second convolution data, the second view data, and code values ​​indicating the cluster to which the first convolution data belongs.

[0024] A method implemented by a processor for neural network training according to one embodiment includes the steps of: determining first backbone feature data and second backbone feature data by individually applying first input data and second input data of different views of the same scene to a first neural network and a second neural network, respectively; determining first projection-based convolutional data and first dropout-based view data based on the first backbone feature data, and second projection-based convolutional data and second dropout-based view data based on the second backbone feature data; determining a first partial loss based on the first convolutional data and a code value indicating one or more clusters to which the first convolutional data belongs; determining a second partial loss based on the second convolutional data and the code value; and training at least one of the first neural network and the second neural network based on the first partial loss and the second partial loss.

[0025] The code value may include a first code value indicating the cluster to which the convolutional data of the first convolutional data belongs, and a second code value indicating the cluster to which other convolutional data of the first convolutional data belongs.

[0026] The step of determining the second partial loss may include the steps of determining a first cross loss between the convolution data of the second convolution data and the second code value, and determining a second cross loss between other convolution data of the second convolution data and the first code value.

[0027] The step of determining the first partial loss may include the steps of determining a first cross loss between the convolution data of the first convolution data and the second code value, and determining a second cross loss between other convolution data of the first convolution data and the first code value. [Effects of the Invention]

[0028] According to the present invention, neural network training can be provided. [Brief explanation of the drawing]

[0029] [Figure 1] This is a flowchart illustrating a training method according to one embodiment. [Figure 2] This figure shows an example of a training system according to one embodiment. [Figure 3] This is a diagram illustrating a training action according to one embodiment. [Figure 4] This is a diagram illustrating a projection model according to one embodiment. [Figure 5] This figure illustrates a drop model according to one embodiment. [Figure 6] This figure illustrates the training operation of multiple neural network models according to another embodiment. [Figure 7] This figure illustrates a training operation using data collected by heterogeneous sensors according to a further embodiment. [Figure 8] This diagram illustrates a training operation corresponding to target hardware according to a further embodiment. [Figure 9] This figure illustrates the generation of inference results using a model that has been trained according to one embodiment. [Figure 10A] This figure illustrates the discriminative power of a neural network model trained according to one embodiment. [Figure 10B] This figure illustrates the discriminative power of a neural network model trained according to one embodiment. [Figure 10C] This figure illustrates the discriminative power of a neural network model trained according to one embodiment. [Figure 10D] This figure illustrates the discriminative power of a neural network model trained according to one embodiment. [Figure 11] This graph illustrates the accuracy of a neural network model trained according to one embodiment. [Figure 12] This block diagram shows the configuration of an electronic device according to one embodiment. [Modes for carrying out the invention]

[0030] The specific structural or functional descriptions disclosed herein are illustrative for the purpose of illustrating embodiments, and embodiments can be carried out in a variety of different forms. The present invention is not limited to the embodiments described herein, and the scope of the present invention includes modifications, equivalents, or substitutions that are included in the technical ideas described in the embodiments.

[0031] Terms such as "first" or "second" may be used to describe multiple components, but such terms should be interpreted solely for the purpose of distinguishing one component from others. For example, the first component may be named the second component, and similarly, the second component may also be named the first component.

[0032] When it is mentioned that one component is “linked” or “connected” to another component, it should be understood that it is directly linked to or connected to the other component, but that other components may be present in between.

[0033] A singular expression includes plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “has” indicate the presence of features, figures, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood not to presuppose the existence or addition of one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0034] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this embodiment belongs. Commonly used, predefined terms should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as ideal or overly formal unless expressly defined herein.

[0035] The embodiments will be described in detail below with reference to the attached drawings. In the description with reference to the attached drawings, the same reference numerals will be used for the same components regardless of the reference numerals in the drawings, and redundant explanations for them will be omitted.

[0036] Figure 1 is a flowchart illustrating a training method according to one embodiment.

[0037] First, in step S110, the electronic device calculates first backbone feature data from input data for the same scene. In one embodiment, the electronic device can calculate first backbone feature data corresponding to each input data by individually applying two or more input data for the same scene to the first neural network model. For example, the electronic device can extract m first backbone feature data by individually inputting m input data into the first neural network model, where m is an integer of 2 or more. The two or more input data may be data relating to the same context, for example, data augmented from one original image for one scene and / or data capturing the same scene with different sensors. As a different example, the two or more input data may be data augmented from one original signal for a certain sound (e.g., speech) and / or data capturing the same sound with different sensors.

[0038] Then, in step S120, the electronic device calculates second backbone feature data from the input data for the same scene. In one embodiment, the electronic device can calculate second backbone feature data corresponding to each input data by individually applying two or more input data to the second neural network model. The electronic device can extract m pieces of second backbone feature data by individually inputting m pieces of input data into the second neural network model.

[0039] A machine learning model is generated through machine learning and may also be referred to as a neural network model. Such learning may be performed, for example, on the electronic device on which the neural network model is executed, or via a separate server. Learning algorithms may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. A neural network model may include multiple artificial neural network layers. A neural network model may be, but is not limited to, a deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted Boltzmann machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of these. In addition to a hardware structure, a neural network model may also include a software structure, either additionally or alternatively. An electronic device can extract backbone feature data by inputting and propagating input data into a neural network model. The backbone feature data, as an abstraction of the input data, may exemplify the form of feature vectors. The backbone feature data can be propagated to subsequent layers of the neural network model to generate inference results corresponding to target tasks (e.g., object recognition, object classification, identity verification, object detection, and language recognition).

[0040] Next, in step S130, the electronic device calculates convolution data and view data from each backbone feature data. In one embodiment, the electronic device calculates first convolution data based on projection and first view data based on dropout from first backbone feature data, and calculates second convolution data based on projection and second view data based on dropout from second backbone feature data. The first convolution data and first view data may, exemplarily, represent feature vectors convolved in the first convolution space of a first neural network model, and the second convolution data and second view data may, exemplarily, represent feature vectors convolved in the second convolution space of a second neural network model. Projection and dropout will be explained with reference to Figures 4 and 5 below, respectively.

[0041] As described above, an electronic device can calculate feature data for various views from data from the same source. For example, m input data may be acquired for a single context (e.g., a scene), and for each of the m input data, backbone feature data may be extracted for each neural network model, and convolutional data and view data may be calculated for each backbone feature data. If the electronic device includes n neural network models, it can determine feature data for m × n × 2 views, where n may be an integer greater than or equal to 2. In this specification, a view indicates the perspective from which the data is being viewed.

[0042] Then, in step S140, the electronic device calculates the loss and performs training. In one embodiment, the electronic device can train at least one of the first and second neural network models based on the loss calculated based on two or more combinations of the first convolutional data, the first view data, the second convolutional data, the second view data, and the clustering results of the convolutional data. For example, the electronic device can update the parameters of the neural network model so that the sum of the partial losses calculated based on the aforementioned first convolutional data, first view data, second convolutional data, and second view data and the clustering results is minimized. For example, the electronic device can update the parameters of the second neural network model until the calculated loss is less than or converges to a threshold loss. The electronic device can repeatedly update the parameters of the neural network model until the calculated loss is less than a threshold.

[0043] An electronic device according to one embodiment can train a neural network model even without a label indicating the class to which the input image belongs. The electronic device can train the neural network model so that the distance in convolution space between extracted feature data increases, and the distance in convolution space between extracted feature data from the same class (e.g., the same scene) increases, by implementing the neural network model on data from different classes (e.g., different scenes) using a method described later. For reference, in order to generate inference results for the target task, post-processing and / or additional network training may be required in addition to the training operation described above.

[0044] Recently, unsupervised visual representation learning has gained attention due to improvements such as large-batch training and the use of heavy networks. Recent methods, such as ResNet-50, have drastically reduced the gap between supervised and unsupervised performance of deep models, but this is relatively limiting for smaller models.

[0045] This specification provides an unsupervised learning framework for small networks that integrates deep self-supervised representation learning and knowledge distillation in a single training process. The teacher model is trained to generate consistent clustering between different views of the same image. Simultaneously, the student model can be trained to mimic the predictions of the on-the-fly self-supervised teacher.

[0046] For effective knowledge transfer, domain classifiers can guide the feature learning of student models with distinctive features that are invariant to distribution shifts. During periods of dropout, multiview generation methods can help capture useful feature information contained within the network itself.

[0047] In a wide range of tests, student models trained by one embodiment are not only state-of-the-art but can also surpass state-of-the-art offline distilled networks from even more powerful self-directed teachers. For example, using the training method described later, a ResNet-18 student model trained with a ResNet-50 teacher model achieved an ImageNet Top-1 accuracy of 68.32% in linear evaluation, which is only 1.44% lower than the supervised baseline.

[0048] In recent years, a growing interest in both unsupervised and self-supervised learning is the efficient acquisition of useful representations from large amounts of unlabeled data. Current self-supervised visual representation learning methods are approaching, or even surpassing, fully supervised methods.

[0049] A significant part of recent success in unsupervised visual representation learning lies in the use of large models trained on powerful computing resources. The smallest model in previous research was ResNet-50, where the network may be trained in large batches (e.g., 4096 images) on multiple specialized hardware devices such as 128 TPU cores. However, this heavy embodiment is not a viable option in resource-constrained environments, and powerful small networks are required for real-world applications. Existing methods for self-supervised learning aim to train deep models more efficiently on general-purpose visual representations using unlabeled data. Furthermore, it has been empirically verified that the predictive performance of larger networks is well-transmitted to smaller networks.

[0050] An electronic device according to one embodiment can enable a lightweight network to learn rich representations by transferring knowledge from a deep self-supervised network without independently training the small network.

[0051] In unsupervised representation learning grammars, most previous distillation methods can be seen as offline training, in that they leverage an already trained self-supervised teacher model to transfer feature information to a student model. Furthermore, this sequential training pipeline often requires post-processing such as k-means clustering to improve performance. In contrast to this access, an electronic device according to one embodiment can use an unsupervised representation learning framework for small networks that combines self-supervised training and knowledge distillation into a single training process. Exemplarily, the teacher model may learn clusters and representations, while simultaneously, the student model may be trained to align with the on-the-fly self-supervised (OSS) teacher model's predictions for clustering. A domain classifier can guide the training of the student model with useful distinguishing features that are invariant to the distribution shift between the teacher and student models. Empirically, increasing the number of different views during self-supervised training results in improved feature quality. There are many accesses for generating multiviews, and most rely on random image transformation schemes. This specification describes training methods for utilizing the rich feature information contained within the network itself.

[0052] The following describes in detail a training method according to one embodiment.

[0053] Figure 2 shows an example of a training system according to one embodiment.

[0054] A training system 200 according to one embodiment includes an input data acquisition unit 210, a first neural network model 221, a second neural network model 222, a first view generation model 231, a second view generation model 232, a clustering unit 240, a first loss calculation unit 251, a second loss calculation unit 252, and a third loss calculation unit 253. However, these are modules shown for the convenience of explanation, and the operation of each configuration of the training system 200 may be performed by a processor.

[0055] The input data acquisition unit 210 acquires input data. For example, the input data acquisition unit 210 generates first input data 211 to the mth input data 212 from the original data 201. The input data acquisition unit 210 can generate various input data by applying different data augmentations to the original data 201. Data augmentation refers to the operation of generating input data that corresponds to different data representations, in other words, various views, from a single sample of the original data 201. For example, the input data acquisition unit 210 can generate two or more input data from the original data 201, which is a color image acquired by a camera sensor, based on data augmentation that includes one or more combinations of random noise addition, image rotation, scaling, image movement, random cropping, and color distortion.

[0056] The first neural network model 221 and the second neural network model 222 are trainable neural network models that can be designed and trained to output backbone feature data from input data. The number of parameters in the first neural network model 221 may be greater than the number of parameters in the second neural network model 222. The first neural network model 221 exhibits relatively higher performance than the second neural network model 222. The second neural network model 222 may be used to generate inference results after training is complete. In this specification, the first neural network model 221 may be referred to as the teacher model, and the second neural network model 222 as the student model, and so on.

[0057] The first view generation model 231 can diversify the view of the first backbone feature data output from the first neural network model 221. For example, the first view generation model 231 includes a first projection model and / or a first drop model, and the electronic device can calculate the first convolution data by projecting the first backbone feature data through the first projection model, and calculate the first view data by projecting the first backbone feature data using the first drop model. The first drop model may include at least one dropout layer. When the electronic device propagates data to the first view generation model 231, it may operate as the first projection model by excluding the dropout layer from the propagation path, or as the first drop model by including the dropout layer in the propagation path. The first convolution data may be data projected to a dimension lower than the dimension of the first backbone feature data, and the first view data may be data in which some features have been dropped out.

[0058] The second view generation model 232 can diversify the view of the second backbone feature data output from the second neural network model 222. For example, the second view generation model 232 includes a second projection model and / or a second drop model, and the electronic device can calculate the second convolution data by projecting the second backbone feature data through the second projection model, and calculate the second view data by projecting the second backbone feature data using the second drop model. The second drop model may include at least one dropout layer. The second convolution data may be data projected at a lower dimension than the second backbone feature data, and the second view data may be data with some features dropped out.

[0059] The first and second projection models mentioned above will be explained with reference to Figure 4 below, and the first and second drop models will be explained with reference to Figure 5 below.

[0060] The clustering unit 240 can determine the cluster to which the convolutional data belongs. For example, the number of elements (e.g., number of clusters) in the prototype cluster vector C is defined as K, and the clustering unit 240 may determine a code value that indicates the cluster to which the convolutional data belongs among the K clusters. Here, K may be an integer greater than or equal to 1. For example, the clustering unit 240 can generate clustering results using the Sinkhorn-Knopp algorithm under equivalent partitioning conditions.

[0061] The first loss calculation unit 251 calculates a first partial loss 291 from the loss 290. The first partial loss 291 may include the cross loss between the probability value of the cluster vector for the convolutional data and / or view data calculated from the first neural network model 221 and the cluster to be assigned.

[0062] The second loss calculation unit 252 calculates the second partial loss 292 from the loss 290. The second partial loss 292 may include the cross-loss between the probability value of the cluster vector for the convolutional data and / or view data calculated from the second neural network model 222 and the cluster to which it is assigned.

[0063] The third loss calculation unit 253 calculates the third partial loss 293 from the loss 290. The third partial loss 293 may include the adversarial loss between the first convolutional data calculated based on the first neural network model 221 and the second convolutional data calculated based on the second neural network model 222. The smaller the adversarial loss, the smaller the difference between the feature distribution of the first neural network model 221 and the feature distribution of the second neural network model 222.

[0064] In unsupervised visual representation learning, rich feature information is used to solve various pretext tasks in which labels are obtained from the image data itself. This method is categorized into various types depending on the type of pretext task. Training according to one embodiment is based on clustering.

[0065] The performance of self-supervised learning is improved by increasing the number of views during training. In one embodiment, training can acquire data from various views of the first backbone feature data of the first neural network model 221 and the second backbone feature data of the second neural network model 222 based on dropout, in addition to data augmentation of the original data 201 as described above.

[0066] In the knowledge distillation from a self - supervised model, in one embodiment, the training utilizes the clustering results for the teacher model to train the student model so as to mimic the relative similarity between data points in the convolutional space of the teacher model, enabling the training of the neural network model in a self - supervised manner without the provision of label information.

[0067] Also, an electronic device according to one embodiment can train a neural network model by making it difficult to distinguish the shift between the domain of the teacher model and the domain of the student model using a domain classifier having a gradient reversal layer. The electronic device can train discriminative features that are invariant to the distribution change between the teacher model and the student model. In FIG. 3 below, an example where the first neural network model is the teacher model and the second neural network model is the student model will be described.

[0068] FIG. 3 is a flowchart for explaining a training operation according to one embodiment.

[0069] In a training system 300 according to one embodiment, an electronic device calculates a loss according to the following formula (1) to train a teacher model (e.g., ResNet - 50) f , , , , and a student model (e.g., ResNet - 18) g η To calculate the loss according to the following formula (1).

Equation

[0070] For example, original image x n By applying data augmentation to the input image x n1 , x n2 It is also possible to generate the following. As mentioned above, the input images x1 and x2 are the original data, the original image x n The input data is shown from different views. The electronic device generates the input image x n1 , x n2 Teacher model f θ and student model g η Feedforwarding can be performed on the following models as shown in equations (2) and (3) below. θ The number of parameters (e.g., number of layers, number of nodes, and / or number of linked weight values) is determined by the student model g η It may be greater than the number of parameters of the training model f. θ The capacity of student models η It can be larger and deeper than its capacity.

number

number

[0071] The electronic device uses the aforementioned first backbone feature data f n1 ,f n2 This is propagated to the first view generation model 331, and the second backbone feature data g n1 , g n2 This can be propagated to the second view generation model 332. For example, the electronic device can use the projection model h of the first view generation model 331. T Based on this, the first backbone feature data f n1 ,f n2 From there, the first convolution data z is obtained as shown in equation (4) below. n1 , z n2 The electronic device calculates the projection model h of the second view generation model 332. S Based on the second backbone feature data g n1 , g n2 From there, the second convolution data w is obtained as shown in the following formula (5). n1 , w n2 Calculate.

number

number

[0072] An electronic device can determine the clustering result for convolutional data. For example, an electronic device may determine which cluster to which the convolutional data belongs among several clusters. Prototype cluster vector C = {c1, ..., c K Defined as}, where K indicates the number of clusters indicated by the prototype cluster vector C. An electronic device can determine which cluster of prototype cluster groups the convolutional data belongs to using the Sinkhorn-Knopp algorithm under equipartition conditions. For example, an electronic device can determine which cluster the convolutional data belongs to using the training model f θ Embedding representation of, for example, convolutional data z n By mapping this with the prototype cluster vector C, the code vector q n Calculate the code vector q. n This may include a code value indicating the cluster assigned to each convolution data. In Figure 3, the electronic device is the training model f θThe first convolution data z calculated based on n1 The code value q indicates the cluster to which it belongs. n1 , First convolution data z n2 The code value q indicates the cluster to which it belongs. n2 It is possible to make a decision.

[0073] An electronic device according to one embodiment exchanges the aforementioned clustering results between input data of various views, similar to SwAV (Swapping Assignments between Views), to create a training model f θ The loss can be predicted. The electronic device calculates the first partial loss L based on a code value that indicates the cluster to which the convolutional data extracted from any one of the two or more input data based on the first neural network model belongs, and the first convolutional data extracted from the other input data based on the first neural network model. T It is possible to calculate the code value q of an electronic device as shown in formula (6) below. n2 and the first convolution data z n1 Cross loss and code value q n1 and the first convolution data z n2 The cross-loss can be calculated.

number

number

number

[0074] The electronic device according to an embodiment can predict the loss of the student model g θ by exchanging the above-described clustering results between input data of various views similar to the cross-loss for the teacher model f η . The electronic device can calculate a code value indicating the cluster to which the first convolution data extracted based on the first neural network model from any one of two or more input data belongs, and a second partial loss L s based on the second convolution data extracted based on the second neural network model from other input data among the two or more input data. For example, the electronic device can calculate the cross-loss between the code value q n2 and the second convolution data w n1 , and the cross-loss between the code value q n1 and the second convolution data w n2 as shown in the following formula (9).

number

[0075] The electronic device has a first partial loss L related to the formula (6) mentioned above. t and the second partial loss L related to formula (9) s To collaboratively minimize the teacher model f θ Parameters and student model g η You can update the parameters.

[0076] In non-trivial online clustering, multiple codes q n However, as mentioned above, it can be calculated using the Sinkhorn-Knopp algorithm under the equi-partition constraint for a mini-batch containing a stored convolution of size B.

number

number

number

[0077] Furthermore, the electronic device has the aforementioned first partial loss L t and second partial loss L s Add a loss that reflects dropout. For example, the electronic device further adds a first partial loss L based on view data and clustering results extracted using a first neural network model from any one of two or more input data. T The electronic device can calculate the first view data Z calculated based on the dropout. ~ n1 , Z ~ n2 The loss between the clustering results and the results can be calculated using the following formula (13).

number

number

[0078] An electronic device according to one embodiment applies adversarial loss to the teacher model f θ and student model g η This can reduce the difference in feature distributions.

[0079] Domain classifier D may include an MLP (multi-layer perceptron) with an output magnitude of 2. The electronic device is based on the training model f θ For the data output from, set to 0, student model g η For the data output from the system, the cross-entropy loss between the feature label and the softmax probability of the domain classifier output can be calculated using the following equations (15) and (16).

number

number

[0080] The electronic device according to one embodiment is the first partial loss L described above. T , second partial loss L S , and third partial loss L D Based on this, the teacher model f θ student model g η The parameters of the remaining layers can be updated. Therefore, electronic devices may not need access to ground truth labels to efficiently learn the visual features of small networks online. As mentioned above, electronic devices can integrate knowledge distillation operations into the on-the-fly self-supervised learning process.

[0081] Figure 4 illustrates a projection model according to one embodiment. Figure 5 illustrates a drop model according to one embodiment.

[0082] According to one embodiment, the electronic device calculates first convolutional data projected from first backbone feature data using a first projection model. The electronic device calculates first view data projected from first backbone feature data using a first drop model including at least one dropout layer. The electronic device calculates second convolutional data projected from second backbone feature data using a second projection model. The electronic device calculates second view data projected from second backbone feature data using a second drop model including at least one dropout layer. Figure 4 illustrates the first projection model and the second projection model, and Figure 5 illustrates the first drop model and the second drop model.

[0083] For example, an electronic device may calculate convolutional data by applying projection model 430h to backbone feature data output from each neural network model (e.g., a teacher model and a student model). Projection model 430h may include a combination of a fully connected layer, a convolutional layer, an activation layer, and a normalization layer. In the example shown in Figure 4, projection model 430h may include a linear layer 431, a BN (batch normalization) layer 433, a ReLU (Rectified Linear Unit) layer 434, a linear layer 435, and a normalization layer 437. However, this is merely an example, and the configuration of projection model 430h is not limited to what is shown.

[0084] In one embodiment, the electronic device uses a drop model 530h on the backbone feature data output from each neural network model. dropBy applying this, view data can be calculated. View data represents data for a view different from the projected convolutional data. Drop model 530h drop This may be a model that further includes a dropout layer 532 in addition to the projection model 430h described above. For example, an electronic device applies linear layers 431 and 435 to the first convolution data, and then applies dropout layers 532 and 536. The electronic device applies linear layers 431 and 435 to the second convolution data, and then applies dropout layers 532 and 536. In Figure 5, the dropout model 530h is shown. drop Although it is shown as including, but not limited to, a dropout layer 532 preceding the activation layer ReLU layer 434 and a dropout layer 536 following the ReLU layer 434, the dropout layers 532 and 536 can generate data for various views.

[0085] Figure 6 illustrates the training operation of multiple neural network models according to another embodiment.

[0086] A training system 600 according to one embodiment includes n neural network models 221, 222, and 629. The training system 600 may also include view generation models 231, 232, and 639 connected to each neural network model. In Figure 6, the second neural network model 222 may be the student model, and the remaining first neural network model 221, the third neural network model, and the nth neural network model 629 may be the teacher models. However, it is not limited to this, and the number of parameters for each neural network model may differ.

[0087] The clustering unit 640 can cluster the convolutional data output from each of the view generation models 231, 232, and 639. The cross-loss calculation unit 650 can calculate the cross-loss in the same manner as the formula (14) described above. In formula (14), the code values ​​used to calculate the cross-loss are determined as follows. For example, the cross-loss calculation unit 650 can determine one of the following: a code value indicating the cluster to which the convolutional data output by the model with the highest recognition performance among a plurality of neural network models, including the first neural network model and the second neural network model, belongs; a code value for each convolutional data output by each model; and a code value indicating the cluster to which the convolutional data output by the model with the smallest clustering loss belongs. The cross-loss calculation unit 650 may calculate the cross-loss between the determined code value and the convolutional data and use it for backpropagation training.

[0088] The discrimination loss calculation unit 660 can calculate the adversarial loss between the convolutional data output from any one of the n neural network models and the convolutional data output from the other neural network models, similar to equation (15). Therefore, the discrimination loss calculation unit 660 can be trained to resemble the feature distribution of the convolutional space of the n neural network models.

[0089] The training system ensembles n neural network models, allowing the student model, the second neural network model 222, to learn from a wider variety of data views.

[0090] Figure 7 illustrates a training operation using data collected with heterogeneous sensors according to a further embodiment.

[0091] An electronic device according to one embodiment can generate one input data from any of two or more input data by capturing a scene using one of a plurality of sensors. The electronic device can generate different input data from any of two or more input data by capturing the same scene using different types of sensors from the plurality of sensors. For example, in the training system 700 shown in Figure 7, the electronic device may include a first sensor 701 and a second sensor 702. While the example described shows the first sensor 701 as a camera sensor and the second sensor 702 as a LiDAR sensor, the device is not limited to this. The first sensor 701 and the second sensor 702 may be different sensors from among image sensors, camera sensors, LiDAR sensors, radar sensors, and infrared sensors.

[0092] For example, when the training system 700 trains a neural network model using camera images and lidar images, it can input the camera images and lidar images, generated by capturing the same scene at the same time, into the first neural network model 221 and the second neural network model 222, respectively. The training system 700 inputs the first input data 211 generated from the camera image into the first neural network model 221. The training system 700 may also input the mth input data 212 generated from the lidar image into the second neural network model 222, where m may be 2.

[0093] The training system 700 can generate first input data 211 for various views by applying size deformation, color distortion, and random cropping to the camera image.

[0094] The training system 700 can generate two or more m-th input data 212 from the original data, which is a lidar image acquired by the lidar sensor, based on data augmentation that includes one or more combinations of detection point augmentation and reflectance value adjustment.

[0095] The first neural network model 221 is a model designed to extract feature vectors from an input image, and may be, for example, ResNet, MobileNet, and RegNet. The second neural network model 222 is a model designed to process point gray sequence data, which is a lidar image, and may be, for example, PointNet and LSTM (Long Short-Term Memory models). The remaining operation is the same as or similar to that described above with reference to Figures 2 and 3.

[0096] Figure 8 illustrates a training operation corresponding to target hardware according to a further embodiment.

[0097] In one embodiment, the training system 800 further includes a data conversion unit 820 and a view conversion unit 839. For example, the data conversion unit 820 can convert two or more input data based on a format defined by the target hardware.

[0098] Exemplary, training through view data generation, training of the teacher model, clustering, and loss calculation may be performed on a server and / or personal computer, while the second neural network model 822 for actual inference operation may be implemented on a lower-performance electronic device (e.g., a mobile terminal). In a different example, multiple neural network models targeting different computing environments may be trained together. For training neural network models on the various hardware environments described above, data transformation for the target hardware is required.

[0099] The data conversion unit 820 may convert the first input data 211 to the mth input data 212 to match the target hardware's computational basis data format (e.g., floating-point format and number of bits). The data conversion unit 820 inputs the converted data to the second neural network model 822. The view conversion unit 839 converts the convolutional data and view data output from the second view generation model 832 to match the computational basis data format of the training system 800 and can provide it to the clustering unit 240 and loss calculation. In addition, the data conversion unit 820 and the view conversion unit 839 may convert the data to match the characteristics of the target hardware based on special sensor features of the target hardware (e.g., color space difference, Bayer pattern image, etc.) and use it for training.

[0100] Figure 9 illustrates the generation of inference results using a model that has been trained according to one embodiment.

[0101] An electronic device according to one embodiment can perform inference operations using a trained neural network model. For example, the electronic device generates an inference result 909 of a target task for the detected data 901 using convolutional data extracted from the detected data 901 based on a trained second neural network model 920. The electronic device can propagate the convolutional data extracted from the second neural network model 920 to an output layer 929 to generate the inference result 909. The output layer 929 may be a layer having parameters designed and further trained to output the inference result 909 for the target task.

[0102] The detected data 901 may be data collected by a sensor, such as image data, and may be input to the second neural network model 920. The target tasks may include, but are not limited to, tasks related to autonomous driving or ADAS (Advanced Driver Assistance Systems), tasks related to IoT (Internet of Things) devices, tasks related to mobile phones, and tasks related to TVs.

[0103] The electronic device may control the operation of the electronic device storing the second neural network model 920 based on the inference result 909. For example, if the electronic device is mounted on a vehicle, the electronic device may estimate the distance between the vehicle and the object based on the object detection result and change at least one or more combinations of the vehicle's speed, acceleration, and steering according to the distance between the vehicle and the object. In a different example, if the electronic device is implemented as a mobile terminal, the electronic device may unlock and release its locked state if it is verified to be a pre-registered user based on the identity verification result. However, the control of the electronic device's operation is not limited to the above, and the electronic device may control various elements, modules, circuits, and / or functions accessible by the electronic device according to the inference result 909.

[0104] As previously mentioned with reference to Figures 1 to 8, the trained second neural network model shows improvements in various aspects of performance, such as accuracy, precision, recall, average precision, average recall, and average intersection for the union.

[0105] Figures 10A to 10D illustrate the discriminative power of a neural network model trained according to one embodiment.

[0106] Figure 10A shows the feature vector distribution 1000a during standalone training of a large model, Figure 10B shows the feature vector distribution 1000b during standalone training of a small model, Figure 10C shows the feature vector distribution 1000c of a teacher model according to one embodiment, and Figure 10D shows the feature vector distribution 1000d of a student model according to one embodiment.

[0107] Figures 10C and 10D show a wider distribution of feature vectors than Figure 10B, retaining features that are more distinctive to other classes. Furthermore, the feature vector distribution 1000d of the student model shown in Figure 10D is similar to the feature vector distribution 1000c of the teacher model, suggesting that knowledge distillation can enable the student model to achieve a similar level of accuracy to the teacher model.

[0108] Figure 11 illustrates the accuracy of a neural network model trained according to one embodiment. [Table 1] Table 1 above shows the Top-1 accuracy (%) of ResNet-18 models for each training method, illustrated as Graph 1100 in Figure 11. As shown in Graph 1100 in Figure 11, the student model according to one embodiment (e.g., OSS) shows excellent accuracy across all epochs. The training according to one embodiment is conceptually simple yet can demonstrate considerable recognition performance even with a typical 256-batch training on a single 8-GPU machine.

[0109] One embodiment of the training device allows for the simultaneous training of multiple neural network models that extract feature vectors or feature maps from unlabeled data via self-supervised learning, enabling each neural network to be trained to produce meaningful and rich representations. Furthermore, the training device according to one embodiment can train the neural network models to output feature vectors with higher discriminative power than when a single network model is trained. The training device can extract feature vectors that demonstrate high recognition accuracy with smaller neural network models and fewer computing resources.

[0110] Figure 12 is a block diagram showing the configuration of an electronic device according to one embodiment.

[0111] A neural network training device (e.g., an electronic device) 1200 according to one embodiment includes a processor 1210 and a memory 1220.

[0112] Processor 1210 individually applies two or more input data for the same scene to the first neural network model and calculates first backbone feature data corresponding to each input data. Processor 1210 individually applies two or more input data to the second neural network model and calculates second backbone feature data corresponding to each input data. From the first backbone feature data, Processor 1210 calculates first convolution data based on projection and first view data based on dropout. From the second backbone feature data, Processor 1210 calculates second convolution data based on projection and second view data based on dropout. Based on the loss calculated from two or more combinations of the first convolution data, first view data, second convolution data, second view data, and code values ​​indicating the cluster to which the first convolution data belongs, Processor 1210 can train at least one of the first and second neural network models. However, the operation of the processor 1210 is not limited to this, and it can also perform the operations described above, as shown in Figures 1 to 11.

[0113] Memory 1220 stores the first neural network and the second neural network. Memory 1220 may also temporarily and / or permanently store data required for training according to one embodiment. Memory 1220 can store a temporary neural network model before training is complete, and can also store the completed neural network model.

[0114] The embodiments described above are embodied in hardware components, software components, or combinations of hardware and software components. For example, the devices and components described in these embodiments are embodied using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPA (field programmable array), PLU (programmable logic unit), microprocessor, or different devices that execute and respond to instructions. The processing device can run an operating system (OS) and one or more software applications run on the OS. The processing device can also access, store, manipulate, process, and generate data in response to software execution. For convenience of understanding, the processing device may sometimes be described as being used as a single unit, but a person with ordinary skill in the art will see that the processing device includes multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0115] Software includes computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or can instruct the processing unit independently or in combination. Software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, in order to be interpreted by a processing unit or to provide instructions or data to a processing unit. Software can be distributed, stored in a distributed manner, or executed on a networked computer system. Software and data can be stored on a recording medium readable by one or more computers.

[0116] The methods according to the embodiments are embodied in the form of program instructions that are implemented via various computer means and recorded on a computer-readable recording medium. The recording medium includes program instructions, data files, data structures, etc., individually or in combination. The recording medium and program instructions may be specifically designed and configured for the purposes of the present invention, or they may be known and usable by those skilled in the art who have technology in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code performed by a computer using an interpreter or the like.

[0117] The hardware device described above may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.

[0118] As described above, embodiments have been illustrated with limited drawings, but a person with ordinary skill in the art can apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or assembled in a different manner than described, or replaced or substituted with other components or equivalents, and still achieve suitable results.

[0119] Therefore, the scope of the present invention is not limited to the disclosed embodiments, but is defined by the claims and equivalents thereof. [Explanation of symbols]

[0120] 200, 300 training systems 430 Projection Model 530 Drop Model 600, 700, 800 Training System Distribution of 1000 feature vectors 1200 Neural Network Training Devices

Claims

1. A neural network training method implemented on a processor, The steps include: applying two or more input data for the same scene individually to the first neural network model to determine the first backbone feature data corresponding to each input data; The steps include individually applying the two or more input data to the second neural network model and determining the second backbone feature data corresponding to each input data, The steps include calculating first convolution data based on projection and first view data based on dropout from the first backbone feature data, and determining second convolution data based on projection and second view data based on dropout from the second backbone feature data, A step of training at least one of the first neural network model and the second neural network model based on a loss determined based on two or more combinations of the first convolutional data, the first view data, the second convolutional data, the second view data, and the clustering results of the convolutional data, It includes, and further, The steps include determining the first view data projected from the first backbone feature data using a first drop model that includes at least one dropout layer, The steps include determining the second view data projected from the second backbone feature data using a second drop model that includes at least one dropout layer, Includes, The step of determining the first view data includes the step of applying a linear layer to the first convolution data, and then applying a dropout layer, The step of determining the second view data includes the step of applying a linear layer to the second convolution data, and then applying a dropout layer. method.

2. The aforementioned training step is, The process includes the step of determining a first partial loss based on a code value indicating the cluster to which the convolutional data of the first convolutional data, extracted from any one of the two or more input data based on the first neural network model, belongs, and the convolutional data of the first convolutional data, extracted from the other input data among the two or more input data based on the first neural network model. The method according to claim 1.

3. The step of determining the first partial loss is: The process includes determining the first partial loss based on the view data of the first view data extracted from any one of the two or more input data using the first neural network model, and the clustering results. The method according to claim 2.

4. The aforementioned training step is, The process includes the steps of determining a second partial loss based on the convolutional data of the second convolutional data extracted from any one of the two or more input data based on the first neural network model, which indicates the cluster to which the convolutional data of the first convolutional data belongs, and determining a second partial loss based on the convolutional data of the second convolutional data extracted from the other input data among the two or more input data based on the second neural network model. The method according to claim 1.

5. The step of determining the second partial loss is: The step of determining the second partial loss based on the view data of the second view data extracted from any one of the two or more input data based on the second neural network model, and the code value, is included. The method according to claim 4.

6. The aforementioned training step is, The process includes the step of determining a first partial loss for the first neural network model and a second partial loss for the second neural network model based on the cross-entropy loss between one of the first convolution data, the first view data, the second convolution data, and the second view data and the code value. The method according to claim 1.

7. The training step includes determining a third partial loss from the first and second convolutional data using a gradient reverse layer (GR layer). The method according to claim 1.

8. The process further includes the step of generating two or more input data from original data, which is a color image acquired by a camera sensor, based on data augmentation including one or more combinations of random noise addition, image rotation, scaling, image movement, random cropping, and color distortion. The method according to claim 1.

9. The process further includes the step of generating two or more input data from original data, which is a lidar image acquired by a lidar sensor, based on data augmentation including one or more combinations of detection point enhancement and reflection value adjustment. The method according to claim 1.

10. The above method further, The steps include: generating one of the two or more input data by capturing a scene using one of the multiple sensors; The steps include generating other input data from the two or more input data by capturing the same scene using a different type of sensor among the multiple sensors, The method according to claim 1, including the method described in claim 1.

11. The above method further, The steps include determining the first convolutional data projected from the first backbone feature data using the first projection model, The steps include determining the second convolutional data projected from the second backbone feature data using the second projection model, including, The method according to claim 1.

12. The number of parameters in the first neural network model is greater than the number of parameters in the second neural network model. The method according to claim 1.

13. The above method further, The method includes the step of determining one of the following: a code value indicating the cluster to which the convolutional data output by the model having the highest recognition performance among a plurality of neural network models, including the first neural network model and the second neural network model, the code value for each convolutional data output by each model, and the code value indicating the cluster to which the convolutional data output by the model with the smallest clustering loss belongs. The method according to claim 1.

14. The step of determining the two backbone feature data mentioned above is: The steps include converting the two or more input data based on a format defined by the target hardware, The steps include inputting the converted data into the second neural network model, including, The method according to claim 1.

15. The training step includes updating the parameters of the second neural network model so that the determined loss is minimized. The method according to claim 1.

16. The aforementioned method, The process includes the step of generating an inference result for a target task on the detection data using convolutional data extracted from the detection data based on the second neural network model that has completed the training, The method according to claim 1.

17. The aforementioned method, The process includes the step of controlling the operation of an electronic device storing the second neural network model based on the inference results, The method according to claim 16.

18. The aforementioned inference results correspond to one or a combination of object recognition, object classification, identity verification, object detection, and language recognition. The method according to claim 16.

19. A computer-readable recording medium storing one or more computer programs including instruction words for performing the method described in any one of claims 1 to 18.

20. A neural network training device, A memory for storing the first neural network model and the second neural network model, It is a processor, Two or more input data for the same scene are individually applied to the first neural network model to determine the first backbone feature data corresponding to each input data. The two or more input data are individually applied to the second neural network model to determine the second backbone feature data corresponding to each input data. Based on the projection-based first convolution data and dropout from the first backbone feature data, the first view data is determined. Based on the projection-based second convolution data and dropout from the aforementioned second backbone feature data, the second view data is determined. Based on the loss determined from two or more combinations of the first convolutional data, the first view data, the second convolutional data, the second view data, and the code value indicating the cluster to which the first convolutional data belongs, at least one of the first neural network model and the second neural network model is trained, and further, The first view data projected from the first backbone feature data is determined using a first drop model that includes at least one dropout layer. The second view data projected from the second backbone feature data is determined using a second drop model that includes at least one dropout layer, and further, The step of determining the first view data involves applying a linear layer to the first convolution data, and then applying a dropout layer. The step of determining the second view data involves applying a linear layer to the second convolution data, and then applying a dropout layer. Processor and A device including a device.

21. A method implemented in a processor for neural network training, The first and second neural network models are individually applied to the first and second neural network models, respectively, to determine the first and second backbone feature data by applying first and second input data of different views of the same scene. The steps include determining first projection substrate convolution data and first dropout substrate view data based on the first backbone feature data, and second projection substrate convolution data and second dropout substrate view data based on the second backbone feature data, A step of determining a first partial loss based on the first projection substrate convolution data and a code value indicating one or more clusters to which the first projection substrate convolution data belongs, The steps include determining a second partial loss based on the second projection substrate convolution data and the code value, A step of training at least one of the first neural network model and the second neural network model based on the first partial loss and the second partial loss, It includes, and further, The steps include determining first view data projected from the first backbone feature data using a first drop model that includes at least one dropout layer, The steps include determining second view data projected from the second backbone feature data using a second drop model that includes at least one dropout layer, Includes, The step of determining the first view data includes the step of applying a linear layer to the first projection substrate convolution data, and then applying a dropout layer. The step of determining the second view data includes applying a linear layer to the second projection-based convolution data, and then applying a dropout layer. method.

22. The code value includes a first code value indicating the cluster to which the convolutional data of the first projection substrate convolutional data belongs, and a second code value indicating the cluster to which other convolutional data of the first projection substrate convolutional data belongs. The method according to claim 21.

23. The step of determining the second partial loss is: The steps include determining a first cross-loss between the convolution data of the second projection substrate convolution data and the second code value, The steps include determining a second cross-loss between the second projection substrate convolution data and the first code value, The method according to claim 22, including the method described in claim 22.

24. The step of determining the first partial loss is: The steps include determining a first cross-loss between the convolution data of the first projection substrate convolution data and the second code value, The steps include determining a second cross-loss between the first projection substrate convolution data and other convolution data and the first code value, The method according to claim 22, including the method described in claim 22.

Images