Methods for configuring neural network models

Abstract

The present invention relates to a computer-implemented method (100) for configuring a neural network model, wherein the method comprises the following steps: providing (102) a neural network model; dividing (104) the neural network model into a first part and a second part, the second part comprising a first head for classifying a first type of classification data and a second head for classifying a second type of classification data; in a training phase, preprocessing (106) the second type of classification data in the first part, processing (108) the preprocessed second type of classification data in the first head and the second head, and determining a first result of processing the first type of classification data in the first head and a second result of processing the first type of classification data in the second head; calculating (110) consistency between the first result and the second result; and configuring (112) the neural network model by updating at least one parameter value of the second head based on the calculated consistency.

Description

Technical Field

[0001] The invention relates to a computer-implemented method for configuring a neural network model, a neural network, a neural network use for configuring a neural network model, a computer program element and a computer-readable medium. Background Art

[0002] Domain adaptation is a field of machine learning that aims to exploit knowledge from a given domain, called the source domain, by transferring said knowledge to another domain, called the target domain. This is a special case of so-called transfer learning, where the given task is the same for both domains, but the target dataset may contain a distribution shift compared to the source dataset. An example of this problem is object recognition in daytime and nighttime images of vehicle surroundings, for example.

[0003] A common learning approach in the source and target domains is to try to modify the internal representation of the data so that they are as similar as possible for both domains. The result of using this approach is that the target labels and source labels are evenly mixed. This approach is also called naive mixing. However, the naive mixing of target and source labels is problematic because it does not indicate to the network during training that the target is more important than the source. For example, due to certain constraints, whenever a specific network must choose its label pattern to remember, it cannot prioritize the target label pattern. In extreme cases where multiple source labels are available, this can severely affect the accuracy of the target. Summary of the invention

[0004] Therefore, it is required to improve the accuracy in transfer learning.

[0005] This object is achieved by the subject matter of the attached independent claims. Embodiments are provided by the dependent claims and the following description and the drawings.

[0006] The embodiments similarly relate to a method for configuring a neural network model, a system for configuring a neural network model, a computer program unit, and a computer readable memory. Although it may not be described in detail, synergistic effects may be generated by different combinations of the embodiments.

[0007] Furthermore, it must be noted that all embodiments of the present invention relate to a method that can be implemented in the described order of steps, however, this is not necessarily the only and necessary order of steps for the method. Unless otherwise explicitly stated below, the methods described herein can be performed in an order other than the published order of steps without departing from the respective method embodiments.

[0008] Technical terms are used according to their common sense. If a particular term expresses a specific meaning, the definition of the term is given below in the context in which the term is used.

[0009] According to a first aspect, a computer-implemented method for classifying a second type of categorical data in a neural network is provided. The method comprises the following steps. In a first step, a neural network model is provided. The neural network model is divided into a first part and a second part, wherein the second part comprises a first head for classifying a first type of categorical data and a second head for classifying a second type of categorical data. In a training phase, the first type of categorical data is preprocessed in the first part. Then, such preprocessed second type of categorical data is processed in the first head and the second head, and a first result of the processing of the second type of categorical data in the first head is determined, and a second result of the processing of the second type of categorical data is determined in the second head.

[0010] In a further step, the consistency between the first result and the second result is calculated, and based on the neural network update and the calculated consistency, the neural network model is configured by at least one parameter value of the second head and at least one parameter value of the first head.

[0011] The method aims to classify a second type of classified data using the knowledge of a neural network about the classification of a first type of classified data. In the present application, the invention is explained using a neural network, which can use any artificial intelligence module that can be divided into a first part and a second part. Therefore, the term "neural network" means such a module for artificial intelligence.

[0012] In this patent document, "classify" or "classification" are interpreted in a broad sense. They include, for example, object detection, semantic segmentation, audio recognition, key point recognition, and other applications of machine learning. Therefore, "classification data" includes the basic input data for the classification process and characterizes the data to be classified according to the above interpretation.

[0013] The neural network model can be a convolutional neural network, a recursive neural network or any other neural network. The neural network includes multiple layers, from an input layer, so-called hidden layers to an output layer. Depending on the type of network, the classified data must also pass through each of these layers (depending on the type of network, more than once), where they are transformed and finally classified. According to the method, these layers are divided into a first part, i.e. a first number of layers, and a second part, i.e. a second number of layers. The data are preprocessed in the first part, which means that the data are output to the second part through the first layer, where further final processing is performed.

[0014] The second part has two separate parallel branches, which are referred to in this patent document as the first head and the second head. Depending on the type of the classified data, i.e., the first type or the second type, the first head or the second head is responsible for classifying the classified data. In a training phase, the first type of classified data is input into the first part in a conventional manner, where it is preprocessed and then input into the first head and the second head for processing. The processing result in one of the heads, for example, indicates the category into which the head has classified the classified data. The results of the two heads are compared using mathematical methods such as statistics, functions, etc., and the settings or configurations of the second head are updated based on the comparison. More complicated is the calculation of the consistency between the first head and the second head, and the correction of one or more values ​​of the internal parameters of the second head or the first head based on the calculated consistency. The results provided by the two heads may be different because, for example, one of the heads has been trained in advance, or because the training data may be different. It is preferred to update each parameter in the network model.

[0015] According to one embodiment, the artificial intelligence module is embodied as a neural network, in particular as a convolutional neural network.

[0016] According to one embodiment, the first part may include an encoder layer, that is, a feature extraction layer of the neural network, and the second layer may include a classification layer of the neural network. In addition, the encoder layer may include an input layer of the neural network, and the classification layer may include an output layer of the neural network.

[0017] According to an embodiment, the first type of classified data is related to the second type of classified data. That is, the first type of data is similar to the second type of data. For example, the images show the same scene, but the first image data set was recorded during the day, while the second image data set was recorded at night. This characteristic illustrates the basis for knowledge transfer from the first domain to the second domain, so that during the training process, the head internal representations converge / converge / aggregate.

[0018] According to one embodiment, the neural network model provides a domain adaptation, and the first type of classification data is the source data and the second type of classification data is the target data. Therefore, the domains are adapted to each other during training, and the second header data, also referred to as the target label in this application, takes precedence over the first header data, also referred to as the source label in this application, based on consistency.

[0019] According to one embodiment, a training of the first part and a training of the first head are performed before a preprocessing step of the second type of classification data in the first part.

[0020] For example, the first head may have been trained with the first type of classified data in another training phase before the present training phase. In addition, during the current training phase, there are also labeled data used in a classical manner to update the configuration of the corresponding head. Since at least some features of the first and second data and the number of data sets differ from each other, the configurations, such as internal weights of the heads, may also differ, resulting in inconsistencies between the two heads for a given data set, such as an image or a plurality of images of an iteration.

[0021] If, for example, the consistency is good, then the influence on the determination of the update value of the second header is almost equal based on the values ​​of the first header and the second header. However, the consistency may be low, for example, because there are a lot of labeled source data, so the source data weights dominate and are mainly based on the influence of the first header on the determination of the update value of the second header. To avoid this, the proposed method prioritizes the second label, i.e., the target label, for example, by using the two headers corresponding to the calculated consistency together for the blending of the calculated update value, such as the weight vector.

[0022] In one example, the parameter value is updated by, for example, determining the consistency loss by a function, calculating a standard loss such as the output of the second head by comparing the correct classification with the wrong classification, and then calculating a weighted average of the two losses, as described below. For example, for each data set or each iteration, the calculation and update can be completed in one step.

[0023] According to an embodiment, a standard loss of the first head is calculated for the first head, and only when the standard loss of the first head is below a predetermined threshold, a step of updating the parameter value of the second head is performed based on the calculated consistency, so that the first head has a sufficient basis for a knowledge transfer to the second head with the standard loss as a criterion. This means that before starting a knowledge transfer from the source to the target, there is a sufficient source knowledge base in a beneficial way. The criterion for having a sufficient knowledge base is the standard loss according to this embodiment.

[0024] According to another example, the accuracy of the first head is calculated for the first head, and only when the accuracy of the first head is higher than a predefined threshold, the configuration step of the neural network is implemented by updating at least one parameter value of the second head and at least one parameter value of the first head based on the calculated consistency, so that the first head has a substantial basis, using accuracy as a criterion, and providing a knowledge transfer to the second head. Of course, standard loss and / or accuracy can also be used as a criterion. Other conditions can also be introduced, such as standard loss and / or accuracy must meet a number of period threshold conditions, etc.

[0025] According to one embodiment, the parameter values ​​to be updated in the first and / or second head are weights of the neural network, ie, internal weights characterizing each layer.

[0026] According to one embodiment, the second type of classification data is partly labeled and partly unlabeled, describing so-called semi-supervision. The labeled first type of classification data, or simply source data, is used to train the first head in the usual way. The labeled second type of classification data, or simply target data, is used to train the second head in the above-mentioned way. The unlabeled data can be classified, for example, based on the initial knowledge obtained about the labeled source data and target data and according to the unlabeled target data during the learning process. In addition, if no labeled target data is available, unsupervised learning can be performed. Learning about the target data starts with the initial knowledge of the labeled source data and continues with the knowledge gradually gained by the estimated classification of the unlabeled target data.

[0027] According to one embodiment, the outputs of the first head and the second head are classified, and the consistency is calculated based on Jensen-Shannon divergence, which is a method for measuring the similarity between two probability distributions. A softmax function or a sigmoid function is used for the classified output.

[0028] According to one embodiment, the outputs of the first head and the second head are continuous, and the consistency calculation is performed based on the mean square error.

[0029] According to one aspect, there is provided a neural network having a configuration for implementing a method as described above.

[0030] According to another aspect, there is provided a use of a neural network for detecting or classifying objects or features in an image of an advanced driver assistance system (ADAS) camera system or an automated driving (AD) camera system. The detected or classified objects or features may be used to control functions of an advanced driver assistance system (ADAS) or automated driving (AD) such as a lane keeping system for a vehicle, an adaptive cruise control system, an autonomous emergency braking system, a construction site assistance function, a traffic jam assistance function, a highway driver function, or an automated driving function. The neural network is intended to classify object patterns, detect a lane, an object, or a key point, for example, for semantic segmentation or classification of text on a traffic sign, or for pose estimation of, for example, a pedestrian. However, the computer-implemented method is not limited to these applications and may also include detection or classification of audio patterns or semantic objects.

[0031] The given method can be used without any problems for any machine learning method with two data sources. The method is most useful in the following cases: First, the target dataset is not sufficient to achieve the required performance, but a sufficiently large similar dataset is given. Areas where the method is applicable in a beneficial way are classification, object recognition, semantic segmentation, audio recognition, and keypoint identification.

[0032] In one example, a data processing system such as a computer is provided, the device being used to perform the steps of the method. The data processing system may include circuits without programmable logic, or may be a microcontroller, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a complex programmable logic device (CPLD) or any other programmable logic device of a type known to a person skilled in the art.

[0033] According to another aspect, a computer program element is provided which comprises instructions which, when executed by a computer, cause the computer to perform the steps of the method. The computer program element may be a part of a computer program, but it may also be a complete program itself. For example, the computer program element may be used to update an existing computer program in order to implement the invention.

[0034] According to another aspect, a computer readable memory comprising instructions is provided, which, when executed by a computer, enables the computer to perform the method steps. The computer readable memory can be regarded as a storage medium, such as a USB flash drive, an optical disk, a DVD, a data storage device, a hard disk or other medium that can store the above program unit.

[0035] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Exemplary embodiments of the present invention are described below with reference to the following drawings:

[0037] Figure 1 This figure shows a flowchart of a computer-implemented method for classifying target patterns in a neural network according to one embodiment;

[0038] Figure 2 This figure shows a block diagram of a neural network for classifying a target pattern in a neural network according to one embodiment.

[0039] Figure 3 This figure shows a first diagram of the classification probability before adaptation according to one embodiment,

[0040] Figure 4 This figure shows a second diagram of classification probabilities after adaptation according to one embodiment,

[0041] Figure 5 This figure shows a graphical representation of the effect of the method according to one embodiment compared with other methods.

[0042] The drawings are schematic diagrams only and serve only to illustrate embodiments of the invention. In principle, identical or equivalent parts have the same reference numerals. DETAILED DESCRIPTION

[0043] Figure 1 A flowchart of a computer-implemented method (100) for classifying a target pattern or target data in a neural network is shown, wherein the neural network is trained using a pattern that partially has a target pattern and partially has a source pattern or partially has source data. The method comprises the following steps: In step 102, a neural network model is provided. In step 104, the neural network model is divided into a first part and a second part, wherein the second part includes a first head for classifying a first type of classified data and a second head for classifying a second type of classified data. As an option, in step 105, the target head is trained to a certain degree first. Step 106 preprocesses the second type of classified data in the first part in a training phase. In step 108, the preprocessed second type of classified data is processed in the first head and the second head, and a first result of processing the first type of classified data in the first head and a second processing result of the first type of classified data processed in the second head are output. In step 110, the consistency between the first result and the second result is calculated. In step 112, based on the calculated consistency, the neural network model is configured by updating at least one parameter value of the second head and at least one parameter value of the first head.

[0044] Figure 2The proposed algorithm architecture is explained. A predefined neural network model implemented in a data processing system is decomposed into an encoder 206 and heads 202 and 204. The encoder 206 contains the layers of the neural network and performs feature recognition. Then, two head instances 202 and 204 are generated so that the source data 232 has a first head 202 and the target data 234 has a second head 204. The head instances 202 and 204 perform classification, wherein they use the results of the encoder 206 as input and provide the classification as output. If, in the next step, a label is available for a given data point, training is performed using a conventional training process. The source data 232 is routed through the encoder 206 and forwarded to the first head 202. The target data 234 is also routed through the encoder 206 and then forwarded to the second head 204. In both head rows, the same loss function is used to calculate the loss 212 or 214, respectively. In the next step, to calculate the consistency between the two, as shown by arrows 224 and 222, the source data 232 is routed via the first head 202 and the second head 204, and the consistency value between the outputs of the two heads is calculated. Similarly, the target data is routed via the first head 202 and the second head 204, and the consistency value between the outputs of the two heads is calculated. That is, both the conventional process and the consistency calculation provide updates, and the weighted average of the two updates is applied to the model parameters. For example, the following function can be used as a consistency function. Depending on the prediction type, if the output is defined as a categorical output, that is, the last activation is a softmax function or a Sigmoid function, the Jensen-Shannon divergence is used. If the output is defined as a continuous output, that is, the last activation is linear, a mean square error function is used.

[0045] exist Figure 3 , four classification probability graphs before adaptation are depicted. Figure 302 shows the classification probability of a target label for the three most likely categories. Because it is a label, the probability of the correct label "classification" 312 is one, while the probability of the other classifications 322 is zero. Figure 304 shows the probabilities of the same category at the output of the neural network. It can be seen that the probability of incorrect classification 324 is relatively high. Figures 306 and 308 show the predictions of the first head (i.e., source head 306) and the second head (i.e., target head 308). It should be noted that first, the probability of the correct target prediction 318 in 308 is higher than the prediction 302 (312), while the probability of other categories of the target label 328 is relatively low. Second, the probability of the correct target prediction 318 is higher than the probability of the correct source prediction 316. This difference represents a consistency loss, which is taken into account in the next update of the second head weights. The effects of these updates are as follows. Figure 4As shown, it can be seen here that the source predictions 402 , 406 and the target predictions 404 , 408 are combined in such a way that even the ranking of the second and third probabilities 422 , 423 is also characterized by the probabilities 426 and 428 of the source prediction 406 or the target prediction 426 , respectively.

[0046] Figure 5 The effect of the proposed method is demonstrated based on a simple classification example to explain the main differences between the proposed method and other related methods.

[0047] In this example, it is assumed that the neural network can only fit two shapes: one is a rectangle and the other is an "L" shape. In addition, it is assumed that the network is working on a vector representation of the input data, which can be described in a simple two-dimensional space. The task is to classify all stars, that is, only stars.

[0048] The difficulty is to also find the specified star in the target validation set. This is called the generalization ability of the model.

[0049] When the model is trained only on the labeled targets 502, the model is only fit to the target samples 512 seen. The two available shapes can cover the star 512 labeled as the target. From a training perspective, both shapes are equally good. Usually, the network will choose the simplest pattern to complete the task. The target verification star 522 will be ignored.

[0050] As can be seen from the source domain examples, the two shapes are not equally good in the proposed methods 100 and 504 (domain consistency). The consistency loss breaks the connection between the shapes and selects the L shape. All validation stars 524 are covered.

[0051] The naive blend 506 may contain information about the labeled target data 516 as well as the labeled source data 536. In this case, the data of the source 536 and the data of the target 516 are considered equally important. The source data 536 is considered a part of the target data 516. Therefore, the shapes available cannot cover all 4 stars of 516 and 536. The network randomly selects three stars that can be covered by the network.

[0052] In the example of 508, unlabeled data is used to change the representation space so that the source and target representations are close to each other. In our example, this is described as a transformation of the representation space to bring the target 518 and the verification star 528 closer to each other. In a lucky case, this transformation also places the verification star 528 closer to the target star 518, resulting in a simpler classification problem that can be covered by a simple rectangle. However, as shown in our example, this is not always the case: one of the verification stars 528 enters the boundary of the selected shape.

[0053] Conventional methods require a lot of research, where the internal representation of the neural network should be selected for the embedding space. If the unlabeled part of the target is involved, all conventional methods use naive mixture (or no mixture at all).

[0054] In semi-supervised domain adaptation, when limited amount of labeled target domain data is used, all baseline methods can be improved if the proposed solution is used instead of naive mixture. Compared with the conventional supervised domain method, the proposed solution has a better performance.

[0055] In one example, the method is used to improve production cameras. In production cameras, the number of images available is about 15,000, which is not enough to achieve predefined key performance indicators. The method is used to improve the generalization ability of public datasets and datasets detected using different camera sensors. This production method can bring several months of advantage because it can solve the problem before a sufficient amount of data is built.

[0056] Other variations of the disclosed embodiments may be understood and implemented by a professional in practicing the claimed invention by studying the drawings, this patent document and the appended claims. In the claims, the word "comprising" does not exclude other parts or steps, and the indefinite article "a" or "an" does not exclude several. A single processor or other unit may implement the functions of given parts or steps in several claims. The fact that specific measures are implemented in mutually independent dependent claims does not mean that a combination of these measures cannot be used in a beneficial manner. A computer program may be stored / distributed on an appropriate medium such as an optical storage medium or solid-state medium provided with or as part of other hardware, but may also be distributed in other forms such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be interpreted as limiting the scope of the claims.

Claims

1. A computer-implemented method (100) for training a neural network model for detecting or classifying objects or features in an image of an ADAS camera system or an AD camera system, in, The method comprises the following steps: Providing (102) a neural network model; Dividing (104) the neural network model into a first part (206) and a second part, wherein the second part includes a first head (202) and a second head (204), the first head is configured to classify a first type of classification data (232), and the second head is configured to classify a second type of classification data (234); In the training phase, the second type classification data (234) is preprocessed (106) in the first part (206), Processing (108) the pre-processed second type classification data (234) in the first header (202) and the second header (204), and determining a first result of processing the second type classification data (232) in the first header (202) and a second result of processing the second type classification data (232) in the second header (204); calculating (110) consistency between the first result and the second result; and configuring (112) the neural network model by updating at least one parameter value of the second head (204) and at least one parameter value of the first head (202) based on the calculated consistency, wherein a first head standard loss is calculated for the first head (202), and only when the first head standard loss is below a predetermined threshold, a step of configuring (112) the neural network model by updating at least one parameter value of the second head (204) and at least one parameter value of the first head (202) based on the calculated consistency is performed.

2. The computer-implemented method (100) of claim 1, wherein: The artificial intelligence module is embodied as a neural network.

3. The computer-implemented method (100) of claim 2, wherein: The neural network is a convolutional neural network.

4. The computer-implemented method (100) of any one of claims 1 to 3, wherein: The first part (206) includes an encoder layer, and the second part includes a classification layer.

5. The computer-implemented method (100) of any one of claims 1 to 3, wherein: The first type of categorical data (232) is associated with the second type of categorical data (234).

6. The computer-implemented method (100) of any one of claims 1 to 3, wherein: The neural network model provides domain adaptation, and the first type of classification data (232) is source data and the second type of classification data (234) is target data.

7. The computer-implemented method (100) of any one of claims 1 to 3, wherein: Prior to the step of preprocessing (106) the second type classification data (234) in the first part (206), training (105) of the first part (206) and training of the first head (202) are performed.

8. The computer-implemented method (100) of any one of claims 1 to 3, wherein: The parameter values ​​to be updated in the first head and / or the second head are weights of the neural network.

9. The computer-implemented method (100) of any one of claims 1 to 3, wherein: The second type of classification data (234) is partially labeled and partially unlabeled.

10. The computer-implemented method (100) of any one of claims 1 to 3, wherein: The outputs of the first head (202) and the second head (204) are classified and consistency is calculated based on Jensen-Shannon divergence.

11. The computer-implemented method (100) of any one of claims 1 to 3, wherein: The outputs of the first head (202) and the second head (204) are continuous and the consistency calculation is based on the mean square error.

12. A neural network (200) having a configuration implemented according to the computer-implemented method of any one of claims 1 to 11.

13. The neural network (200) according to claim 12 is used for detecting lanes, objects or key points, for semantic segmentation, for classification of text, audio patterns or objects or for estimating posture.

14. A computer program product comprising instructions, which, when executed by a computer, cause the computer to perform the steps of the computer-implemented method (100) as claimed in any one of claims 1 to 11.

15. A computer-readable memory comprising instructions, which, when executed by a computer, cause the computer to perform the steps of the computer-implemented method (100) of any one of claims 1 to 11.

Images