Inference method, inference device, and training method using neural networks

Abstract

To provide an inference method, an inference device and a training method using a neural network, which relatively reduce a load applied to a neural network in comparison to a case of restoring the entire training data, and further improve generalization in an important region of training data.SOLUTION: A method comprises receiving input data S210, and determining information inferred from input data based on state information that activates a neural network in response to the input data S220. An embedded vector, which is generated by encoding the input data with at least part of the neural network, contains information that restores a first subregion in the input data with a first accuracy and a second subregion in the input data with a second accuracy. The first subregion is adaptively determined corresponding to at least one of the inferred information and the state information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The following embodiments relate to an inference method, an inference device, a training method, etc., using a neural network. [Background technology]

[0002] It is common to use neural network-based training methods in various fields to improve performance. However, when applying neural network-based algorithms, improving the generality of the training is an important issue. In particular, neural network models that are overfitted to the input data (training data) may fail to train general statistics. Therefore, overfitted neural networks cannot improve the generality of training from limited input data. Summary of the Invention [Problem to be solved by the invention]

[0003] An objective of one embodiment is to improve generality in important regions of the training data while relatively reducing the load placed on the neural network compared to reconstructing the entire training data.

[0004] The objective of one embodiment is to prevent overfitting of the neural network without using a large amount of input data by limiting the reconstruction region by the second decoder, thereby minimizing the additional overhead of the neural network and preventing a decrease in learning performance, thereby improving the overall performance of the neural network.

[0005] An object of one embodiment is to reduce the dimension of data by generating features or codes with reduced dimensions compared to training data and then restoring them, thereby reducing the occurrence of artifacts such as noise.

[0006] The purpose of one embodiment is to use an autoencoder consisting of an encoder and a second decoder to not only generate meaningful features but also improve the versatility of trained features when performing supervised learning, since the resulting data corresponding to the target to be trained is identical to the training data, eliminating the need for additional efforts to obtain training data. [Means for solving the problem]

[0007] According to one embodiment, a method for implementing a processor using a neural network includes receiving input data and determining inferred information from the input data based on state information that activates the neural network in response to the input data, wherein an embedding vector generated by encoding the input data using at least a portion of the neural network includes information that reconstructs a first subregion in the input data with a first accuracy and a second subregion in the input data with a second accuracy, the first subregion being adaptively determined in response to either one or both of the inferred information and the embedding vector.

[0008] The step of determining the inferred information from the input data may include any one of the steps of: determining pixel-wise information for each pixel included in the input data based on the state information; determining a region detected in the input data in the form of a detection window based on the state information; and determining a class classified from the input data based on the state information.

[0009] The input data may include one of video data and audio data.

[0010] The input data may be video data, and the first subregion may include a spatial region of interest within the video data that corresponds to the inferred information.

[0011] The input data may be audio data, and the first sub-region may include a time interval of interest corresponding to the inferred information within the audio data.

[0012] The neural network can infer pixel-wise information or a detection window, and the first sub-region can be adaptively determined to be a sub-region corresponding to the inferred information.

[0013] The neural network infers a class, and the first subregion can be adaptively determined as a subregion corresponding to the embedding vector.

[0014] The first accuracy may have a higher value than the second accuracy.

[0015] According to one embodiment, a method for implementing a processor using a neural network includes receiving training data and a ground truth label corresponding to the training data; encoding the training data to extract an embedding vector corresponding to the training data; decoding the embedding vector using a first decoder to generate inference information for the neural network; decoding the embedding vector using a second decoder to generate reconstruction information for the training data; determining a first loss based on a difference between the inference information and the ground truth label; determining a first subregion of the training data corresponding to one or both of the ground truth label and the embedding vector; determining a second loss based on a difference between the reconstruction information based on the first subregion and the training data; and training the neural network based on the first loss and the second loss.

[0016] The step of training the neural network may include training the neural network to reconstruct the first subregion with a first accuracy and to reconstruct a second subregion in the input data with a second accuracy based on the first loss and the second loss.

[0017] The first accuracy may have a higher value than the second accuracy.

[0018] The step of determining the first subregion may include determining a subregion of the training data corresponding to the ground truth label as the first subregion, so that the neural network is trained to infer either pixel-wise information or a detection window.

[0019] The step of determining the first subregion may include determining a subregion corresponding to the embedding vector as the first subregion, so that the neural network is trained to infer a class.

[0020] Training the neural network may include training the neural network to determine pixel-wise information for each pixel included in the training data based on the first loss and the second loss.

[0021] The step of training the neural network may include training the neural network to determine the region detected in the training data in the form of a detection window based on the first loss and the second loss.

[0022] Training the neural network may include training the neural network to determine classified classes from the training data based on the first loss and the second loss.

[0023] The training data may include one of video data and audio data.

[0024] The step of determining the first partial region may include, when the training data is video data, determining the first partial region to include a spatial region of interest restored within the training data.

[0025] The step of determining the first sub-region may include, when the training data is audio data, determining the first sub-region so that the first sub-region includes a time interval of interest restored within the training data.

[0026] The method further includes receiving input data and determining the inference information from the input data using the trained neural network based on state information for a state in which the neural network is activated in response to the input data, and the first subregion can be adaptively determined based on either one or both of the inference information and the embedding vector.

[0027] In one embodiment, a neural network-based apparatus includes a communications interface that receives input data; and a processor that determines inferred information from the input data based on state information activated by the neural network in response to the input data, wherein an embedding vector generated by encoding the input data using at least a portion of the neural network includes information that reconstructs a first subregion in the input data with a first accuracy and a second subregion in the input data with a second accuracy, the first subregion being adaptively determined in response to either or both of the inferred information and the embedding vector.

[0028] To determine the inferred information, the processor may determine pixel-wise information for each pixel included in the input data based on the state information, determine a region detected in the input data in the form of a detection window based on the state information, or determine a class classified from the input data based on the state information.

[0029] The input data may include one of video data and audio data.

[0030] The input data may be video data, and the first subregion may include a spatial region of interest within the video data that corresponds to the inferred information.

[0031] The input data may be audio data, and the first subregion may include a spatial region of interest that corresponds to the inferred information within the audio data.

[0032] The neural network may be configured to infer pixel-wise information or a detection window, and the first sub-region may be adaptively determined to be a sub-region corresponding to the inferred information.

[0033] The neural network is designed to infer a class, and the first subregion can be adaptively determined as a subregion corresponding to the embedding vector.

[0034] The first accuracy may have a higher value than the second accuracy.

[0035] The inference device may include at least one of a head-up display (HUD) device, a 3D digital information display (DID), a navigation device, a 3D mobile device, and a smart vehicle.

[0036] According to one embodiment, a method for implementing a processor using a neural network includes the steps of: encoding training data to extract an embedding vector; determining a first loss between ground truth data of the training data and one of a class generated based on the embedding vector and inferred information; decoding the embedding vector to generate reconstruction information for the training data; determining a first subregion of the training data corresponding to one of the ground truth data and the embedding vector; determining a second loss between the training data and a portion of the reconstruction information corresponding to the first subregion; and training the neural network based on the first loss and the second loss.

[0037] The method further includes the steps of decoding the embedding vector to generate the inferred information and classifying the inferred information to generate the class, and the first loss may exist between the ground truth data and any one of the inferred information and the class.

[0038] The method further includes classifying the embedding vector to generate the classes, and the first loss may exist between the ground truth data and any one of the inferred information and the classes.

[0039] The first partial region corresponds to an adaptive weight value of the ground truth data, and the portion of the reconstruction information corresponding to the first partial region can be determined by multiplying the reconstruction information by the adaptive weight value.

[0040] Training the neural network may include training the neural network to minimize a weighted sum of the first loss and the second loss. [Effects of the Invention]

[0041] According to one aspect, the load on the neural network can be relatively reduced compared to reconstructing the entire training data, while improving generalization in important regions of the training data.

[0042] According to one aspect, by limiting the reconstruction region by the second decoder, the additional overhead of the neural network is minimized and a decrease in learning performance is prevented, thereby preventing overfitting of the neural network without using a large amount of input data and improving the overall performance of the neural network.

[0043] According to one aspect, the dimensionality of data can be reduced by generating features or codes with reduced dimensions compared to training data and then restoring them, thereby reducing the dimensionality of data and reducing the occurrence of artifacts such as noise.

[0044] According to one aspect, by using an autoencoder consisting of an encoder and a second decoder, not only can meaningful features be generated, but when performing supervised learning, the result data corresponding to the target to be trained is identical to the training data, so additional efforts to obtain training data are not required, and the versatility of the trained features can be improved. [Brief explanation of the drawings]

[0045] [Figure 1] 10A and 10B are diagrams illustrating a method for improving generalization using a training device according to an embodiment. [Figure 2] 1 is a flowchart illustrating an inference method according to an embodiment. [Figure 3] 1 is a flowchart illustrating a training method according to an embodiment. [Figure 4] 1 is a diagram showing a configuration of a training device according to an embodiment; [Figure 5] 10 is a flowchart illustrating a training process of a training device according to an embodiment. [Figure 6]FIG. 10 is a diagram showing the configuration of a training device according to another embodiment. [Figure 7] 1 is a flowchart showing a training process of a training device according to an embodiment. [Figure 8] FIG. 10 is a diagram showing the configuration of a training device according to a further embodiment. [Figure 9] 10 is a flowchart illustrating a training process of a training device according to an embodiment. [Figure 10] FIG. 1 is a block diagram of an inference device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0046] Specific structural or functional descriptions disclosed herein are merely exemplary for purposes of describing embodiments, and the embodiments may be embodied in many different forms and the present invention is not limited to the embodiments described herein.

[0047] Although terms such as "first" or "second" may be used to describe multiple components, such terms should be construed only to distinguish one component from the other components. For example, a first component may be designated as a second component, and similarly, a second component may be designated as a first component. Similarly, the second component can also be named after the first component.

[0048] When a component is referred to as being "coupled" or "connected" to another component, it should be understood that although it is directly coupled or connected to the other component, there may be other components in between.

[0049] On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Expressions describing the relationship between components, such as "between" and "immediately between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0050] The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, the words "comprise" or "have" and the like indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0051] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention pertains. Commonly used predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as having an ideal or overly formal meaning unless expressly defined herein.

[0052] 1 is a diagram illustrating a method for improving generalization in a training apparatus according to an embodiment. Referring to FIG. 1, an example of the configuration of a training apparatus 100 for inference using an artificial neural network according to an embodiment is shown.

[0053] In one embodiment, the training device 100 is configured, for example, in a form in which the neural network 130 is configured to generate features (or codes) via the encoder E110 and perform inference on the encoded features via the first decoder D1120, and further includes a second decoder D2140 for adaptive auto-encoding.

[0054] The encoder E110 generates a feature vector or code corresponding to the training data x. The first decoder D1120 can infer the label of the training data by decoding the feature vector or code generated by the encoder E110. Here, the encoder E110 and the first decoder D1120 may be trained using a first loss that minimizes the difference between the label D1E(x) of the training data inferred by the first decoder D1120 and the ground truth data y.

[0055] The second decoder D2140 can also recover the training data by decoding the feature vectors or codes generated by the encoder E110. The encoder E110 and the second decoder D2140 may be trained with a second loss that minimizes the difference between the training data D2E(x) recovered by the second decoder D2140 and the training data x.

[0056] The second decoder D2140 may be adaptively guided or restricted to restore a partial region (e.g., an important portion r(y)) of the training data x, rather than the entire region corresponding to the training data x. In this case, the training device 100 may be trained using a second loss defined to minimize the difference between the important region restored by the second decoder D2140 and the training data x. This allows the second decoder D2140 to compare the portion of the input data that does not correspond to the important region. In the case The important region can be restored with high accuracy. Here, accuracy means restoration accuracy and is also expressed as "similarity."

[0057] Here, the important part r(y) may be adaptively determined according to the local importance of the training data x, and serves to mask the important part r(y) corresponding to a part of the entire training data x.

[0058] For example, if the training data is video data, the important portion r(y) may be determined by spatial importance. If the training data is video data, the important portion r(y) may correspond to, for example, a user's region of interest (ROI) in the video data. Alternatively, if the training data is audio data such as voice data, the important portion r(y) may be determined by temporal importance. If the training data is voice data, the important portion r(y) may correspond to, for example, a time period of interest in the voice data, which is a time period during which a main speaker speaks.

[0059] In one embodiment, the important portion (e.g., region of interest) r(y) may be defined (or set) based on the following method. The region of interest may be defined by each pixel value of the target image based on pixel-wise information, for example, and may be expressed as r(y). Here, the target image may also be referred to as a "ground truth image." The region of interest may be defined by position information of the target image detected using a detection window, for example.

[0060] Alternatively, in other embodiments, the region of interest may be defined according to, for example, the classification of feature activation regions described below, in which case the region of interest may be denoted as r(E(x)).

[0061] In one embodiment, the training apparatus 100 according to the present embodiment can improve generalization in an important region of the training data x while relatively reducing the load on the artificial neural network compared to a general training apparatus that can restore the entire training data x by using the second loss value additionally defined by the second decoder D2140. Furthermore, in one embodiment, by limiting the region restored by the second decoder D2140, the training apparatus 100 according to the present embodiment can minimize the additional overhead of the neural network of the second decoder D2140 and prevent a decrease in learning performance. As a result, the training apparatus 100 according to the present embodiment can prevent overfitting of the artificial neural network without using a large amount of input data and improve the overall performance of the artificial neural network.

[0062] The training apparatus 100 according to an embodiment reduces the dimension of data by generating features with reduced dimensions compared to the training data x and then restoring them (i.e., encoding and then decoding) to reduce the occurrence of artifacts such as noise. In addition, the autoencoder including the encoder E110 and the second decoder D2140 not only generates generally meaningful features, but also, in the case of supervised learning, the result data corresponding to the target to be trained is the same as the training data, so no additional effort is required to obtain it, thereby improving the versatility of the trained features.

[0063] 2 is a flowchart illustrating an inference method according to an embodiment. Referring to FIG. 2, an inference device according to an embodiment receives input data (S210). The input data may include, for example, video data and audio data, but is not necessarily limited thereto. The inference device may receive the input data via, for example, a communication interface (see 1010 in FIG. 10).

[0064] The inference device determines inferred information from the input data (S220) based on state information that the artificial neural network is activated in response to the received input data (S210).

[0065] Here, "state information" refers to, for example, information indicating the internal state of a node constituting an artificial neural network. Here, the internal state of a node may include information (e.g., spatial information and / or temporal information) that has been accumulated from information processed at a previous point in time by a feedback structure of the neural network. The state information may be information in the form of a vector, such as a hidden state vector. The state information may also be referred to as "hidden state information" or "hidden state information."

[0066] The state information is, for example, a combination of feature activation regions. For example, when an image is input to a convolutional neural network (CNN), a feature map is generated by passing through several filters that filter out lanes, vehicles, pedestrians, etc.

[0067] In one embodiment, an inference device encodes input data using at least a portion of an artificial neural network, generating an embedding vector that includes information for reconstructing a first subregion of the input data with a first accuracy and a second subregion of the input data with a second accuracy. For example, the first accuracy may be higher than the second accuracy. Here, the at least a portion of the artificial neural network may be, for example, a portion of layers that function as encoders in the artificial neural network.

[0068] The first subregion in the input data is adaptively determined in response to inferred information output from the input data by the artificial neural network, state information generated within the artificial neural network during the inference process, or a combination thereof.

[0069] If the artificial neural network is designed for inferring pixel-wise information or a detection window, the first subregion in the input data may be adaptively determined based on the information inferred by the artificial neural network. For example, if the artificial neural network is designed for inferring pixel-wise information or a detection window, the first subregion in the input data may be adaptively determined to be a subregion corresponding to the information (e.g., pixel-wise information or a detection window) inferred by the artificial neural network.

[0070] Alternatively, if the artificial neural network is designed for class inference, the first subregion in the input data may be adaptively determined by an embedding vector generated by the encoder of the artificial neural network. For example, if the artificial neural network is designed for class inference, the first subregion in the input data may be adaptively determined by a subregion corresponding to the embedding vector generated by the encoder. More specifically, by using several filters, several channels are generated, and the artificial neural network can select a region where a large value appears for each channel (in other words, a region of feature activation where a feature is activated). The artificial neural network combines overlapping regions among the regions where a large value appears for each channel and sets them as a region of interest. The state information corresponds, for example, to a pixel-wise map r(E(x)) reflecting a region of interest (ROI), which will be described later.

[0071] The first partial region is a region of interest in the input data, and the second partial region is a remaining region in the input data excluding the region of interest. For example, if the input data is video data, the first partial region may include a spatial region of interest corresponding to inferred information in the video data. Alternatively, if the input data is audio data, the first partial region may include a time interval of interest corresponding to inferred information in the audio data.

[0072] Although not used in the inference process, the information contained in the embedding vector can be confirmed using a second decoder used in the training process. The second decoder corresponds to the first subregion from the embedding vector and can restore input data with higher accuracy than the second subregion.

[0073] In step S220, the inference device provides pixel-wise information for each pixel included in the input data, provides a region detected from the input data in the form of a detection window, and / or provides a class classified from the input data, for example, based on state information in which the artificial neural network is activated. The information inferred from the input data in step S220 may be, for example, a region corresponding to a region of interest such as a lane, a vehicle, a pedestrian, etc., and / or a speech segment of a speaker of interest.

[0074] An inference device according to one embodiment may include, but is not limited to, a head-up display (HUD) device, a 3D digital information display (DID), a navigation device, a 3D mobile device, and / or a smart vehicle.

[0075] 3 is a flowchart illustrating a training method according to an embodiment. Referring to FIG. 3, a training device according to an embodiment receives training data (e.g., training data x) including a ground truth label (e.g., ground truth data y) (S310). The training data may include, for example, either video data or audio data, but is not necessarily limited thereto.

[0076] The training device extracts an embedding vector (eg, a feature vector) corresponding to the training data received in step S310 by encoding the training data using, for example, an encoder (S320).

[0077] The training device decodes the embedding vector extracted in step S320 using a first decoder to generate inferred information (e.g., a label D1E(x)) of the artificial neural network (S330). The training device can generate inferred information by decoding the embedding vector using the first decoder. The training device calculates a first loss based on the difference between the inferred information of the artificial neural network and the correct label included in the training data (S350).

[0078] The training device decodes (S340) the embedding vector extracted in step S320 using a second decoder that generates reconstruction information for the training data received in step S310 (e.g., reconstructed training data D2E(x)). The training device can generate reconstruction information for the training data by decoding the embedding vector using the second decoder.

[0079] The training device determines a first partial region corresponding to a ground truth label in the training data or a first partial region corresponding to an embedding vector generated by the training data (e.g., the embedding vector extracted in step S320) (S360). If the training data is video data, the first partial region may include a spatial region of interest restored in the training data. Alternatively, if the training data is audio data, the first partial region may include a time interval of interest restored in the training data.

[0080] According to one embodiment, when the training data is video data, the training device determines a subregion corresponding to a ground truth label in the training data as a first subregion so that the artificial neural network can infer pixel-wise information or a detection window. Alternatively, when the artificial neural network is designed to infer a class, training may be performed using a target label corresponding to the training video. In this case, since there is no target video corresponding to the training video, a typical training device cannot obtain pixel-wise information corresponding to the target video. Alternatively, according to one embodiment, the training device can train the artificial neural network to infer a class corresponding to a region of interest using activation information (e.g., an embedding vector) of features extracted via an encoder.

[0081] In step S360, the training device restores the first subregion in the training data. The training device can restore the first subregion in the training data, for example, by adaptively decoding a feature vector based on pixel-wise information extracted for each pixel in the training data. For example, see Figures 4 to 7 below for a method in which the training device restores the first subregion in the training data based on pixel-wise information extracted for each pixel.

[0082] The training device can restore the first subregion in the training data by, for example, adaptively decoding the embedding vector (e.g., the embedding vector extracted in step S320) based on the region detected by the detection window in the training data. For example, see the following Figures 4 to 7 for a method in which the training device restores the first subregion in the training data based on the region detected by the detection window.

[0083] In one embodiment, all of Figures 4 to 7 can be used in the pixel-wise information or detection window based method in that both the pixel-wise information or detection window based method can correctly define the region of interest and determine the first partial region. However, Figures 4 to 5 and Figures 6 to 7 differ only depending on whether two decoding networks share at least some decoding layers, and other parts may be applied in the same way.

[0084] Alternatively, the training device can restore the first subregion in the training data by adaptively decoding the feature vector based on a feature map corresponding to the class classified from the training data by the classifier. See the following Figures 6 to 9 for a method in which the training device restores the first subregion in the training data based on a feature map corresponding to the class classified from the training data.

[0085] The training device calculates a second loss (S370) based on the difference between the reconstruction information from the first subregion determined in step S360 (e.g., the reconstruction information generated in step S340) and the training data. As mentioned above, the reconstruction information from the first subregion may correspond to, for example, the pixel-wise information or detection window inferred by the artificial neural network, or may correspond to the class inferred by the artificial neural network.

[0086] The training device trains the artificial neural network based on the first loss calculated in step S350 and the second loss calculated in step S370 (S380). For example, the training device trains a first decoder and encoder based on the first loss and a second decoder and encoder based on the second loss. Thus, the encoder may be trained to generate an embedding vector containing information capable of reconstructing a first subregion with a first accuracy and a second subregion in the training data with a second accuracy. Here, the first accuracy may be higher than the second accuracy.

[0087] The training method according to an embodiment can be applied to pre-train a neural network to ensure generalization even when it is later implemented in various environments when an algorithm or software for, for example, training-based detection, localization, semantic segmentation, etc. is installed in the inference device. Furthermore, the training method according to an embodiment can perform on-device training to ensure generalization without overfitting using minimal device data.

[0088] For the sake of convenience, the following description will be given taking the case where the training data is video data as an example, but the present invention is not necessarily limited to this.

[0089] Fig. 4 is a diagram showing the configuration of a training device according to an embodiment, and Fig. 5 is a flowchart showing a training process of the training device shown in Fig. 4. Referring to Fig. 4, a training device 400 according to an embodiment includes an encoder E410, a first decoder D1420, and a second decoder D1430.

[0090] In one embodiment, the training device 400 can improve generalization by adding a second decoder D2430 to an artificial neural network having a network structure composed of an encoder E410 that outputs pixel-wise values ​​for a training image x to generate a target image, and a first decoder D1420.

[0091] The first decoder D1420 infers specific information (e.g., labels corresponding to training data) corresponding to the embedding vectors generated by the encoder E410. The second decoder D2430 recovers information corresponding to a portion of the training data (e.g., training video x).

[0092] The training device 400 may, for example, extract pixel-wise information from a training video x or train an artificial neural network to detect pixel-wise information.

[0093] Typical pixel-wise information extraction is based on a neural network structure consisting of an encoder and decoder, where training images are passed through the encoder to generate features, which are then passed through the decoder to generate the final result image.

[0094] In the following, for example, a training video is represented as x, a target video (or a correct video) corresponding to the training video is represented as y, an encoder is represented as E, a first decoder is represented as D1, and a second encoder is represented as D2.

[0095] The training device 400 receives a pair {Input x, Target y} consisting of a training video x and a target video (or a ground truth video) y corresponding to the training video x (S510). The training device 400 applies the training video x to an encoder E410 to extract an embedding vector E(x) corresponding to the training video x (S520).

[0096] The training device 400 applies (S530) the embedding vector E(x) to a first decoder D1420 to generate (S540) a result image D1E(x). Depending on the embodiment, the first decoder D1420 may be a detector, or the operations of the first decoder D1420 may be performed by a detector.

[0097] The training device 400 calculates a target loss (S550) corresponding to the difference (eg, Diff(D1E(x), y)) between the result image D1E(x) passed through the first decoder D1420 and the target image y (S560).

[0098] In one embodiment, the adaptive reconstruction decoder may further include a second decoder D2430 in addition to the target loss. The training device 400 passes the embedding vector E(x) generated by the encoder E410 through the second decoder D2430 (S535) to generate a reconstructed image D2E(x) (S545). Multiplying the reconstructed image D2E(x) by an adaptive weight r(y) for each pixel defined from the target image y can have the effect of assigning a weight to the reconstructed image D2E(x) corresponding to important information activated in the target image y. The adaptive weight r(y) may be, for example, a pixel-wise map reflecting a region of interest (ROI) in the training image. The adaptive weight r(y) may be, for example, a binary mask having a value of 0 or 1, or may be a weight (or importance) having a gradually changing real value at the boundary of the ROI.

[0099] The training device 400 calculates a restoration loss defined as the difference (e.g., Diff(D2E(x)·r(y),x)) between the adaptively restored image D2E(x)·r(y) and the training image x using the adaptive weight r(y) (S570). Here, the restored image D2E(x)·r(y) is a partial region (first partial region) of the training image x, such as the ROI region of interest of the training image x.

[0100] The training device 400 trains the artificial neural network so that the reconstruction loss is minimized together with the target loss (S580). For example, the training device 400 can train the artificial neural network so that the sum (e.g., weighted sum) of the target loss and the reconstruction loss for the entire set of training data is minimized.

[0101] Fig. 6 is a diagram showing the configuration of a training device according to another embodiment, and Fig. 7 is a flowchart showing a training process of the training device shown in Fig. 6. Referring to Fig. 6, a training device 600 according to one embodiment may include an encoder E610 and a decoder D1630.

[0102] The decoder D1630 according to the embodiment shown in FIG. 6 may include at least some layers shared between the two separate decoders (first decoder 420 and second decoder 430) described above with reference to FIG. 4. The training device 600 may further include a diverter C633 and an adaptive reconstruction decoder R636 connected to the decoder D1630. For example, the decoder D1630 may operate in conjunction with the diverter C633 while simultaneously operating in conjunction with the adaptive reconstruction decoder R636. Depending on the embodiment, the diverter C633 may operate as a detector. For convenience of explanation, the following description will be given taking a classification operation as an example, but the embodiment may also be applied to the case where the diverter C633 operates as a detector.

[0103] The training device 600 receives a pair {Input x, Target y} consisting of a training video x and a target label (or ground truth label) y corresponding to the training video x (S710). The training device 600 applies the training video x to an encoder E610 to extract an embedding vector E(x) corresponding to the training video x (S720).

[0104] The training device 600 generates a result image D1E(x) by decoding the embedding vector E(x) through the decoder D1630 (S730) (740). Here, the training device 600 passes the result image D1E(x) that has passed through the decoder D1630 through a diverger C633 to obtain a classified class C(D1E(x). The training device 600 calculates a target loss corresponding to the difference between the class C(D1E(x)) and the target label y760 (e.g., Diff(C(D1E(x)),y)) (S750).

[0105] The training device 600 also passes the image D1E(x) resulting from the decoder D1 630 through the adaptive reconstruction decoder R 636 to generate a reconstructed image R(D1E(x)) (S745). Here, multiplying the reconstructed image R(D1E(x)) by the adaptive weight r(y) of each pixel defined from the target label y 760 has the effect of imparting a weight corresponding to important information activated by the target label y to the reconstructed image R(D1E(x)). Here, the adaptive weight r(y) of each pixel may be a pixel-wise map reflecting the region of interest (ROI) as described above, or may be a binary mask having values ​​of 0 or 1, or a weight having real values ​​that gradually change at the boundary of the region of interest.

[0106] The training device 600 calculates a reconstruction loss (S770) defined as the difference (e.g., Diff(R(DIE(x))·r(y),x)) between the adaptively reconstructed image R(DIE(x))·r(y) and the training image x using the weight r(y). Here, the reconstructed image R(DIE(x))·r(y) is a partial region (first partial region) of the training image x, such as the ROI region of interest of the training image x.

[0107] The training device 600 trains the artificial neural network so that the target loss and the reconstruction loss are minimized (S780). For example, the training device 600 may train the artificial neural network so that the sum (e.g., weighted sum) of the target loss and the reconstruction loss for the entire training data set is minimized.

[0108] 6 shares the same decoder 630 and generates only the final results (e.g., classified class C(D1E(x)) and reconstructed image R(D1E(x))) into two types, the number of weights of the artificial neural network that must be trained is smaller than that of a training device that uses an additional second decoder. Therefore, the training device 600 can reduce the load on the artificial neural network.

[0109] FIG. 8 is a diagram showing the configuration of a training device according to a further embodiment, and FIG. 9 is a flowchart showing the training process of the training device shown in FIG.

[0110] 8, a training device 800 according to one embodiment includes an encoder E 810 and a decoder D2 830. Here, at least some layers of the encoder E 810 may operate with a divider C 820.

[0111] The training device 800 receives a pair {Input x, Target y} consisting of a training video x and a target label (or ground truth label) y corresponding to the training video x (S910). The training device 800 passes the training video x through an encoder E810 to extract an embedding vector E(x) corresponding to the training video x (S920).

[0112] The training apparatus 800 classifies the embedding vector E(x) into a class C(E(x)) corresponding to the embedding vector E(x) by passing the embedding vector E(x) through a divider C820 (S930).

[0113] The training device 800 calculates a target loss (S950) corresponding to the difference (e.g., Diff(C(E(x)),y)) between the classified class C(E(x)) and the target label (or ground truth label) y (960) corresponding to the training video x.

[0114] The training device 800 passes the embedding vector E(x) extracted by the encoder E810 through a decoder D2830 (S940) to generate a reconstructed image D2E(x) (S945). The decoder D2830 may operate together with the adaptive reconstruction decoder D2430 described above with reference to FIG. 4. The training device 800 defines a pixel-wise map r(E(x)) from the embedding vector E(x) and multiplies the reconstructed image D2E(x) by the pixel-wise map r(E(x)). This map can indicate the effect of weighting important information such as a region of interest (ROI). The pixel-wise map r(E(x)) is a combination of feature activation regions and may reflect a region of interest (ROI).

[0115] 8 and 9, since a target label (or ground truth label) y (960) corresponding to a training video x is used, a target video may not exist. However, since a target video does not exist, even if weights based on pixel-wise information are not directly defined, the training device 800 can define a pixel-wise map r(E(x)) based on activation information (e.g., feature activation regions) of features extracted for classification in the embodiment shown in FIG.

[0116] The training device 800 calculates a restoration loss defined as the difference (e.g., Diff((D2E(x))·r(E(x)),x)) between the adaptively restored image D2(E(x))·r(E(x)) and the training image x using the pixel-wise map r(E(x)) reflecting the region of interest (ROI) (S970). Here, the adaptively restored image D2(E(x))·r(E(x)) is a partial region (first partial region) of the training image x, such as the region of interest (ROI) of the training image x.

[0117] The training device 800 trains the neural network so that the target loss and the reconstruction loss are minimized (S980). For example, the training device 800 can train the neural network so that the sum (e.g., weighted sum) of the target loss and the reconstruction loss for the entire set of training data is minimized.

[0118] 10 is a block diagram of an inference device according to one embodiment. Referring to FIG. 10, inference device 1000 according to one embodiment includes a communication interface 1010, a processor 1030 (e.g., one or more processors), and a memory 1050 (e.g., one or more processors). Communication interface 1010, processor 1030, and memory 1050 communicate via a communication bus 1005. Inference device 1000 may be, for example, a head-up display (HUD) device, a 3D digital information display (DID), a navigation device, a 3D mobile device, a smart vehicle, or the like.

[0119] The communication interface 1010 receives input data. The input data may be, for example, video data or audio data. The video data may include, for example, a driving video captured in front and / or to the left and right while the vehicle is traveling, but is not necessarily limited thereto. The audio data may include, for example, a voice of a speaker speaking during a certain time period, but is not necessarily limited thereto.

[0120] The processor 1030 provides information inferred from the input data based on state information activated by the artificial neural network in response to the input data received via the communication interface 1010. Here, the inferred information may include, for example, a video including an object corresponding to the region of interest, a voice of a speaker corresponding to the subject of interest, etc.

[0121] Here, by encoding the input data using at least a portion of the artificial neural network, the generated embedding vector includes information for reconstructing a first subregion in the input data with a first accuracy and a second subregion in the input data with a second accuracy, and the first subregion is adaptively determined in response to at least one of information inferred by the processor 1030 and state information activated by the artificial neural network.

[0122] Memory 1050 may store, for example, input data received via communication interface 1010 and / or information inferred from the input data by processor 1030. Memory 1050 may also store state information that the artificial neural network activates in response to the input data and / or embedding vectors generated by processor 1030 by encoding the input data.

[0123] The memory 1050 also stores various information generated during the processing of the processor 1030. The memory 1050 may also store various data and programs. The memory 1050 may include a volatile memory or a non-volatile memory. The memory 1050 may include a large-capacity storage medium such as a hard disk to store various data.

[0124] Furthermore, the processor 1030 may perform at least one of the methods or an algorithm corresponding to at least one of the methods described above with reference to FIGS. 1 to 9. The processor 1030 may be a data processing device implemented in hardware having circuits with physical structures for performing desired operations. For example, the desired operations may include code or instructions included in a program. For example, the data processing device implemented in hardware may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA).

[0125] The above-described embodiments may be implemented using hardware components, software components, or a combination of hardware and software components. For example, the devices and components described herein may be implemented using one or more general-purpose or special-purpose computers, such as a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable array (FPA), programmable logic unit (PLU), microprocessor, or other device that executes and responds to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the operating system. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For ease of understanding, although a single processing device may be described, those skilled in the art will recognize that a processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0126] The methods according to the present invention may be embodied in the form of program instructions that can be executed by various computer means and stored on a computer-readable storage medium. The storage medium may include program instructions, data files, data structures, and the like, alone or in combination. The storage medium and program instructions may be specially designed and constructed for the purposes of the present invention, or they may be well-known and available to those skilled in the art of computer software. Examples of computer-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code, such as produced by a compiler, but also high-level language code that is executed by a computer using an interpreter, for example. A hardware device may be configured to operate as one or more software modules to perform the operations described in the present invention, or vice versa.

[0127] Although the embodiments have been described above using limited drawings, those skilled in the art may apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, or may be replaced or substituted with other components or equivalents to achieve suitable results. Therefore, other implementations, other embodiments, and equivalents of the claims are also within the scope of the following claims. [Explanation of symbols]

[0128] 1000: Reasoning device 1005:Communication bus 1010: Sensor 1030: Processor 1050:Memory

Claims

1. 1. A processor-implemented method using a neural network, the step of training the neural network comprising: receiving training data and ground truth labels corresponding to the training data; extracting an embedding vector corresponding to the training data by encoding the training data; decoding the embedding vector using a first decoder to generate inference information for the neural network; decoding the embedding vector using a second decoder to generate reconstruction information for the training data; determining a first loss based on a difference between the inference information and the ground truth label; determining a first subregion of the training data corresponding to one or both of the ground truth label and the embedding vector; determining a second loss based on a difference between the reconstruction information based on the first subregion and the training data; training the neural network based on the first loss and the second loss; and performing inference using the neural network trained in the step of training the neural network, receiving input data; determining information inferred from the input data based on state information that causes the neural network to be activated in response to the input data; and the trained neural network is capable of reconstructing a first subregion in the input data with a first accuracy and a second subregion in the input data with a second accuracy; The method, wherein the first subregion is adaptively determined in response to either one or both of the inferred information and the embedding vector.

2. determining the inferred information from the input data, determining pixel-wise information for each pixel included in the input data based on the state information; determining, based on the state information, the region detected in the input data in the form of a detection window; determining a class to be classified from the input data based on the state information; The method of claim 1 , comprising any one of:

3. The method of claim 1 or 2, wherein the input data includes one of video data and audio data.

4. the input data is video data, The method of claim 1 , wherein the first subregion comprises a spatial region of interest that corresponds to the inferred information within the video data.

5. the input data is audio data, The method of claim 1 , wherein the first sub-region comprises a time interval of interest that corresponds to the inferred information within the audio data.

6. the neural network infers pixel-wise information or a detection window; The method according to any one of claims 1 to 5, wherein the first sub-region is adaptively determined to be a sub-region corresponding to the inferred information.

7. The neural network infers classes; The method according to any one of claims 1 to 6, wherein the first subregion is adaptively determined as a subregion corresponding to the embedding vector.

8. The method according to any one of claims 1 to 7, wherein the first accuracy has a higher value than the second accuracy.

9. A computer program causing a computer to carry out the method according to any one of claims 1 to 8.

10. 1. A processor-implemented method using a neural network, comprising: receiving training data and ground truth labels corresponding to the training data; extracting an embedding vector corresponding to the training data by encoding the training data; decoding the embedding vector using a first decoder to generate inference information for the neural network; decoding the embedding vector using a second decoder to generate reconstruction information for the training data; determining a first loss based on a difference between the inference information and the ground truth label; determining a first subregion of the training data corresponding to one or both of the ground truth label and the embedding vector; determining a second loss based on a difference between the reconstruction information based on the first subregion and the training data; training the neural network based on the first loss and the second loss; A method comprising:

11. 11. The method of claim 10, wherein training the neural network comprises training the neural network to reconstruct the first subregion with a first accuracy and to reconstruct a second subregion in the training data with a second accuracy based on the first loss and the second loss.

12. The method of claim 11 , wherein the first accuracy has a higher value than the second accuracy.

13. 13. The method of claim 10, wherein the step of determining the first subregion comprises determining a subregion of the training data corresponding to the ground truth label as the first subregion, so that the neural network is trained to infer one of pixel-wise information or a detection window.

14. The method of any one of claims 10 to 13, wherein determining the first subregion comprises determining a subregion corresponding to the embedding vector as the first subregion so that the neural network is trained to infer a class.

15. 15. The method of claim 10, wherein training the neural network comprises training the neural network to determine pixel-wise information for each pixel included in the training data based on the first loss and the second loss.

16. 15. The method of claim 10, wherein the step of training the neural network includes the step of training the neural network to determine regions detected in the training data in the form of a detection window based on the first loss and the second loss.

17. 15. The method of claim 10, wherein training the neural network comprises training the neural network to determine a classified class from the training data based on the first loss and the second loss.

18. The method of any one of claims 10-17, wherein the training data comprises one of video data and audio data.

19. A method according to any one of claims 10 to 18, wherein the step of determining the first partial region includes, when the training data is video data, determining the first partial region to include a spatial region of interest restored within the training data.

20. 19. A method according to any one of claims 10 to 18, wherein the step of determining the first sub-region comprises, when the training data is audio data, determining the first sub-region so that the first sub-region includes a time interval of interest recovered within the training data.

21. receiving input data; determining the inference information from the input data using the trained neural network based on state information for states to which the neural network is activated in response to the input data; further comprising The method according to any one of claims 10 to 20, wherein the first subregion is adaptively determined based on one or both of the inference information and the embedding vector.

22. In a device using a neural network, a communication interface for receiving input data; a processor that determines information inferred from the input data based on state information activated by the neural network in response to the input data; and when training the neural network, the processor: receiving training data and ground truth labels corresponding to the training data; extracting an embedding vector corresponding to the training data by encoding the training data; decoding the embedding vector using a first decoder to generate inference information for the neural network; decoding the embedding vector using a second decoder to generate reconstruction information for the training data; determining a first loss based on a difference between the inference information and the ground truth label; determining a first subregion of the training data corresponding to one or both of the ground truth label and the embedding vector; determining a second loss based on a difference between the reconstruction information based on the first subregion and the training data; training the neural network based on the first loss and the second loss; wherein the trained neural network is capable of reconstructing a first subregion in the input data with a first accuracy and a second subregion in the input data with a second accuracy; The apparatus, wherein the first subregion is adaptively determined in response to either one or both of the inferred information and the embedding vector.

23. To determine the inferred information, the processor: determining pixel-wise information for each pixel included in the input data based on the state information; determining, based on said state information, the detected region of said input data in the form of a detection window; or The apparatus of claim 22 , further comprising: determining a classified class from the input data based on the state information.

24. 24. Apparatus according to claim 22 or 23, wherein the input data comprises one of video data and audio data.

25. the input data is video data, The apparatus of claim 22 , wherein the first subregion comprises a spatial region of interest that corresponds to the inferred information within the video data.

26. the input data is audio data, 23. The apparatus of claim 22, wherein the first subregion comprises a spatial region of interest that corresponds to the inferred information within the audio data.

27. the neural network is configured to infer pixel-wise information or a detection window; The apparatus according to any one of claims 22 to 26, wherein the first subregion is adaptively determined to be a subregion corresponding to the inferred information.

28. the neural network is designed to infer classes; The apparatus according to any one of claims 22 to 27, wherein the first subregion is adaptively determined from a subregion corresponding to the embedding vector.

29. The apparatus of any one of claims 22 to 28, wherein the first accuracy has a higher value than the second accuracy.

30. 30. The device of any one of claims 22-29, wherein the device comprises at least one of a Head Up Display (HUD) device, a 3D Digital Information Display (DID), a navigation device, a 3D mobile device, and a smart vehicle.

31. 1. A processor-implemented method using a neural network, comprising: encoding training data to extract embedding vectors; determining a first loss between the correct data of the training data and any one of the classes generated based on the embedding vector and the inferred information; decoding the embedding vector to generate reconstruction information for the training data; determining a first subregion of the training data corresponding to either the ground truth data or the embedding vector; determining a second loss between the training data and a portion of the reconstruction information corresponding to the first sub-region; training the neural network based on the first loss and the second loss; A method comprising:

32. decoding the embedding vector to generate the inferred information; classifying the inferred information to generate the classes; further comprising 32. The method of claim 31, wherein the first loss exists between the ground truth data and any one of the inferred information and the class.

33. further comprising classifying the embedding vectors to generate the classes; 32. The method of claim 31, wherein the first loss exists between the ground truth data and any one of the inferred information and the class.

34. The first partial area corresponds to an adaptive weight value of the correct data, The method of claim 31 , wherein the portion of the reconstruction information corresponding to the first sub-region is determined by multiplying the reconstruction information by the adaptive weighting value.

35. 32. The method of claim 31 , wherein training the neural network comprises training the neural network to minimize a weighted sum of the first loss and the second loss.

Images