Neural network device and method for selecting an action corresponding to a current state

Abstract

A neural network device and an action selection method using the device select an action corresponding to a current state based on a value return. A method for selecting an action based on deep learning, performed by at least one processor, includes receiving a current state as input, calculating a value distribution corresponding to each of a plurality of actions to be performed on the current state, and selecting an optimal action from the plurality of actions based on the value distribution, wherein the value distribution includes at least one Gaussian graph following a Gaussian distribution.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2019-0058376, filed on May 17, 2019, and Korean Patent Application No. 10-2020-0013731, filed on February 5, 2020, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] The inventive concept relates to a neural network device, and more particularly, to a neural network device and an action selection method using the neural network device, which select an action corresponding to a current state based on a value return. Background Art

[0004] A neural network is a computing architecture modeled after the biological brain. Recent advances in neural network technology have led to active research into using neural network devices that utilize one or more neural network models to analyze input data and extract useful information in various electronic devices.

[0005] Machine learning is a field of artificial intelligence (AI) that uses neural networks, and refers to a technology that inputs data into a computer to allow the computer to learn from the data and generate new knowledge. Specifically, the field of neural networks, a type of machine learning technology, has made great progress, and thus deep learning has been developed.

[0006] Deep learning is a machine learning technology based on artificial neural networks. Although artificial neural networks are designed as multi-layer structures and are therefore deepened, unsupervised learning preprocessing can be performed on data segments used for learning, thereby improving learning efficiency. Specifically, the rapid development of deep learning has been driven by the rapid growth of internet-based big data and the computing power required to process it. Summary of the Invention

[0007] The inventive concept provides a neural network device and an action selection method using the neural network device, which select an optimal action corresponding to a current state based on value return.

[0008] The inventive concept provides a neural network device and an action selection method using the neural network device, which determine kernel weights for selecting an optimal action.

[0009] According to one aspect of the inventive concept, there is provided a method for selecting an action based on deep learning, performed by a device including a neural network device, comprising receiving, by the neural network device, a current state as input, calculating, by the neural network device, a value distribution corresponding to each of a plurality of actions to be performed on the current state, and selecting, by the neural network device, an action from among the plurality of actions based on the value distribution, wherein the value distribution includes at least one Gaussian graph following a Gaussian distribution.

[0010] According to another aspect of the inventive concept, a method for selecting an action based on deep learning performed by a device including a neural network device is provided, the method including receiving a current state as input by the neural network device, performing a convolution operation by the neural network device on an input feature map corresponding to the current state by using a weight kernel, and setting the weight kernel by the neural network device to minimize a distance difference between a first value distribution corresponding to the current state and a second value distribution corresponding to a calculated value of the current state, wherein the first value distribution includes a plurality of first Gaussian maps corresponding to a value return of the current state, and the second value distribution includes a plurality of second Gaussian maps corresponding to a value return of a next state of the current state and a sum of value returns of a plurality of actions.

[0011] According to another aspect of the inventive concept, there is provided a neural network device, comprising: a deep learning module configured to receive a current state and calculate a value distribution corresponding to each of a plurality of actions to be performed on the current state by using a deep learning model; and a post-processing module configured to select an optimal action from among the plurality of actions based on the value distribution, wherein the value distribution includes at least one Gaussian graph following a Gaussian distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Example embodiments of the inventive concept will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 is a block diagram illustrating an electronic system according to an embodiment;

[0014] Figure 2 is a block diagram illustrating an electronic system according to an embodiment;

[0015] Figure 3 is a block diagram illustrating a neural network device according to an embodiment;

[0016] Figure 4 is a flowchart illustrating an operating method of a neural network device according to an embodiment;

[0017] Figure 5 is a diagram illustrating a neural network according to an embodiment;

[0018] Figure 6A and Figure 6B is a diagram for describing a convolution operation of a neural network according to an embodiment;

[0019] Figure 7 is a diagram illustrating a neural network according to an embodiment;

[0020] Figure 8 is a flowchart illustrating an operating method of a neural network device according to an embodiment;

[0021] Figure 9 is a diagram illustrating the operation of a neural network device according to an embodiment;

[0022] Figure 10 is a flowchart illustrating an operating method of a neural network device according to an embodiment;

[0023] Figure 11 is a diagram illustrating the operation of a neural network device according to an embodiment;

[0024] Figure 12 is a flowchart illustrating an operating method of an electronic system according to an embodiment;

[0025] Figure 13 is a block diagram illustrating a neural network device according to an embodiment; and

[0026] Figure 14 is a block diagram illustrating an application processor according to an embodiment. DETAILED DESCRIPTION

[0027] Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings.

[0028] Figure 1 is a block diagram illustrating an electronic system 10 according to an embodiment.

[0029] refer to Figure 1 The electronic system 10 can analyze input data in real time based on a neural network to extract valid information and / or can check a status based on the extracted information and / or can control components of an electronic device equipped with the electronic system 10. For example, the electronic system 10 can be applied to drones, advanced driver assistance systems (ADAS), robotic devices, smart televisions (TVs), smart phones, medical devices, mobile devices, image display devices, measuring devices, Internet of Things (IoT) devices, etc., and can also be equipped in various types of electronic devices.

[0030] The electronic system 10 may include at least one intellectual property (IP) block and a neural network device 100. For example, the electronic system 10 may include first to third IP blocks IP1 to IP3 and a neural network device 100.

[0031] The electronic system 10 may include various types of IP blocks. For example, the IP blocks may include a processing circuit, multiple cores included in the processing circuit, a multi-format codec (MFC), a video module (e.g., a camera interface, a Joint Photographic Experts Group (JPEG) processor, a video processor, a mixer, etc.), a three-dimensional (3D) graphics core, an audio system, a driver, a display driver, a volatile memory, a non-volatile memory, a memory controller, an input and output interface block, a cache memory, etc. Each of the first IP block IP1 to the third IP block IP3 may include at least one of the various types of IP blocks.

[0032] The technology for connecting IP blocks can include a connection method based on the system bus. For example, the Advanced Microcontroller Bus Architecture (AMBA) protocol of Advanced RISC Machine (ARM) can be applied as a standard bus protocol. The bus types of the AMBA protocol can include Advanced High-performance Bus (AHB), Advanced Peripheral Bus (APB), Advanced eXtensible Interface (AXI), AX4 and AXI Coherency Extension (ACE). Among the above bus types, AXI can be an interface protocol between IP blocks and can provide multiple outstanding address functions and data interleaving functions. In addition, other types of protocols, such as SONICs' uNetwork, IBM's CoreConnect or OCP-IP's open core protocol, can also be applied to the system bus.

[0033] The neural network device 100 may be configured to generate a neural network, train and / or learn a neural network, and / or perform arithmetic operations based on input data received thereby, and may generate an information signal based on the execution results and / or may retrain the neural network. The neural network model may include various models such as GoogleNet, AlexNet, Convolution Neural Network (CNN) (such as VGG network), Region with Convolution Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based Deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restricted Boltzmann Machine (RBM), Fully Convolutional Network, Long Short-Term Memory (LSTM) Network, Classification Network, Deep Q-Network (DQN), and Distribution Reinforcement Learning, but is not limited thereto. The neural network device 100 may include one or more processors for performing arithmetic operations based on the neural network model. In addition, the neural network device 100 may include a separate memory for storing a program corresponding to the neural network model. The neural network device 100 may be referred to as a neural network processing device, a neural network integrated circuit, a neural network processing unit (NPU), or a deep learning device.

[0034] The neural network device 100 may be configured to receive various types of input data segments from at least one IP block via a system bus and may generate an information signal based on the input data. For example, the neural network device 100 may perform a neural network operation on the input data to generate the information signal, and the neural network operation may include a convolution operation. Figure 6A and Figure 6B The convolution operation of neural network device 100 is described in more detail.

[0035] The information signal generated by the neural network device 100 may include at least one of various types of recognition signals, such as a voice recognition signal, an object recognition signal, an image recognition signal, and a biometric information recognition signal. For example, the neural network device 100 may receive frame data included in a video stream as input data and may generate a recognition signal corresponding to an object included in an image represented by the frame data from the frame data. However, the present embodiment is not limited thereto, and the neural network device 100 may receive various types of input data and may generate a recognition signal based on the input data.

[0036] According to an embodiment, the electronic system 10 may be configured to use a value distribution to select an action from a plurality of actions corresponding to a current state received as input, and the value distribution may include at least one Gaussian graph. The selected action may be, for example, the optimal action from the plurality of actions based on the value distribution. In addition, the at least one Gaussian graph included in the value distribution may be defined by a value weight, a value average, and a value standard deviation, and the electronic system 10 may output the value weight, the value average, and the value standard deviation as a deep learning result of the neural network device 100, and may select the optimal action based on this.

[0037] Figure 2 1 is a block diagram showing an electronic system 10A according to an embodiment. In detail, Figure 2 Shows the ratio Figure 1 The electronic system 10 is shown in more detail as an example embodiment. Figure 2 When the electronic system 10A is used, the Figure 1 The same or similar description.

[0038] refer to Figure 2 , the electronic system 10A may include a neural network device 100, a random access memory (RAM) 200, a processor 300, a memory 400 and a sensor module 500. The neural network device 100 may be a corresponding Figure 1 Elements of the neural network device 100.

[0039] The RAM 200 may be configured to temporarily store programs, data, and / or instructions. For example, the programs and / or data stored in the memory 400 may be temporarily loaded into the RAM 200 based on the control of the boot code and / or the processor 300. The RAM 200 may be implemented using a memory such as a dynamic RAM (DRAM) or a static RAM (SRAM).

[0040] The processor 300 can be configured to control the overall operation of the electronic system 10A, and for example, the processor 300 may include processing circuitry, such as hardware including logic circuitry; a hardware / software combination, such as a processor that executes software; or a combination thereof. For example, more specifically, the processing circuitry may include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), and the like. The processor 300 may include one processor core (single-core), or may include multiple processor cores (multi-core). The processor 300 may process or execute programs and / or data stored in the RAM 200 and the memory 400, respectively. For example, the processor 300 may execute programs stored in the memory 400 to control the functions of the electronic system 10A.

[0041] The memory 400 may be configured as a storage device for storing data, and for example, may store an operating system (OS), various types of programs, and various data fragments. The memory 400 may be a DRAM, but is not limited thereto. The memory 400 may be at least one of a volatile memory and / or a non-volatile memory. Non-volatile memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable and programmable ROM (EEPROM), a flash memory, a phase-change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), and the like. The volatile memory may include dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), phase change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FeRAM), etc. In addition, in an embodiment, the memory 400 may include at least one of a hard disk drive (HDD), a solid state drive (SSD), a compact flash memory (CF), a secure digital (SD), a micro SD, a mini SD, an extreme digital (xD), and a memory stick.

[0042] The sensor module 500 can be configured to collect peripheral information about the electronic system 10A. The sensor module 500 can sense and / or receive signals from outside the electronic system 10A and convert the sensed and / or received signals into data. For example, the sensor module 500 can sense and / or receive image signals and convert the sensed or received image signals into image data (e.g., image frames). To this end, the sensor module 500 can include a sensing device (e.g., at least one of various types of sensors such as an imaging device, an image sensor, a light detection and ranging (LIDAR) sensor, an ultrasonic sensor, an infrared sensor, a microphone, and / or a tactile sensor) or can receive sensing signals from a sensing device. In an embodiment, the sensor module 500 can provide image frames to the neural network device 100. For example, the sensor module 500 can include an image sensor and can capture the external environment of the electronic system 10A to generate a video stream. Furthermore, the sensor module 500 can sequentially provide consecutive image frames of the video stream to the neural network device 100. Sensor module 500 may be configured to store image frames in memory 400 and / or provide image frames to neural network device 100 .

[0043] The electronic system 10A according to the embodiment may use a value distribution to select an optimal action from a plurality of actions corresponding to a current state received as input, and the value distribution may include at least one Gaussian graph. In addition, the at least one Gaussian graph included in the value distribution may be defined by a value weight, a value average, and a value standard deviation, and the electronic system 10 may output the value weight, the value average, and the value standard deviation as a deep learning result of the neural network device 100, and may select an action, for example, an optimal action, based on the value distribution.

[0044] Figure 3 is a block diagram illustrating a neural network device 100 according to an embodiment.

[0045] refer to Figure 3 , the neural network device 100 may include a deep learning module 120 and a post-processing module 140 .

[0046] The deep learning module 120 may receive the current state CS as an input feature map IFM in the form of data, and may perform deep learning on the current state CS to generate first through nth value distributions VD1 through VDn. In an embodiment, the deep learning module 120 may generate the first through nth value distributions VD1 through VDn, respectively corresponding to a plurality of actions Act1 through Actn that can be performed on the current state CS. In an embodiment, the deep learning module 120 may generate the first through nth value distributions VD1 through VDn, respectively corresponding to the plurality of actions Act1 through Actn, by using distributional reinforcement learning.

[0047] Reinforcement learning can refer to a machine learning method that learns the process of determining the optimal action to take in the current state CS. Each time an action is taken, a reward can be provided in the external environment, and learning can be performed to maximize the reward. The reward can be calculated as a value return, and according to distributional reinforcement learning, the value return can be implemented as a value distribution.

[0048] In reinforcement learning, although the current reward value is small, an action must be selected to maximize the sum of reward values ​​including values ​​to be obtained later. In addition, since the user taking the action does not know the action that allows the sum of reward values ​​to be maximized, the user must determine the optimal choice while taking actions in various ways based on the future.

[0049] In an embodiment, the deep learning module 120 can generate a value distribution corresponding to multiple actions Act1 to Actn corresponding to the current state CS, respectively, so that the value distribution includes one or more Gaussian graphs defined by an average value, a weight value, and a variance value. In addition, the deep learning module 120 can express each of the value distributions as an average value, a weight value, and a variance value of the Gaussian graph to express the result value of the deep learning module 120 as a finite network parameter.

[0050] Herein, the average value of the Gaussian graph constituting the value distribution may be referred to as a value mean, the weight value of the Gaussian graph may be referred to as a probability weight, and the variance value of the Gaussian graph may be referred to as a value standard deviation.

[0051] The deep learning module 120 may output the generated first value distribution VD1 to nth value distribution VDn to the post-processing module 140. In an embodiment, the deep learning module 120 may output network parameters constituting the first value distribution VD1 to nth value distribution VDn (e.g., value averages, probability weights, and value standard deviations of multiple Gaussian graphs) to the post-processing module 140.

[0052] The deep learning module 120 may include a convolution module 122 and a fully connected module 124. The convolution module 122 may receive a weight kernel WK and may perform a convolution operation on the current state CS received as the weight kernel WK and the input feature map IFM to generate an output feature map.

[0053] In an embodiment, the neural network device 100 may determine a weight kernel WK for optimizing the value return based on the distance information between the calculated value and the real value of each value distribution in the first value distribution VD1 to the nth value distribution VDn. This will be referred to below. Figure 12 Provide a description.

[0054] The fully connected module 124 may fully connect the plurality of actions Act1 to Actn to the elements of the output feature map to generate first to nth value distributions VD1 to VDn, respectively. The fully connected module 124 may represent performing arithmetic operations corresponding to all connections between the plurality of actions Act1 to Actn and the elements of the output feature map generated as a result of the convolution, and thus, based on the fully connected module, operation values ​​corresponding to all elements of the output feature map corresponding to the plurality of actions Act1 to Actn, respectively, may be output as return values.

[0055] According to an embodiment, the fully connected module 124 may output at least one Gaussian graph constituting the value distribution as a value return. In addition, in an embodiment, the fully connected module 124 may output the value average, probability weight, and value standard deviation of each Gaussian graph in the at least one Gaussian graph as a value return.

[0056] Post-processing module 140 may receive network parameters corresponding to first through nth value distributions VD1 through VDn and may select an optimal action (Act_sel) from among a plurality of actions (Act1 through Actn) based on the first through nth value distributions VD1 through VDn. In an embodiment, post-processing module 140 may calculate the average of the first through nth value distributions VD1 through VDn based on the network parameters and select the action corresponding to the value distribution with the largest average as the optimal action (Act_sel). In an embodiment, each of the first through nth value distributions VD1 through VDn may have a quality value (q) corresponding to the x-axis and a probability value (p(q)) based on the value (q) corresponding to the y-axis, and the average value (AV) may be expressed as in Equation 1 below.

[0057] [Equation 1]

[0058] AV=∫q·p(q)dq

[0059] Convolution module 122, fully connected module 124, and post-processing module 140 may include: processing circuitry, such as hardware including logic circuitry; a hardware / software combination, such as a processor that executes software; or a combination thereof. For example, more specifically, the processing circuitry may include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), and the like.

[0060] Figure 4 is a flowchart illustrating an operating method of the neural network device 100 according to an embodiment.

[0061] refer to Figure 3 and Figure 4 In operation S10 , the neural network device 100 may receive a current state CS as input and calculate first to nth value distributions VD1 to VDn corresponding to a plurality of actions Act1 to Actn that can be performed on the current state CS, respectively.

[0062] In an embodiment, the neural network device 100 may calculate first to nth value distributions VD1 to VDn by using distribution reinforcement learning, wherein the first to nth value distributions VD1 to VDn include one or more Gaussian graphs defined by value averages, probability weights, and value standard deviations and correspond to multiple actions Act1 to Actn, respectively.

[0063] In operation S30, the neural network device 100 may select an action from a plurality of actions Act1 through Actn using the first through nth value distributions VD1 through VDn. In an embodiment, the neural network device 100 may select an optimal action based on the first through nth value distributions VD1 through VDn. For example, the neural network device 100 may calculate the average value of each of the first through nth value distributions VD1 through VDn and select the action corresponding to the value distribution with the largest average value as the optimal action.

[0064] Figure 5 is a diagram illustrating a neural network according to an embodiment. In detail, Figure 5 The structure of a convolutional neural network is shown as an example of a neural network structure.

[0065] refer to Figure 5, the neural network NN may include a plurality of layers (e.g., the first layer to the nth layer) L1 to Ln. Each of the plurality of layers L1 to Ln may be a linear layer or a nonlinear layer, and in an embodiment, a combination of at least one linear layer and / or at least one nonlinear layer may be referred to as a layer. For example, the linear layer may include a convolutional layer and a fully connected layer, and the nonlinear layer may include a pooling layer and an activation layer.

[0066] For example, the first layer L1 may include a convolutional layer, the second layer L2 may include a pooling layer, and the nth layer Ln may include an output layer and / or a fully connected layer. The neural network NN may also include an activation layer, and further, may include layers for performing different types of operations.

[0067] Each of the plurality of layers L1 to Ln may receive data (e.g., an image frame) input thereto or a feature map generated in a previous layer as an input feature map, and may perform an arithmetic operation on the input feature map to generate a value return VR. In an embodiment, the value return VR may include a value distribution, wherein the value distribution includes at least one Gaussian map or network parameters corresponding to the at least one Gaussian map (e.g., value mean, probability weight, and value standard deviation).

[0068] The feature map may represent data in which various features of the input data are expressed. A plurality of feature maps (e.g., a first feature map, a second feature map, and an nth feature map) FM1, FM2, FM3, and FMn may have, for example, a two-dimensional (2D) matrix form or a 3D matrix (or tensor) form. In an embodiment, the first feature map FM1 of the input may include data corresponding to the current state. The feature maps FM1, FM2, FM3, and FMn may have a width W (e.g., a column), a height H (e.g., a row), and a depth D; and may correspond to the x-axis, y-axis, and z-axis of the coordinates, respectively. The depth D may be referred to as the number of channels.

[0069] The first layer L1 may perform a convolution between the first feature map FM1 and the weight kernel WK to generate a second feature map FM2. The weight kernel WK may filter the first feature map FM1 and may be referred to as a filter or map. The depth (e.g., number of channels) of the weight kernel WK may be the same as the depth (e.g., number of channels) of the first feature map FM1, and a convolution between the weight kernel WK and the same channel of the first feature map FM1 may be performed. The weight kernel WK may be shifted in a crosswise manner using the first feature map FM1 as a sliding window. The amount of shift may be referred to as a stride length or stride.

[0070] When each shift is performed, each of the weight values ​​included in the weight kernel WK is multiplied by all pixel data and summed in the area overlapping with the first feature map FM1. The extracted data segment of the first feature map FM1 in the area where each of the weight values ​​included in the weight kernel WK overlaps with the first feature map FM1 can be referred to as extracted data. When the convolution between the first feature map FM1 and the weight kernel WK is performed, one channel of the second feature map FM2 can be generated. In Figure 3, one weight kernel WK is shown, but in essence, convolution between multiple weight maps and the first feature map FM1 can be performed to generate multiple channels of the second feature map FM2. In other words, the number of channels of the second feature map FM2 can correspond to the number of weight maps.

[0071] The second layer L2 can change the spatial size of the second feature map FM2 by pooling to generate a third feature map FM3. Pooling can be called sampling or downsampling. The 2D pooling window PW can be shifted in the second feature map FM2 in units of the size of the pooling window, and the maximum value (or the average value of the pixel data fragments) among the pixel data fragments in the area overlapping with the pooling window PW can be selected. Therefore, a third feature map FM3 whose spatial size has been changed can be generated from the second feature map FM2. The number of channels of the third feature map FM3 can be the same as the number of channels of the second feature map FM2. In an embodiment, the third feature map FM3 can correspond to the above referenced method performed thereon. Figure 3 The output feature map of the convolution described.

[0072] The nth layer Ln can combine the features of the nth feature map FMn to classify the class CL of the input data. In addition, the nth layer Ln can generate a value return VR corresponding to the class. In an embodiment, the input data can correspond to data corresponding to the current state, and the nth layer Ln can extract classes corresponding to multiple actions from the nth feature map FMn provided from the previous layer to generate a value return VR for determining the optimal action. The nth layer Ln can be referred to above. Figure 3 The fully connected module 124 described (see Figure 3 )implement.

[0073] According to an embodiment, the value return VR may be expressed as a probability distribution of values ​​corresponding to each of a plurality of actions. Herein, as described above, a neural network for calculating the probability distribution of possible value returns for each current state-action pair may be defined as a value distribution network, and in an embodiment, the value distribution network may output network parameters defining the probability distribution of the value returns as a deep learning result.

[0074] Figure 6A and Figure 6Bis a diagram for describing a convolution operation of a neural network according to an embodiment.

[0075] refer to Figure 6A , the input feature map 201 may include D channels, and the input feature map of each of the channels may have a size of H rows and W columns (where D, H, and W are natural numbers). Each of the kernels 202 may have a size of R rows and S columns, and the kernel 202 may include a number of channels corresponding to the depth D of the input feature map 201 (where R and S are natural numbers). The output feature map 203 may be generated by performing a 3D convolution operation between the input feature map 201 and the kernel 202, and may include Y channels (where Y is a natural number) based on the convolution operation.

[0076] Will refer to Figure 6B Describes the process of generating an output feature map by a convolution operation between an input feature map and a kernel, and the above reference can be performed between the input feature map 201 of all channels and the kernel 220 of all channels. Figure 6B The 2D convolution operation described above is performed to generate output feature maps 203 of all channels.

[0077] refer to Figure 6B For ease of description, the input feature map 210 has been shown to have a size of 6×6, the original kernel 220 has been shown to have a size of 3×3, and the output feature map 230 has been shown to have a size of 4×4, but the present embodiment is not limited thereto, and the neural network can be implemented with feature maps and kernels having various sizes. In addition, the values ​​defined in the input feature map 210, the original kernel 220, and the output feature map 230 are merely example values, and the embodiment is not limited thereto.

[0078] The original kernel 220 may perform a convolution operation while sliding in units of a 3×3 window within the input feature map 210. The convolution operation may represent an arithmetic operation that calculates each feature data of the output feature map 230 by summing the values ​​obtained by multiplying each feature data of the window of the input feature map 210 by the weight value of each corresponding position in the original kernel 220. The data segment included in the window of the input feature map 210 and multiplied by the weight value may be referred to as extracted data extracted from the input feature map 210. Specifically, the original kernel 220 may first perform a convolution operation on the first extracted data 211 of the input feature map 210. That is, the feature data "1, 2, 3, 4, 5, 6, 7, 8, and 9" of the first extracted data 211 can be multiplied by the weight values ​​"-1, -3, 4, 7, -2, -1, -5, 3, and 1" of the original kernel 220 corresponding thereto, respectively, thereby obtaining values ​​"-1, -6, 12, 28, -10, -6, -35, 24, and 9". Subsequently, 15, which is a result obtained by summing the obtained values ​​"-1, -6, 12, 28, -10, -6, -35, 24, and 9", can be calculated, and the feature data 231 of the first row and first column of the output feature map 230 can be determined to be 15. Here, the feature data 231 of the first row and first column of the output feature map 230 can correspond to the first extracted data 211. In this way, 4, which is the feature data 232 of the first row and second column of the output feature map 230, can be determined by performing a convolution operation between the second extracted data 212 of the input feature map 210 and the original kernel 220. Finally, 11 as feature data 233 of the fourth row and fourth column of the output feature map 230 may be determined by performing a convolution operation between the original kernel 220 and the sixteenth extracted data 213 as the last extracted data of the input feature map 210 .

[0079] In other words, a convolution operation between an input feature map 210 and an original kernel 220 can be processed by repeatedly performing multiplication of the extracted data of the input feature map 210 and the corresponding weight value of the original kernel 220 and adding the multiplication results, and an output feature map 230 can be generated as a result of the convolution operation.

[0080] Figure 7 is a diagram illustrating a neural network NN according to an embodiment.

[0081] refer to Figure 7 , the neural network NN may include a convolution layer CL, a fully connected layer FCL and a post-processing layer PPL. The convolution layer CL may receive an input feature map IFM corresponding to a current state and may perform a convolution operation on a weight kernel WK to generate an output feature map OFM. The convolution layer CL may also include the above reference Figure 5 The pooling layer described above has been referenced above Figures 5 to 6B The operation of the convolutional layer CL is described, and thus its description is omitted.

[0082] The fully connected layer (FCL) may fully connect multiple actions Act1 to Act5 to elements of the output feature map OFM to calculate multiple value distributions VD1 to VD5 corresponding to the multiple actions Act1 to Act5, and generate multiple network parameters NP1 to NP5 corresponding thereto. In an embodiment, each of the value distributions VD1 to VD5 may include at least one Gaussian graph, and the multiple network parameters NP1 to NP5 may include parameters representing the at least one Gaussian graph. In an embodiment, the network parameters NP1 to NP5 may include the value mean, probability weight, and value standard deviation of each of the at least one Gaussian graph.

[0083] The post-processing layer PPL may determine the optimal action based on the plurality of network parameters NP1 to NP5. In an embodiment, the post-processing layer PPL may calculate the average value of each of the plurality of value distributions VD1 to VD5 based on the plurality of network parameters NP1 to NP5, and select the value distribution with the largest average value ( Figure 7 In the embodiment of FIG5 , an action corresponding to the third value distribution VD3 among the plurality of value distributions VD1 to VD5 may be determined as the optimal action. Figure 7 In the embodiment, the third action Act3 among the multiple actions Act1 to Act5 can be determined as the optimal action.

[0084] Figure 8 is a flowchart illustrating an operating method of a neural network device according to an embodiment. Figure 8 The calculation is shown in detail Figure 4 Graph showing the method (S20) for distributing the values ​​of .

[0085] refer to Figure 3 and Figure 8 In operation S110, the deep learning module 120 may calculate a plurality of Gaussian graphs constituting value distributions corresponding to the plurality of actions, respectively, by using a value distribution network. The value distribution network may output a value distribution representing a probability distribution of value returns based on the action as a result of deep learning, and in an embodiment, the deep learning module 120 may calculate the value distribution to include a plurality of Gaussian graphs.

[0086] In operation S120, the deep learning module 120 may output the network parameters of each of the multiple Gaussian graphs as a result of deep learning (or machine learning). In an embodiment, the deep learning module 120 may output the value average, probability weight, and value standard deviation of each of the multiple Gaussian graphs as the network parameters.

[0087] Figure 9 : is a diagram illustrating the operation of a neural network device according to an embodiment. In detail, Figure 9 A method for calculating the value distribution as a result of deep learning is shown. Figure 9 In the graph of , the horizontal axis represents the value of each action, and the vertical axis represents the probability of having each value.

[0088] refer to Figure 3 and Figure 9 , the deep learning module 120 can calculate the value distribution VD as a result of the first action corresponding to the current state by using the value distribution network. The value distribution VD can be formed by merging the first Gaussian graph GG1 to the third Gaussian graph GG3.

[0089] The first Gaussian graph GG1 may be symmetrical about the first value mean Mn1. In addition, the first Gaussian graph GG1 may be horizontally expanded to correspond to the first value standard deviation STD1, and may have a maximum value corresponding to the first probability weight Wt1.

[0090] The second Gaussian graph GG2 may be symmetrical about the second value mean Mn2. In addition, the second Gaussian graph GG2 may be horizontally expanded to correspond to the second value standard deviation STD2, and may have a maximum value corresponding to the second probability weight Wt2.

[0091] The third Gaussian graph GG3 may be symmetrical about the third value mean Mn3. In addition, the third Gaussian graph GG3 may be horizontally expanded to correspond to the third value standard deviation STD3, and may have a maximum value corresponding to the third probability weight Wt3.

[0092] The value distribution VD may be formed by combining the first to third Gaussian graphs GG1 to GG3 and may therefore be defined by network parameters of the first to third Gaussian graphs GG1 to GG3. In an embodiment, the deep learning module 120 may calculate the value distribution VD using the value distribution network and may output the first to third value averages Mn1 to Mn3, the first to third value standard deviations STD1 to STD3, and the first to third probability weights Wt1 to Wt3 as result values.

[0093] The post-processing module 140 may calculate the action-based value return by using the first to third value averages Mn1 to Mn3 , the first to third value standard deviations STD1 to STD3 , and the first to third probability weights Wt1 to Wt3 .

[0094] Figure 9An embodiment is shown in which the value distribution VD includes three Gaussian graphs GG1 to GG3 , but this is merely an example embodiment, and the value distribution VD may include more or fewer Gaussian graphs than the three shown.

[0095] Figure 10 is a flowchart illustrating an operating method of a neural network device according to an embodiment. Figure 10 It is shown in detail Figure 4 FIG. 1 is a diagram of a method ( S30 ) for selecting an optimal action.

[0096] refer to Figure 3 and 10 In operation S210, the post-processing module 140 may calculate an average value for each of the plurality of value distributions. The average value may correspond to a value returned by an action corresponding to each value distribution. In an embodiment, the post-processing module 140 may receive network parameters corresponding to the plurality of value distributions and may calculate the average value for each value distribution by using the received network parameters.

[0097] According to an embodiment, since each value distribution includes a plurality of Gaussian graphs, the post-processing module 140 may receive a value mean, a probability weight, and a value standard deviation, and may calculate a value return by using the received value mean, probability weight, and value standard deviation.

[0098] In operation S220 , the post-processing module 140 may select an action corresponding to a value distribution having the largest calculated average value as an optimal action.

[0099] Figure 11 : is a diagram illustrating the operation of a neural network device according to an embodiment. In detail, Figure 11 The method of selecting the optimal action is shown in Figure 11 In the graph of , the horizontal axis represents the value of each action, and the vertical axis represents the probability of having each value.

[0100] refer to Figure 3 and Figure 11 , the neural network device 100 may calculate a first value distribution VD1 corresponding to the first action, a second value distribution VD2 corresponding to the second action, and a third value distribution VD3 corresponding to the third action. In an embodiment, each of the first value distribution VD1 to the third value distribution VD3 may include three Gaussian graphs, and the neural network device 100 may calculate the first value distribution VD1, the second value distribution VD2, and the third value distribution VD3 by using network parameters corresponding to the three Gaussian graphs.

[0101] The neural network device 100 may calculate the average value of the first value distribution VD1 to calculate the first value return value VR1. In an embodiment, the first value return value VR1 may be a value obtained by applying Equation 1 to the first value distribution VD1. Depending on the embodiment, the neural network device 100 may calculate the first value return value VR1 by summing the average values ​​of the plurality of Gaussian graphs constituting the first value distribution VD1.

[0102] In a similar manner, the neural network device 100 can calculate a second value return value VR2 corresponding to the second value distribution VD2 and a third value return value VR3 corresponding to the third value distribution VD3 .

[0103] The neural network device 100 may determine an optimal action based on the first to third value return values ​​VR1 to VR3. In an embodiment, the neural network device 100 may determine an action corresponding to a value distribution having the maximum value among the first to third value return values ​​VR1 to VR3 as the optimal action.

[0104] exist Figure 11 In the embodiment, the first value return value VR1 may have a larger value than the second value return value VR2 and the third value return value VR3 , and the neural network device 100 may output a first action corresponding to the first value return value VR1 as the optimal action Act_sel.

[0105] Figure 12 is a flowchart illustrating an operating method of the electronic system 10 according to an embodiment. In detail, Figure 12 The operation of adaptively determining the weight kernel is shown.

[0106] refer to Figure 2 and Figure 13 In operation S310, the electronic system 10 may calculate a first value distribution corresponding to the current state and a second value distribution corresponding to the calculated value of the current state. In an embodiment, the first value distribution may be a graph showing the sum of all value returns after the current state, and the second value distribution may be a graph showing the predicted value returns of all possible actions in the current state and the sum of all value returns after the next state.

[0107] In operation S320, the electronic system 10 may calculate the distance between the first value distribution and the second value distribution by using a predetermined equation. In an embodiment, the electronic system 10 may parameterize the first value distribution and the second value distribution as a mixture of Gaussians (MoG) distribution, and may calculate the distance between the first value distribution and the second value distribution by using a Jensen-Tsallis distance (JTD) defined as a distance criterion as shown in the following equation 2.

[0108] [Equation 2]

[0109]

[0110] Here, f X (r) may represent a first value distribution based on the value r, and f Y (r) may denote a second value distribution based on the value r. Additionally, R may denote a set of possible quality values.

[0111] In operation S330 , the electronic system 10 may determine a weight kernel for minimizing the distance between the first value distribution and the second value distribution. In an embodiment, the electronic system 10 may adaptively determine a weight kernel for minimizing a result value of Equation 2 above.

[0112] According to an embodiment, the value distribution may include a plurality of Gaussian graphs, and thus the distance between the value distributions based on Equation 2 may be calculated by using finite network parameters of the plurality of Gaussian graphs.

[0113] Figure 13 is a block diagram showing a neural network device 100a according to an embodiment. Figure 3 The same or similar description.

[0114] refer to Figure 13 , the neural network device 100a may include a deep learning module 120a and a post-processing module 140a, and the deep learning module 120a may include a convolution module 122a and a fully connected module 124a. The convolution module 122a and the post-processing module 140a may be connected to Figure 3 The convolution module 122 and the post-processing module 140 are the same or similar.

[0115] The fully connected module 124a may receive the number of Gaussian graphs GGN and may calculate a plurality of value distributions VD1 to VDn based on the number of Gaussian graphs GGN. In an embodiment, the plurality of value distributions VD1 to VDn may include one or more Gaussian graphs, and the number of Gaussian graphs GGN may include information about the number of Gaussian graphs included in the plurality of value distributions VD1 to VDn.

[0116] The fully connected module 124 a may determine the number of Gaussian graphs constituting the value distribution based on the number of Gaussian graphs GGN, and may output a plurality of network parameters based on the determined number of Gaussian graphs.

[0117] According to an embodiment, the accuracy of value return and the number of calculations may be adaptively adjusted by adaptively adjusting the number of Gaussian maps.

[0118] Figure 14 1 is a block diagram illustrating an application processor 1000 according to an embodiment. The application processor 1000 may be a semiconductor chip and / or may be implemented as a System on Chip (SoC).

[0119] refer to Figure 14 , the application processor 1000 may include a processor 1010 and a working memory 1020. In addition, although not in Figure 14 Although not shown in FIG, the application processor 1000 may further include one or more IP modules connected to the system bus. The working memory 1020 may store software fragments such as various programs and instructions associated with the operation of the system including the application processor 1000, and may include, for example, an OS 1021, a deep learning module 1022, and a post-processing module 1023. The deep learning module 1022 and the post-processing module 1023 may be configured to perform the processing according to the above reference. Figures 1 to 13 A set of instructions that describe the operations of an embodiment.

[0120] In an embodiment, the processor 1010 may load the instruction set included in the deep learning module 1022 and the post-processing module 1023 loaded into the working memory 1020, and thus may execute the instructions according to the above reference. Figures 1 to 13 In an embodiment, the processor 1010 may load an instruction set included in the deep learning module 1022 to calculate a value distribution including a Gaussian graph for each operation that can be performed in the current state, and may load an instruction set included in the post-processing module 1023 to determine an action corresponding to a value distribution that returns the highest value in the value distribution as the optimal action.

[0121] exist Figure 1410, but the application processor 1000 may include a plurality of processors. In this case, some of the plurality of processors may correspond to general-purpose processors, and the other processors may be dedicated processors for executing a neural network model.

[0122] Although the present inventive concept has been particularly shown and described with reference to embodiments thereof, it will be understood that various changes in form and detail may be made therein without departing from the spirit and scope of the appended claims. It should be understood that other example embodiments may include fewer (such as one) or additional elements or components; particular elements or components may be renamed and / or rearranged; copies of particular elements or components may be omitted or included; such elements or components may be organized in different ways, such as combining the deep learning module 120 and the post-processing module 140 into a single circuit; and / or various technologies may be utilized for each element or component, such as hardware, software, or a combination of hardware and software. Some example embodiments may include multiple components or elements in one device, while other example embodiments may distribute such components or elements among multiple intercommunication devices. Some example embodiments may include sharing resources, such as processors or memory circuits, between several elements or components in series (such as sequentially) and / or in parallel (such as concurrently), while other example embodiments may include different sets of resources for different elements or components. All such variations that are reasonably and logically possible and that do not contradict other statements are intended to be included in the disclosure, the scope of which is to be understood as defined by the claims.

Claims

1. A method for selecting an action based on deep learning, performed by a device including a neural network device, the method comprising: The neural network device receives input data as a current state, wherein the input data includes at least one of image data, audio data, and video data; calculating, by the neural network device, a value distribution corresponding to each of a plurality of actions to be performed for the current state; as well as selecting, by the neural network device, an action from among the plurality of actions based on the value distribution, the selected action including an identification signal of the input data, wherein the value distribution includes a plurality of Gaussian graphs following a Gaussian distribution, Calculating the value distribution includes: receiving, by the neural network device, a number of Gaussian graphs used to generate the value distribution, calculating, by the neural network device, a plurality of Gaussian graphs based on the number of the Gaussian graphs by using a value distribution network, and The value distribution is generated by the neural network device by overlapping the calculated multiple Gaussian graphs.

2. The method according to claim 1, wherein Calculating the value distribution includes calculating the plurality of Gaussian graphs by using a value distribution network, The value distribution network includes a distribution neural network configured to output a plurality of network parameters defining a probability distribution of possible values ​​returned for each current state-action pair, and The value returned includes an estimate of the value obtained as a result of each action performed on the current state.

3. The method according to claim 2, wherein: The plurality of network parameters include at least one of a probability weight, a value mean, and a value standard deviation of each of the plurality of Gaussian graphs.

4. The method according to claim 1, wherein The value distribution includes a curve graph overlapping the first Gaussian graph, the second Gaussian graph, and the third Gaussian graph, and calculating the value distribution includes: calculating, by the neural network device, a first probability weight, a first value mean, and a first value standard deviation of the first Gaussian graph by using a value distribution network; calculating, by the neural network device, a second probability weight, a second value mean, and a second value standard deviation of the second Gaussian graph by using the value distribution network; calculating, by the neural network device, a third probability weight, a third value mean, and a third value standard deviation of the third Gaussian graph by using the value distribution network; and The value distribution is generated by the neural network device based on a calculation result by allowing the first Gaussian graph, the second Gaussian graph, and the third Gaussian graph to overlap with each other.

5. The method according to claim 1, wherein Selecting the actions includes: calculating, by the neural network device, an average value of each of the value distributions corresponding to the plurality of actions, and The neural network device determines an action corresponding to the value distribution with the largest average value as the optimal action, The optimal action is selected as the selected action.

6. The method according to claim 1, wherein Calculating the value distribution includes: performing, by the neural network device, a convolution operation on an input feature map corresponding to the current state using a weight kernel; and A plurality of Gaussian maps are generated by the neural network device based on full connections between each of the plurality of actions and elements of an output feature map generated by a result of the convolution operation. 7 . The method according to claim 6 , further comprising setting the weight kernel to minimize a distance difference between a first value distribution corresponding to the current state and a second value distribution corresponding to a calculated value of the current state.

8. The method according to claim 7, wherein: The first value distribution includes a plurality of first Gaussian graphs corresponding to value returns of the current state, and The second value distribution includes a plurality of second Gaussian graphs corresponding to a value return of a next state of the current state and a sum of value returns of the plurality of actions.

9. The method according to claim 8, wherein Setting the weight kernel includes: calculating distances between the plurality of first Gaussian graphs and the plurality of second Gaussian graphs based on a distance calculation equation; and A weight kernel is determined for minimizing the distance.

10. A method for selecting an action based on deep learning, performed by a device including a neural network device, the method comprising: The neural network device receives input data as a current state, wherein the input data includes at least one of image data, audio data, and video data; performing, by the neural network device, a convolution operation on an input feature map corresponding to the current state using a weight kernel; as well as The weight kernel is set by the neural network device to minimize a distance difference between a first value distribution corresponding to the current state and a second value distribution corresponding to a calculated value of the current state, wherein the first value distribution includes a plurality of first Gaussian graphs corresponding to the value return of the current state, and The second value distribution includes a plurality of second Gaussian graphs corresponding to a value return of a next state of the current state and a sum of value returns of a plurality of actions to be performed on the current state, The selected action includes an identification signal of the input data.

11. The method according to claim 10, wherein: Setting the weight kernel includes: calculating distances between the plurality of first Gaussian graphs and the plurality of second Gaussian graphs based on a distance calculation equation; and A weight kernel is determined for minimizing the distance.

12. A neural network device comprising: The processing circuit is configured to: receiving input data as a current state, the input data comprising at least one of image data, audio data, and video data, The number of received Gaussian graphs, A plurality of Gaussian maps are calculated based on the number of received Gaussian maps by using a value distribution network, calculating a value distribution corresponding to each of a plurality of actions to be performed on the current state by overlapping the calculated plurality of Gaussian graphs using a deep learning model; and An action is selected from among the plurality of actions based on the value distribution, the selected action comprising an identification signal of the input data.

13. The neural network device according to claim 12, wherein: The processing circuit is configured to calculate the plurality of Gaussian maps by using a value distribution network, The value distribution network includes a distribution neural network, wherein the distribution neural network is configured to output a plurality of network parameters defining a probability distribution of possible values ​​returned for each current state-action pair, and The value returned includes an estimate of the value obtained as a result of each action performed on the current state.

14. The neural network device according to claim 13, wherein: The plurality of network parameters include at least one of a probability weight, a value mean, and a value standard deviation of each of the plurality of Gaussian graphs.

15. The neural network device according to claim 12, wherein: The processing circuit is configured to calculate an average value of each of the value distributions respectively corresponding to the plurality of actions, and determine an action corresponding to the value distribution having the largest average value as an optimal action.

16. The neural network device according to claim 12, wherein: The processing circuit is further configured to: performing a convolution operation on an input feature map corresponding to the current state by using a weight kernel; and A plurality of Gaussian maps are generated based on full connections between each of the plurality of actions and elements of an output feature map generated based on the convolution operation.

17. The neural network device according to claim 16, wherein: The processing circuit is configured to set the weight kernel to minimize a distance difference between a first value distribution corresponding to the current state and a second value distribution corresponding to a calculated value of the current state.

18. The neural network device according to claim 17, wherein: The first value distribution includes a plurality of first Gaussian graphs corresponding to value returns of the current state, and The second value distribution includes a plurality of second Gaussian graphs corresponding to a value return of a next state of the current state and a sum of value returns of the plurality of actions.

Images