Multiple UWB radar sensor-based neuromorphic device

Abstract

A neuromorphic device according to the present invention comprises: a communication unit for receiving pieces of ultra-wideband (UWB) radar data measured for each moving object from a plurality of UWB radar sensors; and a processor for imaging and preprocessing each of the received pieces of UWB radar data, spike-encoding each of the images corresponding to the preprocessed pieces of UWB radar data, and determining, on the basis of the spike-encoded data, the type of a prescribed trained spiking neural network (SNN) model to which the spike-encoded data is to be input.

Description

Neuromorphic device based on multiple UWB radar sensors

[0001] The present invention relates to a neuromorphic device, and more particularly, to a neuromorphic device for acquiring data from multiple UWB radar sensors and performing neuromorphic processing or computing.

[0002] This study was conducted as a result of the University ICT Research Center Promotion Project funded by the Ministry of Science and ICT and the National IT Industry Promotion Agency (NIPA). (Project ID 2710007997)

[0003] Ultra-wideband (UWB) is a wireless technology defined by the Federal Communications Commission (FCC) as having an occupied frequency of at least 20% of the center frequency or an occupied bandwidth of at least 500 MHz. UWB utilizes a wide frequency band to enable high-speed, short-range communications, and its low-power signals have the advantage of not interfering with other communication systems. Originally developed for military applications, UWB technology gained approval for civilian use by the FCC in 2002 and is now being utilized in wireless communications and radar.

[0004] Recently, UWB radar has been studied in various fields, including vital sign monitoring, user recognition, and fall detection. Mobile UWB radar sensors are currently primarily used in vehicle smart keys. However, there has been no research on technology for accurately estimating a user's heart rate and respiration rate using UWB radar in a mobile environment.

[0005] To respond to emergencies such as falls requiring caregiver supervision in the home, a method for tracking a person's location using IoT devices capable of continuous, unobstructed monitoring is needed. Cameras can be used to track a person's location, but privacy concerns limit their use for continuous monitoring in the home.

[0006] The present invention aims to provide a neuromorphic device.

[0007] Another technical task to be achieved in the present invention is to provide a neuromorphic processing method.

[0008] Another technical problem to be achieved by the present invention is to provide a computer-readable recording medium having recorded thereon a program for executing a neuromorphic processing method on a computer.

[0009] In order to achieve the above technical task, a neuromorphic device according to the present invention is characterized by including: a communication unit that receives UWB radar data measured for a moving target from a plurality of UWB (Ultra-wideband) radar sensors; and a processor that preprocesses the received UWB radar data by imaging it, spike-encodes each image corresponding to the preprocessed UWB radar data, and determines the type of a predetermined learned spiking neural network (SNN) model to which the spike-encoded data is to be input based on the spike-encoded data.

[0010] The above-described learned SNN model may be any one of an SNN model for tracking the location of the moving object, an SNN model for detecting a fall of the moving object, and an SNN model for monitoring the biosignals of the moving object.

[0011] The processor determines, from the spike-encoded data, that the amount of change in the movement of the moving object is greater than a threshold value for a predetermined first period of time, the type of the SNN model as the SNN model for fall detection, and applies the spike-encoded data to the determined SNN model for fall detection to detect and output whether the moving object has fallen.

[0012] The processor determines, from the spike-encoded data, that the amount of change in the movement of the moving object is below a threshold for a predetermined second period of time, the type of the SNN model as the SNN model for bio-signal monitoring, and applies the spike-encoded data to the determined SNN model for bio-signal monitoring to output a state or value for the bio-signal of the moving object.

[0013] The processor determines the type of the SNN model as the SNN model for position tracking when the change in the movement of the moving object from the spike-encoded data is not greater than a threshold for a predetermined first time period and is not less than a threshold for a predetermined second time period, and applies the spike-encoded data to the determined SNN model for position tracking to track the position of the moving object and output the position, and the second time period may be longer than the first time period.

[0014] The above processor can perform data segmentation on the spike-encoded data according to the type of the SNN model.

[0015] In order to achieve the above-described other technical task, a neuromorphic processing method according to the present invention is characterized by including the steps of: receiving UWB radar data measured for a moving target from a plurality of UWB (Ultra-wideband) radar sensors, respectively; preprocessing the received UWB radar data by imaging it; spike-encoding each image corresponding to the preprocessed UWB radar data; and determining the type of a predetermined learned spiking neural network (SNN) model to which the spike-encoded data is to be input based on the spike-encoded data.

[0016] The above-described learned SNN model may be any one of an SNN model for tracking the location of the moving object, an SNN model for detecting a fall of the moving object, and an SNN model for monitoring the biosignals of the moving object.

[0017] The method may further include a step of determining the type of the SNN model as an SNN model for fall detection when it is determined from the spike-encoded data that the amount of change in the movement of the moving object is greater than a threshold for a predetermined first period of time; and a step of applying the spike-encoded data to the determined SNN model for fall detection to detect and output whether the moving object has fallen.

[0018] The method may include a step of determining the type of the SNN model as an SNN model for bio-signal monitoring when it is determined from the spike-encoded data that the amount of change in the movement of the moving object is below a threshold for a predetermined second period of time; and a step of applying the spike-encoded data to the determined SNN model for bio-signal monitoring to output a state or value for the bio-signal of the moving object.

[0019] The method further includes the steps of: determining the type of the SNN model as an SNN model for position tracking if the change in the movement of the moving object from the spike-encoded data is not greater than a threshold for a predetermined first time period and is not less than a threshold for a predetermined second time period; and applying the spike-encoded data to the determined SNN model for position tracking to track the position of the moving object and output the position, wherein the second time period may be longer than the first time period.

[0020] The method may further include a step of performing data segmentation on the spike-encoded data according to the type of the SNN model.

[0021] The neuromorphic device according to the present invention has the advantage of being able to be expanded to indoor environments (e.g., home environments) by using a UWB radar sensor with less privacy invasion issues, thereby enabling services such as providing guidance on emergency situations by detecting falls of the elderly or other people in need of protection in the home environment, providing healthcare services through bio-signal monitoring, and providing guidance on emergency situations through location tracking.

[0022] In addition, the neuromorphic device (500) according to the present invention has the advantage of requiring fewer resources and being easy to use for a long period of time compared to existing deep learning models by using a model based on the SNN algorithm.

[0023] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention belongs from the description below.

[0024] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide embodiments of the present invention and, together with the detailed description, explain the technical idea of ​​the present invention.

[0025] Figure 1 is a diagram illustrating the layer structure of an artificial neural network.

[0026] Figure 2 is a diagram illustrating an example of a deep neural network.

[0027] Figure 3 is a diagram for explaining a spike encoding method for input data when learning a Spiking Neural Network (SNN) algorithm model.

[0028] Figure 4 is a diagram for explaining the correlation between input data and output data when learning an SNN model according to the present invention.

[0029] FIG. 5 is a block diagram illustrating the function and configuration of a neuromorphic device (500) according to the present invention.

[0030] Figure 6 is a diagram illustrating multiple UWB radar sensors monitoring a person moving indoors.

[0031] FIG. 7 is a diagram showing an example of a neuromorphic device (500) according to the present invention imaging from measurement data of a UWB radar sensor.

[0032] Figure 8 is a diagram illustrating a process of converting an input image by applying spike encoding to a UWB radar data input image that has gone through a preprocessing process.

[0033] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The detailed description set forth below, together with the accompanying drawings, is intended to illustrate exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced. The following detailed description includes specific details to provide a thorough understanding of the present invention. However, one of ordinary skill in the art will appreciate that the present invention may be practiced without these specific details.

[0034] In some cases, to avoid ambiguity in the concepts of the present invention, well-known structures and devices may be omitted or illustrated in block diagram form focusing on the core functions of each structure and device. Furthermore, the same components are described using the same reference numerals throughout this specification.

[0035] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated and described in detail in the drawings. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention.

[0036] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0037] Terms such as first, second, etc. may be used to describe various components, but these components should not be limited by these terms. These terms are used solely to distinguish one component from another.

[0038] The terminology used herein is merely used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0039] In addition, it is to be understood that the components of the embodiments described with reference to each drawing are not limited to the specific embodiments, but may be implemented to be included in other embodiments within the scope in which the technical idea of ​​the present invention is maintained, and that multiple embodiments may be re-implemented as a single integrated embodiment even if a separate description is omitted.

[0040] Additionally, terms such as “part,” “unit,” “module,” and “device” described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware, software, or a combination of hardware and software.

[0041] Deep learning is a type of artificial neural network (ANN) that utilizes the theory of the human neural network (Neural Network). It is a set of machine learning models or algorithms that refer to a deep neural network (DNN) that is structured in a layer structure and has one or more hidden layers (hereinafter referred to as intermediate layers) between the input layer and the output layer. Simply put, deep learning can be said to be an artificial neural network with a deep layer.

[0042] Artificial neural networks (ANNs), a branch of artificial intelligence, are mathematical models modeled after the structure of the biological (typically human) brain. In other words, ANNs mimic the information processing and transmission processes of biological neurons. Similar to how the human brain solves problems, ANNs exhibit excellent parallelism because each neuron operates independently. Furthermore, because information is distributed across numerous connections, problems in a few neurons do not significantly impact the overall system. Consequently, ANNs are robust to a certain level of error and possess the ability to learn from a given environment.

[0043] Deep neural networks (DNNs) can be considered descendants of artificial neural networks (ANNs). They transcend existing limitations and achieve success in areas where numerous AI technologies have failed in the past. They are the latest version of ANNs. Looking at the modeling of ANNs based on biological neural networks, the processing units are modeled as nodes, and the connections are modeled as synapses, which are weights, as shown in Table 1.

[0044] Biological neural network, artificial neural network, cell body, node, dendrite, input, axon, output, synapse, weight

[0045] Figure 1 is a diagram illustrating the layer structure of an artificial neural network.

[0046] Just as human biological neurons are interconnected in multiple layers to perform meaningful tasks, individual neurons in artificial neural networks are interconnected through synapses, creating multiple layers where the strength of connections between layers can be updated using weights. This multilayered structure and connection strengths are utilized in fields such as learning and cognition.

[0047] Each node is connected by weighted links, and the entire model learns by repeatedly adjusting the weights. Weights are the basic means of long-term memory and express the importance of each node. Simply put, an artificial neural network trains the entire model by initializing these weights and updating and adjusting them with the training data set. After training is complete, when a new input value is received, the appropriate output value is inferred. The learning principle of an artificial neural network can be viewed as a process in which intelligence is formed through the generalization of experience and is performed in a bottom-up manner. In Figure 1, when there are two or more intermediate layers (i.e., 5 to 10), it is considered deep and is called a deep neural network. Learning and inference models achieved through such a deep neural network can be referred to as deep learning.

[0048] While artificial neural networks can perform to some extent with a single intermediate layer (commonly referred to as a hidden layer) beyond the input and output layers, as the problem complexity increases, the number of nodes or layers must be increased. While increasing the number of layers to achieve a multilayered model is effective, its application is limited due to the impossibility of efficient learning and the large computational load required to train the network.

[0049] However, by overcoming these limitations, artificial neural networks have been able to achieve deep structures. This has enabled the construction of complex, highly expressive models, leading to groundbreaking results in diverse fields such as speech recognition, face recognition, object recognition, and character recognition.

[0050] Figure 2 is a diagram illustrating an example of a deep neural network.

[0051] A deep neural network (DNN) is an artificial neural network (ANN) with multiple hidden layers between the input layer and the output layer. It is a collection of machine learning models or algorithms that refer to deep neural networks (DNNs) with one or more hidden layers between the input layer and the output layer. The connections in the neural network are made from the input layer to the hidden layer, and from the hidden layer to the output layer.

[0052] Deep neural networks, like typical artificial neural networks, can model complex non-linear relationships. For example, in a deep neural network architecture for object recognition, each object can be represented as a hierarchical structure of basic image elements. Additional layers can then gradually aggregate features from lower layers. This characteristic of deep neural networks allows them to model complex data with a smaller number of units (nodes) compared to similarly implemented artificial neural networks.

[0053] While previous deep neural networks were typically designed as feed-forward neural networks, recent research has successfully applied deep learning structures to recurrent neural networks (RNNs). For example, deep neural network structures have been applied to language modeling. Convolutional neural networks (CNNs) have been successfully applied to computer vision. Furthermore, CNNs have been applied to acoustic modeling for Automatic Speech Recognition (ASR), and are considered more successful than existing models. Deep neural networks can be trained using the standard error backpropagation algorithm, where weights can be updated using stochastic gradient descent.

[0054] Deep learning using artificial neural networks (AINNs) has attracted significant attention in various fields, significantly outperforming existing algorithms. However, the current deep learning methods require significant power consumption, making them difficult to apply to resource-constrained mobile environments. Consequently, interest in spiking neural networks (SNNs), which can operate at low power, is growing. SNNs learn synaptic weights using the STDP algorithm, which adjusts synaptic weights based on the temporal relationship between pre- and post-synaptic spikes. Therefore, SNNs can be trained in various configurations, depending on the number of spikes used in training and the temporal interaction between spikes.

[0055] Neuromorphic chips, AI computing chips, are attracting attention as a next-generation technology because they can solve the power constraints of conventional semiconductor chips and integrate data processing. The core of neuromorphic technology is to mimic the human brain, enabling massive simultaneous memory and computation. Furthermore, neuromorphic chips model how neurons in the brain communicate and learn, using spikes (electrical impulses) and synapses that can be modulated according to the situation. Furthermore, these chips are designed to self-organize and make decisions based on learned patterns and associations.

[0056] Neuromorphic computing

[0057] A key challenge in neuromorphic research is to study the ability to learn from unstructured stimuli with the flexibility of a human brain, at the energy efficiency level of the human brain. The computing components of neuromorphic computing systems are logically analogous to neurons. SNNs are a novel model that arranges these elements to mimic the natural neural networks found in biological brains. Each "neuron" in an SNN operates independently, sending pulses to other neurons in the network and directly altering their electrical states. Encoding information and timing within the signal itself, SNNs dynamically remap the synapses between artificial neurons in response to stimuli, simulating natural learning processes.

[0058] Neuromorphic computing, or neuromorphic engineering, is a field of engineering that aims to mimic the functions of the human brain by creating circuits that mimic the shape of neurons. These circuits and chips are called neuromorphic circuits and neuromorphic chips, respectively. While artificial neural networks are software-based simulations of the human nervous system, neuromorphic chips are hardware-based simulations of neurons. In other words, neuromorphic chips are computer chips that mimic the structure of the biological nervous system (brain). Because these chips consist solely of the circuitry necessary for neural network computation, they offer advantages in power, area, and speed that are hundreds of times greater than those achieved using CPUs and GPUs.

[0059] While ASIC chips specifically designed for abstracted artificial neural network circuits like deep neural networks (DNNs) and convolutional neural networks (CNNs), such as Google's TPU, exist, they are generally not categorized as neuromorphic chips. While TPUs are implemented similarly to conventional DSPs, widely researched neuromorphic chips typically implement individual neurons independently, have higher data locality, and typically employ learning algorithms other than backpropagation. A prime example is an SNN that implements spike-time dependent plasticity (STDP). This approach can ultimately achieve better scalability and higher performance than conventional centrally controlled DSPs. However, due to difficulties in algorithmic and circuit implementation, there have been no significant industrial achievements to date.

[0060] Unlike conventional computers, the human brain processes vast amounts of data while consuming virtually no power. This is due to the parallel structure connecting neurons and synapses. Synapses conserve energy by connecting and disconnecting when working or not. Conventional computers consume significant power when processing data between the CPU and memory, but neuromorphic chips mimic the brain's workings to reduce power consumption.

[0061] Spiking Neural Network (SNN) algorithm model

[0062] SNNs can be called spiking artificial neural networks. The most significant difference from typical artificial neural networks is the presence of a time axis. Rather than simply receiving the values ​​of previous neurons once per neuron, the neuron's values ​​(internal states) continuously change over time. To prevent the network from becoming monotonous and to better simulate the actual brain, the concepts of threshold and spike were developed. When a neuron's internal state exceeds the threshold, it transmits a spike to connected neurons, resetting the internal state. The neuron receiving the spike then experiences an increase or decrease in its internal state depending on the synaptic weights. This spike can trigger another spike in the next neuron. After specifying a time range, the input is the "spike frequency of the input neuron," and the output is measured as the "number of spikes appearing in the output neuron."

[0063] Multi-Spiking Neural Network

[0064] Here, a model called a Multi-Spiking Neural Network (MSN) more accurately simulates the brain. In contrast to the previously introduced model, it's also called a Single-Spiking Neural Network. What's different about this model is that multiple synapses connect the same pair of neurons. Each synapse has a different speed, resulting in different delays in the transmission of spike signals.

[0065] The structure of a neural network itself is similar to a typical ANN: it simply consists of an input layer, a hidden layer, and an output layer. Complex structures such as a Spiking CNN could be drawn, but this is beyond the scope of this discussion. All neurons in each neighboring layer are connected. Now, by dividing a unit of time (in seconds) into timestamps, the process of constructing a new state from the state of the neural network one second prior is repeated for the number of timestamps (usually 60). The state of the neural network consists of only one variable: the internal state of the neuron, which will be discussed below. All other factors are either hyperparameters to be learned or constants.

[0066] Each neuron has a variable called "internal state" (V). This value starts at 0 by default and is a real number less than or equal to a threshold (Vth). Vth is a specific value for each neuron and is one of the learning targets. However, input neurons do not have this value and only fire. The firing period is determined by the corresponding input value (often using the Poisson distribution). Neurons that are not input neurons fire when V≥Vth (threshold voltage), and immediately after firing, V decreases by Vth. Mimicking the refractory period in biology (the period during which a neuron that has fired cannot fire again within a certain period of time), neurons that have fired once do not fire again immediately.

[0067] The value of V continuously decreases (in absolute value) unless a spike (firing) is transmitted from previous neurons. In the paper, V decreases exponentially. If a spike is transmitted through a synapse from a neuron in the previous layer, V increases by the synaptic weight wij. The sign of w determines whether V increases or decreases.

[0068] In this way, SNNs fire only when the membrane potential of a neuron is higher than the threshold voltage, and transmit information between synapses through the fired spikes. This allows for event-driven operation, enabling low-power operation compared to other artificial neural networks. Since neurons and synapses in SNNs are not differentiable, gradient descent and error backpropagation cannot be used to train SNNs. The most widely known training method for SNNs is STDP (Spiking Timing Dependent Plasticity). STDP is a method that learns synaptic weights through the temporal relationship between pre-synaptic and post-synaptic spikes. Therefore, the number of pre- and post-synaptic spikes considered in STDP training and the temporal interaction between spikes affect SNN training.

[0069] Figure 3 is a diagram for explaining a spike encoding method for input data when learning a Spiking Neural Network (SNN) algorithm model.

[0070] Referring to Figure 3, training an SNN model requires encoding input data into spikes. Spike encoding methods include rate coding, latency coding, and delta modulation, and spike encoding methods are categorized based on the input data input method. Spike encoding is possible for image data, 2D tensors, and 1D tensors (time series data).

[0071] Neuromorphic computing learning device

[0072] The SNN algorithm model has an input neuron layer, a first neuron layer (corresponding to an excitatory neuron layer), and a second neuron layer (corresponding to an inhibitory neuron layer). In the present invention, the energy value of the preprocessed UWB radar signal, which is input data, can be input to the input neuron layer in the form of Leaky Integrate-and-Fire (LIF) after spike encoding. As illustrated in Fig. 4, the Spike-Timing-Dependent Plasticity (STDP) algorithm, which is an unsupervised learning algorithm, is a method of connecting the input neuron layer to the excitatory neuron layer in a fully connected manner and learning the weights between them. Spikes fired beyond thresholds are accumulated in the excitatory neuron layer, and the excitatory neuron layer is again connected one-to-one with the inhibitory neuron layer (second neuron layer), thereby transmitting the fact of spike firing. In other words, the spike is transmitted. The inhibitory neuron layer is connected to the excitatory neuron layer in a many-to-many manner, allowing it to transmit a negative inhibitory value (e.g., -50) to all excitatory neurons except the one that transmitted the spike (e.g., the primary excitatory neuron). This is called lateral inhibition. As the series of processes repeats, the specific excitatory neurons that respond to specific data are determined.

[0073] LIF model

[0074] The Leaky Integrate-and-Fire (LIF) model, used in SNNs, is a neuron model designed to mimic the structure of real neurons. In neurons, when spikes received from multiple neurons via dendrites exceed a threshold voltage, they fire spikes to other neurons via the axon. Similarly, in the LIF model, a threshold voltage (Vth) is set. If the accumulated number of spikes (membrane potential) exceeds this threshold voltage, spikes are fired to other neurons, and the membrane potential is reset with Vreset. Furthermore, the neuron enters a refractory period, preventing it from firing spikes for several milliseconds. Since spikes are transmitted between neurons through pathways called synapses, the LIF model uses synapses as a concept similar to weights. A synapse is the gap between the terminal end of the axon, the part of a neuron that conducts impulses out of the cell, and the next neuron through which neurotransmitters pass.

[0075] Spike-Timing-Dependent Plasticity (STDP)

[0076] The Spike-Timing-Dependent Plasticity (STDP) algorithm, an unsupervised learning algorithm, learns the weights between the fully connected input neuron layer and the excitatory neuron layer. STDP is a biologically based learning method and a representative unsupervised learning method for SNNs. The input neuron layer and the excitatory neuron layer are fully connected, similar to the ANN structure. STDP trains each connection, or synapse (=weight). Presynaptic and postsynaptic neurons are established through input neurons, and the difference in spike firing times between neurons is calculated to adjust the weights. The closer the firing times are, the larger the absolute value of the weight increase or decrease. If the pre- and post-neurons fire sequentially, the increase is significant, and if the order is reversed, the increase is significant. Through these established weight values, the relationship between input neurons is recorded as the number of spikes fired in the excitatory neuron layer.

[0077] Inhibition weight update

[0078] Inhibitory weight updating is the process of adjusting inhibitory weights (or inhibitory values) to identify neurons that respond to the same data together. Neuromorphic computing modules can adjust inhibitory weights to identify neurons that respond to the same data together.

[0079] In the excitatory neuron layer, the inhibitory neuron layer is connected one-to-one, for example, excitatory neuron A transmits a spike to inhibitory neuron A. In the inhibitory neuron layer, the excitatory neuron layer is connected many-to-many, so inhibitory neuron A can transmit a negative (-) value (for example, -50) to all excitatory neurons except excitatory neuron A.

[0080] Figure 4 is a diagram for explaining the correlation between input data and output data when learning an SNN model according to the present invention.

[0081] Referring to (a) and (b) of Fig. 4, after spike encoding, the Leaky Integrate-and-Fire (LIF) neuron model is used as an activation function for SNN model training. The nonlinear activation of the ReLU neuron of the existing ANN and the firing rate of the LIF neuron of the SNN model show a high correlation, and the SNN model can be trained based on these characteristics. If the amplitude value of 1D time series data (e.g., energy of UWB radar signal, ECG signal, EEG signal) is large, the characteristics of the spike appear differently depending on the spike encoding method. The energy value of the UWB radar signal is encoded and the corresponding spike is input to the SNN model to train the LIF model-based SNN, and the output value is expressed in the form of a spike.

[0082] By training the SNN model, the SNN model is also utilized for the UWB radar signal proposed in the present invention. For example, if the energy value of the measured UWB radar signal is spike-encoded and the SNN model is trained, the characteristics of the change in magnitude appearing in the radar signal show different patterns in the output spike, and based on this, bio-signals such as respiration rate and heart rate are estimated. Among UWB radar signals, IR-UWB (impulse radio ultra-wideband) radar technology is a short-range positioning technology that transmits and receives impulse signals using a wideband frequency. Its main uses include distance estimation to objects, object position measurement, occupancy detection, people counting, congestion management, and bio-signal measurement.

[0083] The present invention proposes a method for tracking the movement of a moving object (e.g., a person moving indoors) using multiple or more UWB radar sensors placed in various locations in a home environment or indoors, and extracting information by designing a low-power system using neuromorphic hardware and utilizing an artificial intelligence model including an SNN.

[0084] FIG. 5 is a block diagram illustrating the function and configuration of a neuromorphic device (500) according to the present invention.

[0085] Referring to FIG. 5, a neuromorphic device (500) according to the present invention may include a processor (510), a communication unit (520), and a memory (530).

[0086] The processor (510) can perform functions such as tracking the movement of a moving object (such as a person, hereinafter referred to as a "person") to determine its location, detecting a fall, or monitoring a biosignal by performing computational processing (computing) on ​​input data according to the Spiking Neural Networks (SNN) algorithm model according to the present invention. The processor (510) can perform these functions in parallel or independently. The memory (530) stores various information necessary for neuromorphic computing, such as information necessary for the processor (510) to estimate a result, information about the estimated result, etc.

[0087] Figure 6 is a diagram illustrating multiple UWB radar sensors monitoring a person moving indoors.

[0088] Referring to FIG. 6, it can be seen that UWB radar sensors (610, 620, 630, 640) are placed in each room indoors. The UWB radar sensors (610, 620, 630, 640) measure UWB radar signals for each moving person. Thereafter, the UWB radar sensors (610, 620, 630, 640) can transmit the measured UWB radar data (signal) to the neuromorphic device (500). Although FIG. 6 illustrates that the UWB radar sensors (610, 620, 630, 640) are not mounted on the neuromorphic device (500) but are separate devices, the present invention is not limited thereto, and depending on the embodiment, the UWB radar sensors (610, 620, 630, 640) may be mounted on the neuromorphic device (500).

[0089] Each UWB radar sensor (610, 620, 630, 640) transmits and receives radar signals to perform measurements. Each UWB radar sensor (610, 620, 630, 640) can measure the numerical value of the reflected radar signal after transmitting the radar signal. The numerical value (raw data) of the reflected radar signal may be the power (energy) intensity of the radar signal, and the power intensity of the reflected radar signal varies depending on the distance.

[0090] Each UWB radar sensor (610, 620, 630, 640) generates and transmits a UWB (Ultra-wideband) radar signal for a moving person, and can estimate the distance to the moving person based on the numerical value of the reflected radar signal. In the present invention, since the accuracy of estimating the movement and position tracking of a moving object may not be high with only a single UWB radar signal, the neuromorphic device (500) is designed to increase the accuracy by applying the numerical value of the UWB radar signal measured by multiple UWB radar sensors (610, 620, 630, 640) to an artificial intelligence model such as an SNN algorithm model.

[0091] In addition, when used in a home environment, each UWB radar sensor (610, 620, 630, 640) does not operate all the time, but generates impulses at regular intervals to detect movement when a connection with a home environment IOT device is confirmed or a moving person is recognized.

[0092] The communication unit (530) of the neuromorphic device (500) can receive or acquire UWB radar signals measured for each moving target from a plurality of UWB radar sensors. The processor (510) performs preprocessing on the received UWB radar data. Here, the preprocessing may include removing baseline drift and excessive movement elements by applying clutter removal and / or filtering corresponding to external noise. In addition, the preprocessing may include a process of imaging the numerical values ​​(raw data) of UWB radar signals reflected by the plurality of UWB radar sensors. That is, the processor (510) can preprocess by imaging each received UWB radar data. As an example, when imaging the measured UWB radar data, the processor (510) may perform Doppler imaging.

[0093] FIG. 7 is a diagram illustrating an example of a neuromorphic device (500) according to the present invention imaging from measurement data of a UWB radar sensor, and FIG. 8 is a diagram illustrating a process of converting an input image by applying spike encoding to a UWB radar data input image that has gone through a preprocessing process.

[0094] The processor (510) preprocesses the measured UWB radar data by converting it into an image. Fig. 7 (a) is an example of an image generated based on the values ​​measured by the UWB radar sensor (510) when movement occurs. Fig. 7 (b) is a training data sample for training the SNN algorithm model (hereinafter, abbreviated as the SNN model). As shown in the preprocessed image, the white portion can identify the movement path of a moving person.

[0095] Fig. 8 illustrates an example of a spike-encoded image after a UWB radar signal preprocessing process. The processor (510) performs spike encoding on an image (810) corresponding to the preprocessed UWB radar data. The processor (510) converts the input image by applying spike encoding. Fig. 8 is a diagram illustrating image conversion according to the number of steps. Fig. 8 illustrates eight step images (from a 1-step spike image to an 8-step spike image). The spike image changes depending on the number of steps. Here, the reason why there are multiple steps is to configure each step image to be learned by varying the threshold related to the spike, and the learning ability can be improved as spike images with a large number of steps are learned. The above-described learned SNN model can be configured to learn by calculating a loss value from each step image and adjusting the weights.

[0096] The processor (510) inputs, for example, the eight-step spike image in FIG. 8 into a predetermined learned SNN model. In this way, the processor (510) can apply spike encoding to convert the input image and then apply it to the predetermined learned SNN model. At this time, the processor (510) can determine the type of the predetermined learned SNN model to which the spike-encoded data will be input based on the spike-encoded data. The types of the predetermined learned SNN models include an SNN model for tracking the position of a moving target, an SNN model for detecting a fall of a moving target, and an SNN model for monitoring a biosignal (vital signal) of a moving target. The processor (510) can determine at least one SNN model from the types of SNN models based on the characteristics of the spike-encoded data and apply the spike-encoded data.

[0097] As an example, if the processor (510) determines that the change in the movement of a moving object from spike-encoded data is greater than a threshold value for a predetermined first period of time, the processor (510) may determine the type of the SNN model as the SNN model for fall detection. Thereafter, the processor (510) may apply the spike-encoded data to the determined SNN model for fall detection to detect whether the moving object has fallen and output the result.

[0098] As another example, if the processor (510) determines from the spike-encoded data that the amount of change in the movement of a moving object is less than or equal to a threshold for a predetermined second period of time, the processor (510) may determine the type of the SNN model as an SNN model for biosignal monitoring (e.g., blood pressure, heart rate, stress index, heart rate variability, ballistic cardiogram, etc.). Thereafter, the processor (510) may apply the spike-encoded data to the determined SNN model for biosignal monitoring to output a state or value for the biosignal (e.g., blood pressure, heart rate, stress index, heart rate variability, ballistic cardiogram, etc.) of the moving object. Here, the second period of time (second period of time) is longer than the first period of time (first period of time).

[0099] As another example, the processor (510) may determine the type of the SNN model as an SNN model for position tracking if the change in the movement of a moving object from the spike-encoded data is not greater than a threshold for a predetermined first time period and not less than a threshold for a predetermined second time period. Thereafter, the processor (510) may apply the spike-encoded data to the determined SNN model for position tracking to track the position of the moving object and output or identify the position.

[0100] Thus, the SNN model for fall detection may be selected when it exhibits the greatest change in human movement over a relatively short period of time compared to other SNN models. The SNN model for biosignal monitoring may be selected when it exhibits the smallest change in human movement over a relatively long period of time compared to other SNN models. In other cases, the processor (510) may determine the type of SNN model as an SNN model for location tracking.

[0101] The processor (510) can perform data segmentation according to the type of SNN model determined for spike-encoded data. The communication unit (520) can transmit the results output by the processor (510) for the corresponding SNN model to a terminal, server, healthcare system, or integrated system, thereby providing healthcare services to users.

[0102] As described above, the neuromorphic device (500) according to the present invention can be expanded to an indoor environment (e.g., a home environment) by using a UWB radar sensor with less privacy invasion issues, and thus has the advantage of enabling services such as guidance on emergency situations through fall detection for the elderly or the weak who require protection in a home environment, healthcare services through bio-signal monitoring, and guidance on emergency situations through location tracking.

[0103] In addition, the neuromorphic device (500) according to the present invention has the advantage of requiring fewer resources and being easy to use for a long period of time compared to existing deep learning models by using a model based on the SNN algorithm.

[0104] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0105] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed across network-connected computing devices and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0106] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CDROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0107] The embodiments described above are combinations of components and features of the present invention in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented without being combined with other components or features. Furthermore, it is also possible to form an embodiment of the present invention by combining some components and / or features. The order of operations described in the embodiments of the present invention may be changed. Some components or features of one embodiment may be included in another embodiment or may be replaced with corresponding components or features of another embodiment. It is self-evident that claims that do not have an explicit citation relationship in the patent claims may be combined to form an embodiment or may be incorporated as a new claim through a post-application amendment.

[0108] In the present invention, the processor (510) may be implemented by hardware, firmware, software, or a combination thereof. When implementing an embodiment of the present invention using hardware, application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), etc. configured to perform the present invention may be provided in the processor (510).

[0109] It will be apparent to those skilled in the art that the present invention can be embodied in other specific forms without departing from the essential characteristics thereof. Therefore, the above detailed description should not be construed as limiting in any respect, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the scope of equivalents of the present invention are intended to be included within the scope of the present invention.

[0110] The neuromorphic device according to the present invention can be used industrially to provide services such as guidance on emergency situations by detecting falls of the elderly or others in need of protection in a home environment, healthcare services through bio-signal monitoring, and guidance on emergency situations through location tracking.

Claims

1. A communication unit that receives UWB radar data measured for each moving target from multiple UWB (Ultra-wideband) radar sensors; and The UWB radar data received above is imaged and preprocessed, Each image corresponding to the above preprocessed UWB radar data is spike encoded, A neuromorphic device, characterized by including a processor that determines a type of a predetermined learned spiking neural network (SNN) model to which the spike-encoded data will be input based on the spike-encoded data.

2. In paragraph 1, A neuromorphic device, characterized in that the learned SNN model is any one of an SNN model for position tracking of the moving object, an SNN model for fall detection of the moving object, and an SNN model for bio-signal monitoring of the moving object.

3. In paragraph 2, The above processor, If it is determined from the spike encoded data that the amount of change in the movement of the moving object is greater than a threshold value for a predetermined first time period, the type of the SNN model is determined as an SNN model for fall detection. A neuromorphic device that detects whether the moving object has fallen and outputs the result by applying the spike encoding data to the SNN model for fall detection determined above.

4. In paragraph 2, The above processor, If it is determined from the spike encoded data that the amount of change in the movement of the moving object is below a threshold for a predetermined second time period, the type of the SNN model is determined as an SNN model for biosignal monitoring. A neuromorphic device that applies the spike encoding data to the SNN model for monitoring the determined biosignals to output a state or value for the biosignals of the moving object.

5. In paragraph 2, The above processor, If the change in the movement of the moving object from the spike encoded data is not greater than a threshold for a predetermined first time period and not less than a threshold for a predetermined second time period, the type of the SNN model is determined as an SNN model for position tracking. By applying the spike encoding data to the SNN model for the above-determined location tracking, the location of the moving object is tracked and the location is output. A neuromorphic device, characterized in that the second time period is longer than the first time period.

6. In paragraph 2, The above processor, A neuromorphic device characterized by performing data segmentation according to the type of the SNN model on the spike encoded data.

7. A step of receiving UWB radar data measured for each moving target from a plurality of UWB (Ultra-wideband) radar sensors; A step of preprocessing each of the above received UWB radar data by imaging it; A step of spike encoding each image corresponding to the above preprocessed UWB radar data; and A neuromorphic processing method, characterized by comprising a step of determining a type of a predetermined learned spiking neural network (SNN) model to input the spike encoded data based on the spike encoded data.

8. In paragraph 7, A neuromorphic processing method, characterized in that the above-described learned SNN model is any one of an SNN model for position tracking of the moving object, an SNN model for fall detection of the moving object, and an SNN model for bio-signal monitoring of the moving object.

9. In paragraph 8, A step of determining the type of the SNN model as an SNN model for fall detection when it is determined from the spike encoded data that the amount of change in the movement of the moving object is greater than a threshold value for a predetermined first time period; and A neuromorphic processing method, characterized in that it further includes a step of applying the spike encoding data to the determined SNN model for fall detection to detect and output whether the moving object has fallen.

10. In paragraph 8, A step of determining the type of the SNN model as an SNN model for biosignal monitoring when it is determined from the spike encoded data that the amount of change in the movement of the moving object is below a threshold for a predetermined second time period; and A neuromorphic processing method, characterized by including a step of applying the spike encoding data to the SNN model for monitoring the determined biosignal to output a state or value for the biosignal of the moving object.

11. In paragraph 8, A step of determining the type of the SNN model as an SNN model for position tracking if the amount of change in the movement of the moving object from the spike encoded data is not greater than a threshold for a predetermined first time period and not less than a threshold for a predetermined second time period; and Further comprising a step of applying the spike encoding data to the SNN model for the determined location tracking to track the location of the moving object and output the location, A neuromorphic processing method, characterized in that the second time period is longer than the first time period.

12. In paragraph 8, A neuromorphic processing method, characterized in that it further comprises a step of performing data segmentation according to the type of the SNN model for the spike encoded data.

13. A computer-readable recording medium storing a program for executing a neuromorphic processing method described in any one of claims 7 to 12 on a computer.

Images