Apparatus and method for generating images using brain waves

Abstract

The disclosure relates to an apparatus for generating an image using a brainwave signal. The apparatus includes a sensor configured to collect a brainwave signal of at least one passenger in a moving body from a plurality of channels for a predetermined time, and a controller configured to generate a first image using an artificial intelligence model according to the brainwave signal collected from the plurality of channels, select at least one second image included in a predetermined list based on the generated first image, and control the moving body in response to the selected second image.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of Korean Patent Application No. 10-2019-0135834, filed on October 29, 2019, which is incorporated herein by reference. Technical Field

[0003] This disclosure relates to a method and apparatus for controlling a moving body. Background Technology

[0004] The statements in this section provide only background information in connection with this disclosure and may not constitute prior art.

[0005] Vehicles (or mobile bodies) are a very important means and tool in modern life as a means of transportation. In addition, for some people, mobile bodies themselves can be regarded as special things that are endowed with certain meanings.

[0006] With technological advancements, the functions provided by mobile devices have also evolved. For example, in recent years, mobile devices have not only transported passengers to their destinations but have also met passengers' needs for faster and safer arrival. Furthermore, new features have been added to mobile device systems to satisfy passengers' aesthetic preferences and comfort. Additionally, existing features such as steering wheels, transmissions, and accelerator / decelerator mechanisms are being developed to provide users with even more functionality.

[0007] Meanwhile, brain-computer interfaces (BCIs) are a field that uses brainwave signals to control computers or machines according to a person's intentions. ERPs (Event-Related Potentials) are closely related to cognitive function.

[0008] In addition, there has been a growing trend of research into using artificial intelligence models to identify and classify objects in images and generate new images. Summary of the Invention

[0009] This disclosure relates to a method and apparatus for controlling a moving body. Specific embodiments relate to a method and apparatus for controlling a moving body based on error monitoring.

[0010] One embodiment of this disclosure provides an apparatus and method for generating images based on a passenger's brainwave signals.

[0011] Another embodiment of this disclosure provides an apparatus and method for generating images based on a passenger's brainwave signals using an artificial intelligence model and controlling a moving body based on the generated images.

[0012] Embodiments of the present disclosure are not limited to the above-described embodiments, and other embodiments not mentioned will be clearly understood by those skilled in the art through the following description.

[0013] According to an embodiment of the present disclosure, an apparatus for generating an image using a brainwave signal includes a sensor configured to collect brainwave signals of at least one passenger in a moving body from a plurality of channels for a predetermined time, and a controller configured to generate a first image using an artificial intelligence model, based on the brainwave signals collected from the plurality of channels, to select at least one second image included in a predetermined list based on the generated first image, and to control the moving body in response to the selected second image.

[0014] The brainwave signals collected from the plurality of channels can be brainwave signals in at least one of a time domain, a frequency domain, and a spatial domain.

[0015] The artificial intelligence model can be a generative adversarial neural network (GAN) model.

[0016] The first image can be at least one of a service point image representing a service point and a product image, the service point image can be an image representing a service point, and the product image can be an image representing a product provided by the service point.

[0017] The predetermined list can include at least one service point image or at least one product image.

[0018] The service point can be a drive-thru (DT) service providing place within a predetermined range from the moving body.

[0019] The predetermined range can be a range in which transmission and reception are possible through the short-range communication network when transmission and reception are performed between the moving body and the service point based on the short-range communication network.

[0020] The product is at least one of an article provided by the service point, a service provided by the service point, and information about the service point.

[0021] The controller can select the second image based on a similarity determination between the first image and the images included in the predetermined list.

[0022] When the second image is a service point image, the controller can perform at least one of changing a travel route of the moving body to the service point, providing a notification of the travel route to be changed to the passenger, and providing the travel route to be changed to the passenger to guide the passenger to select a route.

[0023] When the second image is a commodity image, the controller can perform at least one of changing a travel route of the moving body to a service point for providing a commodity, providing a notification of the travel route to be changed to the passenger, providing the travel route to be changed to the passenger to guide the passenger to select a route, and transmitting an order signal of the commodity.

[0024] The controller can be further configured to determine whether the selected second image meets an expectation of the passenger.

[0025] According to an embodiment of the disclosure, a method of generating an image using a brainwave signal includes collecting a brainwave signal of at least one passenger in a moving body from a plurality of channels for a predetermined time, generating a first image from the brainwave signal collected from the plurality of channels using an artificial intelligence model, selecting at least one second image included in a predetermined list based on the generated first image, and controlling the moving body in response to the selected second image.

[0026] The brainwave signal collected from the plurality of channels can be a brainwave signal in at least one of a time domain, a frequency domain, and a spatial domain.

[0027] The artificial intelligence model can be a generative adversarial neural network (GAN) model.

[0028] The first image can be at least one of a service point image representing a service point and a commodity image, the service point image can be an image representing a service point, and the commodity image can be an image representing a commodity provided by the service point.

[0029] The predetermined list can include at least one service point image or at least one commodity image.

[0030] The service point can be a drive-through (DT) service providing place within a predetermined range from the moving body.

[0031] When transmission and reception are performed between the moving body and the service point based on a short distance communication network, the predetermined range can be a range in which transmission and reception can be performed through the short distance communication network.

[0032] The commodity can be at least one of an article provided by the service point, a service provided by the service point, and information about the service point.

[0033] Selecting the at least one second image can include selecting the second image based on a similarity determination between the first image and images included in the predetermined list.

[0034] When the second image is a service point image, controlling the moving body can include performing at least one of changing a travel route of the moving body to the service point, providing a notification of the travel route to be changed to the passenger, and providing the travel route to be changed to the passenger to guide the passenger to select a route.

[0035] When the second image is a product image, the controlling the mobile body can include performing at least one of changing a travel route of the mobile body to a service point for providing the product, providing a notification of the changed travel route to the passenger, providing the changed travel route to the passenger to guide the passenger to select a route, and transmitting an order signal of the product.

[0036] The method can further include determining whether the selected second image meets the expectation of the passenger.

[0037] The features briefly described above for the embodiments of the disclosure are merely exemplary aspects of the following detailed description of the embodiments of the disclosure and do not limit the scope of the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order for the disclosure to be better understood, various embodiments of the disclosure will now be described, by way of example, and with reference to the accompanying drawings, in which:

[0039] Figure 1 is a view showing general waveforms of an ERN according to one embodiment of the disclosure;

[0040] Figure 2 is a view showing general waveforms of an ERN and a Pe according to one embodiment of the disclosure;

[0041] Figure 3 is a view showing a deflection characteristic of a Pe according to another embodiment of the disclosure;

[0042] Figure 4A and Figure 4B are views showing measurement regions of an ERP and a Pe, respectively, according to one embodiment of the disclosure;

[0043] Figure 5 is a view showing general waveforms of an ERN and a CRN according to one embodiment of the disclosure;

[0044] Figure 6 is a view showing EEG measurement channels corresponding to brain cortex regions according to one embodiment of the disclosure;

[0045] Figure 7 is a block diagram showing a configuration of an apparatus for generating an image based on a passenger's brainwave signal according to an embodiment of the disclosure;

[0046] Figure 8 is a view showing a range between a mobile body and a service point which transmit and receive each other according to an embodiment of the disclosure;

[0047] is a view showing a range between a mobile body and a service point which transmit and receive each other according to an embodiment of the disclosure;Figure 9 is a flowchart illustrating an operation method of an image generating apparatus according to an embodiment of the disclosure; and

[0048] Figure 10 is a flowchart illustrating an operation method of an image generating apparatus according to an embodiment of the disclosure. DETAILED DESCRIPTION

[0049] The following description is merely exemplary in nature and is not intended to limit the disclosure, application, or uses. It should be understood that throughout the drawings, corresponding reference numerals indicate like or corresponding parts and features.

[0050] Exemplary embodiments of the disclosure will be described in detail, such that one of ordinary skill in the art to which the disclosure pertains will be able to easily implement and practice the apparatus and method provided by the embodiments of the disclosure, with reference to the accompanying drawings. However, the disclosure can be implemented in various forms, and the scope of the disclosure should not be construed to be limited to the exemplary embodiments.

[0051] In describing the embodiments of the disclosure, when well-known functions or configurations can obscure the gist of the disclosure, the well-known functions or configurations will not be described in detail.

[0052] In the embodiments of the disclosure, it will be understood that when an element is referred to as being "connected to", "coupled to" or "joined to" another element, it can be directly connected to or coupled to or joined to the other element, or intervening elements can be present therebetween. It will further be understood that the terms "comprises", "comprising", "includes", "including", "has", "having" and the like, when used in the embodiments of the disclosure, mean the presence of stated features, integers, steps, operations, elements, components and / or combinations thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0053] It will be understood that, although the terms "first", "second", etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element discussed below could be termed a second element without departing from the teachings of the disclosure. Similarly, a second element could be termed a first element.

[0054] In the embodiments of the disclosure, different elements are named to clearly describe the features of various elements, but it does not mean that the elements are physically separated from each other. That is, a plurality of different elements can be combined into a single hardware unit or a single software unit, and conversely, one element can be implemented by a plurality of hardware units or software units. Therefore, although not specifically mentioned, the integrated form of various elements or the separated form of one element can fall within the scope of the disclosure. Also, terms such as "unit" or "module" should be understood as a unit that processes at least one function or operation and can be implemented in a hardware manner (for example, a processor), a software manner, or a combination of a hardware manner and a software manner.

[0055] In the embodiments of the disclosure, all constituent elements described in various forms should not be interpreted as essential elements, and some of the constituent elements can be optional elements. Therefore, embodiments configured in some form by various subsets of constituent elements can also fall within the scope of the disclosure. In addition, embodiments configured by adding one or more elements to various elements can also fall within the scope of the disclosure.

[0056] Hereinafter, embodiments of the disclosure are described with reference to the accompanying drawings.

[0057] A brain wave signal (or brain signal, brain wave) is an electrical activity of a nerve cell constituting a brain, and refers to a biological signal directly or indirectly reflecting a conscious or unconscious state of a person. The brain wave signal can be measured in all areas of a person's scalp, the frequency of the wavelength of the brain wave signal is mainly 30 Hz or less, and the potential difference of the wavelength of the brain wave signal is several tens of microvolts. Various waveforms can appear depending on the brain activity and state. Research is being conducted to control an interface using a brain wave signal according to a person's intention. The brain wave signal can be obtained by using an EEG (Electro Encephalo Graphy) using an electrical signal induced by brain activity, an MEG (Magneto Encephalo Graphy) using a magnetic signal induced together with the electrical signal, and an fMRI (functional Magnetic Resonance Imaging) or fNIRS (functional Near-Infrared Spectroscopy) using a change in oxygen saturation in blood. Although the fMRI and the fNIRS are useful techniques for measuring brain activity, the time resolution of the fMRI is generally low, and the spatial resolution of the fNIRS is low. Due to these limitations, the EEG signal is widely utilized since it has excellent portability and time resolution.

[0058] A brain wave signal varies spatially and temporally according to brain activity. Since a brain wave signal is generally difficult to analyze and a waveform of a brain wave signal is not easily analyzed visually, various processing methods have been proposed.

[0059] For example, according to the number of oscillations (frequency), a brain wave signal can be classified based on a frequency band (power spectral classification). This classification regards a measured brain wave signal as a linear sum of simple signals of each specific frequency, decomposes the signal into each frequency component, and indicates a corresponding amplitude. A brain wave signal of each frequency can be obtained by using pre-processing for removing noise in general, Fourier transform for transforming into a frequency domain, and a band-pass filter (BPF).

[0060] More specifically, according to a frequency band, a brain wave can be classified into a delta (δ) wave, a theta (θ) wave, an alpha (α) wave, a beta (β) wave, and a gamma (γ) wave. The δ wave is a brain wave having a frequency of 3.5 Hz or less and an amplitude of 20 to 200 μV, and mainly occurs in a deep sleep state of a normal person or a newborn. In addition, the δ wave can increase as our awareness of the physical world decreases. In general, the θ wave is a brain wave having a frequency of 3.5 to 7 Hz, and mainly occurs in a state of emotional stability or a sleep state.

[0061] In addition, the θ wave is mainly generated in a parietal cortex and an occipital cortex, and can occur in a calm and focused state of recollection or meditation. In general, the α wave is a brain wave having a frequency of 8 to 12 Hz, and mainly occurs in a relaxed and comfortable state. In addition, the α wave is generally generated in an occipital cortex in a resting state, and can decrease in a sleep state. In general, the β wave is a brain wave having a frequency of 13 to 30 Hz, and mainly occurs in a state of tolerable tension or when causing a certain degree of attention. In addition, the β wave is mainly generated in a frontal cortex, and is related to a state of wakefulness or focused brain activity, a pathological phenomenon, and a drug effect. The β wave can occur in a wide area of the entire brain. In addition, specifically, the β wave can be classified into an SMR wave having a frequency of 13 to 15 Hz, a middle β wave having a frequency of 15 to 18 Hz, and a high β wave having a frequency of 20 Hz or more. Since the β wave occurs more strongly under stress such as anxiety and tension, the β wave is called a stress wave. The γ wave is generally a brain wave having a frequency of 30 to 50 Hz, and mainly occurs in a state of strong excitement or a high-level cognitive information processing process. In addition, the γ wave can occur during a state of consciousness and REM sleep, and can also overlap with the β wave.

[0062] Each of the brain wave signals according to the frequency band is associated with a specific cognitive function. For example, a delta wave is associated with a sleep function, a theta wave is associated with a working memory, and an alpha wave is associated with an attention function or an inhibition function. Accordingly, the characteristics of each of the brain wave signals of the frequency band selectively show a specific cognitive function. In addition, the brain wave signal of each of the frequency bands can show some different aspects in each of the measurement sites on the head surface. The cerebral cortex can be divided into a frontal cortex, a parietal cortex, a temporal cortex, and an occipital cortex. The roles of these parts can be slightly different. For example, the occipital cortex corresponding to the back of the head has a main visual cortex, and thus can be mainly responsible for processing visual information. The parietal cortex located near the top of the head has a somatosensory cortex, and thus can be responsible for processing motor / sensory information. In addition, the frontal cortex can process information related to memory and thinking, and the temporal cortex can process information related to hearing and smell.

[0063] Meanwhile, for another example, the brain wave signal can be analyzed by utilizing an ERP (Event Related Potential). The ERP is an electrical change in the brain associated with an external stimulus or an internal mental process. The ERP refers to a signal including electrical activity of the brain triggered by a stimulus including specific information (e.g., an image, a voice, a sound, an execution command, etc.) after a certain time from the appearance of the stimulus.

[0064] In order to analyze the ERP, a process of separating a signal from noise is required. The averaging method can be mainly utilized. Specifically, by averaging the brain waves measured based on the stimulus appearance time, the brain waves irrelevant to the stimulus can be removed, and only the relevant potential as the brain activity commonly participating in the stimulus processing can be selected.

[0065] Since the time resolution of ERP is high, ERP is closely related to the study of cognitive function. ERP is an electrical phenomenon evoked by external stimulation or associated with internal state. According to the type of stimulation, ERP can be classified into auditory-related potential, visual-related potential, somatosensory-related potential, and olfactory-related potential. According to the nature of the stimulation, ERP can be classified into exogenous ERP and endogenous ERP. Exogenous ERP has a waveform determined by external stimulation, is associated with automatic processing, and mainly occurs in the initial stage of being stimulated. For example, exogenous ERP is brainstem potential and the like. On the other hand, endogenous ERP is determined by internal cognitive processes or psychological processes or states, is independent of stimulation, and is associated with "controlled processing". For example, endogenous ERP is P300, N400, P600, CNV (Contingent Negative Variation), and the like.

[0066] The name of the ERP peak generally includes polarity and latency, and each signal of the peak has a respective definition and meaning. For example, a positive potential is P, a negative potential is N, and P300 indicates a positive peak measured about 300 ms after the occurrence of stimulation. In addition, 1, 2, 3, or a, b, c, and the like are applied according to the order of occurrence. For example, P3 indicates the third positive potential in the waveform after the occurrence of stimulation.

[0067] Hereinafter, various ERPs will be described.

[0068] As one example, N100 is associated with a response to unpredictable stimulation.

[0069] MMN (Mismatch Negativity) can be generated not only by a stimulus of interest but also by a non-stimulus of interest. MMN can be used as an indicator of whether sensory memory (echoic memory) operates before initial attention. P300 to be described hereinafter occurs in the process of paying attention and making a judgment, whereas MMN is analyzed as a process occurring in the brain before paying attention.

[0070] As another example, N200 (or N2) is mainly generated according to visual stimulation and auditory stimulation, and is associated with short-term memory or long-term memory as a form of memory after attention, together with P300 to be described hereinafter.

[0071] As still another example, the P300 (or P3) mainly reflects attention to a stimulus, stimulus cognition, memory search, and uncertainty resolution, and is related to a perceptual decision to distinguish an external stimulus. Since the generation of the P300 is related to cognitive functions, the P300 is generated regardless of the type of the stimulus that occurs. For example, the P300 can be generated in auditory stimuli, visual stimuli, and somatosensory stimuli. The P300 is widely used in research on brain-computer interfaces.

[0072] As still another example, the N400 is related to language processing and is elicited when a sentence with a semantic error or an auditory stimulus with a semantic error occurs. In addition, the N400 is related to a memory process and can reflect a process of retrieving or searching for information from long-term memory.

[0073] As still another example, the P600, which is an indicator representing a reconstruction or recollective process, is related to a process of more accurately processing a stimulus based on information stored in long-term memory.

[0074] As still another example, the CNV is a potential occurring in the latter half of 200 to 300 ms or even several seconds. The CNV is also referred to as a slow potential (SP) and is related to expectancy, preparation, mental priming, association, attention, and motor activity.

[0075] As still another example, the error-related negativity (ERN) or error negativity (Ne) is an event-related potential (ERP) generated by a mistake or an error. The ERN or Ne can be elicited when a subject makes a mistake in a sensorimotor task or the like. More specifically, when a subject recognizes a mistake or an error, the ERN is generated, and a negative peak of the ERN mainly occurs in a frontal region and a central region for about 50 to 150 ms. In particular, the ERN can occur in a situation in which a mistake related to a motor response can occur, and can also be used to indicate a negative self-judgment.

[0076] Hereinafter, main features of the ERN will be described in more detail.

[0077] Figure 1 FIG. 1 is a view showing a general waveform of the ERN according to one embodiment of the disclosure.

[0078] Referring to Figure 1The negative potential value is depicted above the horizontal axis, and the positive potential value is depicted below the horizontal axis. In addition, it can be confirmed that the ERP having the negative peak value is generated within a predetermined time range after the response onset with respect to an arbitrary action. Here, the response can mean a case where an error or a mistake occurs (Error Response). In addition, the predetermined time range can be about 50 to 150 ms. Alternatively, the predetermined time range can be about 0 to 100 ms. Meanwhile, in the case of a correct response, the negative peak value of the generated ERP is relatively smaller than the ERN.

[0079] The ERN, which is an initial negative potential, is time-locked until the response error occurs. In addition, it is known that the ERN reflects reinforcement activity of a dopaminergic system related to action monitoring. The ERN includes a fronto-striatal loop including a rostral cingulate area. Meanwhile, dopamine is related to a brain reward system that generally forms a specific behavior and excites a person to provide a feeling of pleasure and reinforcement. When a behavior that repeatedly obtains an appropriate reward is learned as a habit. In addition, more dopamine is released through emotional learning, and a new behavior is attempted due to the release of dopamine. Accordingly, reward-driven learning is called reinforcement learning.

[0080] In addition, the ERN can be generated between 0 to 100 ms after the error response onset by the frontal cortex electrode during the execution of an interference task (for example, a Go-noGo task, a Stroop task, a Flanker task, and a Simon task).

[0081] In addition, it is known that the ERN, together with the CRN described below, reflects activity of a general action monitoring system that can distinguish between correct behavior and error behavior.

[0082] In addition, it is known that the fact that the ERN reaches the maximum amplitude at the frontal cortex electrode reflects that an intracerebral generator is located in the rostral cingulate area or the dorsal anterior cingulate cortex (dACC) area.

[0083] In addition, the ERN can show a change in amplitude according to a negative emotional state.

[0084] In addition, the ERN can be reported even in the case of action monitoring based on external evaluation feedback processing different from internal action performance, and the ERN can be classified as the FRN described below.

[0085] In addition, the ERN can be generated not only when an error or mistake is recognized, but also before the error or mistake is recognized.

[0086] In addition, the ERN can be generated not only as a response to one's own error or mistake, but also as a response to another person's error or mistake.

[0087] In addition, the ERN can be generated not only as a response to an error or mistake, but also as a response to anxiety or stress about a task or object scheduled to be performed.

[0088] In addition, when a larger ERN peak is obtained, the ERN peak can be considered to reflect a more serious error or mistake.

[0089] Meanwhile, as still another example, the error positivity (Pe) is an event-related potential (ERP) generated after the ERN, and is an ERP having a positive value generated mainly at a frontal cortex electrode between about 150 ~ 300 ms after an error or mistake. The Pe is known to be a reaction of recognizing the error or mistake and paying more attention. In other words, the Pe is related to an indicator of a conscious error information processing process after error detection. The ERN and the Pe are known to be ERPs related to error monitoring.

[0090] Hereinafter, main characteristics of the Pe will be described in more detail.

[0091] Figure 2 FIG. 1 is a view showing a general waveform of an ERN and a Pe according to an embodiment of the disclosure.

[0092] Referring to FIG. 1, Figure 2 a negative potential value is shown above a positive potential value. In addition, it can be confirmed that an ERP having a negative peak, that is, an ERN, is generated within a first predetermined time range after a response to an arbitrary action occurs. Here, the response can mean a situation in which an error or mistake occurs (error response). In addition, the first predetermined time range can be about 50 ~ 150 ms. Alternatively, the first predetermined time range can be about 0 ~ 200 ms.

[0093] In addition, it can be confirmed that an ERP having a positive peak, that is, a Pe, is generated within a second predetermined time range after the ERN occurs. In addition, the second predetermined time range can be about 150 ~ 300 ms after the error occurs. Alternatively, the second predetermined time range can mean about 200 ~ 400 ms.

[0094] Figure 3 FIG. 2 is a view showing a deflection characteristic of a Pe according to an embodiment of the disclosure.

[0095] Referring to FIG. 2, Figure 3Similar to P3, Pe has a large deflection characteristic, and the neural cluster generator includes not only the areas of the posterior cingulate cortex and insular cortex, but also more of the anterior cingulate cortex.

[0096] In addition, Pe can reflect emotional evaluation of an error and attention to a stimulus such as P300. In addition, ERN represents a conflict between a correct response and an error response, and Pe is known to be a response in which an error is realized and attention is more focused. In other words, ERN can be generated in the process of detecting a stimulus, and Pe can be generated according to attention in the process of processing a stimulus. When ERN and / or Pe have relatively large values, respectively, these values are known to be related to adaptive behavior that aims to respond more slowly and more accurately after an error.

[0097] Figure 4A and Figure 4B is a view showing a measurement region of ERN and Pe according to one embodiment of the disclosure.

[0098] ERN and Pe are known to be ERPs related to error monitoring. Regarding the measurement region of ERN and Pe, a maximum negative value and a maximum positive value can be measured in a central region, respectively, in general. However, there can be some differences depending on the measurement conditions. For example, Figure 4A is a main region for measuring ERN, and a maximum negative value of ERN can be measured in a midline frontal or central region (i.e., FCZ), in general. In addition, Figure 4B is a main region for measuring Pe, and a larger positive value of Pe can be measured in a posterior midline region, in general, compared to ERN.

[0099] Meanwhile, as still another example, a feedback-related negativity (FRN) is an event-related potential (ERP) related to error detection based on external evaluation feedback. ERN and / or Pe detect errors based on an internal monitoring process. However, in the case of FRN, when FRN is obtained based on external evaluation feedback, it can be operated similarly to the process of ERN.

[0100] In addition, FRN and ERN can share many electrophysiological characteristics. For example, FRN has a negative peak at a frontal cortex electrode between about 250 ~ 300 ms after negative feedback occurs, and can be generated in a dorsomedial anterior cingulate cortex (dACC) region like ERN.

[0101] In addition, similar to ERN, FRN can reflect reinforcement learning activity of the dopaminergic system. In addition, FRN generally has a larger negative value than positive feedback, and can have a larger value for unpredictable cases than predictable results.

[0102] As still another example, a correct-related negativity (CRN) is an ERP generated from a correct trial, and is a negative value smaller than the ERN. Like the ERN, the CRN can be generated between an initial latency (e.g., 0 ~ 100 ms). Figure 5 FIG. 1 is a view illustrating general waveforms of the ERN and the CRN according to one embodiment of the disclosure.

[0103] As still another example, a positive correct (Pc) is an event-related potential generated after the CRN, and is an event-related potential generated between about 150 ~ 300 ms after a correct trial occurs. The relationship between the CRN and the Pc can be similar to the relationship between the ERN and the Pe.

[0104] Meanwhile, ERPs can be classified into stimulus-locked ERPs and response-locked ERPs. The stimulus-locked ERPs and the response-locked ERPs can be divided according to standards such as a cause of the ERPs and a response time. For example, an ERP generated from the moment when a user is presented with text or a picture from the outside can be referred to as a stimulus-locked ERP. In addition, for example, an ERP generated from the moment when a user speaks or presses a button can be referred to as a response-locked ERP. Thus, based on the above standards, generally, the stimulus-locked ERPs are N100, N200, P2, P3, etc., and the response-locked ERPs are ERN, Pe, CRN, Pc, FRN, etc.

[0105] Meanwhile, brain waves can be classified according to a motive of expression. The brain waves can be classified into spontaneous brain waves (spontaneous potentials) expressed by a user's will and evoked brain waves (evoked potentials) naturally expressed according to external stimuli irrelevant to the user's will. The spontaneous brain waves can be expressed when a user moves or imagines movement by himself or herself, and the evoked brain waves can be expressed by, for example, visual, auditory, olfactory, and tactile stimuli.

[0106] Meanwhile, a brain wave signal can be measured according to the international 10-20 system. The international 10-20 system determines a measurement point of a brain wave signal based on a relationship between an electrode position and a brain cortex region.

[0107] Figure 6 FIG. 1 is a view illustrating general waveforms of the ERN and the CRN according to one embodiment of the disclosure.

[0108] Referring to Figure 6The brain regions (frontal cortex FP1, FP2; frontal cortex F3, F4, F7, F8, FZ, FC3, FC4, FT7, FT8, FCZ; parietal cortex C3, C4, CZ, CP3, CP4, CPZ, P3, P4, PZ; temporal cortex T7, T8, TP7, TP8, P7, P8; occipital cortex O1, O2, OZ) correspond to 32 brain wave measurement channels. For each channel, data can be obtained and analysis can be performed on each brain cortex region by using the data.

[0109] Figure 7 FIG. 1 is a block diagram illustrating a configuration of an apparatus for generating an image based on a brain wave signal of a passenger according to an embodiment of the disclosure.

[0110] A drive-through (DT) service refers to a service that allows a customer to order, pay, and pick up a specific product in a state of riding in a moving body without stopping. Since there is no need to stop or wait in line, the DT service attracts customers as an efficient and convenient service. Recently, the DT service is gradually becoming popular. For example, in daily life, a passenger in a moving body can easily use a DT service provided by a fast food restaurant and a coffee shop on an urban area and an expressway. The moving body can include a vehicle, a mobile / transportation device, etc.

[0111] Meanwhile, in the current DT service, a passenger generally orders a desired item after arriving at a place or a location (hereinafter, referred to as a DT point) where the DT service is provided. In an embodiment of the disclosure, an image generation apparatus and method can be provided that enables selection and ordering of an item (hereinafter, referred to as a DT product) provided by a DT point in a moving body before arriving at the DT point.

[0112] Here, the DT point can be a DT service providing place located within a predetermined range from the moving body.

[0113] In addition, the DT point of an embodiment of the disclosure can include not only a place for providing a DT service but also a business office that can provide a DT product to a moving body and receive information about a product selected and ordered by the moving body without providing a DT service. For example, the DT point of an embodiment of the disclosure can refer to a business office in which a customer in a moving body can select and order a product provided by the DT point but needs to park / stop the moving body at a separate place to pick up the ordered product. That is, the DT point of an embodiment of the disclosure can include a drive-in business office without a drive-through lane.

[0114] The image generation device of the embodiment of the disclosure can generate an image related to a DT point from a passenger's brainwave signal using an artificial intelligence model. For example, an image related to a menu or a product provided by the DT point can be generated. In addition, the image generation device of the embodiment of the disclosure can select an image included in a predetermined list based on the generated image. For example, an image included in a product list or a predetermined menu provided by the DT point can be selected. In addition, the image generation device of the embodiment of the disclosure can control a mobile body or provide predetermined information selected by a passenger to the DT point based on the selected image.

[0115] Meanwhile, artificial intelligence technology enables a computer to learn data like a human being and automatically make a decision. An artificial neural network is a mathematical model inspired by a biological neural network, and can represent an overall model having a problem-solving ability by allowing artificial neurons forming a network through synapse binding to change the synapse binding strength through learning. The artificial neural network can include an input layer, a hidden layer, and an output layer. Neurons included in each layer are connected through weights, and the artificial neural network can have a form of approximating a complex function through a linear combination of weights and neuron values and a nonlinear activation function. The learning purpose of the artificial neural network is to find weights that minimize the difference between the output value calculated at the output layer and the actual output value.

[0116] A deep neural network is an artificial neural network consisting of a plurality of hidden layers between an input layer and an output layer, and can model a complex nonlinear relationship through many hidden layers. A neural network structure capable of performing high-level abstraction by increasing the number of layers is called deep learning. In deep learning, since a large amount of data is learned, and the most probable answer is selected according to the learning result when new data is input, adaptive operation can be performed according to an image, and a characteristic factor can be automatically found in the process of learning a model based on data.

[0117] The deep learning-based model of the embodiments of the disclosure can include at least one of a fully convolutional neural network, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), and a deep belief neural network (DBN), but is not limited thereto. Alternatively, in addition to deep learning, a machine learning method can be included, or a hybrid model combining deep learning and machine learning can be included. For example, features of an image can be extracted by applying a deep learning-based model, and the image can be classified and recognized based on features extracted by applying a machine learning-based model. The machine learning-based model can include a support vector machine (SVM), AdaBoost, or the like, but is not limited thereto. Here, the RNN can include a long short-term memory (LSTM).

[0118] In addition, the learning method of the deep learning-based model of the embodiments of the disclosure can include at least one of supervised learning, unsupervised learning, or reinforcement learning, but is not limited thereto. Supervised learning is performed with a series of learning data and labels (target output values) corresponding thereto, and a neural network model based on supervised learning can be a model for inferring a function from training data. In supervised learning, a series of learning data and target output values corresponding thereto are received, an error is found by learning to compare actual output values of input data with the target output values, and then the model is corrected according to the corresponding result. According to the form of the result, supervised learning can be divided into regression, classification, detection, or semantic segmentation. The function derived through supervised learning can be used again to predict a new result value. The neural network model based on supervised learning optimizes the parameters of the neural network model by learning a large amount of training data.

[0119] Unlike supervised learning, unsupervised learning is a method of performing learning in a case where data does not have a label. That is, unsupervised learning refers to a learning method of teaching a learning algorithm without a known output value or information, and the learning algorithm should extract knowledge from data only using input data. For example, unsupervised transformation can refer to a method of newly representing data so that a human or other machine learning algorithm can more easily interpret the new data than the original data. For example, dimensionality reduction can be varied to include only necessary features while reducing the number of features from many high-dimensional data. As another example, clustering can refer to a method of grouping data having similar features, that is, grouping using common features appearing on a picture without a label.

[0120] As another example, there is a generative adversarial network (GAN) model that has been actively researched in the field of creating or restoring images or the field of imitating actions. The GAN model can refer to a type of artificial neural network or deep learning that performs learning in a manner of competing with each other, and can refer to an algorithm for learning a model created through deep learning to solve a generation problem using an adversarial learning method. The GAN model can learn information through a process in which a generator and a discriminator compete with each other. Specifically, the generator can be used to generate imitation data similar to existing data, and if the imitation data of the generator approaches the actual data, the discriminator can lose the discrimination function. In addition, the discriminator can be used to judge whether the input data is actual data or imitation data, and if the discriminator judges that the imitation data calculated by the generator has a 50% probability of being true, learning can end. Meanwhile, the GAN model of the embodiment of the disclosure can include a deep convolutional GAN (DCGAN) model capable of more stable learning.

[0121] Referring to Figure 7 , the image generation apparatus 700 can include a sensor 710 and / or a controller 720. However, it should be noted that only some components necessary for explaining the present embodiment are illustrated, and the components included in the image generation apparatus 700 are not limited to the above-described examples. For example, two or more constituent units can be implemented in one constituent unit, and the operations performed in one constituent unit can be divided and performed in two or more constituent units. In addition, some constituent units can be omitted, or additional constituent units can be added.

[0122] The image generation device 700 of an embodiment of the present disclosure can collect brainwave signals of at least one passenger in a moving body from a plurality of channels for a predetermined time. Also, the sensor 710 can perform the above-described operation.

[0123] Here, the brainwave signals collected from the plurality of channels can represent the brainwave signals in at least one of a time domain, a frequency domain, and a spatial domain. Here, the spatial domain can represent a brainwave signal measurement channel.

[0124] The image generation device 700 of an embodiment of the present disclosure can generate a first image related to a service point from the brainwave signals collected from the plurality of channels using an artificial intelligence model. Also, the controller 720 can perform the above-described operation.

[0125] Here, the service point can refer to a DT point.

[0126] For example, the service point can be a DT service providing place within a predetermined range from the moving body. Alternatively, the moving body can receive information about a product when the moving body is within a predetermined range from the service point. The number or type of the service point can vary according to the location of the moving body. The image generation device 700 of an embodiment of the present disclosure can further include a receiver (not shown). The receiver can perform the above-described operation.

[0127] Figure 8 is a view showing a range between a moving body and a service point which transmit and receive each other according to an embodiment of the present disclosure.

[0128] Referring to Figure 8 , the first service point 800 can transmit information about a product provided by the first service point 800 to the moving body 820 within a first range 802 from the first service point 800. On the other hand, the second service point 810 does not have a moving body which can transmit information about a product within a second range 812 from the second service point 810. Alternatively, the moving body 820 can receive information about a product from the first service point 800 and the second service point 810 within a third range 822 from the moving body 820.

[0129] As another example, the service point can refer to a place input by a user or a place predetermined in the moving body. Alternatively, the service point can refer to a place automatically detected according to a predetermined condition in a navigation device of the moving body. The service point can be differently set according to a user in the moving body. For example, the service point can be set by inputting a service point that a user likes.

[0130] As another example, the service points can be grouped according to the nature of the service providing place. For example, the service points can be grouped into a fast food restaurant, a coffee shop, a bakery, a convenience store, a bank, or a ticket office, etc. according to the nature or type of goods and services provided, and at least one of the grouped list can be selected according to the selection of the passenger of the moving body. In addition, for example, the grouped list can be displayed on the display of the moving body, and in response to the display, at least one of the grouped list can be selected according to the selection of the passenger of the moving body.

[0131] Here, the predetermined range can refer to a distance of several kilometers or tens of kilometers from the moving body and / or the service point in the radial direction. In addition, the predetermined range can be set based on a communication network. For example, when transmitting and receiving between the moving body and the service point based on a short distance communication network, the predetermined range can be a range in which transmission and reception can be performed through the short distance communication network. At this time, a beacon can be used in the short distance communication network.

[0132] Here, the goods can refer to DT goods. For example, the goods can refer to items such as hamburgers, coffee, and bread. That is, the goods can refer to items to be purchased by the passenger.

[0133] In addition, the goods can include services provided by the service point.

[0134] In addition, the goods can include the name, image, logo, etc. of the service point.

[0135] Here, the information about the goods can refer to information about the DT goods.

[0136] For example, the information about the DT goods can include image, type, price, quantity, and / or name information of the goods.

[0137] As another example, the information about the DT goods can include predetermined information provided by the service point. For example, the information about the DT goods can include new menus, events, discount events, etc. provided by the service point.

[0138] As another example, the information about the DT goods can be set based on the preference of the passenger. To this end, the information about the DT goods can be a result of learning in advance by the passenger. In addition, the information about the DT goods can be updated in real time.

[0139] Meanwhile, the artificial intelligence model can refer to a generative adversarial neural network (GAN) model. The GAN model can include a deep convolutional GAN (DCGAN) model. In addition, the artificial intelligence model can refer to a model obtained by combining an RNN-based model with a GAN model to process a brainwave signal.

[0140] The artificial intelligence model can be pre-trained using a GAN model with respect to the user's brainwave signal and an image of a service point corresponding thereto. Here, the image of the service point can refer to an image representing a DT point or an image representing a product provided by the DT point.

[0141] Here, the first image is at least one of a service point image representing a service point and a product image. The service point image can refer to an image representing a service point, and the product image can refer to an image representing a product provided by the service point. Here, the product image can refer to an image of a product, a menu, an item, etc. provided by a DT point.

[0142] For example, when the first image is the service point image, the first image can refer to a logo or a store image of a fast food restaurant, a coffee shop, a bakery, a convenience store, a bank, or a ticket office.

[0143] As another example, when the first image is the product image, the first image can refer to an image of a hamburger or a beverage, etc. provided by a fast food restaurant.

[0144] As another example, when the first image is the product image, the first image can refer to an image of a product provided by a bakery.

[0145] Meanwhile, in the process of generating the first image, the brainwave signal collected from the plurality of channels can generate an EEG feature vector through a predetermined encoder. Here, the predetermined encoder can include an LSTM layer and a nonlinear layer (e.g., a fully-connected layer including a ReLU nonlinear activation function).

[0146] Referring again to Figure 7 The image generation device 700 of the embodiment of the disclosure can select a second image included in a predetermined list based on the first image. In addition, the controller 720 can perform the above operations.

[0147] Here, the predetermined list can include at least one service point image. Alternatively, the predetermined list can include at least one product image. In addition, the second image can refer to a service point image or a product image.

[0148] For example, the image generation device 700 of the embodiment of the disclosure can select the second image based on a similarity determination between the first image and the images included in the predetermined list. That is, the image generation device 700 of the embodiment of the disclosure can select a second image most similar to the first image from the predetermined list.

[0149] Here, in order to determine the similarity, various similarity determination methods commonly used in the field of image recognition or classification, such as a method of extracting feature points of an input image to determine the similarity, can be applied.

[0150] That is, the image generation device 700 of the embodiment of the disclosure can select a service point judged to be desired by the passenger by selecting the second image. Alternatively, the image generation device 700 of the embodiment of the disclosure can select a commodity judged to be desired by the passenger by selecting the second image.

[0151] In addition, as a response to the selected second image, the image generation device 700 of the embodiment of the disclosure can control the mobile body or provide predetermined information to the service point. In addition, the controller 720 can perform the above-described operation.

[0152] That is, as described above, the image generation device 700 of the embodiment of the disclosure can judge what information the passenger desires or thinks by utilizing the similarity judgment. In addition, according to the judgment, the mobile body can be controlled or predetermined information can be transmitted to the service point according to the purpose.

[0153] For example, the travel route can be changed to a service point corresponding to the selected second image, or the travel route can be provided to the passenger to guide the passenger to select the route. Alternatively, a notification of the changed travel route can be provided to the passenger.

[0154] As another example, an order signal of a commodity corresponding to the selected second image can be transmitted to the service point. For example, when "coffee" is selected as the second image, an order signal of coffee can be transmitted to a coffee shop corresponding to the coffee. Alternatively, the travel route can be changed to the coffee shop corresponding to the coffee, or the travel route can be provided to the passenger to guide the passenger to select the route. Alternatively, a notification of the changed travel route can be provided to the passenger.

[0155] Figure 9 is a flowchart illustrating an operation method of an image generation device according to an embodiment of the disclosure.

[0156] In step S901, a brain wave signal of at least one passenger in a mobile body can be collected from a plurality of channels for a predetermined time.

[0157] Here, the brain wave signal collected from the plurality of channels can refer to a brain wave signal in at least one of a time domain, a frequency domain, and a spatial domain.

[0158] In step S902, a first image can be generated from the brain wave signal collected from the plurality of channels by utilizing an artificial intelligence model.

[0159] Here, the artificial intelligence model can refer to a generative adversarial neural network (GAN) model. The GAN model can include a deep convolutional GAN (DCGAN) model. In addition, the artificial intelligence model can indicate a model obtained by combining an RNN-based model with a GAN model to process a brain wave signal.

[0160] Here, the first image is at least one of a service point image representing a service point and a product image. The service point image can refer to an image representing a service point, and the product image can refer to an image representing a product provided by the service point.

[0161] Meanwhile, the artificial intelligence model can be pre-trained with respect to the brainwave signal of the user and the image of the service point corresponding thereto using the GAN model. Here, the image of the service point can refer to an image representing a DT point or an image representing a product provided by the DT point.

[0162] Meanwhile, in generating the first image, the brainwave signal collected from the plurality of channels can generate an EEG feature vector through a predetermined encoder. Here, the predetermined encoder can include an LSTM layer and a nonlinear layer (e.g., a fully connected layer including a ReLU nonlinear activation function).

[0163] In step S903, at least one second image included in the predetermined list can be selected based on the generated first image.

[0164] Here, the predetermined list can include at least one service point image. Alternatively, the predetermined list can include at least one product image. In addition, the second image can refer to a service point image or a product image.

[0165] For example, the second image can be selected based on a similarity determination between the first image and the image included in the predetermined list. That is, the second image most similar to the first image can be selected from the predetermined list.

[0166] Here, in order to determine the similarity, various similarity determination methods commonly used in the field of image recognition or classification, such as a method of extracting feature points of an input image to determine similarity, can be applied.

[0167] In step S904, in response to the selected second image, the mobile body can be controlled or predetermined information can be provided to the service point.

[0168] That is, it can be determined what information the passenger desires or thinks of by using similarity determination. In addition, according to the determination, the mobile body can be controlled or predetermined information can be transmitted to the service point according to the destination.

[0169] Figure 10 is a flowchart illustrating an operation method of an image generation device according to an embodiment of the disclosure.

[0170] Meanwhile, the image generation device of the embodiment of the present disclosure can update the artificial intelligence model through a process of generating a first image from the passenger's brainwave signal and selecting a second image based on the generated first image. That is, the passenger's brainwave signal and / or the first image can be added as learning data of the artificial intelligence model.

[0171] For example, the image generation device of the embodiment of the present disclosure can further perform a process of determining whether the selected second image meets the passenger's expectation. Also, the probability of generating the first image in the artificial intelligence model can be increased or decreased according to whether the second image meets the passenger's expectation.

[0172] Steps S1001 to S1003 can correspond to steps S901 to S903 of Figure 9 respectively, and detailed processes thereof have been described above with reference to Figure 9 .

[0173] In step S1004, a process of determining whether the selected second image meets the passenger's expectation can be performed. For example, the selected second image or a name corresponding to the second image can be displayed on a predetermined display of the moving body, and as a response to the display, a user input of selecting "Yes" or "No" can be received.

[0174] In step S1005, the artificial intelligence model can be updated according to whether the second image meets the passenger's expectation.

[0175] For example, when the selected second image meets the passenger's expectation, the probability of generating the first image in the artificial intelligence model is increased by a predetermined value, and when the selected second image does not meet the passenger's expectation, the probability of generating the first image in the artificial intelligence model is decreased by a predetermined value.

[0176] According to the embodiment of the present disclosure, an apparatus and a method for generating an image based on a passenger's brainwave signal can be provided.

[0177] According to the embodiment of the present disclosure, an apparatus and a method for generating an image from a passenger's brainwave signal using an artificial intelligence model and controlling a moving body based on the generated image can be provided.

[0178] Effects obtained by the embodiment of the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned above will be clearly understood by those skilled in the art from the above description.

[0179] Although the exemplary methods of the embodiments of the disclosure are described as a series of operational steps for clarity, the disclosure is not limited to the order or sequence of the operational steps described above. The operational steps can be performed simultaneously or in different order. Additional operational steps can be added and / or existing operational steps can be removed or replaced for implementing the methods of the embodiments of the disclosure.

[0180] The various embodiments of the disclosure are not intended to describe all possible combinations, but are intended to describe only representative combinations. The steps or elements in various forms can be used alone or can be combined.

[0181] In addition, the various embodiments of the disclosure can be implemented in the form of hardware, firmware, software, or a combination thereof. When the embodiments of the disclosure are implemented in hardware components, they can be, for example, application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, microcontrollers, microprocessors, etc.

[0182] The scope of the disclosure includes software or machine-executable instructions (e.g., operating system (OS), application, firmware, program) capable of executing various forms of methods on a device or computer and a non-transitory computer-readable medium storing the software or machine-executable instructions so that the software or instructions can be executed on the device or computer.

[0183] The description of the embodiments of the disclosure is merely exemplary in nature and, thus, variations that do not depart from the essence of the disclosure are intended to be within the scope of the disclosure. Such variations are not to be regarded as a departure from the spirit and scope of the disclosure.

Claims

1. An apparatus for generating images using brainwave signals, comprising: Sensors collect brainwave signals from at least one passenger in a moving body from multiple channels within a predetermined time period; as well as The controller, using an artificial intelligence model, generates a first image based on the brainwave signals collected from the plurality of channels, selects at least one second image from a predetermined list based on the generated first image, and controls the moving body in response to the selected second image, wherein the first image is at least one of a service point image representing a service point and an image of goods provided by the service point.

2. The apparatus according to claim 1, wherein, The brainwave signals collected from the plurality of channels are brainwave signals in at least one of the time domain, frequency domain, and spatial domain.

3. The apparatus according to claim 1, wherein, The artificial intelligence model mentioned is a generative adversarial neural network model, also known as a GAN model.

4. The apparatus according to claim 1, wherein, The service point is a location within a predetermined range of the mobile vehicle that provides drive-thru service (DT service).

5. The apparatus according to claim 4, wherein, The reservation list includes at least one service point image or at least one product image.

6. The apparatus according to claim 4, wherein, The goods are at least one of the following: items provided by the service point, services provided by the service point, and information about the service point.

7. The apparatus according to claim 4, wherein, When the second image is a service point image, the controller performs at least one of the following processes: changing the travel route of the mobile body to a service point, providing the passenger with a notification of the travel route to be changed, and providing the passenger with the travel route to be changed to guide the passenger in selecting a route.

8. The apparatus according to claim 4, wherein, When the second image is a product image, the controller performs at least one of the following processes: changing the travel route of the mobile body to a service point for providing the product, providing the passenger with a notification of the change in travel route, providing the passenger with the change in travel route to guide the passenger in selecting a route, and sending an order signal for the product.

9. The apparatus according to claim 1, wherein, The controller selects the second image based on a similarity judgment between the first image and images included in the predetermined list.

10. The apparatus according to claim 1, wherein, The controller further determines whether the selected second image meets the passenger's expectations.

11. A method for generating images using brainwave signals, comprising: Collect brainwave signals from at least one passenger in the moving body from multiple channels within a predetermined time period; Using an artificial intelligence model, a first image is generated based on the brainwave signals collected from the plurality of channels, wherein the first image is at least one of a service point image representing a service point and a product image provided by the service point; Based on the generated first image, select at least one second image included in a predetermined list; and In response to the selected second image, the moving body is controlled.

12. The method according to claim 11, wherein, The brainwave signals collected from the plurality of channels are brainwave signals in at least one of the time domain, frequency domain, and spatial domain.

13. The method according to claim 11, wherein, The artificial intelligence model mentioned is a generative adversarial neural network model, also known as a GAN model.

14. The method according to claim 11, wherein, The service point is a location within a predetermined range of the mobile vehicle that provides drive-thru service (DT service).

15. The method according to claim 14, wherein, The reservation list includes at least one service point image or at least one product image.

16. The method of claim 14, wherein, The goods are at least one of the following: items provided by the service point, services provided by the service point, and information about the service point.

17. The method of claim 14, wherein, When the second image is a service point image, controlling the mobile body includes performing at least one of the following processes: changing the mobile body's route to a service point, providing the passenger with a notification of the route to be changed, and providing the passenger with the route to be changed to guide the passenger in selecting a route.

18. The method according to claim 14, wherein, When the second image is a product image, controlling the mobile body includes performing at least one of the following processes: changing the travel route of the mobile body to a service point for providing the product, providing a notification of the change in travel route to the passenger, providing the change in travel route to the passenger to guide the passenger in selecting a route, and sending an order signal for the product.

19. The method according to claim 11, wherein, Selecting the at least one second image includes selecting the second image based on a similarity judgment between the first image and images included in the predetermined list.

20. The method of claim 11, further comprising: Determine whether the selected second image meets the passenger's expectations.

Images