Method for generating / synthesizing artificial intelligence automation content on basis of neural network signals, and multimedia content generation system for performing same

Abstract

A system for generating multimedia content on the basis of neural network signals, according to one embodiment, comprises: a neural network signal measurement instrument including sensors for measuring at least one type of neural network signal; an electronic device for controlling the neural network signal measurement instrument, identifying a user-intended prompt on the basis of at least one neural network signal acquired from the neural network signal measurement instrument, and outputting multimedia content generated by matching an image resource to the prompt; and a server for acquiring the prompt from the electronic device, and generating multimedia content by matching an image resource to the prompt on the basis of a core keyword for each of paragraphs including one or more sentences included in the prompt.

Description

A method for creating / synthesizing artificial intelligence-based content using neural network signals and a multimedia content creation system that performs the method.

[0001] The present disclosure relates to a method for generating / synthesizing content based on neural network signals. More specifically, the present disclosure relates to a system for generating / synthesizing multimedia content based on neural network signals and its operating method.

[0002] A variety of personalized content is being produced, and video-based multimedia content is growing exponentially, surpassing traditional text-based content. However, as video production, editing, and creation become increasingly expensive, various technologies for creating image- or video-based multimedia content are being unveiled.

[0003] According to one embodiment, techniques for generating prompt-based multimedia content have been disclosed, and a service for producing multimedia content desired by users at low cost has been disclosed through these techniques.

[0004] In order to quickly and accurately create the multimedia content that users want, the development of complex technologies is required, including technologies for understanding user intent, technologies for matching images that fit the identified intent and context, technologies for supplying the generated images to users who want them, and technologies for editing tools for editing the generated images.

[0005] Meanwhile, technologies are being developed to generate text or images using brain-computer interfaces (BCIs) to effectively recognize human thoughts. Typically, a BCI (Brain-Computer Interface) acquires brain signals derived from neural responses induced by visual stimuli. For example, research has been initiated to generate text, images, and videos using electroencephalography (EEG) signals.

[0006] However, there are limitations in the accuracy of signal analysis and the amount of information acquired per hour from existing BCIs, and there is a need for technology development to accurately identify user intent by comprehensively utilizing various types of signals that can be acquired from existing BCIs.

[0007] According to some embodiments, a system and method of operating the same for generating multimedia content based on a user's neural network signal may be provided.

[0008] According to one embodiment, a system and method of operating the same may be provided for generating multimedia content by identifying a prompt based on a user's neural network signal and matching at least one video resource to the prompt.

[0009] According to an embodiment of the present disclosure for achieving the above-described technical problem, a multimedia content generation system may be provided, including: a neural network signal measuring device including sensors measuring at least one type of neural network signal; an electronic device controlling the neural network signal measuring device, identifying a prompt intended by the user based on at least one neural network signal obtained from the neural network signal measuring device, and outputting multimedia content generated by matching a video resource to the prompt; and a server obtaining the prompt from the electronic device and generating multimedia content by matching a video resource to the prompt based on core keywords of paragraphs including one or more sentences included in the prompt.

[0010] According to another embodiment of the present disclosure for solving the above-described technical problem, a method of operating a multimedia content generation system may be provided, including: obtaining at least one type of neural network signal; identifying a prompt intended by a user based on at least one neural network signal; matching a video resource to the prompt based on a core keyword of paragraphs including one or more sentences included in the identified prompt; and generating multimedia content by synthesizing the video resource matched to the prompt.

[0011] According to another embodiment of the present disclosure for achieving the above-described technical problem, a method for generating multimedia content based on a neural network signal by a multimedia content generation system is provided, the method comprising: identifying a prompt intended by a user based on at least one type of neural network signal acquired from an external device; matching a video resource to the prompt based on a core keyword of paragraphs including one or more sentences included in the identified prompt; and generating multimedia content by synthesizing the video resource matched to the prompt; a computer-readable recording medium storing a program for causing the method to be performed.

[0012] In one embodiment, a neural network signal measuring device, an electronic device, and a server can work together to accurately identify a prompt based on a user's intent.

[0013] In one embodiment, virtual input devices may be provided that are controlled according to the fatigue level of a user using a multimedia content creation service.

[0014] FIG. 1 is a diagram schematically illustrating a process of generating and synthesizing multimedia content based on a neural network signal according to one embodiment.

[0015] FIG. 2 is a diagram for explaining the types of neural network signals obtained from a neural network signal measuring device by an electronic device according to one embodiment and signal characteristics that can be determined for each neural network signal.

[0016] FIG. 3 is a flowchart of a method for an electronic device to generate multimedia content based on a neural network signal according to one embodiment.

[0017] FIG. 4 is a diagram illustrating the operation of a neural network-based network used by an electronic device according to one embodiment to generate a text image from a neural network signal.

[0018] FIG. 5 is a flowchart of a specific method for an electronic device to identify a text element or character element corresponding to a keyboard area identified on a virtual input device image according to one embodiment.

[0019] FIG. 6 is a drawing for explaining a process of verifying a virtual input device image output by an electronic device on a display according to one embodiment and a user's intention to select a predetermined keyboard area on the virtual input device image.

[0020] FIG. 7 is a flowchart of a method for verifying a user's intention to select a predetermined keyboard area on a virtual input device image by an electronic device according to one embodiment.

[0021] FIG. 8 is a flowchart of a specific method for an electronic device to verify a user's intention according to one embodiment.

[0022] FIG. 9 is a diagram illustrating a process in which an electronic device according to one embodiment provides a virtual input device and candidate texts to multiple interface areas.

[0023] FIG. 10 is a diagram illustrating an operation of an electronic device according to one embodiment of the present invention to change the frequency band of a visual stimulus according to a user's concentration level.

[0024] FIG. 11 is a block diagram of a content creation system, an electronic device, and a server according to one embodiment.

[0025] Figure 12 is a block diagram of an electronic device according to another embodiment.

[0026] FIG. 13 is a diagram illustrating a process of generating multimedia content by interconnecting a neural network signal measuring device, an electronic device, and a server according to one embodiment.

[0027] A system for generating multimedia content based on a neural network signal according to one embodiment may include: a neural network signal measuring device including sensors for measuring at least one type of neural network signal; an electronic device for controlling the neural network signal measuring device, identifying a prompt intended by the user based on at least one neural network signal obtained from the neural network signal measuring device, and outputting multimedia content generated by matching a video resource to the prompt; and a server for obtaining the prompt from the electronic device and generating multimedia content by matching a video resource to the prompt based on core keywords for each paragraph including one or more sentences included in the prompt.

[0028] According to another embodiment, a method of operating a system for generating multimedia content based on a neural network signal may include: identifying a prompt intended by a user based on at least one type of neural network signal; matching a video resource to the prompt based on a core keyword of paragraphs including one or more sentences included in the identified prompt; and generating multimedia content by synthesizing the video resource matched to the prompt.

[0029] According to another embodiment, a method for generating multimedia content based on a neural network signal by a multimedia content generation system may be provided, the method comprising: identifying a prompt intended by a user based on at least one type of neural network signal acquired from an external device; matching a video resource to the prompt based on a core keyword of paragraphs including one or more sentences included in the identified prompt; and generating multimedia content by synthesizing the video resource matched to the prompt; a computer-readable recording medium storing a program for causing the method to be performed.

[0030] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0031] The terms used in this disclosure have been selected from widely used, current terms, taking into account the functions of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on the meanings of the terms and the overall content of the disclosure.

[0032] When a part of the specification is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used throughout the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0033] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted for clarity of description of the present disclosure, and similar parts are designated with similar reference numerals throughout the specification.

[0034]

[0035] FIG. 1 is a diagram schematically illustrating a process of generating and synthesizing multimedia content based on a neural network signal according to one embodiment.

[0036] According to one embodiment, a multimedia content creation system (10) can identify a user-intended prompt based on a neural network signal and generate and synthesize multimedia content by applying an artificial intelligence model to the identified prompt. According to one embodiment, the multimedia content creation system (10) can include a neural network signal measurement device (120), an electronic device (1000), and a server (2000).

[0037] However, it is not limited to the above-described example, and according to another embodiment, the multimedia content creation system (10) may further include an electronic device (4000) and other bio-signal measuring devices (140) connected via a network (3000). The electronic device (1000) and the electronic device (4000) illustrated in FIG. 1 may be terminals or computer devices used by different users who utilize the multimedia content provision service provided by the multimedia content creation system (10).

[0038] According to one embodiment, the neural network signal measuring device (120) may include at least one of a non-invasive sensor (122) or a brain-invasive sensor (124). According to another example, the neural network signal measuring device (120) may further include a network interface that is connected to an electronic device (1000) to transmit and receive data, a processor that controls the network interface, at least one of the non-invasive sensor (122) or the brain-invasive sensor (124), and a memory that stores instructions executed by the processor. For example, the neural network signal measuring device (120) may obtain at least one type of neural network signal from a user by using at least one type of sensors.

[0039] In one embodiment, the non-invasive sensor (122) may include at least one of an EEG sensor, an MEG sensor, or an infrared-based fNIRS (Functional Near-Infrared spectroscopy) sensor mounted on the scalp surface, and may also include an fMRI device, a PET (Positron Emission Tomography) device, or a SPECT (Single Photon Emission Computed Tomography) device that measure cerebral blood flow changes. In addition, in one embodiment, the brain invasive sensor (124) may include a cortical conduction signal (EcoG) sensor, a single signal measurement sensor, or a spike signal measurement sensor connected to the surface of the cerebral cortex.

[0040] According to one embodiment, the neural network signal may include, but is not limited to, at least one of an electroencephalogram (EEG) signal, an electrocortical conduction signal (ECoG), and a spike signal, and may include brain nerve response signals generated by electrical signals or magnetic signals transmitted between the user's brain nerves. The neural network signal measuring device (120) may obtain neural network signals from a plurality of channels corresponding to at least one type of sensors. In addition, the neural network signal measuring device (120) may measure neural network signals for each event related to the user's behavior or thoughts at a predetermined cycle over time using the plurality of channels.

[0041] The biosignal measuring device (140) can measure the user's biosignal independently of the neural network signal. According to one embodiment, the biosignal measuring device (140) can measure at least one of the user's electrocardiogram (ECG), electromyogram (EMG), and electrooculogram (EOG) signals.

[0042] The electronic device (1000) may, at step S152, acquire at least one type of neural network signal from the neural network signal measuring device (120), and, at step S154, identify a prompt intended by the user based on the acquired at least one type of neural network signal. For example, the electronic device (1000) may control the neural network signal measuring device (120), identify a prompt intended by the user based on the at least one neural network signal acquired from the neural network signal measuring device, and output multimedia content generated by matching a video resource to the prompt.

[0043] According to one embodiment, the electronic device (1000) may include at least one of a camera (130) and a display (131). The electronic device (1000) may identify a prompt based on a neural network signal of a user who has identified a visual stimulus provided through the display and an eye tracking signal determined from a facial image of the user acquired through the camera. According to one embodiment, the prompt may include one or more sentences, paragraphs including one or more sentences, or text including the paragraphs.

[0044] In S156, the server (2000) can match paragraph-by-paragraph video resources included in the prompt obtained from the electronic device (1000). According to one embodiment, the server (2000) can generate multimedia content by obtaining a prompt from the electronic device and matching video resources to the prompt based on core keywords for each paragraph including one or more sentences included in the prompt. For example, when text or a prompt is input, the server (2000) can match video resources using an artificial intelligence-based video generation model that matches video resources based on keywords for each paragraph in the text or summary text. In S158, the server (2000) can generate multimedia content by matching video resources to the prompt. It goes without saying that the above-described S156 to S158 may be performed by the server (2000) or the electronic device (1000).

[0045]

[0046] FIG. 2 is a diagram for explaining the types of neural network signals obtained from a neural network signal measuring device by an electronic device according to one embodiment and signal characteristics that can be determined for each neural network signal.

[0047] According to one embodiment, the electronic device (1000) can obtain at least one type of neural network signals from a neural network signal measuring device (220) mounted externally or internally on the body of the user (210). The electronic device (1000) can measure features such as brain waves (242), SMR (243), ERP (244), or SSEP (246) over time from the neural network signal (230). In addition, although not shown in FIG. 2, the electronic device (1000) can further obtain, in addition to the neural network signal measuring device, an fNIRS feature related to the amount of user cerebral blood flow change based on an infrared sensor from a biosignal measuring device (140), fMRI features obtained from magnetic resonance imaging data, PET features, and SPECT features.

[0048] The electronic device (1000) can identify a prompt intended by a user by combining at least one type of neural network signal obtained from a neural network signal measuring device and at least one type of multiple bio-signals. According to one embodiment, the neural network signal measuring device (220) may include electrode sensors (221) of at least one type of brain invasive or non-invasive, an ADC circuit (222) for converting potential signals obtained from the electrode sensors (221) into digital signals, and a network interface (223) for transmitting the obtained potential signals or ADC conversion values ​​to the electronic device (1000) by wire or wirelessly. In addition, the neural network signal measuring device (220) may include a memory (224) for storing instructions for controlling the operations of the electrode sensors (221), the ADC circuit (222), and the network interface (223), and at least one processor (225) for executing the instructions stored in the memory (224). The neural network signal (230) obtained by the electronic device (1000) from the neural network signal measuring device (220) may include at least one of an electroencephalogram signal (232), a cortical conduction signal (234), and a spike signal (236).

[0049]

[0050] FIG. 3 is a flowchart of a method for an electronic device to generate multimedia content based on a neural network signal according to one embodiment.

[0051] In S310, the electronic device (1000) can identify a prompt intended by the user based on at least one type of neural network signal acquired from an external device. In S320, the electronic device (1000) can match a video resource to the prompt based on core keywords for each paragraph including one or more sentences included in the identified prompt. In S330, the electronic device (1000) can generate multimedia content by synthesizing the video resource matched to the prompt.

[0052] The electronic device (1000) can identify a prompt based on a user's neural network signal using an artificial intelligence model based on an adversarial generative neural network. The operation of the electronic device (1000) to identify a prompt based on at least one type of neural network signal will be described in detail with reference to FIG. 4 below.

[0053] The electronic device (1000) can match video resources based on core keywords of one or more paragraphs included in a prompt. According to one embodiment, the electronic device (1000) can identify target meanings that core keywords may have based on whether the core keywords determined for each paragraph are polysemous or homonyms. The electronic device (1000) can match video resources for each paragraph based on a target meaning that fits the context of the core keyword or a sub-keyword determined based on a priority order according to the number of times it is used in the paragraph among morpheme keywords that are dependent on the target meaning and the core keyword.

[0054] The electronic device (1000) may determine a core keyword for each paragraph based on at least one of a keyword network including a keyword frequency for each paragraph, user feedback accumulated for each frequency, a keyword frequency determined for each paragraph, and information regarding the connection relationship of the keywords. The electronic device (1000) may determine whether a core keyword determined for each paragraph is a homonym or a polysemy based on a preset list of polysemy or a homonym.

[0055] The electronic device (1000) can generate a summary prompt by applying a sequence-to-sequence model including a weight determined based on ATTENTION determined based on the length of the prompt to the prompt, adjust the paragraph division level by merging or splitting paragraphs based on the keyword network similarity of paragraphs in the generated summary prompt, and match video resources to the generated split paragraphs according to the adjusted paragraph division level. The video resources can be labeled by labeling values ​​including at least one of a core keyword, a target meaning, or sub-keywords that are dependent on the target meaning. The electronic device (1000) can generate multimedia content by matching video resources to prompts based on the labeling values ​​of the video resources.

[0056] As another example, it goes without saying that at least one of the operations S310 to S330 illustrated in FIG. 3 can be performed by a server connected to an electronic device or by a linkage between an electronic device and a server.

[0057]

[0058] FIG. 4 is a diagram illustrating the operation of a neural network-based network used by an electronic device according to one embodiment to generate a text image from a neural network signal.

[0059] Referring to Figures (410) to (420) of FIG. 4, a process for an electronic device to identify a text image based on a user's neural network signal will be described. The electronic device (1000) can identify text from a neural network signal using an artificial intelligence model (410) based on a Generative Adversarial Network (GAN). According to one embodiment, the artificial intelligence model (410) based on a Generative Adversarial Network may include a discriminator (412), a generator (414), a latent vector generator (416), and a latent vector transformation network (418).

[0060] According to one embodiment, an artificial intelligence model (410) based on a generative adversarial network can generate a text image whose level of text image obtainable from a neural network signal is gradually improved by using a generator (414) and a discriminator (412). For example, the generator (414) can generate a text image based on a neural network signal. The discriminator (412) can input a real text image and a virtual text image and distinguish between truth and falsehood. The generator (414) can be trained to generate a virtual text image similar to the real text image based on the discrimination of the discriminator (412), and the discriminator (412) can be trained to distinguish between the real text image and the virtual text image. The latent vector generator (416) can sample from a latent space and generate a latent vector to be used as an input of the generator.

[0061] According to one embodiment, the electronic device (1000) may obtain a first neural network signal (411) and transmit the obtained first neural network signal (411) to a generator (414). The generator (414) may generate a first text image 'X' (421) from the first neural network signal (411). According to another example, the generator (414) may obtain a first latent vector generated by a latent vector generator from the first neural network signal (411) and generate a first text image (421) from the obtained first latent vector.

[0062] The electronic device (1000) can output the generated first text image (421) on the display and obtain the second neural network signal (423) of the user who has confirmed the first text image (421) output on the display from the neural network signal measuring device (120). The electronic device (1000) can obtain the second text image (431) by inputting the second neural network signal of the user who has confirmed the first text image (421) into an artificial intelligence model based on an adversarial generative neural network. More specifically, the electronic device (1000) can obtain the second latent vector (425) by inputting the second neural network signal (423) into a latent vector generator (416). The electronic device (1000) can determine the power spectrum characteristics of the second neural network signal in the frequency domain and the pattern represented by the characteristics, and the temporal similarity between the time at which the first text image is visually provided and the ERPs (Event Related Potentials) at which the second neural network signal is acquired.

[0063] The latent vector transformation network (418) can transform the second latent vector (425) into a third latent vector (427) based on the power spectrum characteristics and patterns of the second neural network signal in the frequency domain and the temporal similarity. The electronic device (1000) can obtain a reconstructed second text image (431) by re-inputting the third latent vector (427) into the generator (414) of the artificial intelligence model based on the adversarial generative neural network. The electronic device (1000) can determine the similarity of the second text image (431) to a preset third text image (435), and if the second text image (431) is identified as similar to the third text image, identify a text element or character element corresponding to the second text image, thereby determining a prompt intended by the user.

[0064] As illustrated in FIG. 6 described below, the electronic device (1000) can output at least one type of virtual input device image on the display for prompt input, and obtain a first neural network signal of a user who has confirmed the virtual input device image from a neural network signal measuring device. When a second text image is identified according to the method described above in FIG. 4, the electronic device (1000) can identify a keyboard area including the same text element or keyboard element as the second text image on the virtual input device image.

[0065]

[0066] FIG. 5 is a flowchart of a specific method for an electronic device to identify a text element or character element corresponding to a keyboard area identified on a virtual input device image according to one embodiment.

[0067] In S510, if the electronic device (1000) identifies that the similarity between the second text image generated from the first neural network signal and the third text image is greater than or equal to a threshold similarity, the electronic device (1000) can identify a keyboard area corresponding to the second text image on the at least one type of virtual input device image. For example, the electronic device (1000) can determine the similarity by comparing shape features or neural network (e.g., CNN output value features)-based features of the second text image with shape features or neural network-based features of the text image on the keyboard area included in the virtual input device image, and can identify a keyboard area on the virtual input device image whose similarity is greater than or equal to a threshold similarity.

[0068] Although not shown in FIG. 5, according to another embodiment, the electronic device (1000) generates a first text image in units of words or phrases, rather than a text element or character element corresponding to a keyboard area of ​​a virtual input device, based on a first neural network signal, and inputs a second neural network signal of a user confirming the generated first text image in units of words or phrases into an adversarial generative neural network model, thereby obtaining a second text image in units of words or phrases from the adversarial generative neural network model, and may identify a prompt including a word or phrase corresponding to the second text image based on whether the second text image is similar to a preset third text image.

[0069] In S520, when a keyboard area is identified, the electronic device (1000) can verify the user's intention as to whether to select a text element or character element corresponding to the keyboard area during a preset verification time. For example, when a keyboard area corresponding to a text image generated from a first neural network signal is identified, the electronic device (1000) does not immediately determine the text element corresponding to the keyboard area as the input text, but verifies whether the user has an intention to select the text element or character element corresponding to the keyboard area, thereby improving text input accuracy. In S530, the electronic device (1000) can identify the text element or character element corresponding to the keyboard area based on the verification result.

[0070]

[0071] FIG. 6 is a drawing for explaining a process of verifying a virtual input device image output by an electronic device on a display according to one embodiment and a user's intention to select a predetermined keyboard area on the virtual input device image.

[0072] Referring to FIG. 6, an example of a virtual input device (610) provided by an electronic device (1000) is illustrated. The virtual input device (610) may be provided in a portion of the display, and may be provided on the display separately from an interface area where texts input in real time through the virtual input device are displayed. In addition, according to one embodiment, the virtual input device may be used by users of the image generation / synthesis service according to the present disclosure in the process of performing the image generation / synthesis service in the metaverse space. To this end, the virtual input device interfaces may be provided in the form of images or 3D modeling in the metaverse space where multiple accesses are permitted. The virtual input device (610) may include a plurality of keyboard areas (614, 618). The electronic device (1000) may visually display a keyboard area (612) that is initially identified based on a user's neural network signal differently from other keyboard areas.

[0073] For example, the electronic device (1000) can generate a modulated keyboard area (625) by modulating a keyboard area (623) identified based on a user's neural network signal based on visual characteristics (color characteristics, font size characteristics, keyboard area or size characteristics). The electronic device (1000) can output the modulated keyboard area (625) and the pre-modulated keyboard area (623) alternately at a preset cycle (622) to overlap them on the display or a portion of the keyboard area. The cycle (622) can mean the time from the time when the pre-modulated keyboard area (623) is first displayed to the time when the pre-modulated keyboard area (627) is output again after the modulated keyboard area (625) is displayed.

[0074] The electronic device (1000) can visually modulate a keyboard area identified based on a user's neural network signal, and then alternately display the modulated keyboard area and the pre-modulated keyboard area at a preset cycle, thereby providing the user with information about which keyboard area is identified in the virtual input device image, and indicating that it is waiting for a user input regarding whether or not the user intends to select the corresponding keyboard area.

[0075] In addition, according to one embodiment, the electronic device (1000) may display a verification area (642) including symbol patterns that flicker at different frequencies over a portion of the keyboard area to verify the user's intention for a predetermined verification time. According to one embodiment, when the keyboard area is initially identified based on the first neural network signal, the verification area (642) may be displayed over a predetermined keyboard area on the virtual input device image only for a predetermined verification time. The electronic device (1000) may not always display the verification area (642) on the keyboard areas within the virtual input device image, but may display the verification area only for a predetermined verification time to improve text identification accuracy, thereby minimizing visual discomfort to the user.

[0076] The electronic device (1000) may verify the user's intention to select a text element corresponding to a keyboard area including the verification area based on at least one of the user's third neural network signal for the verification area or the user's first eye tracking signal for the verification area. As illustrated in Figure (640), the verification area (642) according to an embodiment may include a first partial verification area (644) that flickers with a first type of frequency and a second partial verification area (646) that flickers with a second type of frequency. The frequency ranges in which the visual stimuli appearing in the first partial verification area (644) and the second partial verification area (646) flicker may be determined as different frequency ranges within a frequency range between 1 Hz and 40 Hz. According to an embodiment, the first partial verification area (644) may flicker with a first frequency value, and the second partial verification area (646) may flicker with a frequency value different from the first frequency.

[0077] In addition, according to one embodiment, the frequency (e.g., frequency) at which the modulated keyboard area (623) and the modulated keyboard area (625) are alternately displayed may be determined as one of the frequency values ​​lower than the frequency range including the frequency values ​​at which the first partial verification area (644) and the second partial verification area (646) flicker. Through this, the electronic device (1000) can effectively guide the user's focus to the verification area for verifying whether the user has the intention to select the keyboard area while displaying the primarily identified keyboard area to the user.

[0078] For example, when a keyboard area is primarily identified on a virtual input device image based on a first neural network signal of a user who has confirmed a virtual input device image, the electronic device (1000) can verify whether there is an intention to select a text element according to the keyboard area based on the analysis result of a third neural network signal. More specifically, the electronic device (1000) can obtain a third neural network signal of a user who has confirmed a verification area output for a predetermined verification time, and can verify the intention of the user to select a text element corresponding to the keyboard area based on a power spectrum intensity for each frequency domain including different frequencies in a frequency domain domain through Fourier transform of the obtained third neural network signal.

[0079] For example, while the keyboard area (643) is primarily identified and the keyboard area identified according to the drawing (620) is periodically visually modulated and displayed, the electronic device (1000) can determine a first power spectrum intensity of a first frequency domain including a frequency value corresponding to the flickering frequency of the first partial verification area (644) from the third neural network signal, and can determine a second power spectrum intensity of a second frequency domain including a frequency value corresponding to the flickering frequency of the second partial verification area (646) from the third neural network signal. The electronic device (1000) can identify a flickering frequency of the partial verification area corresponding to a frequency value of a frequency domain indicating a power spectrum intensity greater than or equal to a threshold value among the first power spectrum intensity and the second power spectrum intensity.

[0080] For example, if the intensity of the first power spectrum is identified as being greater than or equal to a threshold power spectrum intensity, the electronic device (1000) may determine that the user is focusing on the first partial verification area (644) and determine that the user's intention to select a text element corresponding to the keyboard area (643) has been verified. As another example, if the intensity of the second power spectrum is identified as being greater than or equal to a threshold power spectrum intensity, the electronic device (1000) may determine that the user is focusing on the second partial verification area (646) and determine that the user's intention to select a text element corresponding to the primarily identified keyboard area (643) has not been verified. As another example, the electronic device (1000) may compare the intensity of the first power spectrum and the intensity of the second power spectrum and identify a partial verification area that is flickering at a flickering frequency corresponding to a frequency value in a power spectrum frequency range in which the intensity of the power spectrum is identified as being greater.

[0081]

[0082] FIG. 7 is a flowchart of a method for verifying a user's intention to select a predetermined keyboard area on a virtual input device image by an electronic device according to one embodiment.

[0083] Referring to FIG. 7, a flowchart for verifying a user's intention by the electronic device described above with reference to FIG. 6 is illustrated. In S710, the electronic device (1000) may, during a verification time, alternately output a keyboard area generated by modulating the identified keyboard area based on visual characteristics and the identified keyboard area before modulation on the display at a preset cycle. In S720, the electronic device (1000) may, during a verification time, display a verification area including symbol patterns that flicker at different frequencies by overlapping a portion of the keyboard area in order to verify the user's intention. The verification area may include a plurality of partial verification areas displayed on the display at different flickering frequencies, and the symbol patterns to be flickered may be provided in different patterns.

[0084] In S730, the electronic device (1000) can verify the user's intention to select a text element corresponding to a keyboard area including the verification area based on at least one of the user's third neural network signal for the verification area or the user's first eye tracking signal for the verification area. According to one embodiment, the electronic device (1000) can verify the user's intention to select a text element corresponding to the keyboard area based on the intensity of a power spectrum for each frequency domain including different frequencies included in the third neural network signal. In S740, when the user's intention to select the text element is verified, the electronic device (1000) can identify the text element corresponding to the keyboard area.

[0085] Although not illustrated in FIG. 7, according to another embodiment, the electronic device (1000) may determine that the user's intention to select the text element is not verified according to the third neural network signal if, among the intensities of the power spectra by frequency domain included in the third neural network signal, one power spectrum is identified as being greater than a threshold, and if the energy of the alpha band region of the third neural network signal is identified as decreasing and the beta band energy region is identified as increasing. For example, the electronic device (1000) may determine the power spectrum intensity of the alpha band region (e.g., 8 Hz to 12 Hz) of the third neural network signal and the power spectrum intensity of the beta band region (12 Hz to 30 Hz) for verifying the user's intention to select the corresponding keyboard region during a predetermined verification time.

[0086] If the electronic device (1000) identifies that the intensity of the power spectrum of the alpha region within the third neural network signal decreases and the intensity of the power spectrum of the beta band region increases, the electronic device (1000) may identify the third neural network signal as noise due to motor imagery and determine that the user's intention to select a text element has not been verified based on the third neural network signal. The electronic device (1000) according to the present disclosure detects the occurrence of noise due to the user's imaginary motion focusing on the verification area for the virtual input device, and determines that the user's intention for the text element for the keyboard area that was primarily identified when the noise due to the imaginary motion is generated has the advantage of further improving text accuracy.

[0087]

[0088] FIG. 8 is a flowchart of a specific method for an electronic device to verify a user's intention according to one embodiment.

[0089] In S810, the electronic device (1000) can identify a user focus area on the display based on the first eye tracking signal if the intensities of the power spectrums in the frequency domain corresponding to the different frequencies used to flicker the symbol patterns included in the third neural network signal are all identified as being above a threshold or below a threshold. For example, if the intensities of the first power spectrum in the frequency domain corresponding to the flickering frequency of the first partial verification area and the second power spectrum in the frequency domain corresponding to the flickering frequency of the second partial verification area are both identified as being above a threshold or below a threshold, the user's intention to select a keyboard area may be unclear.

[0090] The electronic device (1000) may acquire a first eye-tracking signal of a user when the intensity of a first power spectrum in a frequency domain corresponding to the flickering frequency of a first partial verification area included in a third neural network signal and the intensity of a second power spectrum in a frequency domain corresponding to the flickering frequency of a second partial verification area are both identified as being above a threshold or below a threshold. The electronic device (1000) may identify a user focus area on the display based on the first eye-tracking signal.

[0091] In S820, the electronic device (1000) can verify the user's intention to select a text element corresponding to the keyboard area based on the ratio of the area occupied by the overlapping area where the focus area and the partial verification area included in the verification area overlap, to the total area including the focus area and the partial verification area determined on the display based on the first eye-tracking signal. For example, as illustrated in FIG. 6, the electronic device (1000) can determine the overlapping area (645) of the partial verification area (644) overlapping the focus area (648) determined based on the first eye-tracking signal, and can identify that the partial verification area that the user intends to select is the first partial verification area (644) based on the ratio of the area of ​​the overlapping area (645) to the total area including the focus area (648) and the partial verification area (644) to which the focus area is partially related.

[0092] In another example, as illustrated in FIG. 6, the electronic device (1000) may determine a first area ratio of the area of ​​the first overlapping area (653) where the focus area (659) according to the first eye tracking signal overlaps the first partial verification area (654) to the total area of ​​the first partial verification area (654) and the focus area (659), and may determine a second area ratio of the area of ​​the second overlapping area (655) where the focus area (659) according to the first eye tracking signal overlaps the second partial verification area (656) to the total area of ​​the second partial verification area (656) and the focus area (659).

[0093] The electronic device (1000) can obtain an eye-tracking signal when both the intensity of the first power spectrum and the intensity of the second power spectrum are within a threshold range, or both the intensity of the first power spectrum and the intensity of the second power spectrum are equal to or greater than a threshold, or both the intensity of the first power spectrum and the intensity of the second power spectrum are equal to or less than a threshold, and can accurately identify a partial verification area that the user intends to select among the partial verification areas based on the first area ratio and the second area ratio. For example, when the first area ratio is identified as being greater than the second area ratio, the electronic device (1000) can identify the user's intention to select the first partial verification area (654) within the verification area, and ultimately determine that the user's intention to select the keyboard area (661) has been verified.

[0094] The process of identifying a focus area based on an eye-tracking signal by the electronic device (1000) may utilize a known technology for identifying a focus area based on an eye-tracking signal. For example, the electronic device (1000) may acquire a facial image through a camera and detect a face area and an eye area from the acquired facial image. The electronic device (1000) may track the position of the pupil in the eye area. For example, the electronic device (1000) may measure the position of the pupil detected in the user's previous frame facial image over time and the change in the position of the pupil detected in the frame facial image acquired after the previous frame, and determine the movement direction and speed of the pupil based on the measured position change. The electronic device (1000) may identify the focus area that the user is currently gazing at on the display based on the movement direction and speed of the pupil. According to one embodiment, the electronic device (1000) may also improve the prediction accuracy of the focus area by correcting the position of the next predicted focus area based on the movement direction and speed of the pupil.

[0095] According to another example, the neural network signal measuring device (120) used by the electronic device (1000) may be provided as a wearable device type that can be worn on the user's head. According to one embodiment, the neural network signal measuring device (120) includes at least one acceleration sensor, and the neural network signal measuring device (120) may measure not only at least one type of neural network signal, but also the movement of the user's head while being worn on the user's head.

[0096] In cases where it is difficult to clearly identify the user's intention for a verification area based on the intensity of the power spectrum of a neural network signal, the electronic device (1000) may more accurately identify the user's intention to select one partial verification area included in the verification area based on an eye tracking signal or an acceleration sensor value obtained from a wearable device.

[0097]

[0098] FIG. 9 is a diagram illustrating a process in which an electronic device according to one embodiment provides a virtual input device and candidate texts to multiple interface areas.

[0099] According to one embodiment, the electronic device may provide a virtual input device interface, a currently identified text guidance interface, and a candidate text guidance interface in different interface areas. For example, the electronic device (1000) may display an image for a virtual input device interface in a first interface area (910), and may display a text guidance interface image (930) that outputs text elements or character elements completed to date as user intent verification is completed in a second interface area (930) adjacent to the first interface area and distinct from the first interface area (910), and a candidate text image (932) related to the text elements or character elements completed to date.

[0100] The first interface area (910) and the second interface area (930) may be separated by interface boundaries (922, 931). The virtual input devices provided in the first interface area (910) may be provided in one screen mode, but may also be provided on the first interface area through a plurality of switchable screen modes consisting of 1 to 3 stages. For example, all English keyboard areas in the first interface area (910) may be provided in one screen mode, but according to another example, keyboard areas such as I, O, P or symbols not shown in FIG. 9 may be provided in a separate screen mode that is outputtable in the first interface area (910).

[0101] According to one embodiment, the electronic device (1000) can detect an eye region on a face image of a user gazing at at least one of the first interface region (910) and the second interface region (930), and identify an eye blink state of the user based on an aspect ratio of the vertical length to the horizontal length of the detected eye region. The electronic device (1000) can determine an eye blink state (e.g., whether the eyes are open or closed) based on the aspect ratio of the eye region, and the time period for which the eyes are closed and the number of eye blinks during a single eye blink. The electronic device (1000) can determine an eye blink motion (or motion) of the user based on the eye blink state, the time period for which the eyes are closed, and the number of eye blinks, and execute a control function corresponding to the determined eye blink motion in advance.

[0102] According to one embodiment, control functions corresponding to the user's eye blinking motion may be set differently for each interface area. For example, the electronic device (1000) may identify an interface area where a focus area based on the user's eye tracking signal remains. When a predetermined eye blinking motion is identified while the user's focus area (924) is identified as remaining in the first interface area (910), the electronic device (1000) may perform one control function from a first function list (962) corresponding to the eye blinking motion, which is preset for the first interface area.

[0103] In another example, the electronic device (1000) may perform one control function from a second function list (942) corresponding to the blinking motion, which is preset for the second interface area, when a predetermined eye blinking motion is identified while the user's focus area (924) is identified as staying in the second interface area (930). For example, the electronic device (1000) may ignore the input when a single eye blinking motion is identified from the user's facial image while the user's focus area (924) is identified as staying in the first interface area (910), but may perform the C function when a single eye blinking motion is identified while the user's focus area (924) is identified as staying in the second interface area (930).

[0104] The electronic device (1000) according to the present disclosure can not only display an area where at least one type of virtual input device images are displayed and an interface area where input text elements or candidate texts are output on a single screen, but also identify a location in the interface area where the current user's focus area remains, and perform control functions corresponding to an eye blinking motion that are set differently for each interface area, thereby improving the user's convenience in entering text.

[0105] According to one embodiment, the electronic device (1000) may output candidate texts (934) in the second interface area (920) by using an artificial intelligence model for recommending candidate texts, including a natural language processing model and an inference engine that recognizes the context of a document, message, conversation, etc. and predicts the user's intention based on the recognized context. For example, the inference engine may be a pre-trained sequence model that generates a user intention table that maps the text (930) input so far and the user's intention, and outputs candidate texts based on the generated user intention table.

[0106] In another example, the electronic device (1000) may acquire fNIR signals from a neural network signal meter and detect changes in blood flow and oxygen concentration in the cerebral cortex region indicated by the fNIR signals to determine the user's emotional state. The electronic device (1000) may also recommend candidate texts by further considering the context of the text identified so far and the user's emotional state, thereby providing candidate texts more suited to the user's intent.

[0107] For example, the electronic device (1000) may recognize the context of a text ('APP') collected through a natural language processing model within an artificial intelligence model, determine one user intent among a plurality of user intents (e.g., intent 1 to input LE after APP, or intent 2 to input LIKE indicating that one likes APP after APP) based on the recognized context using an inference model, and output candidate texts based on the determined user intent. An operation of the electronic device identifying one candidate text among the candidate texts may correspond to an operation of identifying one keyboard area on a first interface area and verifying the user's intention to select the identified keyboard area.

[0108] For example, the electronic device (1000) may identify one of the candidate texts based on the similarity between a fourth text image generated based on a fourth neural network signal of a user who has confirmed the candidate texts output to the second interface area (930) and a predetermined text image displayed on the second interface area. For example, the electronic device (1000) may reconstruct a text image generated based on the fourth neural network signal using an adversarial generative neural network-based artificial intelligence model, and identify a candidate text image (934) based on the similarity between the reconstructed text image and the predetermined images displayed on the second interface area.

[0109] The electronic device (1000) may display a candidate text verification area including candidate text portion verification areas (936, 938) that flicker at different frequencies while visually modulating the identified candidate text image (934) and overlaying it on the candidate text image (934). The electronic device (1000) may verify the user's intention to select the identified candidate text based on at least one of the user's fifth neural network signal or the second eye tracking signal with respect to the candidate text verification area displayed by overlaying the image area where the identified candidate text is displayed. The above-described process may correspond to a process of identifying a keyboard area on a first interface area and verifying the identified keyboard area. When the user's intention to select the candidate text (934) is verified, the electronic device (1000) may identify the selected candidate text.

[0110]

[0111] FIG. 10 is a diagram illustrating an operation of an electronic device according to one embodiment of the present invention to change the frequency band of a visual stimulus according to a user's concentration level.

[0112] In S410, the electronic device (1000) may obtain a user face image gazing at at least one type of virtual input device image. In S420, the electronic device (1000) may determine the user's concentration level based on the aspect ratio of the vertical length to the horizontal length of the eye area detected on the user's face image and the time for which the aspect ratio is maintained above a threshold ratio. For example, the electronic device (1000) may determine the aspect ratio of the eye area on the user's face image using a DTM (Distance Threshold Method) technique, and if the aspect ratio of the eye area is above a first threshold ratio, determine that the user has eyes open, and if the aspect ratio of the eye area is identified as being below a second threshold ratio, determine that the user has eyes closed.

[0113] For example, the electronic device (1000) may determine the user's concentration level to be high if the time for which the aspect ratio is maintained above the threshold ratio is identified as being longer than the threshold time. In another example, the electronic device (1000) may count the number of times the aspect ratio is maintained above the threshold ratio for a threshold time, and determine the user's concentration level to be high if the counted number is longer than the threshold number.

[0114] In S430, if the user's concentration level is determined to be below a threshold level, the electronic device (1000) may upscale a frequency domain used for visual stimulation regarding at least one of the image of the virtual input device displayed on the display, the verification area, and the keyboard area having modulated visual characteristics at a preset ratio. In another example, if the user's concentration level is determined to be below a threshold level, the electronic device (1000) may convert a frequency domain used for visual stimulation into a frequency domain in which the user's fatigue is low at an arbitrary ratio. In another example, if the user's concentration level is determined to be below a threshold level, the electronic device (1000) may replace an image on a first interface area or a second interface area displayed on the display with a predetermined refresh image and provide the replaced image on the monitor at a constant rate, thereby reducing user fatigue.

[0115] Although not illustrated in FIG. 10, the electronic device (1000) determines whether the time for which an aspect ratio greater than or equal to a threshold ratio is maintained on the eye region image is greater than or equal to a threshold time, and if the time for which an aspect ratio greater than or equal to the threshold ratio is maintained is determined to be greater than or equal to a threshold time, the electronic device (1000) can identify the intensity of a neural network signal of a user gazing at the virtual input device image. Even if the aspect ratio greater than or equal to the threshold time is determined to be greater than or equal to a threshold time, if the intensity of a neural network signal of a user gazing at the virtual input device image is identified to be less than or equal to a threshold time, the electronic device (1000) can determine the user's concentration level to be less than a threshold level. Through this, the electronic device (1000) can accurately determine whether the user is dozing off with his or her eyes open or whether the user is in a state of reduced concentration even with his or her eyes open.

[0116] In another example, if it is determined that an aspect ratio higher than a threshold ratio is maintained in the eye region image, the electronic device (1000) identifies the power spectrum intensity of a frequency range corresponding to theta waves (approximately 4 Hz to 8 Hz) appearing in the user's neural network signal, and if the intensity of the identified power spectrum is determined to be lower than the threshold intensity, even if the aspect ratio higher than the threshold ratio is determined to be higher than the threshold time, the user's concentration level may be determined to be lower than the threshold level. In another example, if the intensity of a predetermined frequency band appearing in a neural network signal acquired from a channel mounted near the user's occipital lobe is determined to be lower than the threshold intensity, even if the aspect ratio higher than the threshold ratio is maintained, the electronic device (1000) may determine that the current user is in a fatigued state. Through the above-described method, the electronic device (1000) can accurately measure the current user's fatigue level.

[0117] Also, although not illustrated in FIG. 10, the electronic device (1000) may increase the conversion speed of a visual stimulus displayed in at least one of the first interface area or the second interface area when the user's concentration level is identified as being above a threshold level. According to one embodiment, when the electronic device (1000) identifies the user's concentration level as being above a threshold level, the electronic device (1000) may increase the mode switching speed of a virtual input device displayed on the first interface area illustrated in FIG. 9, or increase the execution speed of a function corresponding to a user's eye blink motion that is set differently for each of the first interface area or the second interface area.

[0118]

[0119] FIG. 11 is a block diagram of a content creation system, an electronic device, and a server according to one embodiment.

[0120] Figure 12 is a block diagram of an electronic device according to another embodiment.

[0121] According to one embodiment, the multimedia content creation system (10) may include an electronic device (1000), a server (2000), and a neural network signal measuring device (120). However, the present invention is not limited to the above-described example, and may further include other bio-signal measuring devices and network devices to create multimedia content as illustrated in FIG. 1. According to one embodiment, the electronic device (1000) may include a first network interface (1500), a display (1210), a memory (1700), and a first processor (1300). However, not all of the illustrated components are essential components. The electronic device (1000) may be implemented with more components than the illustrated components, or may be implemented with fewer components.

[0122] For example, as illustrated in FIG. 12, the electronic device (1000) may further include a user input interface (1100), an output unit (1200), a sensing unit (1400), a network interface (1500), an A / V input unit (1600), and a memory (1700) in addition to a first processor (1300), a first network interface (1500), a display (1210), and a memory (1700). The configuration of the electronic device and the server will be described in detail with reference to FIGS. 11 and 12.

[0123] The user input interface (1100) refers to a means for a user to input data for controlling an electronic device (1000). For example, the user input interface (1100) may include, but is not limited to, a key pad, a dome switch, a touch pad (contact electrostatic capacitance type, pressure resistive film type, infrared detection type, surface ultrasonic conduction type, integral tension measurement type, piezo effect type, etc.), a jog wheel, a jog switch, etc.

[0124] The user input interface (1100) can acquire user input for displaying and controlling a visual response for acquiring a neural network signal. In another example, the output unit (1200) can output an audio signal, a video signal, or a vibration signal, and the output unit (1200) can include a display unit (1210), an audio output unit (1220), and a vibration motor (1230).

[0125] The display unit (1210) includes a screen for displaying and outputting information processed in the electronic device (1000). In addition, the screen can output text inferred by the electronic device (1000) or a rendered image generated by matching a partial video to text. The audio output unit (1220) outputs audio data received from the network interface (1500) or stored in the memory (1700). In addition, the audio output unit (1220) outputs audio signals related to functions performed in the electronic device (1000) (e.g., call signal reception sound, message reception sound, notification sound).

[0126] The first processor (1300) typically controls the overall operation of the electronic device (1000). For example, the first processor (1300) can control the user input interface (1100), the output unit (1200), the sensing unit (1400), the network interface (1500), the A / V input unit (1600), etc., by executing programs stored in the memory (1700). In addition, the first processor (1300) can perform the functions of the electronic device (1000) described in FIGS. 1 to 10 by executing programs stored in the memory (1700).

[0127] According to one embodiment, the first processor (1300) may be provided in multiple units. For example, the plurality of first processors (1300) may execute one or more instructions to identify a prompt intended by the user based on at least one type of neural network signal obtained from an external device, match an image resource to the prompt based on core keywords of paragraphs including one or more sentences included in the identified prompt, and synthesize the image resource matched to the prompt, thereby generating multimedia content.

[0128] The sensing unit (1400) can detect the status of the electronic device (1000) or the status around the electronic device (1000) and transmit the detected information to the processor (1300). The sensing unit (1400) can sense specification information of the electronic device (1000), temperature, humidity, and air pressure information for the monitoring target space, etc.

[0129] For example, the sensing unit (1400) may include, but is not limited to, at least one of a magnetic sensor (1410), an acceleration sensor (1420), a temperature / humidity sensor (1430), an infrared sensor (1440), a gyroscope sensor (1450), a position sensor (e.g., GPS) (1460), a barometric pressure sensor (1470), a proximity sensor (1480), and an RGB sensor (illuminance sensor) (1490). In another example, the sensing unit (1400) may further include sensors for measuring user bio-signals (e.g., electromyogram signals, electrocardiogram signals). Since the function of each sensor can be intuitively inferred from its name by those skilled in the art, a detailed description thereof will be omitted.

[0130] The first network interface (1500) may include one or more components that enable the electronic device (1000) to communicate with other devices (not shown) and a server (2000). The other devices (not shown) may be computing devices such as the electronic device (1000) or sensing devices, but are not limited thereto. For example, the first network interface (1500) may include a wireless communication interface (1510), a wired communication interface (1520), and a mobile communication unit (1530). The wireless communication interface (1510) may include, but is not limited to, a short-range wireless communication unit, a Bluetooth communication unit, a BLE (Bluetooth Low Energy) communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared (IrDA, infrared Data Association) communication unit, a WFD (Wi-Fi Direct) communication unit, a UWB (ultra wideband) communication unit, etc.

[0131] The wired communication interface (1520) may include at least one wired interface for exchanging data with an external device connected to the electronic device via wired communication. The mobile communication unit (1520) transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signals may include various types of data resulting from the transmission and reception of voice call signals, video call signals, or text / multimedia messages.

[0132] The A / V (Audio / Video) input unit (1600) is for inputting audio signals or video signals, and may include a camera (1610) and a microphone (1620). The camera (1610) can obtain image frames, such as still images or moving images, through an image sensor in video call mode or shooting mode. Images captured through the image sensor can be processed through the first processor (1300) or a separate image processing unit (not shown).

[0133] The microphone (1620) receives an external acoustic signal and processes it into electrical voice data. For example, the microphone (1620) can receive an acoustic signal from an external device or a user. The microphone (1620) can receive a user's voice input. The microphone (1620) can utilize various noise removal algorithms to remove noise generated during the process of receiving an external acoustic signal.

[0134] The memory (1700) can store a program for processing and controlling the first processor (1300), and can also store data input to or output from the electronic device (1000). In addition, the memory (1700) can store an artificial intelligence model for learning neural network signal patterns used by the electronic device (1000), an artificial intelligence model for identifying context or core keywords for each text, and an artificial intelligence model for generating a rendered image through text-based image resource matching.

[0135] For example, the memory (1700) may store weight values ​​regarding layers, nodes, and connection strengths of at least one neural network model. Furthermore, the electronic device (1000) may further store learning data generated by the electronic device (1000) to train the neural network model. Furthermore, the memory (1700) may further store information regarding the operating environments of cameras or servers connected to the electronic device.

[0136] The memory (1700) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

[0137] Programs stored in memory (1700) can be classified into multiple modules according to their functions, for example, a UI module (1710), a touch screen module (1720), a notification module (1730), etc.

[0138] The UI module (1710) may provide a UI, GUI, etc. for text identification based on neural network signals, text-based image generation, and editing of generated images by the electronic device (1000). The touch screen module (1720) may detect a user's touch gesture on a touch screen and transmit information about the touch gesture to the processor (1300). The touch screen module (1720) according to some embodiments may recognize and analyze a touch code. The touch screen module (1720) may also be configured as separate hardware including a controller.

[0139] The notification module (1730) may generate a signal to notify the occurrence of an event in the electronic device (1000). For example, when user feedback is obtained for a rendered image provided by the electronic device (1000) or contextual identification of identified text based on a neural network signal is difficult, a notification sound may be provided accordingly. Examples of events according to one embodiment include reception of a call signal, reception of a message, input of a key signal, schedule notification, etc. The notification module (1730) may output a notification signal in the form of a video signal through the display unit (1210), output a notification signal in the form of an audio signal through the audio output unit (1220), or output a notification signal in the form of a vibration signal through the vibration motor (1230).

[0140] According to one embodiment, the server (2000) may include a second network interface (2100), a database (2200), and a second processor (2300). The configuration of the server (2000) illustrated in FIG. 11 may correspond to the configuration of the electronic device described in FIG. 12. According to one embodiment, the second network interface (2100) may obtain prompt or text information from the electronic device (1000) and transmit image information generated based on the obtained prompt or text information to the electronic device.

[0141] In another embodiment, the network interface (2100) may transmit and receive information about the characteristics and patterns of artificial intelligence models, neural network models, and neural network signals learned by the electronic device to and from the electronic device. In another example, the network interface (2100) may also transmit and receive partial video information matching text or prompts, and tag information matching the partial video information.

[0142] Additionally, according to one embodiment, the database (2200) may correspond to the memory described above in FIG. 11. For example, the database (2200) may store neural network signal information, text information, and image or video information that can be matched to the text information obtained from the electronic device (1000).

[0143] According to one embodiment, the second processor (2300) can control the overall operation of the server (2000). For example, the second processor (2300) can perform all or part of the method for generating neural network signal-based multimedia content performed by the electronic device (1000) described in FIGS. 1 to 10 by controlling the network interface (2100) and the database (2200).

[0144] According to one embodiment, the following operations may be performed to accurately match a video resource to a prompt determined from a neural network signal of the server (2000). For example, the second processor (2300) may obtain a prompt from an electronic device, and generate multimedia content by matching the video resource to the prompt based on a target meaning determined based on whether a core keyword of one or more paragraphs included in the prompt is a polysemous or homonymous word, or at least one sub-keyword determined based on whether two or more paragraphs including a core keyword representing the same target meaning are identified within the prompt.

[0145] According to one embodiment, the second processor (2300) may generate multimedia content, characterized in that it summarizes the prompt based on a user input requesting the prompt summary obtained from the electronic device or attention determined based on the length of the prompt, divides the summarized prompt into paragraphs, and determines a core keyword for each divided paragraph. For example, the server (2000) may perform prompt summary using a sequence model that indicates the importance of each text element in the prompt with a predetermined weight according to the attention weight and outputs a summary text when a context vector according to the attention weight and text input is input.

[0146] According to one embodiment, the second processor (2300) determines a first paragraph division point based on the number of core keywords per paragraph, determines a keyword network regarding at least one of the frequencies and connection relationships of the core keywords included in first divided paragraphs divided according to the first paragraph division point, determines a similarity of the keyword network for adjacent paragraphs among the first divided paragraphs, and merges adjacent paragraphs whose similarity of the keyword network is equal to or greater than a threshold similarity, thereby determining a second paragraph division point within the text, and may divide the summarized prompt based on the second paragraph divisions determined based on the second paragraph division point.

[0147] For example, the server (2000) may improve the accuracy of image resource matching by optimizing paragraph segmentation based on the similarity between keyword networks by applying a similarity determination technique based on at least one of cosine similarity, Jaccard similarity, and graph similarity between keyword networks regarding the frequency and connection relationship of keywords per paragraph. According to one embodiment, if the core keyword per paragraph is not a polysemous or homonymous word, the second processor (2300) may determine a single meaning corresponding one-to-one to the core keyword as the target meaning of the core keyword.

[0148] According to one embodiment, when the core keyword for each paragraph is identified as a polysemous word or a homonym, the second processor (2300) determines the core keyword for each paragraph as a target keyword, converts a plurality of meanings related to the determined target keyword into vectors of a predetermined dimension, and determines a sentence analysis range in which sentences adjacent to a sentence including the target keyword, among the sentences to be analyzed to identify the target meaning, are separated from the sentence including the target keyword based on the distance between the converted vectors, identifies a noun for words included in at least one sentence determined based on the sentence analysis range, and determines the target meaning based on a dependency relationship determined according to a syntax tree representing a dominant-subordinate relationship between the identified nouns.

[0149] According to one embodiment, the target meaning may indicate one meaning identified as the most appropriate meaning among a plurality of candidate meanings included in polysemy or homonyms. According to one embodiment, the second processor (2300) may identify whether there is a paragraph having the same target meaning as the core keyword determined for each paragraph in the prompt, and if it is identified that there is a paragraph having the same target meaning, determine at least one sub-keyword for each paragraph including the core keyword, and, based on the target meaning determined for the core keyword for each paragraph and the at least one sub-keyword, match image resources labeled with labeling information including both the target meaning and the sub-keyword for each paragraph, thereby generating the multimedia content.

[0150]

[0151] FIG. 13 is a diagram illustrating a process of generating multimedia content by interconnecting a neural network signal measuring device, an electronic device, and a server according to one embodiment.

[0152] In S1301, the server (2000) can pre-build and train an artificial intelligence-based image generation model. For example, when a prompt is input, the server (2000) can pre-train an artificial intelligence-based image generation model that generates an image by dividing the input prompt into predetermined paragraphs and matching image resources for each divided paragraph. In S1302, the neural network signal measuring device (120) can measure at least one type of neural network signal. In S1304, the neural network signal measuring device (120) can transmit the acquired at least one type of neural network signal to the electronic device (1000). In S1306, the electronic device (1000) can identify the prompt from at least one type of neural network signal.

[0153] In step S1308, the electronic device (1000) may transmit a prompt to the server (2000). In step S1310, the server (2000) may preprocess the prompt obtained from the electronic device (1000). The operation of the server (2000) preprocessing the prompt may include summarizing and removing noise of the prompt. In step S1312, the server (2000) may match a video resource to one or more paragraphs in the preprocessed prompt. In step S1314, the server (2000) may generate multimedia content by matching the video resource to the prompt. In step S1316, the server (2000) may transmit the multimedia content to the electronic device (1000). In step S1318, the electronic device (1000) may output the generated multimedia content.

[0154]

[0155] The method for generating multimedia content based on neural network signals according to the present disclosure may be implemented in the form of program commands that can be executed by various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, and the like, either singly or in combination. The program commands recorded on the medium may be those specifically designed and configured for the present invention, or may be known and usable by those skilled in the art of computer software.

[0156] Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concepts of the present invention defined in the following claims also fall within the scope of the present invention.

Claims

1. In a system for generating multimedia content based on neural network signals, A neural network signal measuring device comprising sensors measuring at least one type of neural network signal; An electronic device that controls the neural network signal measuring device, identifies a prompt intended by the user based on at least one neural network signal obtained from the neural network signal measuring device, and outputs multimedia content generated by matching a video resource to the prompt; and A multimedia content creation system comprising: a server for obtaining the prompt from the electronic device and generating multimedia content by matching a video resource to the prompt based on a core keyword for each paragraph including one or more sentences included in the prompt; 2. In the first paragraph, the electronic device display; First network interface; Memory that stores one or more instructions; and a first processor for executing one or more of the instructions; The first processor obtains a first neural network signal from the external device by executing the one or more instructions, and inputs a second neural network signal of a user that confirms a first text image generated based on the obtained first neural network signal into an artificial intelligence model that outputs a text image based on the input neural network signal when the neural network signal is input, thereby obtaining a second text image from the artificial intelligence model. A multimedia content creation system that identifies the prompt based on the similarity between the second text image and a preset third text image.

3. In the second paragraph, the first processor Outputting at least one type of virtual input device image on the display for the above prompt input, A multimedia content creation system, wherein the neural network signal measuring device obtains a first neural network signal of a user who has confirmed at least one type of virtual input device image output on a display of the electronic device, and transmits the obtained first neural network signal to the electronic device.

4. In the third paragraph, the first processor If the similarity between the second text image and the third text image is identified as being greater than a threshold similarity, a keyboard area corresponding to the second text image is identified on the at least one type of virtual input device image, When the above keyboard area is identified, the user's intention is verified as to whether to select a text element or character element corresponding to the above keyboard area for a preset verification time, A multimedia content creation system that identifies a text element or character element corresponding to the keyboard area based on the verification result of the user's intent.

5. In the fourth paragraph, the electronic device Further comprising a camera for capturing a facial image of the user; The above first processor During the above verification time, the keyboard area generated by modulating the identified keyboard area based on visual characteristics and the identified keyboard area before modulation are alternately output on the display at a preset cycle, During the above verification time, a verification area including symbol patterns flickering at different frequencies is displayed over a portion of the keyboard area to verify the user's intention, Based on at least one of a third neural network signal of a user who has identified a verification area including the flickering symbol patterns obtained from the neural network signal measuring device or a first eye tracking signal determined from a facial image of the user who has identified the verification area, verifying the user's intention to select a text element corresponding to a keyboard area including the verification area, A multimedia content creation system that identifies a text element corresponding to the keyboard area when the user's intention to select the text element is verified.

6. In the fifth paragraph, the first processor From the above neural network signal measuring device, the third neural network signal of the user who confirmed the verification area is obtained, A multimedia content creation system that verifies the user's intention to select a text element corresponding to the keyboard area based on the intensity of the power spectrum for each frequency domain including the different frequencies included in the third neural network signal.

7. In the fifth paragraph, the first processor If the intensities of the power spectrums of each frequency domain corresponding to different frequencies used to flicker the symbol patterns included in the third neural network signal are all identified as being above a threshold or below a threshold, a user focus area is identified on the display based on the first eye tracking signal, A multimedia content creation system that verifies the user's intention to select a text element corresponding to the keyboard area based on the ratio of the area of ​​the overlapping area where the focus area and one partial verification area included in the verification area overlap with the entire area including the focus area and one partial verification area included in the verification area.

8. In the fourth paragraph, the first processor A second interface area adjacent to the first interface area where the virtual input device image is displayed, and distinct from the first interface area, outputting a plurality of candidate texts related to a text element or character element for which the user's intention has been verified, is output on the display, Based on the similarity between the fourth text image generated based on the fourth neural network signal of the user who confirmed the candidate texts output in the second interface area and the predetermined text image displayed on the second interface area, one of the candidate texts is identified, Verifying the user's intention to select the identified candidate text based on at least one of the user's fifth neural network signal or the second eye tracking signal for the candidate text verification area displayed overlapping the image area where the identified candidate text is displayed, A multimedia content creation system that identifies the selected candidate text when the user's intention to select the above candidate text is verified.

9. In the 8th paragraph, the first processor Obtaining a user face image gazing at one of the first interface area or the second interface area, and identifying a user eye blinking motion based on an aspect ratio of the vertical length to the horizontal length of the eye area detected on the user face image, It performs a control function corresponding to a preset eye blinking motion for each of the first interface area and the second interface area, A multimedia content creation system, characterized in that the control function for the preset eye blinking motion for each of the first interface area and the second interface area is set differently.

10. In the 8th paragraph, the first processor Obtaining a user face image gazing at at least one type of virtual input device image, The user's concentration level is determined based on the aspect ratio of the vertical length to the horizontal length of the eye area detected in the user's face image and the time for which the aspect ratio is maintained above a threshold ratio. A multimedia content creation system characterized in that the display is controlled so that, when the user's concentration level is identified as being below a threshold level, a frequency band used for visual stimulation of at least one of an image of the virtual input device displayed on the display, a verification area, and a keyboard area having modulated visual characteristics is upscaled at a preset ratio.

11. In the 10th paragraph, the first processor Determine whether the time for which the aspect ratio is maintained above the critical ratio is greater than or equal to the critical time, If the time for which the aspect ratio above the critical ratio is maintained is determined to be longer than the critical time, the intensity of the neural network signal of the user gazing at the virtual input device image obtained from the neural network signal measuring device is identified, A multimedia content creation system that determines the user's concentration level to be below the threshold level when the neural network signal intensity of the user staring at the virtual input device image is identified as being below the threshold intensity, even if the aspect ratio is determined to be above the threshold time.

12. In the 7th paragraph, the first processor Even if the intensity of one power spectrum among the intensities of the power spectrums by frequency domain included in the third neural network signal is identified as being above the threshold, A multimedia content creation system that determines that the user's intention to select the text element according to the third neural network signal is not verified when the alpha band energy of the third neural network signal is identified as decreasing and the beta band energy area is identified as increasing.

13. In the 11th paragraph, the first processor A multimedia content creation system characterized in that, when the user's concentration level is identified as being above a threshold level, the display is controlled so that the conversion speed of a visual stimulus displayed in at least one of the first interface area or the second interface area increases.

14. In the first paragraph, the server Second network interface; a database storing one or more instructions; and a second processor for executing one or more of the instructions; A multimedia content generation system in which the second processor obtains a prompt from the electronic device and matches the video resource to the prompt based on a target meaning determined based on whether a core keyword of one or more paragraphs included in the prompt is a polysemous or homonymous word or at least one sub-keyword determined based on whether two or more paragraphs including a core keyword representing the same target meaning are identified in the prompt.

15. In the 14th paragraph, the second processor Summarizing the prompt based on user input requesting the prompt summary obtained from the electronic device or attention determined based on the length of the prompt, Split the above summarized prompt into paragraphs, A multimedia content creation system characterized by determining core keywords for each of the above-mentioned divided paragraphs.

16. In the 15th paragraph, the second processor Determine the first paragraph division point based on the number of core keywords per paragraph, Determine a keyword network regarding at least one of the frequencies and connection relationships of the core keywords included in the first divided paragraphs divided according to the first paragraph division point, Among the above first segmented paragraphs, determine the similarity of the keyword network for adjacent paragraphs, The second paragraph division point within the text is determined by merging adjacent paragraphs whose similarity in the keyword network is greater than a threshold similarity, A multimedia content creation system characterized in that the summarized prompt is divided based on second paragraph divisions determined based on the second paragraph division points.

17. In the 15th paragraph, the second processor A multimedia content creation system that determines a single meaning corresponding one-to-one to the core keyword as the target meaning of the core keyword when the core keyword for each paragraph above is not a polysemous or homonymous word.

18. In the 17th paragraph, the second processor If the core keyword of each paragraph above is identified as a polysemous or homonymous word, the core keyword of each paragraph above is determined as the target keyword, Converting multiple meanings related to the above-determined target keyword into vectors of a predetermined dimension, Based on the distance between the transformed vectors, among the sentences adjacent to the sentence containing the target keyword, the sentence analysis range is determined with respect to the range of distances between the sentences to be analyzed to identify the target meaning and the sentence containing the target keyword. Identifying entity names for words included in at least one sentence determined based on the above sentence analysis scope, A multimedia content creation system that determines the target meaning based on a dependency relationship determined according to a syntax tree representing a dominance-subordinate relationship between the identified entity names.

19. In paragraph 17, the second processor Identify whether there is a paragraph with the same target meaning as the core keyword determined for each paragraph within the above prompt, If it is identified that there is a paragraph with the same target meaning as the above, at least one sub-keyword is determined for each paragraph containing the above core keyword, A multimedia content creation system that creates the multimedia content by matching image resources labeled with labeling information that includes both the target meaning and the sub-keyword for each paragraph based on the target meaning determined for the core keyword for each paragraph and the at least one sub-keyword.

Images