Electronic device for supplementing note content on basis of text identified from handwriting input and operation method therefor

Abstract

Disclosed are an electronic device for supplementing note content on the basis of text identified from a handwriting input, and an operation method therefor. The electronic device may comprise a memory for storing instructions, and at least one processor for executing the instructions, wherein when the at least one processor individually and / or collectively executes the instructions, the instructions cause the electronic device to: acquire, through a note application, a handwriting image based on a handwriting input; input the handwriting image to a handwriting recognition model to generate first text data corresponding to the handwriting input; generate first metadata on the basis of the first text data and first supplementary data input in relation to the handwriting image; generate second metadata through a search within the electronic device on the basis of the first metadata; and generate second text data in which the first text data has been supplemented, on the basis of the first text data and the second metadata.

Description

Electronic device for supplementing the contents of a note based on text identified from handwriting input and method of operating the same

[0001] An electronic device and a method of operating the same for supplementing the contents of a note based on identified handwriting are disclosed.

[0002] Handwriting recognition is a technology that allows electronic devices to convert handwritten characters into digital text. For example, handwriting recognition can convert handwriting input using a person's finger or a digital pen into digital text. Handwriting recognition can convert handwriting input into digital text, eliminating the need for separate typing. Improving the accuracy of handwriting recognition can be challenging. For example, handwriting can vary from user to user, and some users may have poor handwriting. However, advances in artificial intelligence and deep learning are significantly improving the accuracy of handwriting recognition.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0004] According to one embodiment, an electronic device may include a memory that stores instructions. The electronic device may include at least one processor that executes the instructions. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to obtain a handwriting image based on a handwriting input through a note application. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to input the handwriting image into a handwriting recognition model to generate first text data corresponding to the handwriting input. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to generate first metadata based on first additional data input in relation to the handwriting image and the first text data. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to generate second metadata through a search within the electronic device based on the first metadata. When said at least one processor individually and / or collectively executes said instructions, said instructions may cause said electronic device to generate second text data supplemented with said first text data based on said first text data and said second metadata.

[0005] According to one embodiment, an electronic device may include a memory that stores instructions. The electronic device may include at least one processor that executes the instructions. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to obtain a handwriting image based on a handwriting input through a note application. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to input the handwriting image into a handwriting recognition model to generate first text data corresponding to the handwriting input. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to generate first metadata based on at least one of text data, image data, auxiliary data, and audio data input in relation to the handwriting image and the first text data. When said at least one processor individually and / or collectively executes said instructions, said instructions may cause said electronic device to generate second metadata corresponding to said first metadata through a search within said electronic device based on said first metadata. When said at least one processor individually and / or collectively executes said instructions, said instructions may cause said electronic device to generate second text data supplemented with said first text data using said second metadata as contextual information for said first text data and display it through said note application.

[0006] According to one embodiment, an operating method of an electronic device may include an operation of acquiring a handwriting image based on a handwriting input through a note application. The operating method may include an operation of inputting the handwriting image into a handwriting recognition model to generate first text data corresponding to the handwriting input. The operating method may include an operation of generating first metadata based on first additional data input in relation to the handwriting image and the first text data. The operating method may include an operation of generating second metadata through a search within the electronic device based on the first metadata. The operating method may include an operation of generating second text data supplemented with the first text data based on the first text data and the second metadata.

[0007] According to one embodiment, an operating method of an electronic device may include an operation of acquiring a handwriting image based on a handwriting input through a note application. The operating method may include an operation of inputting the handwriting image into a handwriting recognition model to generate first text data corresponding to the handwriting input. The operating method may perform an operation of generating first metadata based on at least one of text data, image data, auxiliary data, and audio data input in relation to the handwriting image and the first text data. The operating method may include an operation of generating second metadata corresponding to the first metadata through a search within the electronic device based on the first metadata. The operating method may include an operation of generating second text data supplemented with the first text data by using the second metadata as context information for the first text data and displaying the second text data through the note application.

[0008] According to one embodiment, a non-transitory computer-readable recording medium may store one or more programs including commands that execute an operation of acquiring a handwriting image based on a handwriting input through a note application. The non-transitory computer-readable recording medium may store one or more programs including commands that execute an operation of inputting the handwriting image into a handwriting recognition model to generate first text data corresponding to the handwriting input. The non-transitory computer-readable recording medium may store one or more programs including commands that execute an operation of generating first metadata based on first additional data input in relation to the handwriting image and the first text data. The non-transitory computer-readable recording medium may store one or more programs including commands that execute an operation of generating second metadata through a search within the electronic device based on the first metadata. The non-transitory computer-readable recording medium may store one or more programs including commands that execute an operation of generating second text data supplemented with the first text data based on the first text data and the second metadata.

[0009] According to one embodiment, a non-transitory computer-readable recording medium may store one or more computer programs including commands for executing an operation of acquiring a handwriting image based on a handwriting input through a note application. The non-transitory computer-readable recording medium may store one or more computer programs including commands for executing an operation of inputting the handwriting image into a handwriting recognition model to generate first text data corresponding to the handwriting input. The non-transitory computer-readable recording medium may store one or more programs including commands for executing an operation of generating first metadata based on at least one of text data, image data, auxiliary data, and audio data input in relation to the handwriting image and the first text data. The non-transitory computer-readable recording medium may store one or more programs storing commands for executing an operation of generating second metadata corresponding to the first metadata through a search within the electronic device based on the first metadata. A non-transitory computer-readable recording medium may store one or more programs storing commands that execute an operation of generating second text data supplemented with the first text data using the second metadata as contextual information for the first text data and displaying the second text data through the note application.

[0010] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0011] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0012] FIG. 2 is a diagram illustrating an artificial intelligence system according to one embodiment of the present disclosure.

[0013] Figure 3 is a drawing for explaining conventional handwriting recognition.

[0014] FIG. 4 is a block diagram illustrating a method for generating second text data supplemented with first text data using additional data according to one embodiment of the present disclosure.

[0015] FIG. 5 is a diagram for explaining a first metadata generation model according to one embodiment of the present disclosure.

[0016] FIG. 6 is a diagram illustrating a search engine according to one embodiment of the present disclosure.

[0017] FIG. 7 is a diagram for explaining second text generation using a second metadata generation and text generation model according to one embodiment of the present disclosure.

[0018] FIG. 8 is a diagram for explaining the generation of second text data through a note application according to one embodiment of the present disclosure.

[0019] FIG. 9 is a diagram for explaining a format change of second text data according to one embodiment of the present disclosure.

[0020] FIG. 10 and FIG. 11 are drawings for explaining a system according to one embodiment of the present disclosure.

[0021] FIG. 12 is a flowchart for explaining an operation method of an electronic device according to one embodiment of the present disclosure.

[0022] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0023] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0024] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0025] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include a plurality of artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0026] The number of processors (120) may be one or more. For example, the processor (120) may have a multi-core processor structure such as a dual core, quad core, or hexa core.

[0027] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in the memory (130). For example, the processor (120) can correspond to a plurality of processors that collectively perform a plurality of operations by dividing them among the processors.

[0028] The memory (130) can store various data used by at least one component (e.g., the processor (120) or the sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., the program (140)) and input data or output data for commands related thereto. According to one embodiment, the memory (130) can include one or more memories. The instructions stored in the memory (130) can be stored by one memory. The instructions stored in the memory (130) can be divided and stored by a plurality of memories. The memory (130) can include a volatile memory (132) or a nonvolatile memory (134).

[0029] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0030] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0031] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0032] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0033] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0034] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0035] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0036] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0037] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0038] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0039] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0040] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0041] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0042] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0043] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0044] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0045] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0046] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0047] FIG. 2 is a diagram illustrating an artificial intelligence system according to one embodiment of the present disclosure.

[0048] A User Query / Response Interface (210) can receive user input. The user input can be any type of input, such as natural language, image, audio, and / or video. Additionally, context information can be transmitted together with the user input when the user input is transmitted. The context information can include various side information related to the time when the user input is input into the artificial intelligence system (200). For example, the context information can include information about the application currently being used by the user or information about the user's location. Additionally, the user input can be a mixed type of input of the above-described natural language, image, audio, video, and / or context information. Additionally, the user input can include non-natural language input, such as selecting a menu.

[0049] A user query / response interface (210) can provide output from a generative artificial intelligence system to a user. The output may include a natural language-based response and / or specific content. The output may also include an action requested by the user.

[0050] The AI ​​framework (220) can receive user input. Based on the user input (e.g., the user's query), the AI ​​framework (220) can coordinate and control one or more components necessary to perform an action corresponding to the user's intent.

[0051] User input received from the user query / response interface (210) can be transmitted to a prompt design component (221). The prompt design component (221) can be used to generate a prompt suitable as input to a generative model (e.g., a large language model (LLM) and / or a large multimodal model (LMM)) based on the user input.

[0052] The prompt design component (221) may be an AI component that uses a machine learning algorithm or a neural network. The prompt design component (221) may generate improved prompts through learning over time. The prompt design component (221) may access a knowledge repository (230) to generate prompts based on user input. The knowledge repository (230) may include user preference data, a prompt library, and / or prompt examples. The prompt design component (221) may provide the generated prompts to a generative model (e.g., an LLM and / or an LMM).

[0053] The APIs / Plugins management component (223) can communicate with external information sources based on requests for additional information when user input is transmitted to the generative model.

[0054] The APIs / Plugins management component (223) can establish a communication channel for communication with the outside of the system (200) via the API. The APIs / Plugins management component (223) can enable access to various data sources via the communication channel. The acquired information can be used to generate prompts by the prompt design component (221) along with user input, or can be used as input for the generative model (250).

[0055] The APIs / Plugins management component (223) can request a final action via an API when the final action in response to user input, rather than an intermediate action, must be performed by an application or service.

[0056] The refiner component (225) can fine-tune the output of the generative model (250). For example, the refiner component (225) can determine the relevance (e.g., score) between the output (e.g., content) of the generative model and the user input. For example, the refiner component (225) can determine whether the output contains biased information (e.g., selective information). For example, the refiner component (225) can determine whether the output contains harmful information (e.g., violent content or profanity).

[0057] The refinement component (225) can determine the degree of matching (e.g., score) between the output of the generative model (250) and the user input (e.g., the intent of the user input). If the refinement component (225) determines that the output of the generative model (250) does not correspond to the user input, the refinement component (225) can modify the output so that it corresponds to the user input.

[0058] The refinement component (225) can provide hints (e.g., hints for prompt generation) to the user so that the user can obtain information that matches the user's intention from the generative model (250).

[0059] A generative model (250) may refer to an artificial intelligence neural network that generates new data (e.g., text, images, audio, or video) based on user input (e.g., user utterances). The generative model (250) may include an image generation model and / or a language generation model.

[0060] Image generation models may include generative adversarial networks (GANs) and / or variational autoencoders (VAEs). An example of an image generation model is a diffusion-based generative model with a VAE and transformer architecture.

[0061] A language generation model (e.g., ChatGPT) can be a model trained to generate statistically most appropriate output based on input. A language generation model can include an LMM. An LMM can identify various types of input, such as text, images, audio (e.g., speech), and / or video, and generate new data corresponding to the input.

[0062] Hereinafter, for convenience of explanation, a large language model (LLM) and / or a large vision model (LVM) that can be used in the present disclosure will be described. However, it is obvious that the AI ​​neural network of the present disclosure may include not only a language model, but also various foundation models such as a code model and an image model, and / or other AI neural network models.

[0063] The AI ​​model that can be used in this disclosure may include an LLM, an AI neural network-based language model that has learned a large amount of text data through pre-training. The LLM may contain a relatively larger number of parameters (e.g., approximately 10 billion or more) than existing general language models. The LLM may utilize a transformer AI neural network structure based on an attention mechanism.

[0064] In one embodiment, the training of the LLM may include pre-training and / or fine-tuning. Pre-training may involve training the LLM to acquire general language knowledge using a large amount of text data. For example, pre-training may involve self-supervised learning, which predicts the next word in a text string using a previous word string. Fine-tuning may involve training the LLM to be suitable for a specific domain (e.g., chatbot, AI assistant, translation, summary generation, question answering) and / or task. Fine-tuning may involve further training (e.g., supervised learning, adaptive learning) the LLM using a dataset corresponding to the specific domain and / or task based on the pre-trained model. The LLM may perform a task based on text input containing natural language, referred to as a prompt.

[0065] In one embodiment, fine-tuning can be omitted in LLM learning. Users can control the prompts provided to the LLM to improve performance on a desired task. For example, users can control whether the prompts provide additional examples of tasks and / or guidance for performing the task, such as in-context learning, zero-shot learning, and / or few-shot learning. Publicly available LLMs include Bidirectional Encoder Representations from Transformer (BERT) and generative pre-trained transformer (GPT).

[0066] The term "LLM" can refer to the language neural network model itself, but it can also refer to the model of an LLM-based application (e.g., chatbot, AI assistant, translation, summary generation, text classification, sentence generation). For example, an LLM-based chatbot like ChatGPT or an LLM-based translator can also be referred to as "LLM."

[0067] "LLM" may include an inference engine utilizing the LLM neural network model. For example, "inputting an input prompt to the LLM" may mean "inputting the input prompt to an inference engine based on the LLM." For example, "the output of the LLM for the input prompt" may mean the output information of the last neural network layer of the LLM obtained when the input prompt is input to the LLM-based inference engine, and / or the output information modified through additional processing.

[0068] The attention mechanism is a technique that allows an AI model to focus (attention) on important parts of input data. The attention mechanism can be used to predict output data by predicting the extent to which a portion of time-series input data (e.g., time-series input data such as voice or video, or input data of some layers of a neural network) contributes to the output of the intermediate layers and / or the final output of the neural network. While a recurrent neural network (RNN) structure, which sequentially processes each element of a sequence, may exhibit poor prediction performance when there is information dependence between long time-series distances, the attention mechanism can account for information dependence between long time-series distances by controlling the level of weight concentration (attention) within the entire and / or partial context of the input data. A transformer can be configured as an encoder-decoder structure. The encoder can process the input data and output compressed information (e.g., a contextual representation). The decoder can process compressed information and output data in token units. Each encoder and decoder can include an independent attention network, and may further include a cross-attention network connecting the encoder and decoder.

[0069] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0070] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0071] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0072] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0073] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0074] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0075] Figure 3 is a drawing for explaining conventional handwriting recognition.

[0076] Referring to FIG. 3, a screen (310) of an electronic device (e.g., the electronic device (101) of FIG. 1)) running a note application is illustrated.

[0077] According to one embodiment, an electronic device can obtain handwriting input from a user. Unlike text input, which displays characters mapped to key inputs through key inputs, such as keyboards, touchscreen keyboards, and virtual keyboards, handwriting input may be an input based on the user's handwriting. For example, the electronic device can receive handwriting input through touch input using a digital pen, a touch pad, and / or a part of the user's body (e.g., a finger). The electronic device can display the obtained handwriting input as a handwriting image (312) on a display module (e.g., the display module (160) of FIG. 1 ).

[0078] According to one embodiment, the electronic device may input a handwriting image (312) into a handwriting recognition model (330). When the handwriting recognition model (330) receives the handwriting image (312), the handwriting recognition model (330) may generate text data (322) corresponding to the handwriting image (312). For example, the handwriting recognition model (330) may convert the handwriting image (312) into text data (322).

[0079] According to one embodiment, the handwriting recognition model (330) may include a printed handwriting recognition model that recognizes printed handwriting such as Korean, Japanese, or Chinese, and a cursive handwriting recognition model that recognizes cursive handwriting such as English, a Latin language, or a Cyrillic language.

[0080] According to one embodiment, a handwriting recognition model can analyze input information by matching each character based on graphical features regardless of the handwritten stroke order using a convolutional neural network (CNN) method, which is a type of deep learning method. For example, in the case of a printed language, each character can be divided into strokes, and the shape itself is complexly composed of multiple strokes for each character, so a deep learning method that considers static and graphical elements, such as a CNN method, can obtain highly accurate results.

[0081] According to one embodiment, a handwriting recognition model can analyze input information by selectively considering previously input information based on the stroke input order using a long-short term memory (LSTM) method based on a recurrent neural network (RNN), which is a type of deep learning method. For example, in the case of handwritten languages, since one character is often connected by one stroke and the number of characters themselves is relatively small, a RNN-based LSTM method with excellent performance when analyzing sequential input information in terms of time can obtain results with high recognition rates and recognition speeds.

[0082] In one embodiment, the electronic device can display text data (322) to replace the handwritten image (312). For example, a screen (320) running a note application can include text data (322). For example, the electronic device can display text data corresponding to the handwritten image (312), such as "I listened to music at a cafe with my daughter, the prettiest in the world, yesterday," on the screen (320). The electronic device can display only the text data (322) recognized based on the handwritten image (312).

[0083] In one embodiment, there may be situations where quick note-taking is required, such as lectures, interviews, and meetings. In such situations, it may be difficult for the user to compose complete and / or appropriate sentences using a note-taking application. In such situations, the user may desire the note-taking application to display more content, more accurate content, and / or more complete sentences than the handwritten input. Even in situations other than those described above, the user may desire the note-taking application to display more content, more accurate content, and / or more complete sentences than the handwritten input. For example, the user may desire the note-taking application to display richer content than the handwritten input based on the data entered into the note-taking application.

[0084] Below, we will explain the creation of text data supplemented with handwriting input, rather than simply converting handwriting images into text data based on data entered into a note application along with handwriting input.

[0085] FIG. 4 is a block diagram illustrating a method for generating second text data supplemented with first text data using additional data according to one embodiment of the present disclosure.

[0086] Referring to FIG. 4, an electronic device (e.g., the electronic device (101) of FIG. 1) may include a note application (400), a handwriting recognition model (410), a first metadata generation model (420), a search engine (430), a second metadata generation model (440), and a text generation model (450) (e.g., the generative model (250) of FIG. 2). It will be apparent to those skilled in the art that the above-described models and the search engine (430) may be implemented in software and / or hardware. At least one of the above-described models and the search engine may include an artificial intelligence model trained to perform the operations described in the present disclosure.

[0087] According to one embodiment, the note application (400) may be an application capable of receiving input via a digital pen, a touchpad, a keyboard, a virtual keyboard, a touch screen, and a user's body part (e.g., a finger). The note application (400) may support not only handwriting input and / or text input, but also the addition of audio data (e.g., recording data, music data, etc.). The note application (400) may also support the addition of image data (e.g., image data captured using a camera (e.g., the camera module (180) of FIG. 1), image data stored in a gallery application, captured images, etc.).

[0088] According to one embodiment, the note application (400) can store multiple notes and manage handwritten images, text data, image data, and voice data for each note. The note application (400) can obtain tag data based on specific keywords. The note application can manage auxiliary data for each note, including various data such as tag data, note creation date, note modification date, note author, and note title.

[0089] According to one embodiment, the electronic device may input a handwriting image (e.g., a handwriting image (312) of FIG. 3) based on handwriting input through a note application (400) into a handwriting recognition model (410) (e.g., a handwriting recognition model (330) of FIG. 3). A description of the handwriting recognition model (410) is omitted as it has been described above with reference to FIG. 3.

[0090] According to one embodiment, the handwriting recognition model (410) may generate first text data (e.g., text data (322) of FIG. 3) corresponding to the handwriting image. The first text data may include text data readable by an electronic device, in which handwritten characters included in the handwriting image are converted.

[0091] According to one embodiment, the electronic device may input first text data and first additional data into the first metadata generation model (420). The first additional data may include at least one of image data, audio data, text data, and auxiliary data input in the same note as the handwriting image input into the handwriting recognition model (410). The first additional data may include at least one of text data, image data, auxiliary data, and audio data inserted together with the handwriting input. The text data may be data generated based on text input in which characters mapped to key inputs are displayed through key inputs, such as a keyboard, a touchscreen keyboard, and a virtual keyboard, and may be distinguished from first text data based on handwriting input.

[0092] According to one embodiment, the first metadata generation model (420) can generate first metadata based on the input first text data and first additional data. The first metadata generation model (420) can preprocess the first text data and the first additional data to facilitate search. The first metadata generation model (420) can preprocess the first text data and the first additional data to generate the first metadata. The first metadata generation model (420) and the first metadata will be described later with reference to FIG. 5.

[0093] According to one embodiment, the electronic device may input first metadata into a search engine (430). The search engine (430) may collect data related to the first metadata from among data stored within the note application (400) and / or within the electronic device. Based on the first metadata, the search engine (430) may collect data highly related to the first metadata from among data stored within the note application (400) and / or within the electronic device and determine the collected data as second additional data. The search engine (430) will be described later with reference to FIG. 6.

[0094] According to one embodiment, the electronic device can input second additional data into a second metadata generation model (440). The structure of the second metadata generation model (440) may be the same as the first metadata generation model (420). The operation of the second metadata generation model (440) may be the same as the first metadata generation model (420). The second metadata generation model (440) may have a larger size than the first metadata generation model (420) because it must process second additional data including data collected from the electronic device as well as the note application (400). The second metadata generation model (440) will be described later with reference to FIG. 4.

[0095] In one embodiment, the first text data and the first additional data may be additionally input into the second metadata generation model (440). For example, if there is a blank space in the handwritten image that the user has not yet filled in, the first text data and the first additional data may be input into the second metadata generation model (440) to infer the content of the blank space. However, this is merely an example and the present disclosure is not limited thereto.

[0096] According to one embodiment, the electronic device may input first text data, second metadata, and previously generated second text data into a text generation model (450). The electronic device may input the first text data and the second metadata into the text generation model (450). For example, if there is no history of inputting the second text data, the previously generated second text data may not be input into the text generation model (450).

[0097] According to one embodiment, the text generation model (450) can generate second text data based on first text data, second metadata, and previously generated second text data. The text generation model (450) can generate second text based on the first text data by using at least one of the second metadata and the previously generated second text data as context data. The text generation model (450) can include a generative artificial intelligence model based on LLM and / or LMM. The text generation model (450) will be described later with reference to FIG. 7.

[0098] According to one embodiment, the second text data generated by the text generation model (450) may be displayed on a display module (e.g., the display module (160) of FIG. 1) via the note application (400). The second text data may be displayed instead of a handwritten image corresponding to the first text data. The second text data may be used as the second text data generated later.

[0099] Below, the first metadata creation model (420) will be described.

[0100] FIG. 5 is a diagram for explaining a first metadata generation model according to one embodiment of the present disclosure.

[0101] Referring to FIG. 5, a first metadata generation model (500) (e.g., the first metadata generation model (420) of FIG. 4) is illustrated.

[0102] According to one embodiment, the first metadata generation model (500) can obtain first metadata (503) corresponding to the first additional data (501). The first metadata generation model (500) can include a plurality of models that output the first metadata (503) corresponding to the first additional data (501). The first metadata generation model (500) can include a plurality of models that preprocess data included in the first additional data (501). The first metadata generation model (500) can include at least one transformation model corresponding to each data included in the first additional data (501).

[0103] For example, the plurality of models may include an audio feature extraction model (510), a speech to text model (520), an image feature extraction model (530), an image to text model (540), and a text summarization model (550). The plurality of models may operate in parallel. At least some of the plurality of models may be based on at least some of data processing modules, such as a DNN module, an RNN module, a CNN module, a signal processing module, an image encoder / decoder, and an audio encoder / decoder. The first metadata generation model (500) may process the first additional data (501) so that a search engine (e.g., the search engine (430) of FIG. 4) can easily use it for searching.

[0104] According to one embodiment, the first metadata generation model (500) may include an audio feature extraction model (510) that extracts audio features from audio data and a speech to text model (520) that extracts speech from the audio data and converts it into text (e.g., speech recognition text). However, this is merely an example, and the first metadata generation model (500) may further include a model for processing the audio data into data that can be used for searching. For example, the first metadata generation model (500) may further include a model that generates text describing the audio data from the audio features. For example, the first metadata generation model (500) may further include a model that analyzes the audio data to generate the text "music with an exciting atmosphere in the style of bossa nova." For example, the first metadata generation model (500) may further include a model that generates a characteristic sound based on the extracted audio features.

[0105] According to one embodiment, the first metadata generation model (500) may include an image feature extraction model (530) that extracts image features from image data and an image to text model (540) that outputs a description of the image data (e.g., image description text). The image to text model (540) may be a model pre-trained with training data that pairs images and texts. The image to text model may be a model pre-trained using a pair of image embeddings and text embeddings output by an image encoder and a text encoder. The image encoder may be a ResNet or a VIT (Vision transformer), and the text encoder may be a transformer model. For example, the image to text model (540) may be a CLIP model announced by open AI. However, this is merely an example, and the image to text model (540) may be implemented with various models.

[0106] In one embodiment, the first metadata generation model (500) may further include a model for processing image features into data usable for search. For example, the first metadata generation model (500) may further include a model for generating feature images based on image features.

[0107] According to one embodiment, the first metadata generation model (500) may include a text summary model (550) that summarizes the first text data based on the first text data (e.g., the text data (322) of FIG. 3) and / or auxiliary data. The text summary model (550) may output text summary data based on the first text data and / or the auxiliary data. However, this is merely an example, and the first metadata generation model (500) may further include a model that generates text data based on the first text data and / or the auxiliary data. The auxiliary data may serve to limit the scope when a search engine searches for text associated with the text data. For example, the search engine may perform a text search based on date, time, and tag data based on the auxiliary data.

[0108] According to one embodiment, the first metadata generation model (500) can generate first metadata (503) based on the first additional data (501). The first metadata generation model (500) can preprocess the first additional data (501) to input it into a search engine. For example, the first metadata generation model (500) can convert the first additional data (501) into a format that can be input into the search engine.

[0109] According to one embodiment, the first metadata (503), which is an output of the first metadata generation model (500), may include an output of each model included in the first metadata generation model (500). For example, the first metadata (503) may include audio features, voice recognition text, image features, image description text, and text summary data. However, this is merely an example, and the data included in the first metadata (503) may vary depending on the data included in the first additional data (501) and / or the models included in the first metadata generation model (500).

[0110] The first metadata generation model (500) can transmit the first metadata (503) to a search engine. The search engine that received the first metadata (503) will be described below.

[0111] FIG. 6 is a diagram illustrating a search engine according to one embodiment of the present disclosure.

[0112] Referring to FIG. 6, a search engine (600) (e.g., the search engine (430) of FIG. 4) is illustrated. The search engine (600) can search for at least one of audio data, image data, and test data within an electronic device (e.g., the electronic device (101) of FIG. 1) based on first metadata (603) (e.g., the first metadata (503) of FIG. 5). The search engine (600) may include multiple engines for searching. For example, the search engine (600) may include an audio search engine (610), an image search engine (620), and a text search engine (630). However, this is merely an example, and it will be apparent to those skilled in the art that the search engine (600) may include engines for additional searches.

[0113] According to one embodiment, the audio search engine (610) may perform an audio search within the electronic device based on data related to audio and sound among the first metadata (603). For example, the audio search engine (610) may perform an audio search based on audio features and audio description text. For example, if the text summary data includes content related to audio and sound, the audio search engine (610) may perform a music audio search based on such content. For example, if the text summary data includes content such as "I listened to a piano jazz song with a calm atmosphere," the audio search engine (610) may perform an audio search based on such content.

[0114] In one embodiment, the audio search engine (610) may perform an Internet search based on audio and sound-related data among the first metadata (603) by utilizing the network capabilities of the electronic device (e.g., the first network (198) of FIG. 1 and the second network (199) of FIG. 1). For example, the audio search engine (610) may perform a search through a music streaming application logged into the electronic device.

[0115] According to one embodiment, the search results of the audio search engine (610) may include text data (e.g., attribute data) regarding the searched audio data rather than audio data. For example, the audio search engine (610) may extract attribute data regarding the searched audio data. For example, the audio search engine (610) may extract attribute data such as genre, mood, singer (or performer), and title regarding the searched audio data. The audio search engine (610) may extract the attribute data using an artificial intelligence model trained to extract the aforementioned attribute data regarding the searched audio data.

[0116] According to one embodiment, the audio search engine (610) can output searched audio data and pre-mapped attribute data.

[0117] According to one embodiment, the audio search engine (610) can search a plurality of audio data. The search engine (600) (or, the audio search engine (610)) can determine N audio data having the highest similarity to the audio feature and / or audio description text among the plurality of audio data. N can be a natural number greater than or equal to 1. For example, the search engine (600) can determine the N audio data using the probability value of the classification result of cosine similarity. However, this is merely an example, and the present disclosure is not limited thereto. For example, the search engine (600) can determine the N audio data using at least one of various algorithms and machine learning models that determine the N audio data having high similarity.

[0118] According to one embodiment, the image search engine (620) can perform an image search within the electronic device based on image-related data among the first metadata (603). For example, the image search engine (620) can perform an image search based on image features and image description text. For example, if the text summary data includes content related to the image, the image search engine (620) can perform an image search based on that content. For example, if the text summary data includes content such as "I took a picture of my daughter, the prettiest in the world, in a brightly sunny cafe," the image search engine (620) can perform an image search for a gallery application based on that content.

[0119] In one embodiment, the image search engine (620) may perform an Internet search based on image-related data among the first metadata (603) using the network function of the electronic device. For example, the image search engine (620) may perform a search through an application providing a cloud service that is logged into the electronic device.

[0120] According to one embodiment, the search results of the image search engine (620) may include text data (e.g., attribute data) regarding the searched image data rather than the image data. For example, the image search engine (620) may extract attribute data regarding the searched image data. For example, the image search engine (620) may extract attribute data such as the atmosphere, objects included in the image, the shooting location, and the shooting time regarding the searched image data. The image search engine (620) may extract the attribute data using an artificial intelligence model trained to extract the aforementioned attribute data regarding the searched image data.

[0121] According to one embodiment, the image search engine (620) can output searched image data and pre-mapped attribute data.

[0122] According to one embodiment, the image search engine (620) can search a plurality of image data. The search engine (600) (or, the image search engine (620)) can determine N image data having the highest similarity to the image feature and / or image description text among the plurality of image data. N can be a natural number greater than or equal to 1. For example, the search engine (600) can determine the N image data using the probability value of the classification result of cosine similarity. However, this is merely an example, and the present disclosure is not limited thereto. For example, the search engine (600) can determine the N image data using at least one of various algorithms and machine learning models that determine the N image data having high similarity.

[0123] According to one embodiment, the text search engine (630) may perform a text search within the electronic device based on data (e.g., text summary data) generated based on the first text and / or auxiliary data among the first metadata (603). For example, the text search engine (630) may search for notes with the same tag in the note application based on tag data included in the text summary data. For example, the text search engine (630) may search for notes with similar content in the note application based on the text summary data. For example, the text search engine (630) may search for contacts stored in the contact application based on the text summary data.

[0124] In one embodiment, the text search engine (630) may perform an Internet search based on text summary data among the first metadata (603) using the network function of the electronic device. For example, the text search engine (630) may perform the search using the web browser of the electronic device.

[0125] According to one embodiment, the text search engine (630) can search a plurality of text data. The search engine (600) (or, the text search engine (630)) can determine N text data having the highest similarity to the text summary data among the plurality of text data. N can be a natural number greater than or equal to 1. For example, the search engine (600) can determine the N text data using the probability value of the classification result of cosine similarity. However, this is merely an example, and the present disclosure is not limited thereto. For example, the search engine (600) can determine the N text data using at least one of various algorithms and machine learning models that determine the N text data having high similarity.

[0126] According to one embodiment, data output from the search engine (600) (e.g., search results of the search engine (600)) may be determined as second additional data (605). For example, the second additional data (605) may include at least one of N audio data, N image data, and N text data. For example, the second additional data (605) may include at least one of N audio data, attribute information for the N audio data, N image data, attribute information for the N image data, and N text data.

[0127] According to one embodiment, the second additional data (605) output from the search engine (600) may be transmitted to a second metadata generation model (e.g., the second metadata generation model (440) of FIG. 4).

[0128] Below, the second metadata generation model and the text generation model (e.g., the text generation model (450) of FIG. 4) will be described.

[0129] FIG. 7 is a diagram for explaining second text generation using a second metadata generation and text generation model according to one embodiment of the present disclosure.

[0130] Referring to FIG. 7, a second metadata generation model (700) (e.g., the second metadata generation model (440) of FIG. 4) and a text generation model (710) (e.g., the generative model (250) of FIG. 2 and the text generation model (450) of FIG. 4) are illustrated.

[0131] According to an example, second additional data (705) (e.g., second additional data (605) of FIG. 6) can be input into a second metadata generation model (700).

[0132] According to one embodiment, the second metadata generation model (700) may be structurally identical to the first metadata generation model (e.g., the first metadata generation model (420) of FIG. 4 and the first metadata generation model (500) of FIG. 5). For example, the second metadata generation model (700) may include a plurality of models, and the plurality of models may include an audio feature extraction model (e.g., the audio feature extraction model (510) of FIG. 5), a speech to text model (e.g., the speech to text model (520) of FIG. 5), an image feature extraction model (e.g., the image feature extraction model (530) of FIG. 5), an image to text model (e.g., the image to text model (540)), and a text summary model (e.g., the text summary model (550)). However, this is merely an example, and it will be apparent to those skilled in the art that the second metadata generation model (700) may further include a model for preprocessing the second additional data (705). The description of the first metadata creation model can be equally applied to the second metadata creation model (700), so a detailed description is omitted.

[0133] According to one embodiment, the second metadata generation model (700) may include text data into which each data included in the second additional data (705) is converted. The second metadata may include text data into which the second additional data (705) is converted by the second metadata generation model (700). For example, the second metadata may not include image data and audio data. When the text generation model (710) includes an LLM-based generative model, the second metadata generation model (700) may convert the second additional data (705) into text data for input into the text generation model (710). The second metadata generation model (700) may convert the second additional data (705) into a format (e.g., text data) for input into the text generation model (710). The second metadata generation model (700) can preprocess the second additional data (705) into a format (e.g., text data) for inputting into the text generation model (710). For example, the second metadata generation model (700) can output speech recognition text, image description text, and text summary data output from a speech-to-text model, an image-to-text model, and a text summary model as second metadata. The second metadata may include text data in which each data included in the second additional data (705) is converted. For example, the second metadata generation model (700) may include text output from a model that describes audio data based on extracted audio features. For example, the second metadata may include text data describing the mood and nuance of audio data retrieved based on audio data of the first additional data. For example, the second metadata may include text data describing the mood and objects of image data retrieved based on image data of the first additional data.For example, the second metadata may include text data describing the notes retrieved based on the first text data (e.g., text data (322) of FIG. 3) and auxiliary data.

[0134] In one embodiment, when the text generation model (710) includes an LMM-based generative model (e.g., a multi-modal-based generative model), the second additional data can be input into the text generation model (710). When the text generation model (710) includes an LMM-based generative model, the second additional data can be input into the text generation model (710) without having to convert it into text data.

[0135] According to one embodiment, at least one of the second metadata and the previously generated second text data and the first text data may be input into a text generation model (710). The electronic device (e.g., the electronic device (101) of FIG. 1) may further include a prompt generation model (e.g., the prompt design component (221) of FIG. 2). For example, the prompt generation model may be included in the text generation model (710).

[0136] According to one embodiment, if no previously generated second text data exists (e.g., if no second text data has ever been generated for a particular note), the prompt generation model can generate a prompt including instructions and / or codes that cause the text generation model (710) to generate second text data supplemented with the first text based on the first text and the second metadata. The prompt generation model can generate a prompt including instructions and / or codes that cause the text generation model (710) to generate second text data supplemented with the first text data based on at least one of the second metadata and the previously generated second text data and the first text data.

[0137] According to one embodiment, the text generation model (710) can generate second text data. The text generation model (710) can generate the second text data based on a prompt. The second text data can be input into a display module (e.g., the display module (160) of FIG. 1) through a note application instead of a handwriting image (e.g., the handwriting image (312) of FIG. 3) input into a handwriting recognition model (e.g., the handwriting recognition model (330) of FIG. 3 and the handwriting recognition model (410) of FIG. 4). The second text data can be used as previously generated second text data when a second text image is generated later according to a new handwriting input in the note in which the second text data was generated. The previously generated second text data and second metadata can be used as context information when generating the second text data.

[0138] According to one embodiment, the electronic device may further include an image generation model (not shown). The electronic device may generate a command that causes the image generation model to generate an image related to the second text data based on at least one of the first metadata and the previously generated second text data and the first text data. The image generation model may generate an image related to the second text data based on the command. The electronic device may display the generated image together with the second text data in a note.

[0139] FIG. 8 is a diagram for explaining the generation of second text data through a note application according to one embodiment of the present disclosure.

[0140] Referring to FIG. 8, a screen (800) displaying a note written through a note application is illustrated. The note may include a handwriting image (802) corresponding to handwriting input (e.g., handwriting image (312) of FIG. 3), audio data (804), image data (806), and text data (808).

[0141] According to one embodiment, the conversion of the handwriting image (802) into second text data (812) may be performed for each line of handwriting input entered into the note. The electronic device (e.g., the electronic device (101) of FIG. 1) may perform the conversion into the second text data (812) for each line without waiting for the completion of the entire paragraph. For example, if three lines of handwriting input are entered into the note, the electronic device may start converting the handwriting input of the first line into second text data (812) when the writing of the first line is completed and the writing of the second line begins. The electronic device may display the second text data (812) generated for each line. For example, even if the second line is being written, the electronic device may display the second text data (812) for the first line when the conversion into the second text data (812) for the first line is completed. The electronic device may use the previously generated second text data as contextual information. For example, the electronic device can use the second text data generated for the first line as contextual information when generating the second text data for the second line.

[0142] According to one embodiment, the electronic device can perform conversion into second text data (812) for the entire paragraph at once after the entire paragraph has been written.

[0143] According to one embodiment, when the electronic device identifies a symbol indicating the completion of a sentence (e.g., a period, a question mark, an exclamation mark) during handwriting input, it can perform conversion on the completed sentence into second text data (812).

[0144] According to one embodiment, the electronic device may convert the handwriting image (802) into second text data (812) if it does not receive a user's handwriting input for a threshold time period.

[0145] According to one embodiment, the electronic device may convert the handwritten image (802) into second text data by receiving a command from a user to convert the handwritten image (802) into second text data (812). For example, the electronic device may receive a command to convert the handwritten image (802) into second text data (812) through a conversion button and convert the handwritten image (802) into the second text data.

[0146] According to one embodiment, it is assumed that the handwritten image (802) includes an image based on the handwritten input of “I listened to music at a cafe with my daughter, the prettiest in the world, yesterday.”, the audio data (804) includes fast-tempo music in the style of bossa nova, the image data (806) includes an image taken with the daughter in a bright room, and the text data (808) includes “#OO Cafe” as tag data.

[0147] According to one embodiment, the handwriting image (802) may be input into a handwriting recognition model (e.g., the handwriting recognition model (330) of FIG. 3 and the handwriting recognition model (410) of FIG. 4) and changed into a first text image. Audio data (804), image data (806) and text data (808) may be included in first additional data (e.g., the first additional data (501) of FIG. 5). The audio data (804), image data (806) and text data (808) may be input into a first metadata generation model (e.g., the first metadata generation model (420) of FIG. 4 and the first metadata generation model (500) of FIG. 5). The first metadata generation model may generate first metadata (e.g., the first metadata (503) of FIG. 5 and the first metadata (603) of FIG. 6) in response to the input of the first additional data.

[0148] According to one embodiment, the first metadata may be input into a search engine (e.g., search engine (430) of FIG. 4 and search engine (600) of FIG. 6). The search engine may generate second additional data (e.g., second additional data (605) of FIG. 6 and second additional data (705) of FIG. 7) based on the first metadata. The second additional data may be input into a second metadata generation model (e.g., second metadata generation model (440) of FIG. 4 and second metadata generation model (700) of FIG. 7). The second metadata generation model (700) may generate second metadata based on the second additional data. For example, the second metadata may include “upbeat music in bossa nova style” based on audio data (804), “with my daughter in a bright and pleasant place” based on image data (806), and “cafe” based on text data (808).

[0149] According to one embodiment, the electronic device may input a command to a text generation model (e.g., the generative model (250) of FIG. 2 , the text generation model (450) of FIG. 4 , and the text generation model (710) of FIG. 7 ) to supplement the first text data (e.g., the text data (322) of FIG. 3 ) based on “upbeat music in the bossa nova style,” “in a bright and pleasant place with my daughter,” and “in a café,” to generate second text data (812). The text generation model may generate the second text data (812) based on the input command. For example, the text generation model may generate “I had a great time listening to upbeat music in the bossa nova style with my prettiest daughter in the world in a bright and pleasant café.” as the second text data (812). The electronic device may display the second text data (812) in a note.

[0150] According to one embodiment, the second text data (812) may include text that supplements at least a portion of the first text data and is associated with the first supplementary data, and at least a portion of the first text data.

[0151] Below, the conversion and format change of the handwriting image (802) into second text data (812) will be described.

[0152] FIG. 9 is a diagram for explaining a format change of second text data according to one embodiment of the present disclosure.

[0153] Referring to FIG. 9, a screen (900) is shown displaying notes with a handwriting image (902) corresponding to handwriting input (e.g., handwriting image (312) of FIG. 3) and audio data (904) (e.g., audio data (804) of FIG. 8).

[0154] According to one embodiment, the electronic device may generate second text data (912) (e.g., second text data (812) of FIG. 8) according to the method described above with reference to FIGS. 4 to 8. The electronic device may identify the type of note based on the first text data (e.g., text data (322) of FIG. 3) converted from the handwriting image (902) and audio data (904). For example, the electronic device may identify that the type of note is minutes based on the first text data and audio data (904).

[0155] In one embodiment, the electronic device may modify the second text data (912) based on the type of identified note. For example, the electronic device may modify words and sentences in the second text data (912) to suit the meeting minutes.

[0156] In one embodiment, the electronic device may change the format of the second text data (912) to a format based on the type of identified note. For example, the electronic device may change the format of the second text data (912) to match the format of meeting minutes.

[0157] Below, a system that performs the generation of the second text data (912) described above will be described.

[0158] FIG. 10 and FIG. 11 are drawings for explaining a system according to one embodiment of the present disclosure.

[0159] Referring to FIG. 10, an electronic device (1001) (e.g., the electronic device (101) of FIG. 1) is illustrated.

[0160] According to one embodiment, the electronic device (1001) may include a note application (1000) (e.g., the note application (400) of FIG. 4). The note application (1000) may include a handwriting recognition model (1010) (e.g., the handwriting recognition model (330) of FIG. 3 and the handwriting recognition model (410) of FIG. 4), a first metadata generation model (1020) (e.g., the first metadata generation model (420) of FIG. 4 and the first metadata generation model (500) of FIG. 5), a note search engine (1030), a second metadata generation model (1040) (e.g., the second metadata generation model (440) of FIG. 4 and the second metadata generation model (700) of FIG. 7), and a text generation model (1050) (e.g., the generative model (250) of FIG. 2, the text generation model (450) of FIG. 4, and the text generation model (710) of FIG. 7). Can be.

[0161] According to one embodiment, a search engine (e.g., search engine (430) of FIG. 4 and search engine (600) of FIG. 6) may be implemented as a note search engine (1030) included in the note application (1000) and an external search engine (1060). The note search engine (1030) may perform a search for multiple notes stored in the note application (1000). The external search engine (1060) may perform a search outside the note application (e.g., storage (1070)).

[0162] Referring to FIG. 11, an electronic device (1101) (e.g., the electronic device (101) of FIG. 1 and the electronic device (1001) of FIG. 10) is illustrated.

[0163] According to one embodiment, the electronic device (1101) includes a note application (1110) (e.g., the note application (400) of FIG. 4 and the note application (1000) of FIG. 10), a handwriting recognition model (1120) (e.g., the handwriting recognition model (330) of FIG. 3, the handwriting recognition model (410) of FIG. 4 and the handwriting recognition model (1010) of FIG. 10), a first metadata generation model (1130) (e.g., the first metadata generation model (420) of FIG. 4, the first metadata generation model (500) of FIG. 5 and the first metadata generation model (1020) of FIG. 10), a search engine (1140) (e.g., the search engine (430) of FIG. 4 and the search engine (600) of FIG. 6), a second metadata generation model (1150) (e.g., the second metadata generation model (440) of FIG. 4 and the second metadata generation model (1150) of FIG. 7). It may include a model (700) and a second metadata generation model (1040) of FIG. 10) and a text generation model (1160) (e.g., a generative model (250) of FIG. 2, a text generation model (450) of FIG. 4, a text generation model (710) of FIG. 7, and a text generation model (1050) of FIG. 10).

[0164] According to one embodiment, the note application (1110) may transmit a handwriting image (e.g., a handwriting image (312) of FIG. 3 and a handwriting image (902) of FIG. 9) and first additional data (e.g., first additional data (501)) to a handwriting recognition model (1120) and a first metadata generation model (1130), respectively, and receive second text data (e.g., second text data (812) of FIG. 8 and second text data (912) of FIG. 9) from a text generation model (1160).

[0165] In one embodiment, the search engine (1140) may perform a search on a plurality of notes stored in the note application (1110). The search engine (1140) may perform a search on storage (1170) (e.g., storage (1070) of FIG. 10).

[0166] FIG. 12 is a flowchart for explaining an operation method of an electronic device according to one embodiment of the present disclosure.

[0167] In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Operations (1210) to (1250) may be performed by at least one component of an electronic device (e.g., the electronic device (101) of FIG. 1 , the electronic device (1001) of FIG. 10 , and the electronic device (1101) of FIG. 11 ). For example, instructions stored in a memory (e.g., the memory (130) of FIG. 1 ) may be executed by at least one processor (e.g., the processor (120) of FIG. 2 ), and the instructions may cause the electronic device to perform the following operations (1210) to (1250).

[0168] In operation (1210), the electronic device can obtain a handwriting image (e.g., a handwriting image (312) of FIG. 3 and a handwriting image (902) of FIG. 9) based on handwriting input through a note application (e.g., a note application (400) of FIG. 4, a note application (1000) of FIG. 10 and a note application (1110) of FIG. 11).

[0169] In operation (1220), the electronic device may input a handwriting image into a handwriting recognition model (e.g., handwriting recognition model (330) of FIG. 3, handwriting recognition model (410) of FIG. 4, handwriting recognition model (1010) of FIG. 10, and handwriting recognition model (1120) of FIG. 11) to generate first text data corresponding to the handwriting input (e.g., text data (322) of FIG. 3).

[0170] In operation (1230), the electronic device may generate first metadata (e.g., first metadata (503) of FIG. 5 and first metadata (603) of FIG. 6) based on first additional data (e.g., first additional data (501)) and first text data input in relation to the handwriting image.

[0171] In operation (1240), the electronic device can generate second metadata through a search within the electronic device based on the first metadata.

[0172] In operation (1250), the electronic device may generate second text data (e.g., second text data (812) of FIG. 8 and second text data (912) of FIG. 9) supplemented with the first text data based on the first text data and the second metadata.

[0173] Detailed descriptions of actions (1210) to (1250) are omitted as they have been described above in FIGS. 1 to 11.

[0174] According to one embodiment, an electronic device may include a memory that stores instructions. The electronic device may include at least one processor that executes the instructions. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to obtain a handwriting image based on a handwriting input through a note application. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to input the handwriting image into a handwriting recognition model to generate first text data corresponding to the handwriting input. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to generate first metadata based on first additional data and first text data input in relation to the handwriting image. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to generate second metadata through a search within the electronic device based on the first metadata. When at least one processor individually and / or collectively executes instructions, the instructions may cause the electronic device to generate second text data supplemented with the first text data based on the first text data and the second metadata.

[0175] According to one embodiment, the second text data may include text that supplements at least a portion of the first text data and is associated with the first supplementary data, and at least a portion of the first text data.

[0176] According to one embodiment, the first additional data may include at least one of text data (e.g., text data (808) of FIG. 8), image data (e.g., image data (806) of FIG. 8), auxiliary data, and audio data (e.g., audio data (804) of FIG. 8 and audio data (904) of FIG. 9) inserted together with the handwriting input.

[0177] According to one embodiment, when at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to input first additional data and first text data into a first metadata generation model (e.g., the first metadata generation model (420) of FIG. 4 , the first metadata generation model (500) of FIG. 5 , the first metadata generation model (1020) of FIG. 10 , and the first metadata generation model (1130) of FIG. 11 ). When at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to obtain first metadata corresponding to the first additional data from the first metadata generation model (e.g., the first metadata (503) of FIG. 5 and the first metadata (603) of FIG. 6 ).

[0178] According to one embodiment, the first metadata generation model may include at least one transformation model corresponding to each data included in the first additional data.

[0179] According to one embodiment, when at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to input first metadata into a search engine (e.g., search engine 430 of FIG. 4 , search engine 600 of FIG. 6 , and search engine 1140 of FIG. 11 ) that performs an internal search of the electronic device. When at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to determine a search result of the search engine as second additional data (e.g., second additional data 605 of FIG. 6 and second additional data 705 of FIG. 7 ). When at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to input second additional data into a second metadata generation model (e.g., the second metadata generation model (440) of FIG. 4 , the second metadata generation model (700) of FIG. 7 , the second metadata generation model (1040) of FIG. 10 , and the second metadata generation model (1150) of FIG. 11 ). When at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to obtain second metadata generated from the second metadata generation model.

[0180] According to one embodiment, the second metadata may include text data in which each data included in the second additional data is converted.

[0181] According to one embodiment, when at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to input at least one of the second metadata and the previously generated second text data and the first text data into a text generation model (e.g., the generative model (250) of FIG. 2 , the text generation model (450) of FIG. 4 , the text generation model (710) of FIG. 7 , the text generation model (1050) of FIG. 10 , and the text generation model (1160) of FIG. 11 ). When at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to obtain second text data supplemented with the first text data generated from the text generation model.

[0182] According to one embodiment, when at least one processor individually and / or collectively executes instructions, the instructions may cause the electronic device to generate second text data using at least one of the second metadata and the previously generated second text data as contextual information for the first text data through a text generation model.

[0183] According to one embodiment, an electronic device may include a memory that stores instructions. The electronic device may include at least one processor that executes the instructions. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to obtain a handwritten image based on a handwriting input through a note application. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to input the handwriting image into a handwriting recognition model to generate first text data corresponding to the handwriting input. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to generate first metadata based on at least one of text data, image data, auxiliary data, and audio data input in relation to the handwriting image and the first text data. When the at least one processor individually and / or collectively executes the instructions, the instructions may cause the electronic device to generate second metadata corresponding to the first metadata through a search within the electronic device based on the first metadata. When at least one processor individually and / or collectively executes instructions, the instructions may cause the electronic device to generate second text data supplemented with the first text data using the second metadata as contextual information for the first text data and display the second text data via the note application.

[0184] According to one embodiment, an operating method of an electronic device may include an operation of acquiring a handwriting image based on a handwriting input through a note application. The operating method may include an operation of inputting the handwriting image into a handwriting recognition model to generate first text data corresponding to the handwriting input. The operating method may include an operation of generating first metadata based on first additional data input in relation to the handwriting image and the first text data. The operating method may include an operation of generating second metadata through a search within the electronic device based on the first metadata. The operating method may include an operation of generating second text data supplemented with the first text data based on the first text data and the second metadata.

[0185] According to one embodiment, the second text data may include text that supplements at least a portion of the first text data and is associated with the first supplementary data, and at least a portion of the first text data.

[0186] According to one embodiment, the first additional data may include at least one of text data, image data, auxiliary data, and audio data inserted together with the handwriting input.

[0187] In one embodiment, the operation of generating the first metadata may include inputting the first additional data and the first text data into a first metadata generation model. The operation of generating the first metadata may include obtaining first metadata corresponding to the first additional data from the first metadata generation model.

[0188] According to one embodiment, the first metadata generation model may include at least one transformation model corresponding to each data included in the first additional data.

[0189] In one embodiment, the operation of generating the second metadata may include inputting the first metadata into a search engine that performs an internal search of the electronic device. The operation of generating the second metadata may include determining a search result of the search engine as the second additional data. The operation of generating the second metadata may include inputting the second additional data into a second metadata generation model. The operation of generating the second metadata may include obtaining the second metadata generated from the second metadata generation model.

[0190] According to one embodiment, the second metadata may include text data into which each data included in the second additional data is converted.

[0191] According to one embodiment, the operation of generating the second text data may include inputting at least one of the second metadata and the previously generated second text data and the first text data into a text generation model. The operation of generating the second text data may include obtaining the second text data supplemented with the first text data generated from the text generation model.

[0192] According to one embodiment, the operation of obtaining the second text data may generate the second text data by using at least one of the second metadata and the previously generated second text data as contextual information for the first text data through a text generation model.

[0193] According to one embodiment, a non-transitory computer-readable recording medium may store one or more programs including commands that execute an operation of acquiring a handwriting image based on a handwriting input through a note application. The non-transitory computer-readable recording medium may store one or more programs including commands that execute an operation of inputting the handwriting image into a handwriting recognition model to generate first text data corresponding to the handwriting input. The non-transitory computer-readable recording medium may store one or more programs including commands that execute an operation of generating first metadata based on first additional data input in relation to the handwriting image and the first text data. The non-transitory computer-readable recording medium may store one or more programs including commands that execute an operation of generating second metadata through a search within the electronic device based on the first metadata. The non-transitory computer-readable recording medium may store one or more programs including commands that execute an operation of generating second text data supplemented with the first text data based on the first text data and the second metadata.

[0194] The embodiments of the present invention disclosed in this specification and drawings are merely specific examples presented to easily explain the technical contents according to the embodiments of the present invention and to help understand the embodiments of the present invention, and are not intended to limit the scope of the embodiments of the present invention. Therefore, the scope of the various embodiments of the present invention should be interpreted as including all changes or modified forms derived based on the technical ideas of the various embodiments of the present invention in addition to the embodiments disclosed herein.

Claims

1. In electronic devices (101; 1001; 1101), Memory (130) for storing commands; and At least one processor (120) executing the above instructions Including, When the at least one processor (120) individually and / or collectively executes the instructions, the instructions cause the electronic device (101; 1001; 1101) to: Obtaining handwriting images (312; 902) based on handwriting input through a note application (400; 1000; 1110), The above handwriting image (312; 902) is input into a handwriting recognition model (330; 410; 1010; 1120) to generate first text data (322) corresponding to the handwriting input, Generate first metadata (503; 603) based on the first additional data (501) and the first text data (322) input in relation to the above handwritten image (312; 902), Generating second metadata through a search within the electronic device (101; 1001; 1101) based on the first metadata (503; 603), To generate second text data (812; 912) supplemented with the first text data (322) based on the first text data (322) and the second metadata. Electronic devices (101; 1001; 1101).

2. In paragraph 1, The above second text data (812; 912) is Text that supplements at least a portion of the first text data (322) in association with the first additional data (501) and includes at least a portion of the first text data (322). Electronic devices (101; 1001; 1101).

3. In either of paragraphs 1 and 2, The above first additional data (501) is, At least one of text data (808), image data (806), auxiliary data and audio data (804; 904) inserted together with the above handwriting input, Electronic devices (101; 1001; 1101).

4. In any one of paragraphs 1 to 3, When the at least one processor (120) individually and / or collectively executes the instructions, the instructions cause the electronic device (101; 1001; 1101) to: Inputting the first additional data (501) and the first text data (322) into the first metadata generation model (420; 500; 1020; 1130), and obtaining the first metadata (503; 603) corresponding to the first additional data (501) generated from the first metadata generation model (420; 500; 1020; 1130). Electronic devices (101; 1001; 1101).

5. In paragraph 4, The above first metadata generation model (420; 500; 1020; 1130) is Including at least one transformation model corresponding to each data included in the above first additional data (501), Electronic devices (101; 1001; 1101).

6. In paragraph 1, When the at least one processor (120) individually and / or collectively executes the instructions, the instructions cause the electronic device (101; 1001; 1101) to: Inputting the first metadata (503; 603) into a search engine (430; 600; 1140) that performs an internal search of the electronic device (101; 1001; 1101), determining the search result of the search engine (430; 600; 1140) as second additional data (605; 705), inputting the second additional data (605; 705) into a second metadata generation model (440; 700; 1040; 1150), and obtaining second metadata generated from the second metadata generation model (440; 700; 1040; 1150). Electronic devices (101; 1001; 1101).

7. In paragraph 6, The above second metadata is, Each data included in the above second additional data (605; 705) includes converted text data. Electronic devices (101; 1001; 1101).

8. In paragraph 1, When the at least one processor (120) individually and / or collectively executes the instructions, the instructions cause the electronic device (101; 1001; 1101) to: Inputting at least one of the second metadata and the previously generated second text data (812; 912) and the first text data (322) into a text generation model (250, 450, 710, 1050, and 1160), and obtaining the second text data (812; 912) supplemented with the first text data (322) generated from the text generation model (250, 450, 710, 1050, and 1160). Electronic devices (101; 1001; 1101).

9. In paragraph 8, When the at least one processor (120) individually and / or collectively executes the instructions, the instructions cause the electronic device (101; 1001; 1101) to: Generating the second text data (812; 912) by using at least one of the second metadata and the previously generated second text data (812; 912) as context information for the first text data (322) through the text generation model (250, 450, 710, 1050, and 1160). Electronic devices (101; 1001; 1101).

10. In the operating method of an electronic device (101; 1001; 1101), An action of obtaining a handwriting image (312; 902) based on handwriting input through a note application (400; 1000; 1110); An operation of inputting the above handwriting image (312; 902) into a handwriting recognition model (330; 410; 1010; 1120) to generate first text data (322) corresponding to the handwriting input; An operation of generating first metadata (503; 603) based on the first additional data (501) and the first text data (322) input in relation to the above handwritten image (312; 902); An operation of generating second metadata through a search within the electronic device (101; 1001; 1101) based on the first metadata (503; 603); and An operation of generating second text data (812; 912) supplemented with the first text data (322) based on the first text data (322) and the second metadata. including, How it works.

11. In paragraph 10, The above second text data (812; 912) is Text that supplements at least a portion of the first text data (322) in association with the first additional data (501) and includes at least a portion of the first text data (322). How it works.

12. In paragraph 10, The above first additional data (501) is, At least one of text data (808), image data (806), auxiliary data and audio data (804; 904) inserted together with the above handwriting input, How it works.

13. In paragraph 10, The operation of generating the above first metadata (503; 603) is: An operation of inputting the first additional data (501) and the first text data (322) into the first metadata generation model (420; 500; 1020; 1130); and An operation of obtaining first metadata (503; 603) corresponding to the first additional data (501) generated from the first metadata generation model (420; 500; 1020; 1130) including, How it works.

14. In any one of the 13 clauses, The above first metadata generation model (420; 500; 1020; 1130) is Including at least one transformation model corresponding to each data included in the above first additional data (501), How it works.

15. In paragraph 10, The operation of generating the above second metadata is: An action of inputting the above first metadata (503; 603) into a search engine (430; 600; 1140) that performs an internal search of the electronic device (101; 1001; 1101); An action of determining the search result of the above search engine (430; 600; 1140) as the second additional data (605; 705); An operation of inputting the second additional data (605; 705) into the second metadata generation model (440; 700; 1040; 1150); and An operation of obtaining second metadata generated from the second metadata generation model (440; 700; 1040; 1150) including, How it works.

Images