Electronic device, method, and non-transitory computer-readable storage medium for verifying information acquired by using language model

Abstract

A processor of an electronic device according to an embodiment may receive a prompt sentence from an external electronic device via a communication circuit. The processor may execute a first model for natural language processing to generate a first natural language sentence related to the prompt sentence. The processor may acquire a second natural language sentence by changing a portion of the first natural language sentence. The processor may execute a second model linked to evidence information to determine scores indicating accuracies of the first natural language sentence and the second natural language sentence. The evidence information may be used as a criterion for the second model to evaluate the accuracy of a natural language sentence. The processor may determine the first natural language sentence as a response message to be transmitted to the external electronic device by using the scores.

Description

Electronic device, method and non-transitory computer-readable storage medium for verifying information obtained using a language model

[0001] The present disclosure relates to an electronic device, a method, and a non-transitory computer-readable storage medium for verifying information obtained using a language model.

[0002] Natural language (or ordinary language) refers to the language used in human daily life. Electronic devices supporting functions related to natural language are being developed. For example, electronic devices can detect natural language contained in audio data and / or text.

[0003] In one embodiment, a non-transitory computer-readable storage medium storing instructions may be provided. The instructions, when executed by an electronic device including communication circuitry, may cause the electronic device to receive a prompt sentence from an external electronic device via the communication circuitry. The instructions, when executed by the electronic device, may cause the electronic device to execute a first model for natural language processing to generate a first natural language sentence related to the prompt sentence. The instructions, when executed by the electronic device, may cause the electronic device to modify a portion of the first natural language sentence to obtain a second natural language sentence. The instructions, when executed by the electronic device, may cause the electronic device to execute a second model linked to evidence information to determine scores representing the accuracy of each of the first natural language sentence and the second natural language sentence. The above evidence information may be used as a criterion for evaluating the accuracy of a natural language sentence by the second model. The instructions, when executed by the electronic device, may cause the electronic device to use the scores to determine the first natural language sentence as a response message to be transmitted to the external electronic device.

[0004] According to one embodiment, an electronic device may include a communication circuit, a memory storing instructions, and a processor. The instructions, when executed by the processor, may cause the electronic device to receive a prompt sentence from an external electronic device via the communication circuit. The instructions, when executed by the processor, may cause the electronic device to execute a first model for natural language processing to generate a first natural language sentence related to the prompt sentence. The instructions, when executed by the processor, may cause the electronic device to modify a portion of the first natural language sentence to obtain a second natural language sentence. The instructions, when executed by the processor, may cause the electronic device to execute a second model linked to evidence information to determine scores representing the accuracy of each of the first natural language sentence and the second natural language sentence. The evidence information may be used as a criterion for evaluating the accuracy of the natural language sentence by the second model. The instructions, when executed by the processor, may cause the electronic device to use the scores to determine the first natural language sentence as a response message to be transmitted to the external electronic device.

[0005] In one embodiment, a method of an electronic device including a communication circuit may be provided. The method may include receiving a prompt sentence from an external electronic device via the communication circuit. The method may include transmitting, to the external electronic device, a first signal related to a natural language sentence associated with the prompt sentence, obtained from a first model for natural language processing. The method may include determining whether to modify the natural language sentence using a second model for evaluating the natural language sentence using evidence information. The method may include transmitting, to the external electronic device, a second signal for modifying the natural language sentence based on the determination to modify the natural language sentence.

[0006] FIG. 1 is a block diagram of an electronic device according to one embodiment.

[0007] FIG. 2 illustrates one or more models executed by an electronic device, according to one embodiment.

[0008] FIG. 3 is a flowchart of the operation of an electronic device according to one embodiment.

[0009] Figures 4a, 4b, 4c, 4d, 4e and 4f illustrate exemplary operations of an electronic device for verifying natural language sentences obtained by executing a language model.

[0010] FIGS. 5A, 5B, 5C, 5D, 5E, and 5F illustrate exemplary operations of an electronic device in relation to a prompt sentence received from an external electronic device.

[0011] Hereinafter, various embodiments of this document are described with reference to the attached drawings.

[0012] The various embodiments of this document and the terminology used therein are not intended to limit the technology described in this document to a specific embodiment, but should be understood to include various modifications, equivalents, and / or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expression may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and / or B", "A, B, or C", or "at least one of A, B, and / or C" may include all possible combinations of the items listed together. Expressions such as "first", "second", "first", or "second" may modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).

[0013] The term "module" as used in this document includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimal unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0014] FIG. 1 is a block diagram of an electronic device (101), according to one embodiment. FIG. 1 illustrates an exemplary system including the electronic device (101) and / or an external electronic device (109). While the electronic device (101) and / or the external electronic device (109) are illustrated as being directly connected, the embodiment is not limited thereto. For example, the electronic device (101) and the external electronic device (109) may be connected to each other via one or more routers and / or an access point (AP).

[0015] The external electronic device (109) of FIG. 1 may be a terminal owned by a user. The terminal may include, for example, a personal computer (PC) such as a laptop or desktop, a smartphone, a smartpad, or a tablet PC. The terminal may include a portable gaming device and / or a gaming console. The terminal may include a smart accessory such as a smartwatch, a smart ring, and / or a head-mounted device (HMD). The terminal may also be referred to as a mobile device, a user terminal, a user equipment (UE), a multi-function device, a portable communication device, and / or a portable device.

[0016] Referring to FIG. 1, the electronic device (101) may include at least one of a processor (110-1), a memory (115-1), and a communication circuit (120-1). The processor (110-1), the memory (115-1), and the communication circuit (120-1) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (102-1). Hereinafter, the electronic components being operably coupled may mean that a direct connection or an indirect connection is established between the electronic components, either wired or wireless, such that a second electronic component is controlled by a first electronic component among the electronic components. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the electronic components of FIG. 1 (e.g., at least a portion of the processor (110-1), the memory (115-1), and the communication circuit (120-1)) may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of electronic components included in the electronic device (101) is not limited to that illustrated in FIG. 1. For example, the electronic device (101) may include only some of the electronic components illustrated in FIG. 1. One or more clusters in which a plurality of electronic devices including the electronic device (101) are grouped may be referred to as servers.

[0017] Referring to FIG. 1, according to one embodiment, an external electronic device (109) may include at least one of a processor (110-2), a memory (115-2), a communication circuit (120-2), a display (125), a microphone (126), and a speaker (127). Similar to the electronic device (101), the processor (110-2), the memory (115-2), the communication circuit (120-2), the display (125), the microphone (126), and / or the speaker (127) may be electrically and / or operatively connected to each other by electronic components such as a communication bus (102-2). In the present disclosure, among the descriptions of the processor (110-2), memory (115-2), and communication circuit (120-2) of the external electronic device (109), descriptions that overlap with the descriptions of the processor (110-1), memory (115-1), and communication circuit (120-1) of the electronic device (101) may be omitted.

[0018] In one embodiment, a processor (e.g., processors 110-1 and 110-2) may include circuitry (e.g., processing circuitry) for processing data based on one or more instructions. The circuitry for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and / or an application processor (AP). For example, the number of processors may be one or more. The processing circuitry of the processor that loads (or fetches) instructions and performs calculations corresponding to the loaded instructions may be referred to as or referred to as a core circuit (or core). For example, the processor may have a multi-core processor architecture including a plurality of core circuits, such as a dual core, a quad core, a hexa core, or an octa core. The functions and / or operations described with reference to the present disclosure may be collectively performed by one or more processing circuits included in a processor.

[0019] In one embodiment, memory (e.g., memories (115-1, 115-2)) may include circuitry for storing data and / or instructions input to and / or output from a processor (e.g., processors (110-1, 110-2)). The memory may include, for example, volatile memory, such as random-access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, a hard disk, a compact disc, a solid state drive (SSD), and an embedded multi media card (eMMC). The processor (110-1) of the electronic device (101) can execute instructions of the memory (115-1) within the electronic device (101) to perform functions and / or operations indicated by the instructions.

[0020] In one embodiment, the communication circuit (e.g., the communication circuits 120-1, 120-2) may include hardware for supporting transmission and / or reception of electrical signals between the electronic device (101) and the external electronic device (109). The communication circuit may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), near field communication (NFC), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), fifth generation (5G) new radio (NR), sixth generation (6G), and / or above-6G.

[0021] In one embodiment, the display (125) of the external electronic device (109) can output visualized information to the user. For example, the display (125) can be controlled by a controller, such as a graphic processing unit (GPU) and / or a processor (110-2), to output visualized information to the user. The display (125) can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs). The display (125) can include a flat panel display (FPD) and / or electronic paper. The embodiment is not limited thereto, and the display (125) can have an at least partially curved shape or a deformable shape. A display (125) having a deformable shape can be referred to as a flexible display.

[0022] In one embodiment, the external electronic device (109) may include a sensor (e.g., a touch sensor panel (TSP)) for detecting an external object (e.g., a user's finger) on the display (125). For example, using the TSP, the external electronic device (109) may detect an external object that is in contact with the display (125) or floating on the display (125). In response to detecting the external object, the external electronic device (109) may execute a function associated with a particular visual object among visual objects displayed within the display (125) that corresponds to a location of the external object on the display (125).

[0023] In one embodiment, a processor (110-2) of an external electronic device (109) may obtain an electrical signal representing vibration of the atmosphere from a microphone (126). The processor (110-2) may transmit an audio signal to a speaker (127), thereby causing the speaker (127) to cause vibration of the atmosphere. The external electronic device (109) may include one or more microphones and / or one or more speakers.

[0024] Referring to FIG. 1, one or more instructions (or commands) indicating operations and / or actions to be performed on data by a processor (e.g., processors (110-1, 110-2)) may be stored in a memory (e.g., memories (115-1, 115-2)). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, a program, and / or a software application (hereinafter, an application). Hereinafter, installation of an application may mean that one or more instructions provided in the form of an application are stored in a memory, and that the one or more applications are stored in a format executable by the processor (e.g., a file having an extension designated by the operating system of the electronic device (101)).

[0025] Referring to FIG. 1, an exemplary program (e.g., a server agent application (141)) installed in an electronic device (101) is illustrated. The processor (110-1) of the electronic device (101) may execute the server agent application (141) to execute a function related to natural language. The function may include a function of generating or transmitting at least one natural language sentence expressing information related to one or more natural language sentences (e.g., one or more natural language sentences received from a user of an external electronic device (109). The function may include a function of executing or calling another function supported by the electronic device (101) and expressed by one or more natural language sentences. The server agent application (141) may be installed in the electronic device (101) to support a service related to conversational interaction (e.g., a chatbot service).

[0026] Referring to FIG. 1, an exemplary program (e.g., a client agent application (151)) installed in an external electronic device (109) is illustrated. A processor (110-2) of the external electronic device (109) may execute the client agent application (151) to communicate with the electronic device (101). For example, the processor (110-2) executing the client agent application (151) may control a communication circuit (120-2) to establish a communication link between the electronic device (101) and the external electronic device (109). Through the communication link, the processor (110-2) may transmit one or more natural language sentences to the electronic device (101). The one or more natural language sentences may be recognized or identified from a user input received through a display (125) and / or a microphone (126). For example, using speech-to-text (STT), the processor (110-2) can detect or obtain one or more natural language sentences from an audio signal received through a microphone (126).

[0027] According to one embodiment, the electronic device (101) may receive at least one natural language sentence received by an external electronic device (109) executing a client agent application (151) through a communication link established between the electronic device (101) and the external electronic device (109) using a communication circuit (120-1). Hereinafter, the at least one prompt sentence may include the at least one natural language sentence received by the electronic device (101) from the external electronic device (109). The at least one prompt sentence may include one or more natural language sentences processed by the electronic device (101) executing the server agent application (141). The electronic device (101) that receives the at least one prompt sentence may execute the server agent application (141) to generate one or more natural language sentences related to the at least one prompt sentence.

[0028] One or more natural language sentences generated by executing a server agent application (141) may be verified by a processor (110-1) executing the server agent application (141) before being transmitted to an external electronic device (109). Hereinafter, the operation of verifying the natural language sentence may include an operation of obtaining a parameter indicating at least one of the fluency and / or accuracy of the natural language sentence. For example, the parameter may include a numeric value determined one-to-one for the natural language sentence. For example, the parameter may include one or more numeric values ​​corresponding to each of portions of the natural language sentence (e.g., portions of the natural language sentence distinguished by words and / or tokens).

[0029] In order to verify one or more natural language sentences, the electronic device (101) may include evidence information (142). An exemplary operation of the electronic device (101) for verifying one or more natural language sentences using the evidence information (142) is described with reference to FIG. 2. The electronic device (101) executing the server agent application (141) may perform calculations referred to as a neural network, an artificial neural network, a deep learning model, a machine learning model, a model, and / or an agent to generate one or more natural language sentences. The model may include a software application (e.g., the server agent application (141)) designed to simulate neural activity (e.g., reasoning, and / or learning) of a living organism, including a human. The processor (110-1) of the electronic device (101) may execute the software application to perform calculations related to the model. To perform the above calculations, a software application may be installed on the electronic device (101) together with information related to the calculations (e.g., types of calculations included in the model, and / or filters and / or weights related to layers included in the model).

[0030] A model for generating natural language sentences may include a generative model. From a probability distribution perspective, the generative model may be trained to generate second data from first data that follows a specific probability distribution (e.g., a probability distribution used to train the generative model). The generative model may include a neural language model (NLM) and / or a pretrained language model (PLM) (e.g., bidirectional encoder representations from transformers (BERT), text-to-text transfer transformer (T5), pathways language model (PaLM), and / or a generative pre-trained transformer (GPT)). Since the model is trained to fluently generate natural language sentences, the processor (110-1) executing the model may generate natural language sentences that include inaccurate information. The phenomenon of generating natural language sentences that include inaccurate information, and / or the natural language sentences, may be referred to as hallucination.

[0031] According to one embodiment, the electronic device (101) may, while executing a model for generating natural language sentences, verify the natural language sentences obtained by executing the model. For example, the electronic device (101) may verify the natural language sentences in order to reduce hallucination. The operation of the electronic device (101) verifying the natural language sentences obtained by executing the model is described with reference to FIGS. 3 to 4A to 4F. Based on the verification, the electronic device (101) may determine whether to provide the natural language sentences generated by the model to the external electronic device (109) (or the user of the external electronic device (109). If it is determined to provide the generated natural language sentences to the external electronic device (109), the electronic device (101) may transmit a signal representing the generated natural language sentences to the external electronic device (109) via the communication circuit (120-1). Based on the above verification, if it is decided not to provide the generated natural language sentence to the external electronic device (109), the electronic device (101) can re-generate the natural language sentence using the model. After transmitting a signal including the generated natural language sentence to the external electronic device (109), if it is decided not to provide the generated natural language sentence to the external electronic device (109), the electronic device (101) can transmit another signal to the external electronic device (109) to remove or retrieve the generated natural language sentence. The operation of the electronic device (101) determining or adjusting a user interface (UI) displayed through the external electronic device (109) based on the above verification is described with reference to FIGS. 5A to 5F.

[0032] Hereinafter, with reference to FIG. 2, one or more exemplary models executed by the electronic device (101) described with reference to FIG. 1 are described.

[0033] FIG. 2 illustrates one or more models executed by an electronic device, according to one embodiment. The electronic device of FIG. 2 may be an example of the electronic device (101) of FIG. 1.

[0034] Referring to FIG. 2, one or more models (e.g., a language model (210), a perturbation model (220), and / or a probability measurement model (230)) executed by an electronic device are illustrated, according to one embodiment. Some of the models illustrated in FIG. 2 (e.g., the language model (210)) may be executed by an external electronic device different from the electronic device (e.g., a server within a network that runs the language model (210). Each of the models illustrated in FIG. 2 may provide an application programming interface (API) for invoking or calling functionality related to the model. The API may be provided together with the model to run or execute the model without additional modification, such as a white box.

[0035] Referring to FIG. 2, by executing the language model (210), the electronic device can generate or obtain a natural language sentence (B1) related to a prompt sentence (e.g., “The Merchant of Venice is a Shakespearean comedy”). For example, the electronic device can execute the language model (210) using at least one prompt sentence to obtain the natural language sentence (B1). Referring to FIG. 2, the electronic device can execute the language model (210) using the exemplary prompt sentence “Tell me about The Merchant of Venice” to obtain the natural language sentence (B1). When the language model (210) is executed by an external electronic device (e.g., a server) different from the electronic device, the electronic device can communicate with the external electronic device to obtain or receive the natural language sentence (B1) generated by the language model (210).

[0036] Referring to FIG. 2, by executing the corruption model (220), the electronic device (101) can alter or change a natural language sentence (B1) obtained from the language model (210). The corruption model (220) may include T5 (or T5-b3). According to one embodiment, the electronic device (101) may execute the corruption model (220) using the natural language sentence (B1) to obtain or generate a plurality of natural language sentences (C1, C2, C3, ...) in which at least a portion of the natural language sentence (B1) is changed. In order to execute the corruption model (220) using the natural language sentence (B1), the electronic device (101) may obtain tokens corresponding to the natural language sentence (B1). A token may be matched one-to-one with a word (or morpheme). A token may include a vector (or matrix) in which a word corresponding to the token is parameterized. A token may include a vector corresponding to a word and / or morpheme corresponding to the token within a vector space representing semantic similarity between words and / or morphemes. A token may be referred to as token information and / or a token vector. One or more tokens corresponding to a natural language sentence (B1) may be acquired or generated to perform a calculation related to the natural language sentence (B1).

[0037] Referring to the exemplary case of FIG. 2, the electronic device (101) can execute the corruption model (220) using a natural language sentence (B1) with one token masked. For example, if the electronic device (101) has masked (or obscured) a portion of the natural language sentence (B1) (e.g., a portion corresponding to “Shakespeare” among the natural language sentences (B1), the electronic device (101) can execute the corruption model (220) to infer or generate a token (or word, and / or morpheme) corresponding to the masked portion. Referring to FIG. 2, natural language sentences (C1, C2, C3, ...) obtained by executing the corruption model (220) together with the natural language sentence (B1) with the portion masked are exemplarily illustrated.

[0038] In the exemplary case where the part corresponding to “Shakespeare” in the natural language sentence (B1) is masked, the natural language sentences (C1, C2, C3, ...) obtained by executing the corruption model (220) may include a word corresponding to a token inferred by the corruption model (220) at a position corresponding to the part. For example, the natural language sentence (C1) may include a different word (e.g., “Newton”) from the word in the masked part (e.g., “Shakespeare”) at a position corresponding to the masked part. For example, the natural language sentence (C2) may include a word (e.g., “Kim Man-jung”) filled in by the corruption model (220) at a position corresponding to the part. For example, the natural language sentence (C3) may include a word (e.g., “apple”) corresponding to a token generated by the corruption model (220) at a position corresponding to the part.

[0039] Referring to FIG. 2, the electronic device (101) may obtain information for verifying the natural language sentences (B1, C1, C2, C3, ...) by executing a probability measurement model (230) using at least one of a natural language sentence (B1) obtained using a language model (210) and natural language sentences (C1, C2, C3, ...) obtained using a corruption model (220). The information obtained by executing the probability measurement model (230) may include numerical values ​​(e.g., log probability and / or log likelihood) representing the naturalness of each of the natural language sentences (B1, C1, C2, C3, ...). In order to calculate the log probability, the probability measurement model (230) may include GPT2 (or GPT2-XL).

[0040] In one embodiment, the probability measurement model (230) may be trained to evaluate the naturalness of a natural language sentence. For example, a numerical value calculated using the probability measurement model (230) may represent a probability that a natural language sentence corresponding to the numerical value is generated by the language model (210). In one embodiment, the language model (210) may be trained so that the probability calculated using the probability measurement model (230) (e.g., the probability indicating the naturalness of a natural language sentence obtained using the language model (210)) increases.

[0041] In one embodiment, the electronic device (101) can use the evidence information (142) to execute a probability measurement model (230) to evaluate the naturalness of a natural language sentence more accurately than other models that retrieve information from a network without the evidence information (142). The evidence information (142) can be implicitly stored in the electronic device to prevent inaccurate evaluation of the natural language sentence due to inaccurate explicit information (e.g., information retrieved from the network). The evidence information (142) can be stored in the electronic device to prevent additional delays that occur while retrieving information from the network.

[0042] In one embodiment, the evidence information (142) may include information for evaluating the accuracy of natural language sentences. For example, the evidence information (142) may include document files (e.g., manual files), image files, and / or video files. The evidence information (142) may include a plurality of natural language sentences used to evaluate the probability of the probability measurement model (230). In one embodiment, the probability measurement model (230) may be trained to output a relatively high probability for natural language sentences that are similar to the plurality of natural language sentences included in the evidence information (142).

[0043] According to one embodiment, the electronic device can verify a natural language sentence (B1) generated from a language model (210) using probabilities corresponding to natural language sentences (B1, C1, C2, C3, ...) calculated by a probability measurement model (230). By normalizing the probabilities, the electronic device can verify the natural language sentence (B1). Verifying the natural language sentence (B1) may include an operation of determining whether to provide the natural language sentence (B1) as a response message to an external electronic device (e.g., the external electronic device (109) of FIG. 1). For example, in order to reduce a delay that occurs during generation and verification of the natural language sentence (B1), the natural language sentence (B1) generated from the language model (210) may be transmitted to the external electronic device during verification of the natural language sentence (B1) (or before verifying the natural language sentence (B1)). The electronic device may transmit a signal (e.g., a signal including binary values ​​of the natural language sentence (B1) encoded in Unicode) and / or a packet related to the natural language sentence (B1) to an external electronic device. One or more natural language sentences provided to the external electronic device prior to verification may be referred to as a candidate response message transmitted from the electronic device to the external electronic device.

[0044] Referring to the exemplary natural language sentences (B1, C1, C2, C3, ...) of FIG. 2, if the natural language sentence (B1) obtained by executing the language model (210) includes accurate information (e.g., Shakespeare, the author of The Merchant of Venice), the natural language sentence (B1) can be understood as more natural than the natural language sentences (C1, C2, C3, ...). In other words, the probabilities for each of the natural language sentences (B1, C1, C2, C3, ...) obtained by executing the probability measurement model (230) may have a value in which the probability corresponding to the natural language sentence (B1) is relatively higher than other probabilities. In other words, if the first natural language sentence obtained by executing the language model (210) contains inaccurate information, the probability for the first natural language sentence obtained by executing the probability measurement model (230) may have a lower value than the probabilities for one or more second natural language sentences obtained from the first natural language sentence by executing the corruption model (220) (e.g., the probabilities obtained by executing the probability measurement model (230). The distribution of the probabilities obtained by executing the probability measurement model (230) is exemplarily described with reference to FIGS. 4A to 4F.

[0045] By verifying the natural language sentence (B1), the electronic device can determine a response message to be provided to an external electronic device. For example, when a plurality of natural language sentences including the natural language sentence (B1) are obtained by executing a language model (210), the electronic device can obtain probabilities for each of the plurality of natural language sentences using a probability measurement model (230). Using the probabilities for each of the plurality of natural language sentences obtained using the probability measurement model (230), the electronic device can determine whether to provide each of the plurality of natural language sentences obtained using the language model (210) as a response message.

[0046] As described above, using a plurality of models (e.g., language model (210), corruption model (220), and / or probability measurement model (230)), the electronic device can verify one or more natural language sentences to be provided to the user. By verifying one or more natural language sentences, the electronic device can generate a response message to be provided to an external electronic device from a candidate response message generated from the language model (210). The electronic device can transmit one or more natural language sentences containing more accurate information than the candidate response message to the external electronic device as the response message. Since pre-trained models (e.g., language model (210), corruption model (220), and / or probability measurement model (230)) are used, the electronic device can verify natural language sentences (zero-shot) without an additional operation of training the model (e.g., fine tuning).

[0047] Hereinafter, with reference to FIG. 3, an exemplary operation of an electronic device verifying a natural language sentence according to one embodiment is described.

[0048] FIG. 3 is a flowchart illustrating the operation of an electronic device according to one embodiment. The operation of the electronic device described with reference to FIG. 3 may be performed by the electronic device (101) and / or the processor (110-1) of FIG. 1.

[0049] Referring to FIG. 3, in operation (310), according to one embodiment, a processor of an electronic device may receive a prompt sentence. The processor may receive a signal including at least one prompt sentence from an external electronic device (e.g., the external electronic device (109) of FIG. 1) via a communication circuit (e.g., the communication circuit (120-1) of FIG. 1). The signal may be generated or transmitted by the external electronic device to request that the external electronic device provide a response message corresponding to the at least one prompt sentence. For example, the processor may receive the prompt sentence from the external electronic device via the communication circuit.

[0050] Referring to FIG. 3, in operation (320), according to one embodiment, a processor of an electronic device may obtain a first natural language sentence related to a prompt sentence. The processor may execute a first model for natural language processing (e.g., language model (210) of FIG. 2) to generate a candidate response message including the first natural language sentence related to at least one prompt sentence of operation (310). For example, the processor may perform tokenization on portions of the prompt sentence (e.g., portions corresponding to one or more words and / or one or more morphemes) to obtain a plurality of tokens from the prompt sentence. The processor may execute a first model with the plurality of tokens to obtain a plurality of tokens related to the first natural language sentence. For example, the processor, having identified the plurality of tokens obtained using the first model, may obtain a first natural language sentence represented by the plurality of tokens.

[0051] The processor that obtains the first natural language sentence by performing operation (320) may transmit at least a portion of the first natural language sentence to an external electronic device while performing other operations of FIG. 3 (or before performing other operations). For example, the processor that obtains at least one word included in the first natural language sentence by executing the first model may transmit a signal representing the at least one word to the external electronic device. The first natural language sentence transmitted to the external electronic device before performing other operations of FIG. 3 may be referred to as a candidate response message, or may be included in a candidate response message.

[0052] Referring to FIG. 3, in operation (330), a processor of an electronic device according to an embodiment may generate one or more second natural language sentences in which a portion of a first natural language sentence is changed. The processor may execute a second model (e.g., a corruption model (220) of FIG. 2) to change a portion of the first natural language sentence, thereby obtaining one or more second natural language sentences. Having obtained a set of n tokens representing the first natural language sentence by performing operation (320), the processor may, in operation (330), mask each of the n tokens. Masking a token may include replacing the token with a designated token (or vector) indicating that the change by the second model is permitted.

[0053] By masking each of n tokens representing a first natural language sentence, the processor can obtain n sets of tokens (e.g., sets of tokens in which one of the n tokens is masked). Referring to FIG. 2, for a natural language sentence (B1) "The Merchant of Venice is a comedy by Shakespeare," which contains four words, the processor can obtain sets of four tokens in which one of the four words is masked by masking each of the tokens corresponding to the four words. For example, for the English sentence "The Merchant of Venice is a comedy by Shakespeare," the processor can obtain sets of nine tokens in which one of the nine words is masked by masking each of the tokens corresponding to nine words.

[0054] Using each of the sets of multiple tokens in which one token is masked, the processor can execute a second model. By executing the second model, the processor can replace a masked token with another token within the set of tokens, using the meaning represented by other tokens and / or the position of the masked token within the natural language sentence. For example, by executing the second model, the processor can obtain another set of tokens from the set of tokens in which one token included in the set is replaced with another token. The other set can include tokens representing words and / or morphemes included in the second natural language sentence of operation (330).

[0055] In one embodiment, the processor may repeatedly execute the second model k times using a particular set of tokens to obtain sets of k tokens from the particular set. For example, in an exemplary case where n sets of tokens are obtained by masking each of n tokens representing a first natural language sentence, the processor may perform operation (330) to obtain or generate n Х k sets of tokens. The k sets of tokens may each represent the second natural language sentences of operation (330). The number of times the second model is repeatedly executed (e.g., k times) may be heuristically determined to verify the first natural language sentence.

[0056] Referring to FIG. 3, in operation (340), according to one embodiment, a processor of an electronic device may determine scores of a first natural language sentence and one or more second natural language sentences. When n Х k second natural language sentences are obtained using a first natural language sentence including n tokens, the processor may calculate or determine scores of all of the first natural language sentences and the second natural language sentences (e.g., n Х (k + 1) scores). For example, the processor may determine scores for each of the first natural language sentence of operation (320) and the one or more second natural language sentences of operation (330) using a third model (e.g., the probability measurement model (230) of FIG. 2) for evaluating natural language sentences using evidence information (e.g., the evidence information (142) of FIG. 1 and / or FIG. 2). Each of the scores may include a probability obtained by executing the third model using a natural language sentence corresponding to the score. For example, using the third model, the processor can determine scores (e.g., probabilities) indicating the extent to which the evidence information matches the first natural language sentence and one or more second natural language sentences.

[0057] Referring to FIG. 3 , in operation (350), according to one embodiment, a processor of an electronic device may determine a first natural language sentence as a response message corresponding to a prompt sentence using the determined scores. The processor may perform operation (340) and use the determined scores to determine a candidate response message including the first natural language sentence as a response message to be transmitted to the external electronic device that transmitted the prompt sentence of operation (310). For example, the processor that determined the scores of operation (340) may determine whether a word matches evidence information using scores corresponding to one or more second natural language sentences obtained by modifying a portion of the first natural language sentence corresponding to one word of the first natural language sentence of operation (320). For example, using a probability that the scores for one or more second natural language sentences of operation (330) are higher than the score for the first natural language sentence, the processor may determine the candidate response message including the first natural language sentence as the response message of operation (350). An exemplary operation in which the processor verifies a first natural language sentence using the distribution of scores of the operation (340) is described with reference to FIGS. 4a to 4f.

[0058] Based on the determination of the operation (350) related to the response message, the processor may transmit the response message to the external electronic device. In one embodiment where the first natural language sentence is transmitted to the external electronic device prior to operation (350), the processor may transmit a signal to remove, correct, or change the first natural language sentence displayed by the external electronic device. If the first natural language sentence is transmitted to the external electronic device prior to operation (350) and the first natural language sentence is not determined to be the response message by operation (350), the processor may transmit a signal to change the first natural language sentence transmitted to the external electronic device (or a signal indicating that the first natural language sentence is not the response message). The operation of the processor transmitting a signal to the external electronic device based on the determination of operation (350) is described with reference to FIGS. 5A to 5F.

[0059] Although the operation of verifying a single first natural language sentence has been described, the embodiment is not limited thereto. In operation (320), if the processor acquires a plurality of natural language sentences related to the prompt sentence, each of the acquired plurality of natural language sentences can be understood as the first natural language sentence of operation (320), and the processor can perform the operations of FIG. 3 (e.g., operations (330, 340, 350)) for each of the plurality of natural language sentences.

[0060] In one embodiment, the operations of the electronic device performed to verify the first natural language sentence are not limited to the embodiment of FIG. 3. For example, the operations of the electronic device may be summarized by the pseudo-algorithm (or pseudo-code) of Table 1 and / or Table 2.

[0061]

[0062]

[0063] Referring to Tables 1 and 2, evidence E may represent natural language sentences included in the evidence information. Claim C may represent the first natural language sentence of the action (320). The large-language model (LLM) of Table 1 may include one or more models (e.g., the corruption model (220) of FIG. 2 and / or the probability measurement model (230)) executed to verify the first natural language sentence.

[0064] Hereinafter, exemplary operations of an electronic device for verifying natural language sentences are described with reference to FIGS. 4a to 4f.

[0065] FIGS. 4A, 4B, 4C, 4D, 4E, and 4F illustrate exemplary operations of an electronic device for verifying (or evaluating) natural language sentences obtained by executing a language model. The operations of the electronic device described with reference to FIGS. 4A to 4F may be performed by the electronic device (101) and / or the processor (110-1) of FIG. 1. The operations of the electronic device described with reference to FIGS. 4A to 4F may be related to at least one of the operations of FIG. 3 (e.g., operations (340, 350)).

[0066] Referring to FIG. 4A, the electronic device (101) can evaluate a natural language sentence (412) (e.g., "Douglas M. North is the head of a clown school located in Albany, New York") using English evidence information (411). The natural language sentence (412) can be obtained based on the execution of the language model (210) of FIG. 2. For example, the electronic device (101) can determine the probabilities that each of the words (or tokens) included in the natural language sentence (412) matches the evidence information (411). In the exemplary case of FIG. 4A, if the probability that a token (e.g., a token corresponding to "a clown school") of the natural language sentence (412) matches the evidence information (411) is relatively low (e.g., less than a specified threshold probability), the electronic device (101) can determine that the token is generated by a hallucination of the language model.

[0067] Referring to FIG. 4A, when a signal for displaying a natural language sentence (412) is transmitted to an external electronic device, the external electronic device can display a natural language sentence (413) based on the signal. When it is determined that a token (e.g., a token corresponding to "a clown school") of the natural language sentence (412) is generated by a hallucination, the electronic device (101) can transmit another signal to the external electronic device to visually highlight a portion (414) corresponding to the token within the natural language sentence (413). The external electronic device, upon receiving the other signal, can visually highlight the portion (414) within its display. For example, the external electronic device can change the font color of the portion (414) and / or display a shape with a specified transparency and a specified color as an overlay. For example, the external electronic device can display a numerical value representing a probability that the token corresponding to the portion (414) matches the evidence information (411) on the display. The above numerical value may be included in the other signal transmitted from the electronic device (101).

[0068] Referring to FIG. 4B, the electronic device (101) can evaluate a natural language sentence (422) (e.g., "NCSoft plans to release a new mobile TL. The TL will be released domestically in December and will target the global market next year.") using evidence information (421) of Korean. An embodiment of obtaining a natural language sentence (422) by executing a language model (210) of FIG. 2 is illustrated, but the embodiment is not limited thereto. According to one embodiment, the electronic device (101) can execute a model (e.g., a corruption model (220) and / or a probability measurement model (230) of FIG. 2) for evaluating tokens (or words) of a natural language sentence (422) using evidence information (421). By executing the above model, the electronic device (101) can identify tokens (e.g., tokens corresponding to “new mobile product”) that do not match the evidence information (421) within the natural language sentence (422).

[0069] The electronic device (101) that has identified a token that does not match the evidence information (421) may generate or output information indicating at least a portion of the natural language sentence (422) that does not match the evidence information (421) as a result of evaluating the accuracy of the natural language sentence (422). Referring to FIG. 4B, the electronic device (101) that displays the natural language sentence (423) based on the information may visually highlight a portion (424) of the natural language sentence (423) corresponding to the token indicated by the information. The operation of visually highlighting the portion (424) may be performed similarly to the operation described above with reference to FIG. 4A. The electronic device (101) may display, together with the natural language sentence (423), a numerical value indicating a probability that the token corresponding to the portion (424) matches the evidence information (421).

[0070] Referring to FIG. 4C, a table (433) including a plurality of natural language sentences generated from a natural language sentence (432) by an electronic device (101) that is executed using evidence information (431) is illustrated. The electronic device (101) may execute the corruption model (220) of FIG. 2 to replace words included in the natural language sentence (432) with other words, thereby generating perturbed sentences included in the table (433). Referring to the table (433) of FIG. 4C, a specific token (e.g., a token corresponding to "a clown school") of the natural language sentence (432) may be repeatedly replaced multiple times to generate perturbed sentences. According to one embodiment, the electronic device (101) may perform the operation described with reference to FIG. 3 to determine scores for each of the perturbed sentences. Referring to the table (433) of FIG. 4C, the scores may be included in the "log prob" column. The electronic device (101) can use the distribution of the above scores to determine whether a natural language sentence (432) matches evidence information (431).

[0071] Referring to FIG. 4d, a table (443) including a plurality of natural language sentences generated when an electronic device (101) running using evidence information (441) evaluates a natural language sentence (442) is illustrated. The natural language sentences included in the table (443) may be generated by executing the corruption model (220) of FIG. 2. Referring to the table (443) of FIG. 4d, the electronic device (101) may repeatedly replace a specific token (e.g., a token corresponding to “new mobile product”) of the natural language sentence (442) with another token to generate four perturbed sentences. The electronic device (101) may execute the probability measurement model (230) of FIG. 2 to calculate probabilities that each of the natural language sentence (442) and the perturbed sentences matches the evidence information (441). The scores described in the “log prob” column of the table (443) of FIG. 4d may be calculated. Using the distribution of the above scores, the electronic device (101) can evaluate the degree of correlation between a natural language sentence (442) and evidence information (431). Hereinafter, exemplary operations of the electronic device (101) for analyzing the distribution of scores are described with reference to FIGS. 4E to 4F.

[0072] Referring to the first column of the table (450) of FIG. 4e, exemplary natural language sentences obtained by an electronic device executing the language model (210) and / or the corruption model (220) of FIG. 2 are illustrated. In an exemplary state where a prompt sentence (e.g., “I want to know the principal of a school located in Albany”) obtained from an external electronic device (e.g., the external electronic device (109) of FIG. 1) is received, the electronic device may execute the language model using the prompt sentence to obtain the first natural language sentence (e.g., “Douglas M. North is the head of a clown school located in Albany, New York”) described in the second row of the table (450).

[0073] An electronic device can acquire one or more second natural language sentences by executing a corruption model using the first natural language sentence. Referring to table (450) of FIG. 4e, an electronic device that executes a corruption model based on masking of a specific word (e.g., "clown") in the first natural language sentence can acquire four second natural language sentences in rows 3 to 6. In an exemplary case where the electronic device uses a corruption model to replace one token of a first natural language sentence (e.g., a token corresponding to "clown") with four different tokens, the electronic device can obtain 44 (= 4 Х 11) second natural language sentences from the corruption model by masking each of the 11 tokens (e.g., tokens corresponding to "Douglas M. North", "is", "the", "head", "of", "a", "clown", "school", "located", "in", "Albany, New York", respectively).

[0074] An electronic device may estimate a probability for each of a first natural language sentence and a plurality of second natural language sentences. The probability may represent a probability that a natural language sentence matches evidence information and / or a similarity, obtained by executing a probability measurement model. The probability may be a log probability. In an exemplary case where four second natural language sentences are obtained by masking a specific word (e.g., “clown”), the electronic device may perform normalization of five probabilities corresponding to each of the second natural language sentences and the first natural language sentences. Referring to table (450) of FIG. 4E, probabilities (hereinafter, normalized probabilities) obtained by normalizing the probabilities corresponding to each of the five exemplary natural language sentences are described in the second column of table (450).

[0075] Referring to FIG. 4E, a normal distribution curve (451) is illustrated. According to one embodiment, the electronic device may use the position (p1) of the normalized probability (-1.531 in the exemplary case of Table (450)) of the first natural language sentence on the normal distribution curve (420) to determine whether the first natural language sentence matches the evidence information and / or whether to provide the first natural language sentence as a response message. For example, the electronic device may obtain a numerical value x used to determine whether the first natural language sentence matches the evidence information using Equation 1.

[0076]

[0077] Referring to mathematical expression 1, z_orig can correspond to the normalized probability of the first natural language sentence (-1.531 in the exemplary table (410)). z in mathematical expression 1 can correspond to any one of the normalized probabilities of the second natural language sentences obtained using the corruption model (-1.315, -1.671, -1.445, -1.682 in the exemplary table (410)). In mathematical expression 1, , the normalization probabilities of the second natural language sentences may be greater than the normalization probabilities of the first natural language sentences. For example, in mathematical expression 1, , on the normal distribution curve (451), may be the area of ​​the region larger than the position (p1) of the normalized probability corresponding to the first natural language sentence. In mathematical expression 1, The coefficient 2 of Equation 1 is can be empirically determined to amplify the numerical value of .

[0078] As described above with reference to FIGS. 1 to 3, each of the probabilities described in the second column of the table (450) is a probability obtained by executing a model that utilizes evidence information (e.g., the probability measurement model (230) of FIG. 2), and a natural language sentence may be a natural probability. If a first natural language sentence matches the evidence information, that is, if it is a natural language sentence that conforms to the facts, the normalized probability corresponding to the first natural language sentence will have a higher value than the normalized probabilities corresponding to second natural language sentences in which at least one word of the first natural language sentence has been replaced, because the more the first natural language sentence is damaged, the less it conforms to the facts. For example, on a normal distribution curve (451), the position (p1) of the normalized probability of the first natural language sentence will be located in the direction of the + x-axis from the center (c) of the normal distribution curve (451). The higher the normalized probability of the first natural language sentence, the farther the position (p1) will be from the center (c) along the + x-axis. For example, the higher the normalization probability of the first natural language sentence, the higher the normalization probability of Equation 1. As is increased, x in Equation 1 can be increased.

[0079] For example, if the first natural language sentence does not match the evidence information, that is, if it is a natural language sentence that does not correspond to the facts, the normalized probability corresponding to the first natural language sentence may have a value similar to or smaller than the normalized probabilities corresponding to the second natural language sentences in which at least one word of the first natural language sentence is replaced, because the first natural language sentence may not correspond to the facts even if it is damaged. For example, on the normal distribution curve (420), the position (p1) of the normalized probability of the first natural language sentence will be close to the center (c) of the normal distribution curve (420) or will be located in the direction of the - x-axis from the center (c). The lower the normalized probability of the first natural language sentence, the more the position (p1) will move along the direction of the - x-axis. For example, the lower the normalized probability of the first natural language sentence, the more the position (p1) of Equation 1 Since is reduced, x in equation 1 can be reduced.

[0080] The numerical value x in mathematical expression 1 is a numerical value used to determine whether the first natural language sentence matches the evidence information, and may be referred to as a score for the first natural language sentence (or a specific word of the first natural language sentence). In an exemplary case where the word "clown" of the first natural language sentence is masked to obtain second natural language sentences, the electronic device may determine or calculate a score corresponding to the word "clown" using the normalized probabilities described in the second column of table (450).

[0081] For example, if the probability that the word "clown" matches the evidence information is determined to be 80% by mathematical expression 1, the electronic device can compare the probability of 80% with a specified threshold to determine whether the first natural language sentence including the word "clown" matches the evidence information. In an exemplary case where 11 tokens can be obtained from the first natural language sentence, the electronic device can obtain scores corresponding to each of the 11 tokens. If all of the obtained scores exceed the specified threshold, the electronic device can provide the first natural language sentence as a response message to an external electronic device (e.g., the external electronic device (109) of FIG. 1).

[0082] Referring to FIG. 4F, an exemplary case of a wearable device that evaluates an exemplary first natural language sentence (e.g., "NCSoft is planning to release a new mobile TL") obtained using a language model is illustrated. As shown in Table (460), the wearable device can obtain a plurality of second natural language sentences by replacing or changing at least one word (or at least one token) of the first natural language sentence. Referring to Table (460) of FIG. 4F, the plurality of second natural language sentences are illustrated together with the first natural language sentence. The electronic device can execute a probability measurement model to calculate probabilities that each of the first natural language sentence and the plurality of second natural language sentences matches evidence information. For example, the electronic device can determine similarities of natural language sentences (e.g., the first natural language sentence and / or the second natural language sentences) to evidence information. Referring to FIG. 4f, within table (460), natural language sentences and the probabilities of each of the natural language sentences are exemplarily illustrated.

[0083] According to one embodiment, the electronic device can determine whether the first natural language sentence matches the evidence information by using the distribution of the probabilities on the normal distribution curve (461) of the probabilities. As described above, if the first natural language sentence matches the evidence information, the second natural language sentences generated by at least partially modifying the first natural language sentence may include information that does not match the evidence information. For example, within the normal distribution curve (461), the probability for the first natural language sentence may be relatively greater than the probabilities for the second natural language sentences. On the other hand, if the first natural language sentence does not match the evidence information, the second natural language sentences generated by at least partially modifying the first natural language sentence may not match the evidence information, or may match the evidence information. For example, within the normal distribution curve (461), the probability for the first natural language sentence may be substantially similar to the probabilities for the second natural language sentences. Using the probability distribution illustrated above, the electronic device can detect hallucinations included in the first natural language sentence and caused by the language model.

[0084] Referring to FIG. 4F, the electronic device may calculate or determine probabilities that each of the words and / or tokens of the first natural language sentence matches with the evidence information. For example, the electronic device may calculate or determine a probability (e.g., 80%) that a specific word (e.g., "clown") of the first natural language sentence matches with the evidence information. By comparing the probability with a designated probability, the electronic device may determine whether the specific word matches with the evidence information. If a word that does not match with the evidence information is detected from the first natural language sentence, the electronic device may execute a language model to discard the first natural language sentence or generate a new natural language sentence to replace the first natural language sentence.

[0085] Hereinafter, with reference to FIGS. 5A to 5F, an exemplary UI (user interface) provided by an electronic device performing the operations of FIGS. 1 and 4A to 4F is described.

[0086] FIGS. 5A, 5B, 5C, 5D, 5E, and 5F illustrate exemplary operations of an electronic device (101) in relation to a prompt sentence received from an external electronic device (109). The electronic device (101) and the external electronic device (109) of FIGS. 5A to 5F may correspond to the electronic device (101) and the external electronic device (109) of FIG. 1 , respectively. The electronic device (101) and the external electronic device (109) of FIGS. 5A to 5F may perform the operations of the electronic device (101) and the external electronic device (109) described with reference to FIGS. 1 and 4A to 4F , respectively.

[0087] Referring to FIGS. 5A to 5F, exemplary states of an electronic device (101) and an external electronic device (109) connected to each other to provide a service for conversational interaction are illustrated. When the client agent application (151) of FIG. 1 is executed, the external electronic device (109) can display the screen (500) of FIGS. 5A to 5F. The client agent application (151) may include a software application executable by the external electronic device (109). The client agent application (151) may include a plug-in executed in a web browser application for viewing web pages. The embodiment is not limited thereto, and independently of the client agent application (151), the external electronic device (109) may display a web page provided from the electronic device (101) through a network to display the screen (500). The screen (500) may occupy at least a portion of a display controlled by an external electronic device (109) (or a processor (110-2) of the external electronic device (109) of FIG. 1).

[0088] Within the exemplary screen (500) of FIG. 5A, the external electronic device (109) may display a text box (511) for receiving text, and / or a button (512) for receiving an input for transmitting the received text to the electronic device (101) using the text box (511). While one embodiment of displaying the text box (511) and / or the button (512) at the bottom of the screen (500) is illustrated, the embodiment is not limited thereto. With the text box (511) in focus, the external electronic device (109) may, in response to a text input received via a user's voice input (e.g., a voice input based on an audio signal received via the microphone (126) of FIG. 1) and / or a keyboard (e.g., a keyboard device electrically connected to the external electronic device (109) and / or a software keyboard occupying at least a portion of the display), display one or more characters represented by the voice input and / or the text input within the text box (511). Referring to FIG. 5A, an exemplary state of an external electronic device (109) is illustrated that displays at least one natural language sentence (e.g., “Tell me about The Merchant of Venice”) within a text box (511) in response to a user’s voice input and / or text input.

[0089] In the exemplary state of FIG. 5a, in response to an input indicating a selection of a button (512), the external electronic device (109) may transmit a first signal indicating one or more characters included in a text box (511) to the electronic device (101). The first signal may include a request to execute a function of the electronic device (101) associated with the one or more characters (e.g., a function of identifying a prompt sentence included in the one or more characters using natural language processing). Referring to FIG. 5b, a screen (500) displayed by the external electronic device (109) after receiving an input indicating a selection of the button (512) is illustrated. Referring to the exemplary state of FIG. 5b, the external electronic device (109) may remove one or more characters that were displayed in the text box (511) prior to the state of FIG. 5b, or may reset the text box (511). An external electronic device (109) may display a visual object (521) representing one or more characters corresponding to the input within the screen (500). The visual object (521) may be referred to as a bubble. An exemplary screen (500) is illustrated in which the visual object (521) is positioned along the right edge of the screen (500), but the location of the visual object (521) is not limited thereto.

[0090] In one embodiment, an electronic device (101) that receives a first signal associated with a prompt sentence (e.g., an exemplary natural language sentence such as “Tell me about the Merchant of Venice” in FIG. 5A) may execute a first model for natural language processing (e.g., a language model (210) in FIG. 2) using the prompt sentence. The electronic device (101) may transmit one or more natural language sentences obtained from the first model and associated with the prompt sentence to an external electronic device (109). For example, the electronic device (101) may transmit a second signal associated with the one or more natural language sentences to the external electronic device (109). For example, the one or more natural language sentences may be transmitted to the external electronic device (109) as a candidate response message before completing verification of the one or more natural language sentences.

[0091] Referring to FIG. 5c, the external electronic device (109) that received the second signal may display a visual object (531) including one or more natural language sentences (e.g., “The Merchant of Venice is a Shakespearean comedy”) indicated by the second signal within the screen (500). The visual object (531) may be referred to as a bubble. In order to distinguish the source of the visual objects (521, 531) (e.g., the user of the external electronic device (109) and / or the electronic device (101)), the visual object (531) may be displayed on the left edge of the screen (500), which is different from the right edge of the screen, which is the reference for the location of the visual object (521). The location at which the visual object (531) is displayed is not limited thereto.

[0092] In one embodiment, the external electronic device (109) may display an indicator (532) in the form of a cursor within the visual object (531) until it receives a designated signal from the electronic device (101) indicating the end of transmission of a response message for a prompt sentence. The position of the indicator (532) within the visual object (531) may indicate the position of a natural language sentence to be received from the electronic device (101). When the external electronic device (109) receives a signal corresponding to at least one token from the electronic device (101), the external electronic device (109) may add the at least one word and one or more words corresponding to the at least one token to the visual object (531).

[0093] Referring to FIG. 5d, an exemplary state of an external electronic device (109) that receives a signal related to a second natural language sentence (e.g., “Antonio and Romeo are the main characters”) following the first natural language sentence is illustrated in the state of FIG. 5c, which displays a visual object (531) including a first natural language sentence (e.g., “The Merchant of Venice is a Shakespearean comedy”). The second natural language sentence can be acquired by the electronic device (101) executing the first model. The electronic device (101) that identifies the second natural language sentence can transmit a third signal related to the second natural language sentence to the external electronic device (109). The external electronic device (109) that receives the third signal can further display the second natural language sentence indicated by the third signal within the visual object (531) of the screen (500). In the exemplary state of FIG. 5c where the second natural language sentence is displayed after the first natural language sentence, the external electronic device (109) can display an indicator (532) after the second natural language sentence.

[0094] According to one embodiment, the electronic device (101) may determine that the candidate response message is a response message by using scores for natural language sentences (e.g., the first natural language sentence and / or the second natural language sentence) included in the candidate response message while transmitting at least a portion (e.g., the first natural language sentence and / or the second natural language sentence) of the candidate response message through a communication circuit. The electronic device (101) may execute one or more second models (e.g., the corruption model (220) and / or the probability measurement model (230) of FIG. 2) for evaluating the natural language sentence using evidence information to determine whether to change the natural language sentence included in the candidate response message (e.g., the natural language sentence displayed in the visual object (531) of FIG. 5C). The electronic device (101) may verify one or more natural language sentences included in the candidate response message to perform the operation described with reference to FIG. 1 and FIG. 4A to FIG. 4F.

[0095] In one embodiment, when the verification of one or more natural language sentences included in the candidate response message is completed, the electronic device (101) may determine the one or more natural language sentences as a response message to be provided to the external electronic device (109). Based on determining the one or more natural language sentences as the response message, the electronic device (101) may transmit a designated signal indicating the determination of the response message to the external electronic device (109). The external electronic device (109) that has received the designated signal may stop displaying the indicator (532) within the visual object (531). The external electronic device (109) that has stopped displaying the indicator (532) may display one or more visual objects (e.g., a like button) for receiving feedback related to the response message expressed by the visual object (531) within the visual object (531) (or on a portion of the screen (500) adjacent to the visual object (531).

[0096] In one embodiment, when one or more natural language sentences included in a candidate response message are transmitted to an external electronic device (109), if the one or more natural language sentences are not determined as a response message, the electronic device (101) may execute a function of retrieving the one or more natural language sentences. Referring to FIG. 5C, it is assumed that a first natural language sentence included in a visual object (531) includes factual information that matches evidence information, and a second natural language sentence includes factual information that does not match evidence information (e.g., Romeo is a character in The Merchant of Venice). The electronic device (101) that performs verification on a candidate response message including the first natural language sentence and the second natural language sentence may determine the first natural language sentence as a response message and exclude the second natural language sentence from the response message.

[0097] Based on excluding the second natural language sentence from the response message, the electronic device (101) may transmit a fourth signal to the external electronic device (109), which causes removal of at least a portion of the second natural language sentence displayed on the display of the external electronic device (109). As described above with reference to FIGS. 1 and 4A to 4F , the electronic device (101) may verify the second natural language sentence based on tokens (or words, and / or morphemes). For example, using the probabilities that each of the tokens included in the second natural language sentence matches the evidence information, the electronic device (101) may determine whether the words (or morphemes) expressed by each of the tokens match the evidence information. In the exemplary second natural language sentence of FIG. 5D , the electronic device (101) may identify, among the tokens of the second natural language sentence, a token that does not match the evidence information (e.g., a token corresponding to the word "Romeo"). The fourth signal transmitted by the electronic device (101) to the external electronic device (109) may include a command to remove the entire second natural language sentence and / or the word “Romeo” corresponding to the identified token.

[0098] Referring to FIG. 5E, in an exemplary state in which the fourth signal is received, the external electronic device (109) may remove at least a portion (e.g., the word “Romeo”) of the second natural language sentence corresponding to the fourth signal within the visual object (531). The external electronic device (109) that has removed at least a portion may display an indicator (532) within the visual object (531) at a position where the at least portion was displayed. In an exemplary state in which the indicator (532) is displayed at a position from which at least a portion is removed, the external electronic device (109) may, in response to a signal additionally received from the electronic device (101), insert one or more words indicated by the received signal into the position in which the indicator (532) is displayed.

[0099] The electronic device (101) that transmitted the fourth signal may perform an operation to replace or change at least a portion of the second natural language sentence that is determined not to match the evidence information. For example, the electronic device (101) may execute the first model using a prompt sentence for replacing at least a portion of the second natural language sentence. The electronic device (101) that obtains the third natural language sentence (or at least one word) obtained by executing the first model may transmit a fifth signal related to the third natural language sentence to an external electronic device (109). For example, the electronic device (101) may transmit a fifth signal indicating that the second natural language sentence is changed to the third natural language sentence to the external electronic device (109). The fourth signal and / or the fifth signal may be referred to as signals for changing the second natural language sentence that does not match the evidence information.

[0100] Referring to FIG. 5F, the external electronic device (109) that receives the fifth signal may display a third natural language sentence (e.g., “Antonio and Shylock are the main characters”) indicated by the fifth signal. For example, using the fifth signal, the electronic device (101) may cause the external electronic device (109) to change the second natural language sentence into a third natural language sentence. For example, the external electronic device (109) may replace the second natural language sentence with the third natural language sentence within the visual object (531), or display a portion of the third natural language sentence that is different from the second natural language sentence (e.g., the word “Shylock”). In response to the fifth signal, the external electronic device (109) that displays the third natural language sentence may move the position of the indicator (532) within the visual object (531) to the end of one or more natural language sentences included in the visual object (531).

[0101] In the state of FIG. 5f, when receiving a signal indicating an additional natural language sentence from the electronic device (101), the external electronic device (109) may further display the natural language sentence at the location of the visual object (531) where the indicator (532) is displayed. As the natural language sentence is added within the visual object (531), the external electronic device (109) may increase the size (e.g., width, height, and / or area) of the visual object (531) within the screen (500). When receiving a signal from the electronic device (101) indicating that the generation of the natural language sentence is terminated or that the transmission of the response message is completed, the external electronic device (109) may stop displaying the indicator (532) within the visual object (531).

[0102] Although the operation of the electronic device (101) to determine a response message by modifying one or more natural language sentences included in the candidate response message after transmitting at least a portion of the candidate response message has been described, the embodiment is not limited thereto. For example, the electronic device (101) may complete the operation of determining a response message from the candidate response message without transmitting one or more natural language sentences included in the candidate response message. The electronic device (101) that has determined the response message may transmit a signal to the external electronic device (109) indicating all of the natural language sentences included in the response message. In the above example, the electronic device (101) may, prior to transmitting the signal, transmit another signal to the external electronic device (109) to display a visual object (e.g., a progress bar) indicating the generation and / or determination of the response message.

[0103] As described above, according to one embodiment, the electronic device (101) can provide an interactive user experience based on at least one prompt sentence. When generating one or more natural language sentences related to the at least one prompt sentence, the electronic device (101) can verify the one or more natural language sentences. Based on the verification, the electronic device (101) can replace or change the one or more natural language sentences. When the electronic device (101) transmits the one or more natural language sentences to an external electronic device (109), the electronic device (101) can transmit a signal to the external electronic device (109) to at least partially change the one or more natural language sentences. Based on the verification, hallucination occurring in the one or more natural language sentences can be prevented or reduced.

[0104] In one embodiment, a method for reducing hallucination caused by the execution of a language model may be required. As described above, in one embodiment, a non-transitory computer-readable storage medium storing instructions may be provided. The instructions, when executed by an electronic device including a communication circuit, may cause the electronic device to receive a prompt sentence from an external electronic device via the communication circuit. The instructions, when executed by the electronic device, may cause the electronic device to execute a first model for natural language processing to generate a first natural language sentence related to the prompt sentence. The instructions, when executed by the electronic device, may cause the electronic device to modify a portion of the first natural language sentence to obtain a second natural language sentence. The instructions, when executed by the electronic device, may cause the electronic device to execute a second model linked to evidence information to determine scores representing the accuracy of each of the first natural language sentence and the second natural language sentence. The evidence information may be used as a criterion for evaluating the accuracy of the natural language sentence by the second model. The instructions, when executed by the electronic device, may cause the electronic device to determine, using the scores, the first natural language sentence as a response message to be transmitted to the external electronic device. According to one embodiment, the electronic device may prevent or reduce hallucination caused by the language model.

[0105] For example, the instructions, when executed by the electronic device, may cause the electronic device to determine whether a word matches the evidence information using a score corresponding to the second natural language sentence obtained by changing the portion corresponding to a word of the first natural language sentence.

[0106] For example, the instructions, when executed by the electronic device, may cause the electronic device to determine, using the second model, scores indicating the degree to which the evidence information and the first natural language sentence and the second natural language sentence match, respectively.

[0107] For example, the instructions, when executed by the electronic device, may cause the electronic device to determine the first natural language sentence as the response message using the scores while transmitting at least a portion of the first natural language sentence through the communication circuit.

[0108] For example, the instructions, when executed by the electronic device, may cause the electronic device to determine scores for a plurality of third natural language sentences that include the second natural language sentence in which the word is replaced with one of different words. The instructions, when executed by the electronic device, may cause the electronic device to determine the candidate response message as the response message using a probability that the scores for the plurality of third natural language sentences are higher than the scores for the first natural language sentence.

[0109] For example, the instructions, when executed by the electronic device, may cause the electronic device to generate the plurality of third natural language sentences by replacing the word in the first natural language sentence a plurality of times.

[0110] For example, the instructions, when executed by the electronic device, may cause the electronic device to change the first natural language sentence into a fourth natural language sentence generated using the first model based on determining that the first natural language sentence is different from the response message.

[0111] For example, the instructions, when executed by the electronic device, may cause the electronic device to transmit a signal to the external electronic device indicating that the first natural language sentence is changed to the fourth natural language sentence based on determining that the candidate response message is different from the response message after transmitting the first natural language sentence through the communication circuit.

[0112] For example, the instructions, when executed by the electronic device, may cause the electronic device to transmit a signal to the external electronic device, causing the first natural language sentence displayed on the display of the external electronic device to be removed, and causing the fourth natural language sentence to be displayed at a location within the display where the first natural language sentence was removed.

[0113] For example, the instructions, when executed by the electronic device, may cause the electronic device to determine, based on a determination that the first natural language sentence is different from the response message after transmitting the first natural language sentence through the communication circuit, using word-by-word accuracies of the first natural language sentence represented by the scores, at least one word of the first natural language sentence as a word associated with a hallucination generated by the first model. The instructions, when executed by the electronic device, may cause the electronic device to transmit, to the external electronic device, a signal for visually emphasizing the at least one word within the first natural language sentence displayed on the external electronic device.

[0114] According to one embodiment, an electronic device as described above may include a communication circuit, a memory storing instructions, and a processor. The instructions, when executed by the processor, may cause the electronic device to receive a prompt sentence from an external electronic device via the communication circuit. The instructions, when executed by the processor, may cause the electronic device to execute a first model for natural language processing to generate a first natural language sentence related to the prompt sentence. The instructions, when executed by the processor, may cause the electronic device to modify a portion of the first natural language sentence to obtain a second natural language sentence. The instructions, when executed by the processor, may cause the electronic device to execute a second model linked to evidence information to determine scores representing the accuracy of each of the first natural language sentence and the second natural language sentence. The above evidence information may be used as a criterion for evaluating the accuracy of a natural language sentence by the second model. The instructions, when executed by the processor, may cause the electronic device to use the scores to determine the first natural language sentence as a response message to be transmitted to the external electronic device.

[0115] For example, the instructions, when executed by the processor, may cause the electronic device to determine whether a word matches the evidence information using a score corresponding to the second natural language sentence obtained by changing the portion corresponding to a word of the first natural language sentence.

[0116] For example, the instructions, when executed by the processor, may cause the electronic device to determine, using the second model, scores indicating the degree to which the evidence information and the first natural language sentence and the second natural language sentence each match.

[0117] For example, the instructions, when executed by the processor, may cause the electronic device to determine the first natural language sentence as the response message using the scores while transmitting at least a portion of the first natural language sentence through the communication circuit.

[0118] For example, the instructions, when executed by the processor, may cause the electronic device to determine scores for a plurality of third natural language sentences that include the second natural language sentence in which the word is replaced with one of different words. The instructions, when executed by the processor, may cause the electronic device to determine the candidate response message as the response message using a probability that the scores for the plurality of third natural language sentences are higher than the scores for the first natural language sentence.

[0119] For example, the instructions, when executed by the electronic device, may cause the electronic device to generate the plurality of third natural language sentences by replacing the word in the first natural language sentence a plurality of times.

[0120] For example, the instructions, when executed by the processor, may cause the electronic device to change the first natural language sentence into a fourth natural language sentence generated using the first model based on determining that the first natural language sentence is different from the response message.

[0121] For example, the instructions, when executed by the processor, may cause the electronic device to transmit a signal to the external electronic device indicating that the first natural language sentence is changed to the fourth natural language sentence based on a determination that the first natural language sentence is different from the response message after transmitting the first natural language sentence through the communication circuit.

[0122] For example, the instructions, when executed by the processor, may cause the electronic device to transmit a signal to the external electronic device, causing the first natural language sentence displayed on a display of the external electronic device to be removed, and causing the fourth natural language sentence to be displayed at a location within the display where the first natural language sentence was removed.

[0123] For example, the instructions, when executed by the processor, may cause the electronic device to determine, based on a determination that the first natural language sentence is different from the response message after transmitting the first natural language sentence through the communication circuit, using word-by-word accuracies of the first natural language sentence represented by the scores, at least one word of the first natural language sentence as a word associated with a hallucination generated by the first model. The instructions, when executed by the processor, may cause the electronic device to transmit, to the external electronic device, a signal for visually emphasizing the at least one word within the first natural language sentence displayed on the external electronic device.

[0124] As described above, in one embodiment, a method of an electronic device including a communication circuit may be provided. The method may include an operation of receiving a prompt sentence from an external electronic device through the communication circuit. The method may include an operation of transmitting, to the external electronic device, a first signal related to a natural language sentence related to the prompt sentence, the first signal obtained from a first model for natural language processing. The method may include an operation of determining whether to modify the natural language sentence using a second model for evaluating the natural language sentence using evidence information. The method may include an operation of transmitting, to the external electronic device, a second signal for modifying the natural language sentence based on the determination to modify the natural language sentence.

[0125] For example, the act of transmitting the second signal may include an act of transmitting the second signal that causes removal of the natural language sentence displayed on the display of the external electronic device.

[0126] For example, the operation of transmitting the second signal may include an operation of obtaining another natural language sentence different from the natural language sentence using the first model based on a decision to change the natural language sentence. The operation of transmitting the second signal may include an operation of transmitting a third signal related to the other natural language sentence to the external electronic device.

[0127] For example, the determining action may include an action of obtaining a score indicating the degree to which the evidence information and the natural language sentence match using the second model.

[0128] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0129] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0130] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0131] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0132] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A non-transitory computer-readable storage medium storing instructions, wherein the instructions, when executed by an electronic device including a communication circuit, cause the electronic device to: Receive a prompt sentence from an external electronic device through the above communication circuit; Executing a first model for natural language processing to generate a first natural language sentence related to the prompt sentence; By changing a part of the first natural language sentence, a second natural language sentence is obtained; By executing a second model linked to evidence information, scores representing the accuracy of each of the first natural language sentence and the second natural language sentence are determined, and the evidence information is used as a criterion for evaluating the accuracy of the natural language sentence by the second model; and Using the above scores, causing the first natural language sentence to be determined as a response message to be transmitted to the external electronic device. Non-transitory computer-readable storage medium.

2. In claim 1, the instructions, when executed by the electronic device, cause the electronic device to: By using a score corresponding to the second natural language sentence obtained by changing the part corresponding to one word of the first natural language sentence, it is determined whether the word matches the evidence information. Non-transitory computer-readable storage medium.

3. In claim 2, the instructions, when executed by the electronic device, cause the electronic device to: Using the second model, determine the scores indicating the degree to which the evidence information and the first natural language sentence and the second natural language sentence match each other. Non-transitory computer-readable storage medium.

4. In claim 1, the instructions, when executed by the electronic device, cause the electronic device to: While transmitting at least a part of the first natural language sentence through the communication circuit, causing the first natural language sentence to be determined as the response message using the scores, Non-transitory computer-readable storage medium.

5. In claim 2, the instructions, when executed by the electronic device, cause the electronic device to: Determining scores for a plurality of third natural language sentences containing the second natural language sentence, wherein the word is replaced with one of different words; The scores for the plurality of third natural language sentences cause the candidate response message to be determined as the response message using a higher probability than the score for the first natural language sentence. Non-transitory computer-readable storage medium.

6. In claim 5, the instructions, when executed by the electronic device, cause the electronic device to: By replacing the word of the first natural language sentence multiple times, a plurality of third natural language sentences are generated. Non-transitory computer-readable storage medium.

7. In claim 1, the instructions, when executed by the electronic device, cause the electronic device to: Based on the determination that the first natural language sentence is different from the response message, causing the first natural language sentence to be changed to a fourth natural language sentence generated using the first model. Non-transitory computer-readable storage medium.

8. In claim 7, the instructions, when executed by the electronic device, cause the electronic device to: After transmitting the first natural language sentence through the communication circuit, a signal indicating that the first natural language sentence is changed to the fourth natural language sentence is transmitted to the external electronic device based on the determination that the first natural language sentence is different from the response message. Non-transitory computer-readable storage medium.

9. In claim 8, the instructions, when executed by the electronic device, cause the electronic device to: To cause the external electronic device to transmit the signal, which causes the first natural language sentence displayed on the display of the external electronic device to be removed, and causes the fourth natural language sentence to be displayed at the location within the display where the first natural language sentence was removed. Non-transitory computer-readable storage medium.

10. In claim 1, the instructions, when executed by the electronic device, cause the electronic device to: After transmitting the first natural language sentence through the communication circuit, based on determining that the first natural language sentence is different from the response message: Using the word-by-word accuracies of the first natural language sentence represented by the above scores, at least one word of the first natural language sentence is determined as a word related to the hallucination generated in the first model; causing said external electronic device to transmit a signal for visually highlighting said at least one word within said first natural language sentence displayed on said external electronic device; Non-transitory computer-readable storage medium.

11. In electronic devices, communication circuit; Memory that stores instructions; and A processor, wherein the instructions, when executed by the processor, cause the electronic device to: Receive a prompt sentence from an external electronic device through the above communication circuit; Executing a first model for natural language processing to generate a first natural language sentence related to the prompt sentence; By changing a part of the first natural language sentence, a second natural language sentence is obtained; By executing a second model linked to evidence information, scores representing the accuracy of each of the first natural language sentence and the second natural language sentence are determined, and the evidence information is used as a criterion for evaluating the accuracy of the natural language sentence by the second model; and Using the above scores, causing the first natural language sentence to be determined as a response message to be transmitted to the external electronic device. Electronic devices.

12. In claim 11, the instructions, when executed by the processor, cause the electronic device to: By using a score corresponding to the second natural language sentence obtained by changing the part corresponding to one word of the first natural language sentence, it is determined whether the word matches the evidence information. Electronic devices.

13. In claim 12, the instructions, when executed by the processor, cause the electronic device to: Using the second model, determine the scores indicating the degree to which the evidence information and the first natural language sentence and the second natural language sentence match each other. Electronic devices.

14. In claim 11, the instructions, when executed by the processor, cause the electronic device to: While transmitting at least a part of the first natural language sentence through the communication circuit, causing the first natural language sentence to be determined as the response message using the scores, Electronic devices.

15. In claim 12, the instructions, when executed by the processor, cause the electronic device to: Determining scores for a plurality of third natural language sentences containing the second natural language sentence, wherein the word is replaced with one of different words; The scores for the plurality of third natural language sentences cause the candidate response message to be determined as the response message using a higher probability than the score for the first natural language sentence. Electronic devices.

16. In claim 15, the instructions, when executed by the electronic device, cause the electronic device to: By replacing the word of the first natural language sentence multiple times, a plurality of third natural language sentences are generated. Electronic devices.

17. In claim 11, the instructions, when executed by the processor, cause the electronic device to: Based on the determination that the first natural language sentence is different from the response message, causing the first natural language sentence to be changed to a fourth natural language sentence generated using the first model. Electronic devices.

18. In claim 17, the instructions, when executed by the processor, cause the electronic device to: After transmitting the first natural language sentence through the communication circuit, a signal indicating that the first natural language sentence is changed to the fourth natural language sentence is transmitted to the external electronic device based on the determination that the first natural language sentence is different from the response message. Electronic devices.

19. In claim 18, the instructions, when executed by the processor, cause the electronic device to: To cause the external electronic device to transmit the signal, which causes the first natural language sentence displayed on the display of the external electronic device to be removed, and causes the fourth natural language sentence to be displayed at the location within the display where the first natural language sentence was removed. Electronic devices.

20. In claim 11, the instructions, when executed by the processor, cause the electronic device to: After transmitting the first natural language sentence through the communication circuit, based on determining that the first natural language sentence is different from the response message: Using the word-by-word accuracies of the first natural language sentence represented by the above scores, at least one word of the first natural language sentence is determined as a word related to the hallucination generated in the first model; causing said external electronic device to transmit a signal for visually highlighting said at least one word within said first natural language sentence displayed on said external electronic device; Electronic devices.

Images