Method for recognizing voice and electronic device supporting the method

Abstract

The present disclosure relates to a method for recognizing speech and an electronic device supporting the method. The electronic device includes a microphone, a display, a camera, a processor and a memory. The processor is configured to receive a first speech input through a microphone. The processor is also configured to obtain first data from a first image displayed on a display or stored in a memory. The processor is also configured to store the first speech input in association with the first data when the obtained first data matches the first speech input. In addition, the processor is configured to activate the camera when the first data does not match the first speech input. The processor is also configured to obtain second data from a second image collected by the camera, and when the obtained second data matches the first speech input, the first speech input is stored in association with the second data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2019-0095777 filed in the Korean Intellectual Property Office on August 6, 2019, the disclosure of which is incorporated herein in its entirety by reference. Technical Field

[0003] The present disclosure relates to a method for recognizing a user's voice and an electronic device supporting the method. Background Art

[0004] In addition to the input method using a keyboard or a mouse, recent electronic devices can also support various input methods such as voice input. For example, electronic devices such as smart phones or tablets provide speech recognition services for receiving user speech and performing operations corresponding to the input user speech.

[0005] Such a speech recognition service is developed based on a technology that processes natural languages ​​such as daily conversations of users. According to the technology that processes natural languages, the intention of user input (speech) is recognized, and a result corresponding to the recognized intention is calculated to provide services to the user. Summary of the invention

[0006] An electronic device according to the related art provides a voice recognition service corresponding to a speech input and a service (visual service) for analyzing an image to provide relevant information, respectively. In this case, the electronic device cannot process the speech input and the image input for performing a single task in association with each other, and thus may not be able to provide an effective artificial intelligence (AI) service expected by the user.

[0007] An electronic device according to an embodiment of the present disclosure may provide a voice recognition service for centrally processing a user speech input and an image input in association with each other.

[0008] According to one aspect of the present disclosure, an electronic device is provided. The electronic device includes a microphone, a display, a camera, a processor, and a memory operably connected to the processor, wherein the memory stores instructions that, when executed, cause the processor to: receive a first voice input through the microphone; obtain first data from a first image displayed on the display or stored in the memory; when the obtained first data matches the first voice input, store the first voice input in association with the first data; when the recognized first data does not match the first voice input, activate the camera; obtain second data from a second image collected by the camera; and when the obtained second data matches the first speech input, store the first speech input in association with the second data.

[0009] Before proceeding to the following detailed description, it may be helpful to set forth the definitions of certain words and phrases used throughout the patent document: the terms "include" and "comprising" and their derivatives mean including but not limited to; the term "or" is inclusive, meaning and / or; the phrases "associated with" and "associated with" and their derivatives may mean including, included in, interconnected with, containing, contained in, connected to or connected with, coupled to or coupled with, communicable with, cooperating with, intertwined, juxtaposed, adjacent to, tied to or bound with, having or having the property of, etc.; and the term "controller" means any device, system or part thereof that controls at least one operation, and such a device may be implemented in hardware, firmware or software or some combination of at least two of them. It should be noted that the functionality associated with any particular controller may be centralized or distributed, local or remote.

[0010] In addition, the various functions described below may be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and recorded in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, related data, or a portion thereof, suitable for implementation in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disk (CD), a digital video disk (DVD), or any other type of memory. "Non-transitory" computer-readable media do not include wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transitory computer-readable media include media that can permanently store data and media that can store data and then overwrite it, such as rewritable optical disks or erasable storage devices.

[0011] Definitions for certain words and phrases are provided throughout this patent document, those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following written description taken in conjunction with the accompanying drawings, in which like reference numerals represent like parts:

[0013] Figure 1 A block diagram of an integrated intelligent system according to an embodiment of the present disclosure is shown;

[0014] Figure 2 A diagram showing a format in which relationship information between concepts and actions is stored in a database according to an embodiment of the present disclosure;

[0015] Figure 3 A user terminal showing a screen in which a received voice input is processed by a first application (eg, a smart application) according to an embodiment of the present disclosure is shown;

[0016] Figure 4 A configuration diagram of an application for processing speech input and image input according to various embodiments of the present disclosure is shown;

[0017] Figure 5A A flow chart of a speech recognition method according to an embodiment of the present disclosure is shown;

[0018] Figure 5B A flow chart of a speech recognition method according to another embodiment of the present disclosure is shown;

[0019] Figure 6 A method for voice recognition using screenshots according to various embodiments of the present disclosure is shown;

[0020] Figure 7 A method for voice recognition using a camera according to various embodiments of the present disclosure is shown;

[0021] Figure 8 A diagram illustrating a method of using stored identified data according to various embodiments of the present disclosure;

[0022] Fig. 9 A diagram showing an example of information processing using an image input and a user speech input according to various embodiments of the present disclosure;

[0023] Fig.10 A diagram showing an example of a voice recognition service based on image input through a memo or message application;

[0024] Fig.11A A view showing an example of a screen in which recognized data and additional information are displayed based on an image according to one occurrence of a user speech input according to various embodiments of the present disclosure;

[0025] Fig. 11B A view showing an example of a screen in which recognized data and a result obtained according to an additional condition are displayed based on an image according to one occurrence of a user utterance input according to various embodiments of the present disclosure; and

[0026] Fig.12A view illustrating an example of a screen related to using stored recognized data in response to different utterance inputs according to various embodiments of the present disclosure. DETAILED DESCRIPTION

[0027] Discussed below Figures 1 to 12 The various embodiments used to describe the principles of the present disclosure in this application document are merely exemplary and should not be interpreted in a manner that limits the scope of the present disclosure. Those skilled in the art will appreciate that the principles of the present disclosure can be implemented in any appropriately arranged system or device.

[0028] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to these specific embodiments, but may include various modifications, equivalents and / or alternative forms of the embodiments of the present disclosure. With respect to the description of the accompanying drawings, the same reference numerals may refer to the same elements.

[0029] Figure 1 A block diagram of an integrated intelligent system according to an embodiment of the present disclosure is shown.

[0030] refer to Figure 1 , the integrated intelligent system according to an embodiment of the present disclosure includes a user terminal 100 , an intelligent server 200 and a service server 300 .

[0031] The user terminal 100 of an embodiment of the present disclosure may be a terminal device (or electronic device) connectable to the Internet, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a television, a white appliance, a wearable device, an HMD, or a smart speaker.

[0032] According to the illustrated embodiment, the user terminal 100 may include a communication interface 110, a microphone 120, a speaker 130, a display 140, a memory 150, or a processor 160. The above elements may be operably connected or electrically connected to each other. According to an embodiment of the present disclosure, the user terminal 100 may also include a camera. The camera may obtain image data related to surrounding objects.

[0033] The communication interface 110 of the embodiment of the present disclosure can be connected to an external device to send / receive data. The microphone 120 of the embodiment of the present disclosure can receive sound (e.g., user speech) and can convert the sound into an electrical signal. The speaker 130 of the embodiment of the present disclosure can output an electrical signal as sound (e.g., voice). The display 140 of the embodiment of the present disclosure can be configured to display an image or video. The display 140 of the embodiment of the present disclosure can also display a graphical user interface (GUI) of an executed application (or application program).

[0034] The memory 150 of the embodiment of the present disclosure may store a client module 151, a software development kit (SDK) 153, and a plurality of applications 155. The client module 151 and the SDK 153 may constitute a framework (or solution program) for performing general functions. In addition, the client module 151 or the SDK 153 may constitute a framework for processing voice input.

[0035] The plurality of applications 155 may be programs for performing specified functions. According to an embodiment of the present disclosure, the plurality of applications 155 may include a first application 155_1 and a second application 155_3. According to an embodiment of the present disclosure, each of the plurality of applications 155 may include a plurality of operations for performing specified functions. For example, the applications may include an alarm clock application, a message application, and / or a calendar application. According to an embodiment of the present disclosure, the plurality of applications 155 may be executed by the processor 160 to sequentially perform at least a portion of the plurality of operations.

[0036] The processor 160 of an embodiment of the present disclosure may control the overall operation of the user terminal 100. For example, the processor 160 may be electrically connected to the communication interface 110, the microphone 120, the speaker 130, and the display 140 to perform designated operations.

[0037] The processor 160 of an embodiment of the present disclosure may execute a program stored in the memory 150 to perform a designated function. For example, the processor 160 may execute at least one of the client module 151 or the SDK 153 to perform the following operation for processing a voice input. The processor 160 may control the operation of the plurality of applications 155 through, for example, the SDK 153. The following operation described as the operation of the client module 151 or the SDK 153 may be performed by the processor 160.

[0038] The client module 151 of an embodiment of the present disclosure may receive a voice input. For example, the client module 151 may receive a voice input corresponding to a user's speech detected by the microphone 120. The client module 151 may send the received voice input to the intelligent server 200. The client module 151 may send state information about the user terminal 100 together with the received voice input to the intelligent server 200. The state information may be, for example, execution state information about an application.

[0039] The client module 151 of the embodiment of the present disclosure can receive the result corresponding to the received voice input. For example, when the intelligent server 200 is able to calculate the result corresponding to the received voice input, the client module 151 can receive the result corresponding to the received voice input. The client module 151 can display the received result on the display 140.

[0040] The client module 151 of an embodiment of the present disclosure may receive a plan corresponding to the received voice input. The client module 151 may display the results of executing multiple operations of the application according to the plan on the display 140. The client module 151 may, for example, sequentially display the results of executing multiple operations on the display 140. For another example, the user terminal 100 may display only a portion of the results of executing multiple operations (e.g., the result of executing the last operation) on the display 140.

[0041] According to an embodiment of the present disclosure, the client module 151 may receive a request for acquiring information for calculating a result corresponding to the voice input from the smart server 200. According to an embodiment of the present disclosure, the client module 151 may send information to the smart server 200 in response to the request.

[0042] The client module 151 of the embodiment of the present disclosure may send result information about the results of executing the plurality of operations according to the plan to the smart server 200. The smart server 200 may use the result information to confirm that the received voice input has been correctly executed.

[0043] The client module 151 of an embodiment of the present disclosure may include a voice recognition module. According to an embodiment of the present disclosure, the client module 151 may recognize a voice input for executing a limited function through the voice recognition module. For example, the client module 151 may execute an intelligent application for processing a voice input for executing an operation of the system through a specified input (e.g., wake up!).

[0044] The intelligent server 200 of an embodiment of the present disclosure may receive information related to a user voice input from the user terminal 100 through a communication network. According to an embodiment of the present disclosure, the intelligent server 200 may change the data related to the received voice input into text data. According to an embodiment of the present disclosure, the intelligent server 200 may generate a plan for executing a task corresponding to the user voice input based on the text data.

[0045] According to an embodiment of the present disclosure, the plan may be generated by an artificial intelligence (AI) system. The artificial intelligence system may be a rule-based system or a neural network-based system (e.g., a feedforward neural network (FNN), a recurrent neural network (RNN)). Alternatively, the artificial intelligence system may be a combination of the foregoing systems or other artificial intelligence systems. According to an embodiment of the present disclosure, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the artificial intelligence system may select at least one plan from a plurality of predetermined plans.

[0046] The intelligent server 200 of the embodiment of the present disclosure may send the result according to the generated plan to the user terminal 100, or may send the generated plan to the user terminal 100. According to the embodiment of the present disclosure, the user terminal 100 may display the result according to the plan on the display 140. According to the embodiment of the present disclosure, the user terminal 100 may display the result of performing the operation according to the plan on the display 140.

[0047] The intelligent server 200 of the embodiment of the present disclosure may include a front end 210 , a natural language platform 220 , a capsule database 230 , an execution engine 240 , a terminal user interface 250 , a management platform 260 , a big data platform 270 or an analysis platform 280 .

[0048] The front end 210 of an embodiment of the present disclosure may receive a voice input from the user terminal 100. The front end 210 may send a response corresponding to the voice input.

[0049] According to an embodiment of the present disclosure, the natural language platform 220 may include an automatic speech recognition (ASR) module 221 , a natural language understanding (NLU) module 223 , a planner module 225 , a natural language generator (NLG) module 227 , or a text-to-speech (TTS) module 229 .

[0050] The automatic speech recognition module 221 of the embodiment of the present disclosure can convert the speech input received from the user terminal 100 into text data. The natural language understanding module 223 of the embodiment of the present disclosure can use the text data of the speech input to identify the user's intention. For example, the natural language understanding module 223 can identify the user's intention by performing grammatical analysis or semantic analysis. The natural language understanding module 223 of the embodiment of the present disclosure can use the language features (e.g., syntactic elements) of morphemes or phrases to identify the meaning of words extracted from the speech input, and can match the identified meaning with the intention to determine the user's intention.

[0051] The planner module 225 of an embodiment of the present disclosure may generate a plan using the intent and parameters determined by the natural language understanding module 223. According to an embodiment of the present disclosure, the planner module 225 may determine multiple domains for performing tasks based on the determined intent. The planner module 225 may determine multiple operations included in each of the multiple domains determined based on the intent. According to an embodiment of the present disclosure, the planner module 225 may determine parameters for performing multiple determined operations or result values ​​output by performing multiple operations. Parameters and result values ​​may be defined as concepts of a specified form (or class). Therefore, the plan may include multiple operations and multiple concepts determined according to the user's intent. The planner module 225 may determine the relationship between multiple operations and multiple concepts in stages (or hierarchically). For example, the planner module 225 may determine the execution order of multiple operations determined based on the user's intent based on multiple concepts. That is, the planner module 225 may determine the execution order of multiple operations based on the parameters for performing multiple operations and the results output by performing multiple operations. Therefore, the planner module 225 may generate a plan including association information (eg, ontology) between a plurality of operations and a plurality of concepts. The planner module 225 may generate the plan using information stored in the capsule database 230, which stores a set of relationships between concepts and operations.

[0052] The natural language generator module 227 of the embodiment of the present disclosure can change the specified information into text form. The information changed into text form can have the form of natural language speech. The text-to-speech conversion module 229 of the embodiment of the present disclosure can change the information in text form into information in speech form.

[0053] According to an embodiment of the present disclosure, part or all of the functions of the natural language platform 220 may be implemented in the user terminal 100 .

[0054] The capsule database 230 may store information about the relationship between a plurality of operations and concepts corresponding to a plurality of domains. According to an embodiment of the present disclosure, a capsule may include a plurality of action objects (or action information) and a concept object (or concept information). According to an embodiment of the present disclosure, the capsule database 230 may store a plurality of capsules in the form of a concept action network (CAN). According to an embodiment of the present disclosure, a plurality of capsules may be stored in a function registry included in the capsule database 230.

[0055] The capsule database 230 may include a policy registry storing policy information used when determining a plan corresponding to the voice input. The policy information may include reference information for determining one plan when there are multiple plans corresponding to the voice input. According to an embodiment of the present disclosure, the capsule database 230 may include a follow-up registry storing follow-up operation information for suggesting a follow-up operation to a user in a specified situation. The follow-up operation may include, for example, a follow-up utterance. According to an embodiment of the present disclosure, the capsule database 230 may include a layout registry storing layout information about information output through the user terminal 100. According to an embodiment of the present disclosure, the capsule database 230 may include a vocabulary registry storing vocabulary information included in the capsule information. According to an embodiment of the present disclosure, the capsule database 230 may include a dialogue registry storing dialogue information about a dialogue (or interaction) with a user. The capsule database 230 may update the stored objects through a developer tool. The developer tool may include, for example, a function editor for updating an action object or a concept object. The developer tool may include a vocabulary editor for updating a vocabulary. The developer tool may include a policy editor for generating and registering a policy for determining a plan. The developer tool may include a dialog editor for generating a dialog with a user. The developer tool may include a follow-up editor for editing a follow-up speech that activates a follow-up target and provides a prompt. The follow-up target may be determined based on a currently set target, a user's preference, or an environmental condition. In an embodiment of the present disclosure, the capsule database 230 may also be implemented in the user terminal 100.

[0056] The execution engine 240 of the embodiment of the present disclosure can calculate the result using the generated plan. The terminal user interface 250 can send the calculated result to the user terminal 100. Therefore, the user terminal 100 can receive the result and provide the received result to the user. The management platform 260 of the embodiment of the present disclosure can manage the information used in the intelligent server 200. The big data platform 270 of the embodiment of the present disclosure can collect the user's data. The analysis platform 280 of the embodiment of the present disclosure can manage the quality of service (QoS) of the intelligent server 200. For example, the analysis platform 280 can manage the elements and processing speed (or efficiency) of the intelligent server 200.

[0057] The service server 300 of an embodiment of the present disclosure may provide a specified service (e.g., a food order or a hotel reservation) to the user terminal 100. According to an embodiment of the present disclosure, the service server 300 may be a server operated by a third party. The service server 300 of an embodiment of the present disclosure may provide information for generating a plan corresponding to the received voice input to the intelligent server 200. The provided information may be stored in the capsule database 230. In addition, the service server 300 may provide result information to the intelligent server 200 according to the plan.

[0058] In the above integrated intelligent system, the user terminal 100 can provide various intelligent services to the user in response to the user input. The user input may include, for example, input through a physical button, touch input, or voice input.

[0059] In an embodiment of the present disclosure, the user terminal 100 may provide a voice recognition service through an intelligent application (or voice recognition application) stored in the user terminal 100. In this case, for example, the user terminal 100 may recognize a user's speech or voice input received through a microphone, and may provide a service corresponding to the recognized voice input to the user.

[0060] In an embodiment of the present disclosure, the user terminal 100 can perform a specified operation based on the received voice input, alone or in combination with the intelligent server and / or the service server. For example, the user terminal 100 can execute an application corresponding to the received voice input, and the specified operation can be performed by the executed application.

[0061] In an embodiment of the present disclosure, when the user terminal 100 provides a service in conjunction with the intelligent server 200 and / or the service server 300, the user terminal 100 may detect user speech using the microphone 120, and may generate a signal (or voice data) corresponding to the detected user speech. The user terminal 100 may use the communication interface 110 to send the voice data to the intelligent server 200.

[0062] The intelligent server 200 according to an embodiment of the present disclosure can generate a plan for performing a task corresponding to a voice input or a result of performing an operation according to a plan as a response to a voice input received from a user terminal 100. The plan may include, for example, a plurality of operations for performing a task corresponding to a user voice input and a plurality of concepts related to the plurality of operations. These concepts may be definitions of input parameters for performing a plurality of operations, or definitions of result values ​​output by performing a plurality of operations. The plan may include association information between a plurality of operations and a plurality of concepts.

[0063] The user terminal 100 of the embodiment of the present disclosure may receive the response using the communication interface 110. The user terminal 100 may output a voice signal generated in the user terminal 100 to the outside using the speaker 130, or may output an image generated in the user terminal 100 to the outside using the display 140.

[0064] Figure 2 A diagram illustrating a format in which relationship information between concepts and operations is stored in a database according to various embodiments of the present disclosure.

[0065] The capsule database (eg, capsule database 230 ) of the smart server 200 may store capsules in the form of CANs. The capsule database may store operations for performing tasks corresponding to the user voice input and parameters for the operations in the form of CANs 400 .

[0066] The capsule database may store a plurality of capsules A 401 and a plurality of capsules B 404 corresponding to a plurality of domains (e.g., applications), respectively. According to an embodiment of the present disclosure, a single capsule (e.g., capsule A 401) may correspond to a single domain (e.g., location (geographic location), application). In addition, at least one service provider (e.g., CP 1 402 or CP 2 403) for performing a function of a domain related to one capsule may correspond to one capsule. According to an embodiment of the present disclosure, a single capsule may include at least one operation 410 and at least one concept 420 for performing a specified function.

[0067] The natural language platform 220 may use capsules included in the capsule database to generate a plan for performing a task corresponding to the received speech input. For example, the planner module 225 of the natural language platform 220 may generate a plan using capsules stored in the capsule database. For example, the planner module 225 may generate a plan 407 using operations 4011 and 4013 and concepts 4012 and 4014 of capsule A 410, and operations 4041 and concepts 4042 of capsule B 404.

[0068] Figure 3 A view illustrating a screen in which a voice input received through a smart application is processed by a user terminal according to various embodiments of the present disclosure.

[0069] The user terminal 100 may execute a smart application for processing user input through the smart server 200 .

[0070] According to an embodiment of the present disclosure, in screen 310, when the user terminal 100 has recognized a specified voice input (e.g., wake up!) or has received an input through a hardware key (e.g., a dedicated hardware key), the user terminal 100 may execute a smart application for processing the voice input. The user terminal 100 may, for example, execute the smart application in a state where a scheduling application has been executed. According to an embodiment of the present disclosure, the user terminal 100 may display an object (e.g., an icon) 311 corresponding to the smart application on the display 140. According to an embodiment of the present disclosure, the user terminal 100 may receive a voice input caused by a user's speech. For example, the user terminal 100 may receive a voice input "Tell me about this week's schedule!". According to an embodiment of the present disclosure, the user terminal 100 may display a user interface (UI) 313 (e.g., an input window) of the smart application on the display, on which text data of the received voice input is displayed.

[0071] According to an embodiment of the present disclosure, in screen 320, the user terminal 100 may display a result corresponding to the received voice input on the display. For example, the user terminal 100 may receive a plan corresponding to the received user input, and may display “this week’s schedule” on the display according to the plan.

[0072] Figure 4 A configuration diagram of an application for processing speech input and image input according to various embodiments of the present disclosure is shown. Figure 4 The illustrations are merely examples, and embodiments of the present disclosure are not limited thereto.

[0073] refer to Figure 4 , the application 451 may include a first application 460 (eg, a smart application), a plurality of capsules 470 , an image providing application 480 , and a visual engine 490 .

[0074] According to an embodiment of the present disclosure, the first application 460 (e.g., a smart application) may process the user's speech input 455 in association with the image providing application 480. When the target or object in the speech input 455 is not determined, the first application 460 (e.g., a smart application) may analyze the image provided by the image providing application 480 to determine the target or object to perform a task corresponding to the speech input 455. For example, when the user generates the speech input 455 by saying the word "this" or "here" with respect to the picture that the user is viewing (an actual target in the real world in a mobile screen or a screen captured by a camera), the first application 460 (e.g., a smart application) may map the core keywords to the target identified by analyzing the image (see FIGS. 5 to 6). Fig.12 ).

[0075] According to an embodiment of the present disclosure, the first application 460 (eg, a smart application) may process the speech input 455 through a capsule corresponding to the speech input 455 among the plurality of capsules 470. According to an embodiment of the present disclosure, the user's speech input 455 may be sent to an external server (eg, Figure 1 According to an embodiment of the present disclosure, the user's speech input 455 may be sent to surrounding electronic devices (e.g., IoT devices) for analysis. According to an embodiment of the present disclosure, the user's speech input 455 may be analyzed by a separate speech analysis module or system in the electronic device.

[0076] According to an embodiment of the present disclosure, when the parameter in the user's speech input 455 is not determined, the plurality of capsules 470 may send a signal (e.g., a visual prompt) for performing a process associated with the image input (or visual input) to the first application 460 (e.g., a smart application). According to an embodiment of the present disclosure, the parameter may include data determined to identify the user's intention, behavior, or command in the input data included in the speech input 455. For example, the parameter may represent an additional word for extracting a word from the user's voice input and identifying the user's intention, behavior, or command from the word.

[0077] According to an embodiment of the present disclosure, the first application 460 (eg, smart application) may receive an image from the image providing application 480 in response to a signal. For example, the image may include a screenshot image, a preview image or an image captured by a camera, and an image pre-stored in a memory.

[0078] According to an embodiment of the present disclosure, the first application 460 (e.g., a smart application) may analyze an image through the visual engine 490. The visual engine 490 may recognize text or objects included in the image, or may analyze a region of interest (ROI). The first application 460 (e.g., a smart application) may transmit data recognized by the visual engine 490 (hereinafter referred to as recognized data) to the capsule. The capsule may provide the recognized data to the user, or may provide the user with a result of processing the recognized data automatically or based on additional input from the user.

[0079] Figure 5A A flow chart of a speech recognition method according to an embodiment of the present disclosure is shown.

[0080] refer to Figure 5A At operation 510, an electronic device (eg, Figure 1The processor 160 of the present disclosure may execute a first application (e.g., a smart application or a voice recognition application). In an embodiment of the present disclosure, the first application may be executed by various conditions. For example, the first application may be activated by an activation speech in a user speech (e.g., "Wake up!" or "Hello, Bixby"). For example, the first application may be executed by a user input (e.g., a touch input or a hardware key input). For example, the first application (e.g., a smart application) may be activated according to the user's location, a specified time, or a request from another application. The processor 160 may display a user interface related to the first application (e.g., a smart application). When the first application (e.g., a smart application) is executed, the processor 160 may wait to receive a user's speech input.

[0081] According to an embodiment of the present disclosure, an electronic device (e.g., processor 160) may execute a first application (e.g., smart application) based on an image obtained by the electronic device. For example, when an image is captured by using a camera, the processor 160 may analyze the captured image. When the captured image corresponds to a specific situation, the processor 160 may execute the first application (e.g., smart application) to activate a conversation with the user.

[0082] For example, when the captured image includes an area number of a parking lot, the processor 160 may execute a first application (eg, a smart application), and may output or display “Do you want to store the image as parking lot information?” to the user.

[0083] For another example, when the captured image includes a human face, the processor 160 may execute the first application and may output a voice message “Do you want to store the image as a passport photo” to the user through the speaker 130 .

[0084] In step 520, when the first application (e.g., smart application) is executed, the processor 160 may receive the user's first speech input through the microphone 120. The processor 160 may analyze the received first speech input. According to an embodiment of the present disclosure, the operation 510 of the first application (e.g., smart application) and the operation 520 of receiving the first speech input may be performed simultaneously. For example, while waiting for the user's speech to execute the first application (e.g., smart application), the electronic device may detect a specific activation speech in the user's speech, may analyze the user speech included after the activation speech, and may determine a portion of the user's speech as the first speech input.

[0085] In an embodiment of the present disclosure, the processor 160 may send the first speech input to the external intelligent server 200 to request analysis of the first speech input. In another embodiment of the present disclosure, the processor 160 may analyze the first speech input by performing calculations or data processing or using an analysis system in an electronic device. In another embodiment of the present disclosure, the processor 160 may analyze a portion of the first speech input by internal calculations, and may request the intelligent server 200 to analyze another portion of the first speech input.

[0086] According to an embodiment of the present disclosure, the first speech input may be a speech in which a parameter for performing a task is expressed as (replaced with) a specified keyword. The parameter may be data determined to identify the user's intention, behavior, or command in the input data included in the first speech input. Keywords may include demonstrative pronouns such as "this", "this place", "this person", or "this thing" or terms that do not specify a target. For example, when the first speech input reads "Store this as a parking location", the keyword may be "this", and the parameter corresponding to the keyword "this" may be "J4, underground floor" determined by image analysis.

[0087] According to another embodiment of the present disclosure, the first speech input may be a speech that does not include a target for performing a task. For example, the first speech input may read “store” or “store as a parking location”.

[0088] At operation 530, the processor 160 may obtain first recognized data (or first recognized object) by analyzing the first image. According to an embodiment of the present disclosure, the first recognized data may be text, an icon, or object information having a preset format. According to an embodiment of the present disclosure, the processor 160 may perform image analysis on the first image, and may extract object information included in the first image as the first recognized data based on the result of performing the image analysis. For example, the processor 160 may perform image analysis on the first image through an object recognition algorithm, and may obtain information about the object included in the first image as the first recognized data. According to an embodiment of the present disclosure, the processor 160 may extract text included in the first image as the first recognized data.

[0089] According to an embodiment of the present disclosure, the processor 160 may generate multiple data using the first image, and may remove unnecessary data by primary filtering. For example, the processor 160 may generate multiple data using the first image, and may obtain first recognized data from the multiple data by an additional algorithm. For example, when there are multiple objects obtained using the first image, the processor 160 may select one object from the multiple objects as the first recognized data based on the status information. For example, when the objects extracted from the first image include a parking lot, a vehicle, a person, or a parking lot floor height, the processor 160 may determine the parking lot floor height information as the first recognized data by an additional algorithm for selecting multiple objects. According to an embodiment of the present disclosure, the first recognized data may include multiple objects. The processor 160 may obtain all multiple objects as the first recognized data through the first image.

[0090] According to an embodiment of the present disclosure, the format of the first recognized data may be defined in various forms according to the domain processed by the first application (e.g., smart application). The processor 160 may obtain the first recognized data by performing internal calculations or analyzing the first image using an external server (e.g., smart server 200 or an additional server for analyzing images).

[0091] According to an embodiment of the present disclosure, the first image may be an image included in the content displayed on the display. For example, the processor 160 may store a content segment displayed on the display as an image. For example, when executing the first application (e.g., a smart application), the processor 160 may store a screenshot of the execution screen of the most recently executed application as the first image. For another example, regardless of whether the first application (e.g., a smart application) is executed, the processor 160 may store a screenshot of the execution screen of the most recently executed application as the first image. For another example, when receiving a first speech input, the processor 160 may store a screenshot of the content displayed on the display as the first image. According to an embodiment of the present disclosure, the first image may include information stored as an image of multiple pieces of content provided by an electronic device. For example, when a user is viewing an image through an image application, the electronic device may store the image content in the executed image application as the first data. According to an embodiment of the present disclosure, the electronic device may combine and store multiple images as a first image. For example, when a user uses an image application and a web browser at the same time, the screen (e.g., image, web screen) of the content used in each application may be extracted and combined into one first image.

[0092] According to an embodiment of the present disclosure, the processor 160 may obtain a first image from an application based on situation information about the electronic device. For example, the situation information may include the user's location information, time information, schedule information, or multiple preset information. For example, when the location information about the electronic device is determined to be a parking lot, the processor 160 may store the screen of an application related to the parking lot as the first image. For example, when the user's schedule information indicates that the user is in a meeting, the processor 160 may store the screen displayed by various application office programs (e.g., PowerPoint, Excel, or Word processor) of the electronic device as the first image.

[0093] According to an embodiment of the present disclosure, the processor 160 may confirm the first image in the most recently executed application. For example, the processor 160 may determine the image most recently enlarged and displayed in the gallery application as the first image.

[0094] According to an embodiment of the present disclosure, the type of the first recognized data can be determined as a specified type according to the domain of the first speech input. For example, when the domain of the first speech input is "storage parking location", the type of the first recognized data can be determined as a combination of characters and numbers. For example, according to the speech input "parking location", "14th floor, area B" which is information representing the location of the parking lot in the first image can be obtained as the first recognized data.

[0095] At operation 540, the processor 160 may confirm whether the obtained first recognized data matches the first utterance input. According to an embodiment of the present disclosure, the matching operation may include an operation of confirming by the processor 160 whether an attribute of the first recognized data matches a domain of the first utterance input.

[0096] For example, when the attribute of the extracted first recognized data is a combination of characters and numbers, and the domain of the first utterance input is “store parking place”, the processor 160 may determine that the extracted first recognized data matches the first utterance input.

[0097] For example, when the attribute of the extracted first recognized data is a combination of numbers and symbols, and the domain of the first utterance input is “processing of passport information”, the processor 160 may determine that the extracted first recognized data matches the first utterance input.

[0098] According to an embodiment of the present disclosure, the operation 540 of matching with the first speech input is not limited to the operation of confirming based on the attribute of the first recognized data. For example, the processor 160 can identify the meaning of the first speech input, and can determine that the first recognized data matches the first speech input when the meaning of the first speech input is determined to be associated with the first recognized data.

[0099] According to various embodiments of the present disclosure, operation 530 and operation 540 may be combined. For example, the processor 160 may obtain first recognized data matching the first utterance input from data obtained by analyzing the first image.

[0100] In operation 550, when the extracted first recognized data matches the first speech input, the processor 160 may store the extracted first recognized data in association with the first speech input. The processor 160 may store the first recognized data or the first image by marking the first recognized data or the first image with the keyword of the first speech input. For example, during the operation of marking and storing, the processor 160 may generate a keyword corresponding to the first speech input, and may store the first recognized data with the generated keyword as a tag value. According to an embodiment of the present disclosure, the electronic device may store the marked keywords and the first recognized data in the form of a database, and may store it so that the electronic device can obtain or use the first recognized data based on the marking information. When the first image is stored by marking the first image, the electronic device may store the first image together with the keyword information by mapping the first image and the keyword information, and may search or use the first image based on the marking information.

[0101] According to an embodiment of the present disclosure, when storing the obtained image or the first identified data, the processor 160 may further determine whether the obtained image or the first identified data is information related to personal information about the user or information that should be protected (hereinafter referred to as sensitive information). When the first identified data is sensitive information (e.g., personal information, security information, or important information), the processor 160 may store the first identified data in a separate storage area (e.g., a security area) with enhanced security. For example, when the image is determined to be information that should be protected or information for which preset security is enhanced, the processor 160 may store the first identified data in a storage area with enhanced security (e.g., a trust zone) instead of an ordinary storage area. According to an embodiment of the present disclosure, when the image includes sensitive information, the processor 160 may store the first identified data in a separate secure folder with restricted access.

[0102] According to an embodiment of the present disclosure, the processor 160 may use the first recognized data as a necessary parameter of the first speech input to perform a task corresponding to the first speech input. For example, when the first speech input reads "store parking lot information", the task corresponding to the first speech input is a task of obtaining parking lot information and storing the obtained information. In order to perform the task corresponding to the first speech input, the processor 160 may confirm whether the first recognized data obtained from the first image is the parking lot information corresponding to the first speech input. When the first recognized data is parking lot information, the processor 160 may store the first recognized data as parking lot information.

[0103] According to an embodiment of the present disclosure, the processor 160 may display the extracted first identified data through the user interface of the first application (eg, smart application). The processor 160 may allow the user to confirm or change the first identified data obtained through the user interface of the first application (eg, smart application).

[0104] According to an embodiment of the present disclosure, the processor 160 may provide the first recognized data to another application, and may support so that an operation associated with the first recognized data is performed by the other application.

[0105] According to an embodiment of the present disclosure, the processor 160 may provide the first recognized data to the surrounding electronic devices so as to perform an operation associated with the first recognized data together with the surrounding electronic devices.

[0106] According to various embodiments of the present disclosure, when the second speech input requests processing of the stored first recognized data, the processor 160 may perform the task for the second speech input using the stored first recognized data. For example, when the first recognized data is stored as "J1, underground four floors" in response to the first speech input reading "store parking location", when the second speech input reading "store parking location" is received, the processor 160 may display "J1, underground four floors" as the first recognized data on the display.

[0107] According to various embodiments of the present disclosure, the processor 160 may store the first identified data in association with a first application (e.g., a smart application). Alternatively, the processor 160 may provide the first identified data to an application associated with the first identified data. The processor 160 may store the first identified data in a storage format available in an associated application. For example, when the first identified data is a parking position, the processor 160 may provide the first identified data to a reminder application. For example, the processor 160 may store the first identified data so that the first identified data may be used in a reminder application.

[0108] At operation 560, when the extracted first recognized data does not match the first speech input, the processor 160 may obtain a second image by executing a second application (e.g., a camera application). For example, the processor 160 may obtain the second image using a camera application. For example, when the first image does not include information corresponding to the first speech input, the second image may be obtained by further executing the camera application.

[0109] According to an embodiment of the present disclosure, when the first image does not include information corresponding to the first speech input, the processor 160 may obtain the second recognized data by using the second image obtained by an external camera of the electronic device as the second application or an external electronic device or server. For example, the processor 160 may obtain location information related to the electronic device as the second application and the second image through a map application based on the location information.

[0110] According to an embodiment of the present disclosure, the second image may be a preview image collected by a camera. The processor 160 may obtain the second recognized data by analyzing the preview image in real time.

[0111] According to an embodiment of the present disclosure, the second image may be an image captured automatically or in response to a user input.

[0112] According to an embodiment of the present disclosure, the second application may include a web browser application. The processor 160 may obtain a second image related to the user utterance from the web browser application.

[0113] According to an embodiment of the present disclosure, the processor 160 may determine an image obtained by various sensors (eg, an infrared sensor, an optical sensor, a depth sensor, a location sensor, a GPS, or a biometric sensor) of the electronic device as the second image.

[0114] At operation 570, the processor 160 may extract the second recognized data by analyzing the second image obtained by the second application (e.g., a camera application). The type of the second recognized data may be text, icon, or object information. According to an embodiment of the present disclosure, the format of the second recognized data may be defined in various forms according to the domain processed by the first application (e.g., a smart application). The processor 160 may obtain the second recognized data by performing internal calculations or analyzing the second image using an external server (e.g., the smart server 200 or an additional server for analyzing images).

[0115] At operation 580, the processor 160 may confirm whether the extracted second recognized data matches the first speech input. According to an embodiment of the present disclosure, the processor 160 may confirm whether the attribute of the second recognized data matches the domain of the first speech input. According to an embodiment of the present disclosure, the operation of matching with the first speech input is not limited to the operation of confirming the attribute of the second recognized data. For example, the processor 160 may identify the meaning of the first speech input, and may determine that the second recognized data matches the first speech input when the meaning of the first speech input is determined to be associated with the second recognized data.

[0116] According to various embodiments of the present disclosure, operation 570 and operation 550 may be combined. For example, the processor 160 may obtain second recognized data matching the first utterance input from data obtained by analyzing the second image.

[0117] At operation 590, when the extracted second recognized data matches the first speech input, the electronic device (e.g., processor 160) may store the extracted second recognized data in association with the first speech input. The processor 160 may store the second recognized data or the second image by tagging the second recognized data or the second image with a keyword of the first speech input.

[0118] In operation 591, when the extracted second recognized data does not match the first utterance input, the processor 160 may issue an alarm to the user. For example, the alarm may be a message notifying the lack of data recognized by the camera.

[0119] Figure 5B A flow chart of a speech recognition method according to another embodiment of the present disclosure is shown.

[0120] refer to Figure 5B At operation 595, an electronic device (e.g., Figure 1 The processor 160 of the processor can execute a first application (for example, a smart application or a voice recognition application).

[0121] At operation 596, the processor 160 may receive the first utterance input. The processor 160 may analyze the first utterance input. In an embodiment of the present disclosure, the processor 160 may send the first utterance input to the external intelligent server 200 to request analysis of the first utterance input.

[0122] According to an embodiment of the present disclosure, the first utterance input may be an utterance in which a parameter for performing a task is expressed as (replaced with) a specified keyword (eg, “this”, “this place”, “this person”, or “this thing”).

[0123] At operation 597 , the processor 160 may obtain first recognized data associated with a first utterance input from a first image provided through a first application.

[0124] According to an embodiment of the present disclosure, the first application may be a recently executed application or an application that has been executed before. The processor 160 may obtain first recognized data associated with the first utterance input by analyzing a first image (eg, a screenshot) provided from the first application.

[0125] According to another embodiment of the present disclosure, the first application may be a camera application. The processor 160 may obtain first recognized data associated with the first utterance input by analyzing a first image (eg, a preview image or a captured image) using the camera application.

[0126] At operation 598, the processor 160 may store the obtained first recognized data in association with the first speech input. The processor 160 may store the first recognized data or the first image by tagging the first recognized data or the first image with the keyword of the first speech input. For example, during the tagging and storing operation, the processor 160 may generate a keyword corresponding to the first speech input, and may store the first recognized data with the generated keyword as a tag value.

[0127] At operation 599, the processor 160 may provide the first recognized data so that the first recognized data may be used by the second application. The processor 160 may provide the first recognized data to an application associated with the first recognized data or the first speech input. For example, when the first recognized data is a parking location, the processor 160 may provide the first recognized data to a reminder application.

[0128] According to various embodiments of the present disclosure, the processor 160 may extract the first identified data by analyzing the screen information about the application. The processor 160 may notify the user of the processing of the first identified data based on the format or attribute of the first identified data. For example, when parking lot information is identified from the screen, the processor 160 may output or display a sound or text "Do you want to store as parking lot information?". When additional input from the user occurs, the processor 160 may provide the first identified data so that the first identified data can be used by the second application.

[0129] According to various embodiments of the present disclosure, when identification data related to personal information about a user is obtained, the processor 160 may inform the user that the identification data may be stored in a separate secure folder, or the processor 160 may automatically store the identified data in the secure folder. For example, when a passport image or passport information is recognized through an image, the processor 160 may store the passport image or passport information in a secure folder that can be verified through additional authentication.

[0130] Figure 6 A voice recognition method using screen information (eg, image content) of an application according to various embodiments of the present disclosure is illustrated. Figure 6 The illustrations are merely examples, and embodiments of the present disclosure are not limited thereto.

[0131] refer to Figure 6 According to an embodiment of the present disclosure, the processor 160 may display at least one piece of image content of at least one application in the first screen 610. For example, the first screen 610 may be a screen in which one image is enlarged and displayed in a gallery application. Figure 6 The gallery application is shown exemplarily, but the embodiments of the present disclosure are not limited thereto. For example, the first screen 610 may be an execution screen of a message application for exchanging messages with the other party. For another example, the first screen 610 may be a network search screen including images or texts.

[0132] In the second screen 620, the processor 160 may execute a first application (eg, a smart application). The first application (eg, a smart application) may be executed by an activation speech (eg, "Wake up!" or "Hello, Bixby") or a designated user input (eg, a touch input or a hardware key input).

[0133] According to various embodiments of the present disclosure, when executing a first application (e.g., a smart application), the processor 160 may automatically store image content (e.g., a screenshot or an image file) of an application that has been recently executed before executing the first application (e.g., a smart application). For example, in the case of a gallery application, the processor 160 may store the image content displayed by the gallery application as an image file. According to an embodiment of the present disclosure, the processor 160 may store at least a portion of a screen provided by the application being executed as an image file. According to an embodiment of the present disclosure, the processor 160 may store at least a portion of image information captured by a camera application as an image file. According to an embodiment of the present disclosure, the processor 160 may store an image included in a web page displayed in a web browser as an image file. When the stored image content is used to process the user's first utterance input 621, the stored image content may be used. For example, the processor 160 may obtain data for processing the first utterance input 621 from the stored image content, and may confirm the association relationship between the data and the first utterance input 621. The processor 160 may store the associated data as the first identified data. According to an embodiment of the present disclosure, the processor 160 may provide associated data so that the associated data may be used in the service of another application. The processor 160 may display the result of executing the task on the display.

[0134] The processor 160 may receive a first speech input 621 in which a parameter for performing a task is represented as a specified keyword (replaced with the specified keyword). The parameter may be data determined to identify the user's intention, behavior, or command in the input data included in the first speech input 621. The keyword may include a demonstrative pronoun, such as "this", "this place", "this person", or "this thing" or a term that does not specify a target. For example, when the first speech input 621 reads "store this as a parking location", the keyword may be "this", and the parameter corresponding to the keyword "this" may be determined by image analysis.

[0135] After receiving the first utterance input 621 , the processor 160 may confirm whether there is data corresponding to a format of parking data (eg, a combination of characters and numbers) among data recognized by analyzing the image of the first screen 610 (first recognized data).

[0136] At screen 630, when data corresponding to the format of parking information (e.g., a combination of characters and numbers) is included in the image, the processor 160 may display the first recognized data 631 (“J1, fourth underground floor”) or may store the first recognized data by mapping the first recognized data to a keyword of the first utterance input 621. In addition, the processor 160 may display a result of processing the first utterance input 621 using the first recognized data 631 as a necessary parameter (e.g., “Yes, stored. Parking location: J1, fourth underground floor”).

[0137] According to various embodiments of the present disclosure, the processor 160 may further display additional information 632 , such as an image and map information related to the first recognized data 631 .

[0138] According to various embodiments of the present disclosure, the processor 160 may also display an interface 633 such as a touch button for allowing the user to confirm whether the first recognized data 631 is correct.

[0139] In the fourth screen 640, the processor 160 may store the first recognized data 631 or additional information 632 extracted by another application associated with the first recognized data 631 or the domain of the first speech input 621. For example, when the first recognized data 631 is a parking location, the first recognized data 631 or the additional information 632 may be stored by a reminder application. According to an embodiment of the present disclosure, the additional information 632 may include location information, address information, or homepage information.

[0140] According to various embodiments of the present disclosure, the processor 160 may compile the first recognized data 631 or the additional information 632 into a database. The processor 160 may store statistical information about floors / areas where users frequently park.

[0141] Figure 6 It is shown that the first screen 610 is executed earlier than the first application (e.g., smart application), but the embodiments of the present disclosure are not limited to this. For example, the first application (e.g., smart application) may request a screen or image from an application that is being executed in the background or processed and executed in parallel. According to an embodiment of the present disclosure, the processor 160 may request a screen or image from an application to be executed after the first application (e.g., smart application) is executed. For example, when there is no application corresponding to the first speech input among the currently executed applications after receiving the first speech input, the processor 160 may further execute the application corresponding to the first speech input, or when the application corresponding to the first speech input is executed after waiting for a fixed time, the processor 160 may obtain the screen or image.

[0142] Figure 7A voice recognition method using a camera according to various embodiments of the present disclosure is illustrated. Figure 7 The illustrations are merely examples, and embodiments of the present disclosure are not limited thereto.

[0143] refer to Figure 7 , in the first screen 710, the processor 160 may display an execution screen of at least one application. For example, the first screen 710 may be an execution screen of a network search application.

[0144] In the second screen 720, the processor 160 may execute a first application (eg, a smart application). The first application (eg, a smart application) may be executed by a user's activation speech (eg, "Wake up!" or "Hello, Bixby") or a designated user input (eg, a touch input or a hardware key input).

[0145] The processor 160 may receive a first utterance input 721 of a user in a state of executing a first application (eg, a smart application).

[0146] After receiving the first utterance input 721, the processor 160 may confirm whether there is data corresponding to the format of the parking information in the first recognized data extracted by analyzing the image (e.g., screenshot) of the first screen 710. The image for the first screen 710 may be pre-stored before executing the first application (e.g., smart application), or may be collected after receiving the first utterance input 721.

[0147] In the third screen 730 , when data corresponding to the format of parking information does not exist in the image of the first screen 710 , the processor 160 may execute a camera application.

[0148] The processor 160 may confirm whether the second identified data corresponding to the format of the parking information exists in the image collected by the camera application. For example, the processor 160 may confirm whether the second identified data corresponding to the format of the parking information exists in the data extracted from the preview image or the captured image. According to various embodiments of the present disclosure, the processor 160 may provide guidance to the user so that the parking position information having a predetermined format enters the field of view of the camera.

[0149] When data corresponding to the format of parking information (e.g., a combination of characters and numbers) is included in the preview image or the captured image, the processor 160 may display the extracted second recognized data 741 (“J1, fourth underground floor”) or may store the first recognized data by mapping the second recognized data to a keyword of the first utterance input 721. In addition, the processor 160 may display a result of processing the first utterance input 721 using the first recognized data 741 as a necessary parameter (e.g., “Yes, stored. Parking location: J1, fourth underground floor”).

[0150] According to various embodiments of the present disclosure, the processor 160 may store the second recognized data 741 or the additional information 742 extracted by another application associated with the second recognized data 741 or the first speech input 721. For example, when the second recognized data 741 is a parking location, the second recognized data 741 or the additional information 742 may be stored by a reminder application. For example, when the obtained second recognized data 741 is location information, the second recognized data 741 may be provided to a map display application.

[0151] Figure 8 A diagram illustrating a method of using stored recognized data according to various embodiments of the present disclosure. Figure 8 The illustrations are merely examples, and embodiments of the present disclosure are not limited thereto.

[0152] refer to Figure 8 , in the first screen 810, the processor 160 may receive the second utterance input 811 and the first utterance input ( Figure 6 of 621, Figure 7 721). For example, the processor 160 may Figure 6 or Figure 7 The illustrated process receives a second utterance input 811 of “Where do I park?” in a state where the parking location is stored.

[0153] In the second screen 820, the processor 160 may display the stored recognition data 821 in response to the second speech input 811. The processor 160 may provide not only the recognized data 821 but also additional information 822, such as map information, path finding information, or time information. According to an embodiment of the present disclosure, when the stored recognition data 821 is displayed in response to the second speech input 811, the processor 160 may confirm whether the second speech input 811 has been input due to the first speech input (for example, the same speaker (or user) as Figure 6For example, when it is determined through voice analysis that the second speech input 811 has been spoken by the same user as the first speech input, additional information 822 may be displayed. According to an embodiment of the present disclosure, the processor 160 may perform additional authentication (e.g., biometric information, password input) after obtaining the second speech input 811, and may provide additional information 822.

[0154] According to various embodiments of the present disclosure, in the third screen 830, even if no additional second speech input is received, the processor 160 may pre-display the image input (eg, a screenshot or an image collected by a camera) and the first speech input ( Figure 6 of 621, Figure 7 721). For example, in the execution screen of the first application (e.g., smart application), the recognition data 831 stored based on the image and the first speech input may be displayed in the form of a card. The processor 160 may determine whether to display the stored recognition data 831 based on the user's location and commuting time.

[0155] Fig. 9 A diagram illustrating an example of information processing using an image input and a user speech input according to various embodiments of the present disclosure.

[0156] refer to Fig. 9 , the processor 160 may obtain the recognition data using the image 910. The image 910 may be a screenshot or an image collected by a camera. The processor 160 may use the extracted recognition data to perform a task corresponding to the first speech input 920. The processor 160 may store the extracted recognition data, or may store the result of performing the task.

[0157] For example, the image 910 may be an image of a user's prescription, and the first speech input 920 may read “This is my prescription.”

[0158] The processor 160 may analyze the image 910 by performing internal calculations or calculations through an external server. For example, the processor 160 may recognize information such as “patient information (name)”, “drugstore information (manufacturer number, company name, address, or phone number)”, “doctor information (name, address, or phone number)”, “drug ingredient information”, and “drug taking time” included in the image 910. The processor 160 may store a plurality of pieces of data extracted through the image 910. The processor 160 may store a plurality of pieces of data so that the plurality of pieces of data may be processed in different applications according to the attributes of the plurality of pieces of data. For example, the processor 160 may store “doctor information” so that the “doctor information” may be processed in a contact application, and may store “drug taking time” so that the “drug taking time” may be processed in a schedule application.

[0159] According to various embodiments of the present disclosure, the processor 160 may store additional information related to the extracted identification data. The additional information not included in the prescription may be information that the user may refer to, such as other explanations about each ingredient of the drug, similar drugs, precautions when taking the drug, and foods that should not be taken with the drug.

[0160] According to various embodiments of the present disclosure, the processor 160 can manage the extracted identification data using a database. When multiple prescriptions are stored for the same person, the ingredients, dosage, and frequency of administration of each prescription can be recorded. The processor 160 can notify or warn the user according to a specified condition.

[0161] According to various embodiments of the present disclosure, the processor 160 may notify the user in various ways based on the attributes of each extracted identification data. For example, in the result screen 930, the processor 160 may store "patient information (name)" and "medication time" as schedule information, and may notify the user of the medication time when the specified time arrives.

[0162] According to various embodiments of the present disclosure, in a state where the recognized data is stored, the processor 160 may receive a second utterance input 940 related to the recognized data or the first utterance input 920. For example, the second utterance input 940 may read out “Tell me when I should take medicine,” “What medicine is this?” or “Which medicine have I taken in the past three months?”.

[0163] In the result screen 950, the processor 160 may use the stored recognition data to perform a task corresponding to the second utterance input 940. For example, when the second utterance input 940 reads “What medicine is this?”, the processor 160 may display the result of searching “drug ingredient information” from a website.

[0164] Fig.10 A view showing an example of a voice recognition service based on an image input through a memo or message application. Fig.10 The illustrations are merely examples, and embodiments of the present disclosure are not limited thereto.

[0165] refer to Fig.10, the recognized data may be extracted based on the screen or image provided by the application. For example, the image 1005 provided from the application may be image information about the content of the conversation on the message application. In order to obtain the image information displayed on the message application, the processor 160 may store the image added on the screen being viewed by the user as the image information. According to an embodiment of the present disclosure, the processor 160 may analyze the text information, and may extract the recognized data from the analyzed text information. For example, the recognized data may be extracted from the text information related to the text exchanged between users of the message application. The preview image 1010 may be an image collected in real time by a camera. The processor 160 may first extract the recognition data related to the first speech input 1020 by the image 1005 provided from the application, and when there is no matching recognition data, the recognition data related to the first speech input 1020 may be extracted for the second time by the preview image 1010. The processor 160 may use the extracted recognition data to perform a task corresponding to the first speech input 1020. The processor 160 may store the extracted recognition data, or may store the result of performing the task.

[0166] For example, the image 1005 or the preview image 1010 provided from the application may include information about the user's schedule, and the first speech input 1020 may read “This is my son's play schedule.”

[0167] The processor 160 may analyze the image 1005 or the preview image 1010 provided from the application by performing internal calculations or calculations by an external server. For example, the processor 160 may recognize information such as “schedule title”, “date (year / month / day)”, “time (start time / end time)”, “place”, and “participants” included in the image 1005 or the preview image 1010 provided from the application. The processor 160 may store the recognition data extracted by the image 1005 or the preview image 1010 provided from the application. The processor 160 may store the recognized data in the form of a schedule application (1050).

[0168] Fig.11A A view illustrating an example of a screen in which recognized data and additional information are displayed based on an image according to one occurrence of a user speech input according to various embodiments of the present disclosure.

[0169] According to an embodiment of the present disclosure, the processor 160 may provide the user with a screen 1105 of an SNS application, or may provide an image 1110 obtained by capturing an object in real time. Here, the processor 160 may obtain multiple pieces of recognized data from the screen 1105 or the image 1110. According to an embodiment of the present disclosure, the processor 160 may obtain multiple pieces of recognized data based on the first utterance input 1120, and may store only one piece of previously obtained recognition data corresponding to the first utterance input 1120 as recognition data.

[0170] According to an embodiment of the present disclosure, the processor 160 may store or display data based on the first speech input 1120. For example, when the first speech input 1120 reads out "Where is it?", the processor 160 may obtain recognized data corresponding to the first part 1120a "it" included in the first speech input 1120, such as "landmark name", "area name" or "object", from the screen 1105 or the image 1110. The recognized data "landmark name" and additional information such as location information, traffic information, and surrounding tour route information may be displayed based on the second part 1120b "where" included in the first speech input 1120, or may be provided so that a related application may provide a service (1150).

[0171] Fig. 11B A view illustrating an example of a screen on which recognized data and a result obtained according to an additional condition are displayed based on an image according to one occurrence of a user utterance input according to various embodiments of the present disclosure.

[0172] According to an embodiment of the present disclosure, the processor 160 may provide a screen 1160 including project information to the user, or may provide an image 1180 obtained by capturing an object in real time. Here, the processor 160 may obtain multiple pieces of recognized data from the screen 1160 or the image 1180. According to an embodiment of the present disclosure, the processor 160 may obtain multiple pieces of recognized data based on the first utterance input 1170, and may store only one piece of previously obtained recognition data corresponding to the first utterance input 1170 as recognition data.

[0173] According to an embodiment of the present disclosure, the processor 160 may store or display data based on the first utterance input 1170. For example, when the first utterance input 1170 reads “display from the lowest price”, the processor 160 may display the recognition data and additional information corresponding to the first part 1170a of the first utterance input 1170 based on the condition “lowest price” included in the second part 1170b of the first utterance input 1170, or may provide the recognition data and additional information so that a related application can provide a service (1190).

[0174] Fig.12 A view illustrating an example of a screen related to using stored recognized data in response to different utterance inputs according to various embodiments of the present disclosure. Fig.12 The illustrations are merely examples, and embodiments of the present disclosure are not limited thereto.

[0175] refer to Fig.12 , the processor 160 may store the first recognized data based on the first image 1210 and the first speech input 1215. For example, the first recognized data may be passport information.

[0176] According to various embodiments of the present disclosure, identification data related to the user's personal information can be stored in a secure folder. For example, the processor 160 can store a passport image or passport information in a secure folder that can be verified by additional authentication.

[0177] The processor 160 may store the second recognized data based on the second image 1220 and the second speech input 1225. For example, the second recognized data may be flight information.

[0178] In a state where the first identified data and the second identified data are stored, the processor 160 may use the first identified data and the second identified data in another application according to the properties of the first identified data and the second identified data. For example, a duty-free shop application may be executed on the first screen 1230 and the second screen 1240, and a user interface including a first field 1241 for inputting passport information or a second field 1242 for inputting flight information may be displayed.

[0179] When the third speech input 1245 occurs, the processor 160 may match the attributes of the first recognized data (e.g., passport information) with the attributes of the first field 1241 to fill in the first field 1241 with the first recognized data. The processor 160 may match the attributes of the second recognized data (e.g., flight information) with the attributes of the second field 1242 to fill in the second field 1242 with the second recognized data.

[0180] According to an embodiment of the present disclosure, when identified data related to the user's personal information is input in the first field 1241 or the second field 1242, the processor 160 can input the identified data into the first field 1241 or the second field 1242 after performing an additional authentication process (e.g., fingerprint authentication, iris information, face authentication).

[0181] According to various embodiments of the present disclosure, an electronic device (eg, Figure 1 The user terminal 100 may include a display (eg, Figure 1Display 140), camera, processor ( Figure 1 processor 160) and operably connected to the processor (e.g., Figure 1 of the processor 160) (e.g., Figure 1 Memory 150), wherein the memory (e.g., Figure 1 The memory 150 of the embodiment may store instructions that, when executed, cause the processor (e.g., Figure 1 Processor 160): receiving a first speech input; obtaining first recognized data from a first image; when the obtained first recognized data matches the first speech input, storing the first recognized data in association with the first speech input; activating a camera when the obtained first recognized data does not match the first speech input; acquiring second recognized data from a second image collected by the camera; and when the obtained second recognized data matches the first speech input, storing the second recognized data in association with the first speech input.

[0182] According to various embodiments of the present disclosure, the instruction may cause a processor (eg, Figure 1 The processor 160 analyzes the first image or the second image when the first voice input includes a designated keyword or a parameter for performing a task corresponding to the first voice input is omitted.

[0183] According to various embodiments of the present disclosure, instructions may enable a processor (e.g., Figure 1 The processor 160 executes a speech recognition application and processes the first speech input through the speech recognition application.

[0184] According to various embodiments of the present disclosure, the instruction may enable a processor (eg, Figure 1 The processor 160 determines a format of the first recognized data or a format of the second recognized data based on an attribute of the first utterance input.

[0185] According to various embodiments of the present disclosure, the instruction may enable a processor (eg, Figure 1 The processor 160 ) determines a screenshot of an execution screen of a most recently executed application as a first image.

[0186] According to various embodiments of the present disclosure, the instruction may cause a processor (eg, Figure 1 The processor 160 of the embodiment determines the first image by receiving the first image from a most recently executed application.

[0187] According to various embodiments of the present disclosure, the instruction may enable a processor (eg, Figure 1 The processor 160) determines the preview screen of the camera as the second image.

[0188] According to various embodiments of the present disclosure, the instruction may enable a processor (eg, Figure 1 When the processor 160 of the embodiment of the present invention does not recognize the second recognized data corresponding to the attribute of the first utterance input in the preview screen, the display (eg, Figure 1 User notification or image capture guidance is displayed on the display 140).

[0189] According to various embodiments of the present disclosure, the instruction may enable a processor (eg, Figure 1 The processor 160 of the embodiment of the present invention takes the image captured in response to the image capture input from the user as the second image.

[0190] According to various embodiments of the present disclosure, the instruction may enable a processor (eg, Figure 1 The processor 160 stores the additional information associated with the first identified data or the second identified data in association with the first identified data or the second identified data.

[0191] According to various embodiments of the present disclosure, the instruction may enable a processor (eg, Figure 1 The processor 160 stores a portion of the first identified data in association with the first application and stores another portion of the first identified data in association with the second application.

[0192] According to various embodiments of the present disclosure, the instruction may enable a processor (eg, Figure 1 The processor 160 ) displays the first recognized data or the second recognized data automatically or when receiving the second speech input.

[0193] According to various embodiments of the present disclosure, the instruction may enable a processor (eg, Figure 1 The processor 160 executes an application associated with a task corresponding to the first speech input and allows the application to enter a specified state using the first recognized data or the second recognized data.

[0194] According to various embodiments of the present disclosure, the instruction may enable a processor (eg, Figure 1 The processor 160 of the embodiment fills in fields included in the user interface of the application using the first recognized data or the second recognized data.

[0195] According to various embodiments of the present disclosure, electronic devices (eg, Figure 1 The user terminal 100 may include a display (eg, Figure 1 display 140), a camera, a processor (e.g., Figure 1 processor 160) and operably connected to the processor (e.g., Figure 1 of the processor 160) (e.g., Figure 1Memory 150), wherein the memory (e.g., Figure 1 The memory 150) may store instructions that, when executed, cause the processor to: execute a first application; obtain a first speech input using the first application; obtain first recognized data associated with the first speech input from a first image; store the first recognized data in association with the first speech data; and provide the obtained first recognized data so that the first recognized data can be used by a second application.

[0196] According to various embodiments of the present disclosure, the instruction may enable a processor (eg, Figure 1 The processor 160 of the present invention analyzes the first image, executes the first application based on the result of executing the first image, and provides a query corresponding to the analysis result to the user through the first application.

[0197] According to various embodiments of the present disclosure, the instruction may enable a processor (eg, Figure 1 The processor 160 of the embodiment of the present invention stores in a memory (for example, Figure 1 The first identified data is stored in a secure area of ​​the memory 150 ).

[0198] The speech recognition method according to various embodiments of the present disclosure may be used in an electronic device (e.g., Figure 1 The method is executed in a user terminal 100) and may include: receiving a first voice input; extracting first recognized data from a first image; when the extracted first recognized data matches the first voice input, storing the first recognized data in association with the first voice input; when the extracted first recognized data does not match the first voice input, activating a camera; extracting second recognized data from a second image collected by the camera; and when the extracted second recognized data matches the first speech input, storing the second recognized data in association with the first speech input.

[0199] According to various embodiments of the present disclosure, the extracting of the first recognized data may include determining a screenshot of an execution screen of a most recently executed application as the first image.

[0200] According to various embodiments of the present disclosure, the extracting of the second recognized data may include determining a preview screen of the camera as the second image.

[0201] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0202] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or alternative forms for the corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “coupled with another element (e.g., the second element)”, “coupled to another element (e.g., the second element)”, “connected with another element (e.g., the second element)”, or “connected to another element (e.g., the second element)” with or without the terms “operably” or “communicatively” being used, it means that the element may be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0203] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0204] The various embodiments described herein may be implemented as software (e.g., multiple applications 155) including one or more instructions stored in a storage medium readable by a machine (e.g., user terminal 100). For example, under the control of a processor, a processor (e.g., processor 160) of the machine (e.g., user terminal 100) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be run by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0205] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be published in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or the computer program product may be published online (e.g., downloaded or uploaded) via an application store (e.g., Play Store), or the computer program product may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a manufacturer's server, an application store's server, or a memory of a forwarding server).

[0206] The electronic device according to various embodiments of the present disclosure may provide a voice recognition service for integrally and collectively processing data analyzed through an image input and a user speech input.

[0207] The electronic device according to various embodiments of the present disclosure may determine a target of a user utterance input by analyzing a screenshot or an image collected through a camera input for an untargeted item in the user utterance input.

[0208] The electronic device according to various embodiments of the present invention can provide the user with additional information related to the user's speech input and image input.

[0209] According to various embodiments, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities. According to various embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., a module or a program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as a corresponding one of the multiple components performing one or more functions before integration. According to various embodiments, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more of the operations may be run or omitted in a different order, or one or more other operations may be added.

[0210] Although the present disclosure has been described with various embodiments, various changes and modifications may be suggested to one skilled in the art. The present disclosure is intended to include such changes and modifications as fall within the scope of the appended claims.

Claims

1. An electronic device for recognizing speech and comprising: microphone; monitor; camera; Memory; as well as processor; The processor is configured to: receiving a first speech input through the microphone, obtaining first data from a first image displayed on the display or stored in the memory, wherein the first data includes information of an object or text included in the first image, determining whether the first data matches the first utterance input by confirming whether an attribute of the first data matches a domain of the first utterance input, When the first data matches the first speech input, storing the first speech input in association with the first data, When the first data does not match the first speech input, activating the camera, obtaining second data from a second image collected by the camera, wherein the second data includes information of an object or text included in the second image, determining whether the second data matches the first utterance input by confirming whether an attribute of the second data matches a domain of the first utterance input, and When the second data matches the first speech input, storing the first speech input in association with the second data, Wherein, when the first speech input includes a specified keyword or a parameter for performing a task corresponding to the first speech input is omitted, the first image or the second image is analyzed.

2. The electronic device according to claim 1, wherein: The processor is further configured to: executing a speech recognition application, and The first utterance input is processed by the speech recognition application.

3. The electronic device according to claim 1, wherein: The processor is further configured to determine a format of the first data or a format of the second data based on a property of the first speech input.

4. The electronic device according to claim 1, wherein: The processor is further configured to determine a screenshot of an execution screen of a most recently executed application as the first image.

5. The electronic device according to claim 1, wherein: The processor is further configured to determine the first image based on an image received from a most recently executed application.

6. The electronic device according to claim 1, wherein: The processor is further configured to determine a preview screen of the camera as the second image.

7. The electronic device according to claim 6, wherein: The processor is further configured to display a user notification or an image capturing guide on the display when the second data corresponding to the attribute of the first utterance input is not recognized in the preview screen.

8. The electronic device according to claim 1, wherein: The processor is further configured to determine an image captured in response to an image capture input from a user as the second image.

9. The electronic device according to claim 1, wherein: The processor is further configured to store additional information associated with the first data or the second data together with the first data or the second data.

10. The electronic device according to claim 1, wherein: The processor is further configured to: storing a portion of the first data in association with a first application, and Another portion of the first data is stored in association with a second application.

11. The electronic device according to claim 1, wherein: The processor is further configured to display the first data or the second data automatically or upon receiving a second speech input.

12. The electronic device according to claim 1, wherein: The processor is further configured to: executing an application associated with a task corresponding to the first speech input, and The application is allowed to enter a specified state based on the first data or the second data.

13. The electronic device according to claim 12, wherein: The processor is further configured to fill in a field included in a user interface of the application based on the first data or the second data.

14. A speech recognition method performed in an electronic device, the speech recognition method comprising: Receiving a first speech input through a microphone of the electronic device; obtaining first data from a first image displayed on a display of the electronic device or stored in a memory of the electronic device, wherein the first data includes information of an object or text included in the first image; determining whether the first data matches the first utterance input by confirming whether an attribute of the first data matches a domain of the first utterance input; When the first data obtained from the first image matches the first speech input, storing the first speech input in association with the first data; activating a camera in the electronic device when the first data recognized from the first image does not match the first speech input; obtaining second data from a second image collected by the camera, wherein the second data includes information of an object or text included in the second image; determining whether the second data matches the first utterance input by confirming whether an attribute of the second data matches a domain of the first utterance input; and When the second data obtained from the second image matches the first utterance input, the first utterance input is stored in association with the second data.

Images