Electronic device and method for character recognition

Abstract

An electronic device according to an embodiment disclosed in this document may include: an imaging device for generating image data; a communication circuit; at least one processor operably connected to the imaging device and the communication circuit; and a memory operably connected to the processor for storing commands.

Description

Technical Field

[0001] The embodiments disclosed herein relate to character recognition technology. Background Art

[0002] Optical character recognition (OCR) refers to the technology that converts characters written by people or printed by machines into data that can be recognized by electronic devices (such as smartphones). For example, when an electronic device faces a sign, the electronic device can recognize the characters written on the sign through OCR technology.

[0003] OCR technology can be divided into embedded OCR technology performed in electronic devices and server OCR technology performed in servers. In embedded OCR technology, the processing speed is fast, but the accuracy may be reduced due to the performance limitations of electronic devices. On the other hand, in server OCR technology, the electronic device needs to send data to the server, resulting in low processing speed but achieving high accuracy. Summary of the invention

[0004] Technical issues

[0005] Recently, as more and more users require complex character recognition, the interest and demand for server OCR technology is also increasing. However, as mentioned above, in server OCR technology, the processing speed may be slow because the electronic device needs to send data to the server. In particular, when there is a large amount of data, it may take a long time to recognize the characters.

[0006] The embodiments disclosed in the present disclosure provide an electronic device for solving the above-mentioned problems and the problems proposed in the present disclosure.

[0007] Technical Solution

[0008] According to an embodiment disclosed herein, an electronic device may include an imaging device that generates image data, a communication circuit, at least one processor operably connected to the imaging device and the communication circuit; and a memory operably connected to the processor to store instructions, wherein the instructions, when executed, may cause the processor to receive first image data including a first image from the imaging device, send the first image data to a first server through the communication circuit, receive first text data including a first text recognized from the first image data from the first server through the communication circuit, receive second image data including a second image from the imaging device, the second image including a portion of the first image, and send the second text data to the first server through the communication circuit, the second text data including at least a portion of the first text data and a portion of the second image data, but not all of the second image data.

[0009] In addition, according to the embodiments disclosed herein, a method for recognizing characters in an electronic device may include: receiving first image data including a first image from an imaging device, sending the first image data to a first server through a communication circuit, receiving first text data including a first text recognized from the first image data from the first server through the communication circuit, receiving second image data including a second image from the imaging device, the second image including a portion of the first image, and sending second text data to the first server through the communication circuit, the second text data including at least a portion of the first text data and a portion of the second image data, but not all of the second image data.

[0010] In addition, according to the embodiments disclosed herein, the electronic device may include a camera, a display that outputs an image recognized by the camera, a communication circuit that sends the image to a first server, a processor electrically connected to the camera, the display, and the communication circuit; and a memory operably connected to the processor to store instructions and an application for capturing images, wherein the instructions, when executed, may cause the processor to capture a first image output through the display in response to user input for executing the application, send the first image to the first server, receive first text extracted from the first image from the first server, and output the first text through the display, send second text as first type data, the second text being located on an area of ​​the second image that overlaps with the first image and including at least a portion of the first text, and when a second image including at least a portion of the first image is output through the display, an area of ​​the second image that does not overlap with the first image is sent to the first server as second type data, and receive third text extracted from the first type data and the second type data from the first server, and output the third text to the display.

[0011] In addition, according to the embodiments disclosed herein, a method for recognizing characters in an electronic device may include: in response to user input, capturing a first image output through a display, for sending the first image to a first server, receiving a first text extracted from the first image from the first server, outputting the first text through the display, sending a second text as a first type of data, the second text being located on an area of ​​the second image that overlaps with the first image and including at least a portion of the first text, and when outputting a second image including at least a portion of the first image through the display, sending an area of ​​the second image that does not overlap with the first image as a second type of data to the first server, receiving a third text extracted from the first type of data and the second type of data from the first server, so as to output the third text to the display.

[0012] Beneficial Effects

[0013] According to the embodiments disclosed herein, character recognition time can be shortened.

[0014] Furthermore, various effects directly or indirectly understood by the present disclosure can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0016] Figure 2 is a block diagram of a process according to various embodiments.

[0017] Figure 3 A text recognition system according to an embodiment of the present disclosure is shown.

[0018] Figure 4A is a flowchart of the operation of a text recognition system according to an embodiment of the present disclosure.

[0019] Figure 4B is a flowchart of an operation of an electronic device according to an embodiment of the present disclosure.

[0020] Figure 5 A screen and a first image output through a display according to an embodiment of the present disclosure are shown.

[0021] Fig. 6A A first text according to an embodiment of the present disclosure is shown.

[0022] Figure 6B A first text according to another embodiment of the present disclosure is shown.

[0023] Fig. 7A is an example diagram for determining whether an image has changed in an electronic device according to an embodiment of the present disclosure.

[0024] Figure 7B is an example diagram for determining whether an image has changed according to another embodiment of the present disclosure.

[0025] Fig. 8A A first image according to an embodiment of the present disclosure is shown.

[0026] Figure 8B A second image according to an embodiment of the present disclosure is shown.

[0027] Fig. 9 An operating environment of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0028] Figure 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0029] refer to Figure 1In the network environment 100, the electronic device 101 may communicate with the electronic device 102 through the first network 198 (e.g., short-range wireless communication), or may communicate with the electronic device 104 or the server 108 through the second network 199 (e.g., long-range wireless communication). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input device 150, a sound output device 155, a display device 160, an audio module 170, a sensor module 176, an interface 177, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identification module 196, and an antenna module 197. According to some embodiments, at least one of the components of the electronic device 101 (e.g., the display device 160 or the camera module 180) may be omitted, or other components may be added to the electronic device 101. According to some embodiments, some components may be integrated and implemented as a case of a sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) embedded in a display device 160 (eg, a display).

[0030] The processor 120 may operate, for example, software (e.g., program 140) to control at least one of other components (e.g., hardware or software components) of the electronic device 101 connected to the processor 120, and may process and calculate various data. The processor 120 may load a command set or data received from other components (e.g., sensor module 176 or communication module 190) into the volatile memory 132, may process the loaded command or data, and may store the result data into the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit or an application processor) and an auxiliary processor 123 (e.g., a graphics processing device, an image signal processor, a sensor hub processor, or a communication processor), which operates independently of the main processor 121, additionally or alternatively uses less power than the main processor 121, or is designated as a designated function. In this case, the auxiliary processor 123 may be operated separately from the main processor 121 or embedded.

[0031] In this case, when the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 may control at least some functions or states associated with, for example, at least one component (e.g., display device 160, sensor module 176, or communication module 190) among the components of the electronic device 101 instead of the main processor 121, or when the main processor 121 is in an active (e.g., application execution) state, the auxiliary processor 123 controls together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., image signal processor or communication processor) may be implemented as a part of another component (e.g., camera module 180 or communication module 190) functionally related to the auxiliary processor 123. The memory 130 may store various data used by at least one component (e.g., processor 120 or sensor module 176) of the electronic device 101, for example, software (e.g., program 140) and input data or output data about commands associated with the software. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0032] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or applications 146 .

[0033] The input device 150 may be a device for receiving a command or data for a component (eg, the processor 120 ) of the electronic device 101 from outside (eg, a user) of the electronic device 101 , and may include, for example, a microphone, a mouse, or a keyboard.

[0034] The sound output device 155 may be a device for outputting sound signals to the outside of the electronic device 101, and may include, for example, a speaker for general purposes (such as multimedia playback or recording playback), and a receiver only for receiving calls. According to an embodiment, the receiver and the speaker may be implemented integrally or separately.

[0035] The display device 160 may be a device for visually presenting information to a user of the electronic device 101, and may include, for example, a display, a holographic device or a projector, and a control circuit for controlling a corresponding device. According to an embodiment, the display device 160 may include a touch circuit or a pressure sensor for measuring the pressure intensity of a touch.

[0036] The audio module 170 can bidirectionally convert sound and electrical signals. According to an embodiment, the audio module 170 can obtain sound through the input device 150, or can output sound through a wired or wireless connection to an external electronic device (e.g., electronic device 102 (e.g., speaker or earphone)) of the sound output device 155 or the electronic device 101.

[0037] The sensor module 176 may generate an electrical signal or data value corresponding to an operating state (e.g., power or temperature) inside the electronic device 101 or an external environmental state. The sensor module 176 may include, for example, a posture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0038] The interface 177 may support a specified protocol for wired or wireless connection to an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, an HDMI (high-definition multimedia interface), a USB (universal serial bus) interface, an SD card interface, or an audio interface.

[0039] The connection terminal 178 may include a connector that physically connects the electronic device 101 to an external electronic device (eg, the electronic device 102 ), such as an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

[0040] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation sensed by the user through tactile or kinesthetic sense. The haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0041] The camera module 180 may capture still images or video images. According to an embodiment, the camera module 180 may include, for example, at least one lens, an image sensor, an image signal processor, or a flash.

[0042] The power management module 188 may be a module for managing power supplied to the electronic device 101 and may function as at least a part of a power management integrated circuit (PMIC).

[0043] The battery 189 may be a device for supplying power to at least one component of the electronic device 101 , and may include, for example, a non-rechargeable (primary) battery, a rechargeable (secondary) battery, or a fuel cell.

[0044] The communication module 190 can establish a wired or wireless communication channel between the electronic device 101 and an external electronic device (e.g., the electronic device 102, the electronic device 104, or the server 108), and support communication execution through the established communication channel. The communication module 190 may include at least one communication processor that operates independently of the processor 120 (e.g., the application processor) and supports wired communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module 194 (e.g., a LAN (local area network) communication module or a power line communication module), and can communicate with an external electronic device through a first network 198 (e.g., a short-range communication network such as Bluetooth, WiFi direct connection, or IrDA (infrared data association)) or a second network 199 (e.g., a long-range wireless communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or WAN)) using a corresponding communication module therein. The various communication modules 190 described above may be implemented in one chip or in separate chips, respectively.

[0045] According to an embodiment, the wireless communication module 192 may identify and authenticate the electronic device 101 using user information stored in the subscriber identification module 196 in the communication network.

[0046] The antenna module 197 may include one or more antennas to send signals or power to an external source or receive signals or power from an external source. According to an embodiment, the communication module 190 (e.g., the wireless communication module 192) may send signals to an external electronic device or receive signals from an external electronic device through an antenna suitable for a communication method.

[0047] Some of the components can be connected to each other through a communication method used between peripheral devices (e.g., a bus, GPIO (general purpose input / output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) to exchange signals (e.g., commands or data) with each other.

[0048] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 by a server 108 connected to the second network 199. Each of the electronic devices 102 and 104 may be of the same or different type as the electronic device 101. According to an embodiment, all or some of the operations performed by the electronic device 101 may be performed by another electronic device or a plurality of external electronic devices. When the electronic device 101 automatically or by request performs some functions or services, in addition to performing the functions or services by itself or in place of performing the functions or services by itself, the electronic device 101 may request the external electronic device to perform at least some of the functions related to the functions or services. The external electronic device receiving the request may perform the requested functions or additional functions and send the results to the electronic device 101. The electronic device 101 may provide the requested functions or services as is or after additionally processing the received results based on the received results. To this end, for example, cloud computing, distributed computing or client-server computing technology may be used.

[0049] Figure 2 is a block diagram illustrating a procedure according to various embodiments.

[0050] According to an embodiment, the program 140 may include an operating system 142 for controlling one or more resources of the electronic device 101, middleware 144, or an application 146 executable in the operating system 142. The operating system 142 may include, for example, an Android TM 、iOS TM , Windows TM , Symbian TM 、Tizen TM or Bada TM For example, at least some of the programs 140 may be preloaded on the electronic device 101 when manufactured, or downloaded from an external electronic device (eg, the electronic device 102 or 104, or the server 108) or updated by the external electronic device when used by the user.

[0051] The operating system 142 may control the management (e.g., allocation or retrieval) of one or more system resources (e.g., processes, memory, or power) of the electronic device 101. The operating system 142 may additionally or alternatively include one or more drivers for driving other hardware devices of the electronic device 101, such as an input device 150, a sound output device 155, a display device 160, an audio module 170, a sensor module 176, an interface 177, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identification module 196, or an antenna module 197.

[0052] The middleware 144 may provide various functions to the application 146 so that functions or information provided by one or more resources of the electronic device 101 may be used by the application 146. The middleware 144 may include, for example, an application manager 201, a window manager 203, a multimedia manager 205, a resource manager 207, a power manager 209, a database manager 211, and a package manager 213, a connection manager 215, a notification manager 217, a location manager 219, a graphics manager 221, a security manager 223, a telephony manager 225, or a speech recognition manager 227.

[0053] For example, the application manager 201 can manage the life cycle of the application 146. The window manager 203 can manage, for example, one or more GUI resources used in the screen. For example, the multimedia manager 205 can identify one or more formats required for the reproduction of the media file, and encode or decode the corresponding media file among the media files by using a codec corresponding to the selected format among the formats. The resource manager 207 can manage, for example, the source code of the application 146 or the memory space of the memory 130. The power manager 209 can manage, for example, the capacity, temperature, or power of the battery 189, and use corresponding information among these information to determine or provide relevant information required for the operation of the electronic device 101. According to an embodiment, the power manager 209 can interoperate with a basic input / output system (BIOS) (not shown) of the electronic device 101.

[0054] For example, the database manager 211 can create, search or change the database to be used by the application 146. For example, the package manager 213 can manage the installation or update of applications distributed in the form of package files. The connection manager 215 can manage, for example, a wireless connection or a direct connection between the electronic device 101 and an external electronic device. For example, the notification manager 217 can provide a function for notifying the user of the occurrence of a specified event (e.g., an incoming call, a message, or an alarm). For example, the location manager 219 can manage the location information of the electronic device 101. The graphic manager 221 can manage, for example, one or more graphic effects to be provided to the user, or a user interface related thereto.

[0055] For example, the security manager 223 can provide system security or user authentication. The phone manager 225 can manage, for example, a voice call function or a video call function provided by the electronic device 101. The voice recognition manager 227 can send, for example, the user's voice data to the server 108, and receive commands corresponding to the functions to be executed in the electronic device based on at least a portion of the voice data or text data generated from the conversion of at least a portion of the voice data. According to an embodiment, the middleware 244 can dynamically delete some of the existing components or add new components. According to an embodiment, at least a portion of the middleware 144 can be included as part of the operating system 142, or can be implemented as separate software different from the operating system 142.

[0056] Application 146 may include, for example, home page 251, dialer 253, SMS / MMS 255, instant message (IM) 257, browser 259, camera 261, alarm 263, contact 265, voice recognition 267, email 269, calendar 271, media player 273, album 275, watch 277, health 279 (e.g., measurement of biological information such as exercise volume or blood sugar) or environmental information 281 (e.g., measurement of atmospheric pressure, humidity, or temperature information). According to an embodiment, application 146 may also include an information exchange application (not shown) capable of supporting information exchange between electronic device 101 and external electronic devices. Information exchange applications may include, for example, a notification relay application configured to transmit specified information (e.g., call, message or alarm) to an external electronic device, or a device management application configured to manage an external electronic device. For example, a notification relay application may transmit notification information corresponding to a specified event (e.g., mail reception) occurring in another application (e.g., email application 269) of electronic device 101 to an external electronic device. Additionally or alternatively, the notification relay application may receive notification information from an external electronic device and provide the notification information to a user of electronic device 101 .

[0057] The device management application may control, for example, power (e.g., turning on or off) or functionality (e.g., brightness, resolution, or focus of the display device 160 or the camera module 180) of an external electronic device that communicates with the electronic device 101 or a component thereof (e.g., the display device 160 or the camera module 180). The device management application may additionally or alternatively support installation, deletion, or updating of applications operating on the external electronic device.

[0058] The electronic device according to various embodiments disclosed in the present disclosure may be various types of devices. The electronic device may include, for example, at least one of a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a mobile medical appliance, a camera, a wearable device, or a household appliance. The electronic device according to an embodiment of the present disclosure should not be limited to the above-mentioned devices.

[0059] It should be understood that the various embodiments of the present disclosure and the terms used in the embodiments are not intended to limit the technology disclosed in the present disclosure to the specific form disclosed herein; on the contrary, the present disclosure should be interpreted as covering various modifications, equivalents and / or substitutes of the embodiments of the present disclosure. With regard to the description of the drawings, similar components may be assigned similar reference numerals. As used herein, the singular form may also include the plural form, unless the context clearly indicates otherwise. In the present disclosure disclosed herein, the expressions "A or B", "at least one of A or / and B", "A, B or C" or "one or more of A, B or / and C" etc. used herein may include any and all combinations of one or more associated listed items. The expressions "first", "second", "the first" or "the second" used herein may refer to various components, regardless of their order and / or importance, but do not limit the corresponding components. The above expressions are only used to distinguish one component from other components. It should be understood that when a component (e.g., a first component) is referred to as being (operably or communicatively) “connected” or “coupled” to another component (e.g., a second component), the component may be directly connected or coupled to the other component, or any other component (e.g., a third component) may be interposed between the component and the other component.

[0060] The term "module" as used herein may refer to a unit including, for example, one or more combinations of hardware, software, and firmware. The term "module" may be used interchangeably with the terms "logic," "logic block," "part," and "circuit." A "module" may be the smallest unit of an integrated part, or may be a part thereof. A "module" may be the smallest unit for performing one or more functions or a part thereof. For example, a "module" may include an application-specific integrated circuit (ASIC).

[0061] Various embodiments of the present disclosure may be implemented by software (e.g., program 140) including instructions stored in a machine-readable storage medium (e.g., internal memory 136 or external memory 138) readable by a machine (e.g., a computer). A machine may be a device that calls instructions from a machine-readable storage medium and operates depending on the called instructions, and may include an electronic device (e.g., electronic device 101). When the instructions are executed by a processor (e.g., processor 120), the processor may perform functions corresponding to the instructions directly or using other components under the control of the processor. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" used herein is a limitation on the medium itself (i.e., tangible, not a signal), not a limitation on the persistence of data storage.

[0062] According to an embodiment, the method according to various embodiments disclosed in the present disclosure may be provided as part of a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be available only through an application store (e.g., Play Store TM In the case of online distribution, at least a portion of the computer program product may be temporarily stored or generated in a storage medium (such as a memory of a manufacturer's server, an application store's server, or a relay server).

[0063] Each component (e.g., module or program) according to various embodiments may include at least one of the above-mentioned components, and may omit a portion of the above-mentioned subcomponents, or may further include additional subcomponents. Alternatively or additionally, some components (e.g., modules or programs) may be integrated in one component, and may perform the same or similar functions performed by each corresponding component before integration. The operations performed by the modules, programming or other components according to various embodiments of the present disclosure may be performed sequentially, in parallel, repeatedly or in a heuristic manner. In addition, at least some operations may be performed, omitted or other operations may be added in different orders.

[0064] While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

[0065] Figure 3 A text recognition system according to an embodiment of the present disclosure is shown.

[0066] refer to Figure 3 , the text recognition system 300 may include an electronic device (e.g., Figure 1 electronic device 101), a first server 310, a second server 320, a third server 330 and a fourth server 340.

[0067] The electronic device 101 can capture an image and send the image to the first server 310. For example, a camera (e.g., Figure 1 The camera module 180 may be provided on one surface (eg, front surface or rear surface) of the electronic device 101, and the user may photograph the user's surroundings through the camera module 180. The electronic device 101 may transmit the photographed image to the first server 310.

[0068] The first server 310 may extract text from the captured image and send the extracted text to the electronic device 101. For example, when the captured image is a signboard, the first server 310 may extract a store name, a phone number, etc. written on the signboard. In another embodiment, when the captured image represents a blackboard, the first server 310 may extract letters written on the blackboard. The extracted store name, phone number, letters, etc. may be sent to the electronic device 101. In the present disclosure, the first server 310 may be referred to as an optical character recognition (OCR) server.

[0069] The electronic device 101 may send the text received from the first server 310 to at least one of the second server 320, the third server 330, and the fourth server 340. For example, when letters are included in the text, the electronic device 101 may send the text to the second server 320. In another embodiment, when physical quantity units are included in the text, the electronic device 101 may send the text to the third server 330. In yet another embodiment, when currency units are included in the text, the electronic device 101 may send the text to the fourth server 340. The above embodiments are merely embodiments, and the configuration included in the text, the server to which the electronic device 101 sends the text, and the like are not limited to the above embodiments.

[0070] When the electronic device 101 sends a text to the second server 320, the second server 320 may translate the letters included in the text. The translated letters may be sent back to the electronic device 101. For example, when Chinese is included in the text, the second server 320 may translate Chinese into Korean and send the text to the electronic device 101. In another embodiment, when English is included in the text, the second server 320 may translate English into Korean and send the text to the electronic device 101. In the present disclosure, the second server 320 may be referred to as a translation server.

[0071] The third server 330 may change the physical quantity unit. The changed physical quantity unit may be sent back to the electronic device 101. For example, when the text includes miles, the third server 330 may change the miles to kilometers and send the converted kilometers to the electronic device 101. In another embodiment, when the text includes gallons, the third server 330 may change the gallons to liters (liters, l) and send the converted liters to the electronic device 101. In the present disclosure, the third server 330 may be referred to as a unit conversion server.

[0072] The fourth server 340 may change the currency unit. The changed currency unit may be sent back to the electronic device 101. For example, when the text includes US dollars ($), the fourth server 340 may change the US dollars into Korean won. The changed Korean won is sent to the electronic device 101. In another embodiment, when the text includes Japanese yen (¥), the fourth server 340 may change Japanese yen into Korean won. And the changed Korean won is sent to the electronic device 101. In the present disclosure, the fourth server 340 may be referred to as a currency conversion server.

[0073] When at least one of the translated letters, the changed physical quantity units, and the changed currency units is received from the second to fourth servers 320 to 340, the electronic device 101 may output the received text. For example, when the image captured by the camera module 180 includes "Hello", the electronic device 101 may output It's the translated letters, not "Hello".

[0074] In another embodiment, the area where the received text is output may be adjacent to the image captured by the camera module 180. For example, when the image captured by the camera module 180 includes "Hello", the electronic device 101 may output the received text in an area adjacent to the image captured by the camera module 180.

[0075] In this disclosure, reference is made to Figures 1 to 3 The content described can be applied to Figures 1 to 3 Components of the electronic device 101 and the text recognition system 300 are shown with the same reference numerals.

[0076] Figure 4A is a flowchart of the operation of a text recognition system according to an embodiment of the present disclosure. Figure 4A is used to describe in detail the Figure 3 A diagram of the operations of the components in . Figure 4AIn operation 411, the electronic device 101 (e.g., the processor 120) may capture a first image. For example, the electronic device 101 may execute an application for recognizing text in response to a user input. When the application is executed, the electronic device 101 may capture the user's surroundings through the camera module 180. In addition, the electronic device 101 may capture the surroundings. In the present disclosure, the first image may refer to a first image captured through a display (e.g., Figure 1 The image captured by the display device 160).

[0077] In operation 413, the electronic device 101 (eg, Figure 1 The communication module 190) can send the captured first image to the first server 310.

[0078] In operation 415, the first server 310 may extract the first text from the first image. For example, when a signboard is included in the first image, the first server 310 may extract a store name, a phone number, etc. written on the signboard.

[0079] In operation 417, the first server 310 may transmit the extracted first text to the electronic device 101. In the above embodiment, the first server 310 may transmit the extracted store name, phone number, etc. to the electronic device 101. According to an embodiment, the first server 310 may transmit the extracted first text to the electronic device 101 as data in JavaScript object notation (JSON) format.

[0080] In operation 419, the electronic device 101 may receive the first text from the first server 310 and transmit the received first text to the second server 320. Figure 4A The electronic device 101 is shown as sending the first text to the second server 320 , but the electronic device 101 may send the first text to the third server 330 and / or the fourth server 340 .

[0081] In operation 421, the second server 320 may translate the first text received from the electronic device 101. In the above embodiment, when the first text includes a Chinese store name, the second server 320 may translate the Chinese store name into a Korean store name.

[0082] In operation 423, the second server 320 may transmit the translated first text to the electronic device 101. In the above embodiment, the second server 320 may transmit the Korean store name to the electronic device 101.

[0083] In operation 425, the electronic device 101 may output the translated first text. In one embodiment, the electronic device 101 may output the translated first text to replace the first text. For example, the electronic device 101 may output the Korean store name by replacing the Chinese store name.

[0084] In operation 427, the electronic device 101 may determine whether the image output through the display (e.g., display device 160) changes. For example, when the user moves the electronic device 101, the image output through the camera module 180 may also change. Therefore, the second image may be output instead of the first image.

[0085] In the present disclosure, the second image may refer to an image that includes at least a portion of the first image. For example, when a signboard is included in the first image, at least a portion of the signboard may be included in the second image. In other words, when the user moves the electronic device 101 within a specific range, a second image that includes at least a portion of the first image may be output. However, when the user rotates the electronic device 101 in the opposite direction, only the environment on the opposite side of the first image may be output through the display (e.g., display device 160), and the first image and the second image may not be output.

[0086] When outputting the second image, in operation 429 , the electronic device 101 may capture the second image.

[0087] In operation 431, the electronic device 101 may compare the first image and the second image. The electronic device 101 may send a second text to the first server 310, the second text including a portion of the first text corresponding to an area in the second image that overlaps with the first image. In addition, the electronic device 101 may send a portion of the second image to the first server 310, the portion of the second image corresponding to an area in the second image that does not overlap with the first image. For example, the first image may include a signboard "A", and the second image may include a portion of the signboard "A" and a signboard "B". The electronic device 101 may send a portion of the signboard "A" as the first type of data, the portion of the signboard "A" being an area in the second image that overlaps with the first image. The electronic device 101 may send a signboard "B" as the second type of data, the signboard "B" being an area in the second image that does not overlap with the first image. In the present disclosure, the first type of data and the second type of data have different formats, and the first type of data may include the second text. The second type of data may include an area in the second image that does not overlap with the first image.

[0088] According to an embodiment, the size of the first type of data may be smaller than the size of the second type of data. For example, the first type of data may be sent to the first server 310 as data having a javascript object notation (JSON) format. The second type of data may be sent to the first server 310 as image data (e.g., JPG data).

[0089] at the same time, Figure 4A The embodiments shown are merely embodiments, and the embodiments of the present disclosure are not limited to Figure 4A For example, the electronic device 101 may not transmit the area of ​​the second image that overlaps with the first image, and may transmit the area of ​​the second image that does not overlap with the first image.

[0090] According to an embodiment of the present disclosure, the first type data may have a data format having a smaller size, thereby reducing the size of data sent by the electronic device 101 to the first server 310. Therefore, the time required for the electronic device 101 to output text may be shortened.

[0091] In operation 433, the first server 310 may extract the third text from the first type data and the second type data. For example, when the first image includes a signboard "A" and the second image includes a portion of the signboard "A" and a signboard "B", the first server 310 may extract the third text from the first type data and the second type data corresponding to a store name, a telephone number, etc. written on a portion of the signboard "A" and the signboard "B".

[0092] In operation 435, the first server 310 may transmit the extracted third text to the electronic device 101. In the above embodiment, the first server 310 may transmit the store name, phone number, etc. extracted from a part of the sign "A" and the sign "B" to the electronic device 101.

[0093] In another embodiment, in operation 435, the first server 310 may send the extracted third text to the second server 320. The second server 320 may translate the third text and send the translated third text to the electronic device 101. The electronic device 101 may send the translated third text to the third server 330 and / or the fourth server 340. The third server 330 and / or the fourth server 340 may change the physical quantity unit and / or the monetary unit and send the changed physical quantity unit and / or the monetary unit to the electronic device 101.

[0094] In operations 437 to 441, the electronic device 101 may transmit the third text to the second server 320, and the second server 320 may translate the third text. The translated third text may be transmitted to the electronic device 101. The description of operations 419 to 423 may also be applied to operations 437 to 441.

[0095] In operation 443, the electronic device 101 may receive the translated third text from the second server 320. The electronic device 101 may output the third text. In the above embodiment, the electronic device 101 may output the store name written on a portion of the sign "A" and the store name written on the sign "B".

[0096] Figure 4B is a flowchart of an operation of an electronic device according to an embodiment of the present disclosure. Figure 4B It is used to describe in detail Figure 3 A diagram illustrating the operation of electronic device 101 is shown.

[0097] refer to Figure 4B In operation 451, the processor 120 may receive first image data from an imaging device. In the present disclosure, the imaging device may be referred to as a camera module 180 as a device capable of generating first image data. The first image data may refer to image data including a first image output through the display device 160.

[0098] In operation 453, the processor 120 may transmit the first image data to the first server 310 through the communication module 190. The first server 310 may extract the first text from the first image data and transmit the first text data including the extracted first text back to the electronic device 101. For example, when a signboard is included in the first image, the first server 310 may extract the store name written on the signboard, etc. The first server 310 may be referred to as an optical character recognition (OCR) server.

[0099] In operation 455, the processor 120 may receive the first text data through the communication module 190. When the first text data is received, the processor 120 may output the first text through the display device 160. For example, when a signboard is included in the first image, the processor 120 may output the store name written on the signboard through the display device 160.

[0100] In operation 457, the processor 120 may receive second image data from the imaging device. The second image data may refer to image data including a second image at least partially overlapping the first image.

[0101] In operation 459, the processor 120 may transmit a portion of the second text data and the second image data to the first server 310 through the communication module 190. In this document, the second text data may refer to text data including at least a portion of the first text data in an area where the first image and the second image overlap each other. That is, in operation 459, the processor 120 may transmit an area of ​​the second text data and the second image data that does not overlap with the first image data to the first server 310.

[0102] Figure 5 A screen and a first image output through a display according to an embodiment of the present disclosure are shown. Figure 5 is a diagram for describing operations 411 and 413 shown in FIG. 4 in detail.

[0103] refer to Figure 5 , electronic devices (e.g. Figure 1 The electronic device 101 may execute an application for recognizing text in response to user input. When the application is executed, the electronic device 101 may use a camera (e.g., Figure 1 The camera module 180) captures the user's surroundings. Figure 5 As shown, the user can take a photo of a signboard 510 in a travel destination through the electronic device 101. The electronic device 101 can output the signboard 510 through a display screen.

[0104] According to an embodiment, the electronic device 101 may capture a first image 511. The first image 511 may refer to an image output through a display screen. When the first image 511 is captured, the electronic device 101 may send the first image 511 to the first server 310.

[0105] Fig. 6A A first text according to an embodiment of the present disclosure is shown. Figure 6B A first text according to another embodiment of the present disclosure is shown. Fig. 6A and Figure 6B is a diagram for describing operation 415 shown in FIG. 4 in detail.

[0106] refer to Fig. 6A and Figure 6B , the first server (for example, Figure 3 The first server 310 may obtain the first image (eg, Figure 5 The first image 511 of FIG. 5 is used to extract the first text 610 having different formats. The extracted first text 610 may be sent to an electronic device (eg, Figure 1 For example, Fig. 6AAs shown, the first server 310 can extract the first text 610, the position information of the first text 610, and the position information of each of the words included in the first text 610 from the first image, and send these to the electronic device 101. In this case, the first text 610, the position information, and the position information of each word can be sent to the electronic device in a first type data format (e.g., JSON format).

[0107] In another embodiment, if Figure 6B As shown, the first server 310 can extract the first text 610 from the first image and separately display the first text 610. For example, the first server 310 can analyze the first text 610 in sentence elements and display the first text 610 in units of blocks 621, lines 622, words 623, and / or letters 624.

[0108] According to an embodiment, when sending the first image, the electronic device 101 may send a command for determining the format of the first text 610 to the first server 310 together. The first server 310 may extract the first text 610 based on the command. For example, the first server 310 may extract the first text 610, the position information, and the position information of each word based on the command and send them to the electronic device 101. In another embodiment, the first server 310 may divide the first text 610 into blocks 621, lines 622, words 623, and / or letters 624, and send the first text 610 to the electronic device 101.

[0109] Fig. 7A is an example diagram for determining whether an image has changed in an electronic device according to an embodiment of the present disclosure. Figure 7B is an example diagram for determining whether an image has changed according to another embodiment of the present disclosure. Fig. 7A and Figure 7B It is used to describe in detail Figure 4A A diagram of operation 427 is shown.

[0110] refer to Fig. 7A , electronic devices (e.g. Figure 1 The electronic device 101 may compare the images outputted through the display. When there is no significant change as a result of the comparison, the electronic device 101 may not send the changed image to the first server (eg, Figure 3 For example, when comparing the left image 710 and the right image 720, the text included in the left image 710 may be substantially the same as the text included in the right image 720. The electronic device 101 may not send the right image 720 to the first server 310 because the text included in the left image 710 may be substantially the same as the text included in the right image 720.

[0111] refer to Figure 7B , when viewed through a display (e.g., Figure 1 When the image output by the display device 160 of the electronic device 101 changes, the electronic device 101 can send the changed image to the first server 310. For example, the user can move the electronic device 101 from the left direction 730 to the right direction 740. As the electronic device moves, the screen output by the display can also change. For example, as the electronic device 101 moves, the first image 731 can be output in the left direction 730, and the second image 741 can be output in the right direction 740. The second image 741 may include an area 741a overlapping with the first image and an area 741b not overlapping with the first image.

[0112] Fig. 8A A first image according to an embodiment of the present disclosure is shown. Figure 8B A second image according to an embodiment of the present disclosure is shown. Fig. 8A and Figure 8B is a diagram for describing operation 431 shown in FIG. 4 in detail.

[0113] refer to Fig. 8A and Figure 8B , electronic devices (e.g. Figure 1 The electronic device 101) may compare the first image 810 (eg, Fig. 7A 731) and the second image 820 (eg, Figure 7B The second image 741 of FIG. 820 is generated, and the area 820a where the second image 820 overlaps with the first image 810 is sent to the first server (eg, Figure 3 The electronic device 101 may send the area 820b of the second image 820 that does not overlap with the first image 810 to the first server 310 as the second type data 820-2. For example, the left area 820a of the second image 820 may overlap with the first image 810, but the right area 820b of the second image 820 may not overlap with the first image 810. Therefore, the electronic device 101 may send the left area 820a as the data 820-1 having the JSON format to the first server 310. The electronic device 101 may send the right area 820b as the image data 820-2 (e.g., JPG data) to the first server 310.

[0114] at the same time, Fig. 8A and Figure 8B The embodiments shown are merely embodiments, and the embodiments of the present disclosure are not limited to Fig. 8A and Figure 8BFor example, the electronic device 101 may not send the left area 820 a of the second image 820 , but may send the right area 820 b of the second image 820 to the first server 310 .

[0115] According to an embodiment of the present disclosure, the image that has been sent to the first server 310 can be sent to the server in a small data format, or not sent to the server, thereby reducing the size of the data sent by the electronic device 101. Therefore, the time required for the electronic device 101 to output text can be shortened.

[0116] According to an embodiment, after receiving the second image 820, the first server 310 may extract the third text from the area 820b where the second image 820 does not overlap with the first image 810. For example, because "Baby I am sorry" in the second image 820 is extracted from the first image 810, the first server 310 may not extract "Baby I am sorry" but "but I love you". According to the text recognition system based on the comparative example, even when there is an overlapping area in the image, all text included in the image may be extracted. However, according to the text recognition system based on the embodiment of the present disclosure (e.g., Figure 3 The text recognition system 300 of the present invention can shorten the text recognition time by extracting only the text in the non-overlapping area (eg, 820b).

[0117] Fig. 9 An operating environment of an electronic device according to an embodiment of the present disclosure is shown. Fig. 9 is a diagram for describing operation 431 shown in FIG. 4 in detail.

[0118] refer to Fig. 9 , users can use electronic devices (e.g. Figure 1 The electronic device 101 of the present invention can capture an environment in which a lecturer (hereinafter referred to as subject 911) is giving a lecture in front of a blackboard during class time. In this case, a portion of the blackboard may be blocked by the subject 911. The electronic device 101 can capture the captured environment as a first image 910. The captured first image 910 can be sent to a first server (e.g., Figure 3 The first server 310), and the first server 310 can extract the text in the area 910a that is not blocked by the object 911.

[0119] According to an embodiment, as the object 911 moves, the area 910b blocked by the object 911 may change. In this case, the electronic device 101 may capture the screen as the second image 920 in a state where the object 911 has moved. The electronic device 101 may compare the first image 910 and the second image 920, and as a result of the comparison, send the area 920a where the first image 910 overlaps with the second image 920 as the first type of data (e.g., data in JSON format) to the first server 310. The electronic device 101 may send the non-overlapping area 920b obtained according to the comparison to the first server 310 as the second type of data (e.g., JPG data). The first server 310 may extract text from the non-overlapping area 920b.

[0120] at the same time, Fig. 9 The embodiment shown is only one embodiment, and the embodiments of the present disclosure are not limited to Fig. 9 For example, an area where the first image 910 overlaps with the second image may not be sent to the first server 310 , and an area where the first image 910 does not overlap with the second image may be sent to the first server 310 .

[0121] According to an embodiment of the present disclosure, an image that has been sent to a server may be sent to the server in a small data format, or may not be sent to the server, thereby reducing the size of data sent by the electronic device 101. Therefore, the time required for the electronic device 101 to output text may be shortened.

Claims

1. An electronic device, comprising: an imaging device configured to generate image data; Communication circuits; at least one processor operably connected to the imaging device and the communication circuit; as well as a memory operatively connected to the processor to store instructions, Wherein, when the instructions are executed, the processor: receiving first image data including a first image from the imaging device; sending the first image data to a first server via the communication circuit; receiving first text data from the first server via the communication circuit, the first text data comprising first text recognized from the first image data; receiving, from the imaging device, second image data including a second image, the second image including a first area overlapping a portion of the first image and a second area not overlapping the first image; transmitting the second image data to the first server through the communication circuit by transmitting the first area as first type data and transmitting the second area as second type data different from the first type data; as well as Third text data including third text recognized from the first type data and the second type data of the transmitted second image data is received from the first server through the communication circuit.

2. The electronic device according to claim 1, in, The first type of data includes javascript object notation JSON format; and The second type of data includes image format.

3. The electronic device according to claim 1, wherein: The size of the first type of data is smaller than the size of the second type of data.

4. The electronic device according to claim 3, wherein: The first text data includes data regarding at least one coordinate associated with the first text in the first image.

5. The electronic device according to claim 1, further comprising: a display configured to output an image recognized by the imaging device, wherein the imaging device is a camera, Wherein, when the instructions are executed, the processor: In response to a user input for executing an application for capturing an image, the first image output through the display is captured.

6. The electronic device according to claim 5, wherein: The instructions, when executed, cause the processor to: The third text data is received from the first server to output the third text through the display.

7. The electronic device according to claim 5, wherein: The instructions, when executed, cause the processor to: sending the first text to a second server; and A first translated text obtained by converting the first text into a designated language is received from the second server, and the first translated text is output through the display.

8. The electronic device according to claim 7, wherein: The instructions, when executed, cause the processor to: sending the third text to the second server; and A second translated text obtained by converting the third text into a designated language is received from the second server, and the second translated text is output through the display.

9. The electronic device according to claim 5, wherein: The first text includes a first physical quantity unit, and Wherein, when the instructions are executed, the processor sends the first text to a second server, receives from the second server a second physical quantity unit obtained by changing the first physical quantity unit to a specified unit, and outputs the second physical quantity unit through the display.

10. The electronic device according to claim 5, wherein: The first text includes a first currency unit, and Wherein, the instructions, when executed, cause the processor to send the first text to a second server, receive from the second server a second currency unit obtained by changing the first currency unit into a designated unit, and output the second currency unit through the display.

11. The electronic device according to claim 5, wherein: The first image includes an image captured at a first point in time, the second image includes an image captured at a second point in time, and the second image data includes data visible in the second image and obscured by an external object in the first image.

12. A method for recognizing characters in an electronic device, comprising: receiving first image data including a first image from an imaging device; sending the first image data to a first server via a communication circuit; receiving first text data from the first server via the communication circuit, the first text data comprising first text recognized from the first image data; receiving, from the imaging device, second image data including a second image, the second image including a first area overlapping a portion of the first image and a second area not overlapping the first image; transmitting the second image data to the first server through the communication circuit by transmitting the first area as first type data and transmitting the second area as second type data different from the first type data; as well as Third text data including third text recognized from the first type data and the second type data of the transmitted second image data is received from the first server through the communication circuit.

13. The method according to claim 12, in, The first type of data includes javascript object notation JSON format; and The second type of data includes image format.

14. The method according to claim 12, in, The size of the first type of data is smaller than the size of the second type of data.

15. The method according to claim 14, wherein: The first text data includes data regarding at least one coordinate associated with the first text in the first image.

Images