Method and electronic device for providing text translation management data related to application

Abstract

Certain embodiments of the present disclosure relate to an apparatus and method for translating text included in an image in an electronic device using an external electronic device. The method includes: displaying a picture on a display, the picture including an object carrying text at a location within the picture; extracting the text; generating another text from the extracted text; and automatically overlaying the other text on the object in another picture on the display, the other picture including the object at another location within the other picture.

Description

Technical Field

[0001] Certain embodiments of the present disclosure relate to an apparatus and method for providing a translation service of text included in an image obtained through a camera in an electronic device. Background Art

[0002] With the enhancement of information and communication technology and semiconductor technology, various types of electronic devices are developing into multimedia devices that provide various multimedia services. For example, the multimedia services may include at least one of cellular phone services, Voice over IP (VOIP) services, messaging services, broadcast services, wireless Internet services, cameras, electronic payment, or media playback.

[0003] The above information is presented as background information only to assist with understanding the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with respect to the present disclosure. Summary of the Invention

[0004] Solution to the problem

[0005] Electronic devices can use cameras to provide various services to enhance user convenience. For example, an electronic device can search for information about a product captured by a camera and output that information. It can also provide translation services for text included in images captured by the camera.

[0006] The electronic device can translate text included in an image obtained by a camera into a language desired by the user and can display the text. For example, the electronic device can recognize text included in a preview image obtained by the camera through optical character recognition (OCR) for the preview image. The electronic device can translate the text recognized from the preview image into the language desired by the user through a translation engine and can render the translated text into a text area of ​​the preview image.

[0007] Because the electronic device drives the OCR and translation engines in the electronic device, the performance of the translation service may be limited. The electronic device can download the database and translation engine from an external device for text translation and store them in the electronic device to enhance the performance of the translation service. This may consume a large amount of memory in the electronic device.

[0008] Certain embodiments of the present disclosure provide an apparatus and method for providing a translation service of text included in an image obtained through a camera by using an external electronic device in an electronic device.

[0009] According to certain embodiments of the present disclosure, an electronic device includes a camera, a display, a transceiver, a memory, and a processor, wherein the processor is configured to: display, through the display, a first image including one or more external objects obtained by using the camera; identify, during at least a portion of the time when the first image is displayed, at least one external object corresponding to text among the one or more external objects; send, through the transceiver, a partial image of the first image corresponding to the at least one external object to an external electronic device; receive, through the transceiver, text corresponding to the text from the external electronic device; identify a movement of the electronic device or at least one external object generated during the time of sending the partial image and receiving the corresponding text from the external electronic device; and, when displaying, through the display, a second image including the at least one external object, display the corresponding text on the at least one external object by compensating for the movement.

[0010] According to certain embodiments of the present disclosure, a method for operating an electronic device includes: displaying a first image on a display of the electronic device, the first image being obtained by using a camera operably connected to the electronic device and including one or more external objects; identifying at least one external object corresponding to text among the one or more external objects during at least a portion of the time the first image is displayed; sending a partial image of the first image corresponding to the at least one external object to an external electronic device; receiving text corresponding to the text from the external electronic device; identifying movement of the electronic device or at least one external object generated during the time of sending the partial image and receiving the corresponding text from the external electronic device; and displaying the corresponding text on the at least one external object by compensating for the movement when displaying a second image including the at least one external object through the display.

[0011] According to certain embodiments of the present disclosure, an electronic device includes a camera, a display, a transceiver, a memory, and a processor, wherein the processor is configured to: display a first image obtained by using the camera through the display; send at least one partial image and an overall image of the first image to an external electronic device through the transceiver; when text information corresponding to the partial image or the overall image is received from the external electronic device through the transceiver, improve an area for displaying the text information based on the partial image or the overall image; calibrate a position of the text information based on motion information of the electronic device or at least one external object included in the first image; and, based on the improved area information and position information, display the text information on a second image.

[0012] According to certain embodiments, there is a method for annotating a picture, the method comprising: displaying a picture including an object carrying text at a location within the picture; extracting the text; and displaying another text on the object in another picture, the other picture including the object at another location within the other picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0014] Figure 1 is a block diagram of an electronic device for managing data related to an application in a network environment according to certain embodiments of the present disclosure;

[0015] Figure 2 is a block diagram illustrating a camera according to certain embodiments of the present disclosure;

[0016] Figure 3 is a flowchart for providing a translation service in an electronic device by using an external electronic device according to certain embodiments of the present disclosure;

[0017] Figure 4 is a flowchart for selectively transmitting an image for a translation service to an external electronic device in an electronic device according to certain embodiments of the present disclosure;

[0018] Figure 5 is a flowchart for extracting a partial image of an image obtained by a camera in an electronic device according to some embodiments of the present disclosure;

[0019] Figure 6 is a diagram illustrating a configuration of an image obtained by a camera according to certain embodiments of the present disclosure;

[0020] Figure 7 is a flowchart for displaying a translated text received from an external electronic device in an electronic device according to certain embodiments of the present disclosure;

[0021] Figure 8A is a view showing a screen for a translation service of an image obtained through a camera according to certain embodiments of the present disclosure;

[0022] Figure 8B is a view showing a screen for a translation service of an image obtained through a camera according to certain embodiments of the present disclosure;

[0023] Figure 8C is a view showing a screen for a translation service of an image obtained through a camera according to certain embodiments of the present disclosure;

[0024] Figure 8D is a view showing a screen for a translation service of an image obtained through a camera according to certain embodiments of the present disclosure;

[0025] Figure 8Eis a view showing a screen for a translation service of an image obtained through a camera according to certain embodiments of the present disclosure;

[0026] Figure 8F is a view showing a screen for a translation service of an image obtained through a camera according to certain embodiments of the present disclosure;

[0027] Figure 9 is a flowchart for setting a display position of translated text in an electronic device according to certain embodiments of the present disclosure.

[0028] Figure 10A is a diagram illustrating a configuration for setting a display position of a translated text in an external electronic device according to certain embodiments of the present disclosure;

[0029] Figure 10B is a diagram illustrating a configuration for setting a display position of a translated text in an external electronic device according to certain embodiments of the present disclosure;

[0030] Figure 10C is a diagram illustrating a configuration for setting a display position of a translated text in an external electronic device according to certain embodiments of the present disclosure; and

[0031] Figure 11 is a flowchart for displaying translated text corresponding to a partial image in an electronic device according to certain embodiments of the present disclosure. DETAILED DESCRIPTION

[0032] Certain embodiments of the present disclosure will be described herein with reference to the accompanying drawings. In the following description, well-known functions or configurations are not described in detail because they would obscure the present disclosure in unnecessary detail. Furthermore, the terms used herein are defined based on the functionality of the present disclosure. Therefore, these terms may vary depending on the intent and usage of the user or operator. In other words, the terms used herein must be understood based on the description provided herein.

[0033] Figure 1 is a block diagram illustrating an electronic device 101 for managing data associated with applications in a network environment 100 according to some embodiments. Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or communicate with the electronic device 104 or the server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input device 150, a sound output device 155, a display device 160 (touch screen display), an audio module 170, a sensor module 176 (motion sensor), an interface 177, a haptic module 179, a camera 180, a power management module 188, a battery 189, a communication module 190 (which may include, among other things, a wireless communication circuit including a transmitter / receiver (transceiver), a modulator / demodulator (MODEM), and an oscillator), a subscriber identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the components described above (e.g., display device 160 or camera 180) may be omitted from electronic device 101, or one or more other components may be added to electronic device 101. In some embodiments, some of the components described above may be implemented as a single integrated circuit. For example, sensor module 176 (e.g., a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented as embedded in display device 160 (e.g., a display). The term "transceiver" may refer to a single transmitter / receiver, or a combination of a transmitter and a receiver.

[0034] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component of the electronic device 101 (e.g., a hardware component or a software component) connected to the processor 120, and may perform various data processing or calculations. The term "processor," although used in the singular, should be understood to mean one or more processors. According to one embodiment, as at least part of the data processing or calculation, the processor 120 may load commands or data received from another component (e.g., sensor module 176 or communication module 190) into the volatile memory 132, process the commands or data stored in the volatile memory 132, and store the resulting data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operationally independent of or combined with the main processor 121. Additionally or alternatively, the auxiliary processor 123 may be adapted to consume less power than the main processor 121 or adapted to be specifically used for a designated function. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of the main processor 121.

[0035] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., the display device 160, the sensor module 176, or the communication module 190) among the components of the electronic device 101 (not the main processor 121), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., the display device 160, the sensor module 176, or the communication module 190) together with the main processor 121. Depending on the embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera 180 or the communication module 190) that is functionally related to the auxiliary processor 123.

[0036] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134.

[0037] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0038] The input device 150 may receive commands or data to be used by other components of the electronic device 101 (eg, the processor 120) from outside the electronic device 101 (eg, a user). The input device 150 may include, for example, a microphone, a mouse, or a keyboard.

[0039] The sound output device 155 can output sound signals to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or records, and the receiver can be used for incoming calls. Depending on the embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0040] The display device 160 can visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. Depending on the embodiment, the display device 160 may include a touch circuit adapted to detect a touch or a sensor circuit adapted to measure the strength of the force caused by the touch (e.g., a pressure sensor).

[0041] The audio module 170 can convert sound into an electrical signal, and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input device 150, or output sound via the sound output device 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0042] The sensor module 176 can detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a user's state) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. Depending on the embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0043] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly to an external electronic device (e.g., the electronic device 102). Depending on the embodiment, the interface 177 may include, for example, a High-Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital (SD) card interface, or an audio interface.

[0044] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. Depending on the embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0045] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0046] The camera 180 may capture still images or moving images. Depending on the embodiment, the camera 180 may include one or more lenses, image sensors, image signal processors, or flashes.

[0047] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0048] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0049] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. Depending on the embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip), or these various types of communication modules can be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 192 can identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0050] The antenna module 197 can transmit or receive signals or power to or from the outside of the electronic device 101 (e.g., an external electronic device). Depending on the embodiment, the antenna module 197 may include one or more antennas, and thus, for example, the communication module 190 (e.g., the wireless communication module 192) may select at least one antenna suitable for the communication scheme used in the communication network (such as the first network 198 or the second network 199) from the plurality of antennas. Signals or power can then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna.

[0051] At least some of the above components can be connected to each other via an inter-peripheral communication scheme (e.g., a bus, general-purpose input output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0052] According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 and the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type than the electronic device 101. According to an embodiment, all or some operations to be executed on the electronic device 101 may be executed on one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or execute another function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial response to the request, either by further processing the result or without further processing the result. To this end, for example, cloud computing technology, distributed computing technology, or client-server computing technology may be used.

[0053] The electronic device according to certain embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to embodiments of the present disclosure, the electronic device is not limited to those described above.

[0054] It should be understood that certain embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but rather include various changes, equivalents or alternative forms for corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things, unless the relevant context clearly indicates otherwise. As used herein, each phrase in the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include all possible combinations of the items listed together with the corresponding phrases in the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish corresponding components from another component, and do not limit the components in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “combined with another element (e.g., a second element)”, “combined to another element (e.g., a second element)”, “connected with another element (e.g., a second element)”, or “connected to another element (e.g., a second element)”, when the term “operably” or “communicatively” is used or when the term “operably” or “communicatively” is not used, it means that the element can be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0055] As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or the smallest unit or portion of the single integrated component. For example, depending on an embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0056] Certain embodiments described herein may be implemented as software (e.g., program 140) comprising one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to be operable to perform at least one function according to the called at least one instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" merely means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0057] According to an embodiment, the method according to certain embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be published in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be published online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™), or may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a manufacturer's server, an application store's server, or a memory of a forwarding server).

[0058] According to certain embodiments, each component (for example, module or program) in the above-mentioned components may include a single entity or multiple entities. According to certain embodiments, one or more components in the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (for example, module or program) may be integrated into a single component. In this case, according to certain embodiments, the integrated component may still perform the one or more functions of each component in the multiple components in the same or similar manner as a corresponding component in the multiple components before integration. According to certain embodiments, the operation performed by module, program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more operations in the operation may be run or omitted in different orders, or one or more other operations may be added.

[0059] Figure 2 FIG2 is a block diagram 200 illustrating a camera 180 according to some embodiments. Figure 2 , the camera 180 may include a lens assembly 210, a flash 220, an image sensor 230, an image stabilizer 240, a memory 250 (e.g., a buffer memory), or an image signal processor 260. The lens assembly 210 may collect light emitted or reflected from an object whose image is to be captured. The lens assembly 210 may include one or more lenses. Depending on the embodiment, the camera 180 may include a plurality of lens assemblies 210. In this case, the camera 180 may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies 210 may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from those of another lens assembly. The lens assembly 210 may include, for example, a wide-angle lens or a telephoto lens.

[0060] The flash 220 can emit light, wherein the emitted light is used to enhance light reflected from an object. Depending on the embodiment, the flash 220 may include one or more light-emitting diodes (LEDs) (e.g., red, green, and blue (RGB) LEDs, white LEDs, infrared (IR) LEDs, or ultraviolet (UV) LEDs) or a xenon lamp. The image sensor 230 can capture an image corresponding to the object by converting light emitted from or reflected from the object and transmitted through the lens assembly 210 into an electrical signal. Depending on the embodiment, the image sensor 230 may include one image sensor selected from a plurality of image sensors having different properties (e.g., an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor), a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor 230 may be implemented using, for example, a charge-coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

[0061] The image stabilizer 240 can move the image sensor 230 or at least one lens included in the lens assembly 210 in a specific direction, or control the operational properties of the image sensor 230 (e.g., adjust the readout timing) in response to the movement of the camera 180 or the electronic device 101 including the camera 180. This allows for compensating for at least a portion of the negative effects (e.g., image blur) that may be generated by the movement of the image being captured. Depending on the embodiment, the image stabilizer 240 can use a gyroscope sensor (not shown) or an acceleration sensor (not shown) disposed within or outside the camera 180 to sense such movement of the camera 180 or the electronic device 101. Depending on the embodiment, the image stabilizer 240 can be implemented as, for example, an optical image stabilizer.

[0062] Memory 250 can at least temporarily store at least a portion of an image acquired via image sensor 230 for subsequent image processing tasks. For example, if multiple images are captured quickly or image capture is delayed due to shutter lag, the acquired original image (e.g., a Bayer pattern image, a high-resolution image) can be stored in memory 250, and its corresponding duplicate image (e.g., a low-resolution image) can be previewed via display device 160. Then, if a specified condition is met (e.g., through user input or system command), at least a portion of the original image stored in memory 250 can be acquired and processed by, for example, image signal processor 260. Depending on the embodiment, memory 250 can be configured as at least a portion of memory 130, or memory 250 can be configured as a separate memory that operates independently of memory 130.

[0063] The image signal processor 260 may perform one or more image processing operations on images acquired via the image sensor 230 or stored in the memory 250. The one or more image processing operations may include, for example, depth map generation, three-dimensional (3D) modeling, panoramic image generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor 260 may control at least one of the components included in the camera 180 (e.g., image sensor 230) (e.g., exposure time control or readout timing control). Images processed by the image signal processor 260 may be stored back in the memory 250 for further processing, or may be provided to an external component outside of the camera 180 (e.g., memory 130, display device 160, electronic device 102, electronic device 104, or server 108). Depending on the embodiment, the image signal processor 260 may be configured as at least a portion of the processor 120, or may be configured as a separate processor that operates independently of the processor 120. If the image signal processor 260 is configured as a separate processor from the processor 120 , the at least one image processed by the image signal processor 260 may be displayed as it is by the processor 120 via the display device 160 , or may be displayed after being further processed.

[0064] Depending on the embodiment, the electronic device 101 may include multiple cameras 180 having different properties or functions. In this case, at least one of the multiple cameras 180 may form, for example, a wide-angle camera, and at least another camera 180 of the multiple cameras 180 may form a telephoto camera. Similarly, at least one of the multiple cameras 180 may form, for example, a front-facing camera, and at least another camera 180 of the multiple cameras 180 may form a rear-facing camera.

[0065] The electronic devices according to certain embodiments of the present disclosure may include various types of electronic devices. The electronic devices may include, for example, at least one of a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. The electronic devices according to embodiments of the present disclosure are not limited to the above devices.

[0066] It should be understood that certain embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but rather include various changes, equivalents or alternative forms for corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things, unless the relevant context clearly indicates otherwise. As used herein, each phrase in the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include all possible combinations of the items listed together with the corresponding phrases in the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish corresponding components from another component, and do not limit the components in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “combined with another element (e.g., a second element)”, “combined to another element (e.g., a second element)”, “connected with another element (e.g., a second element)” or “connected to another element (e.g., a second element)”, whether or not the term “operably” or “communicatively” is used, it means that the element can be connected to the other element directly or via a third element.

[0067] As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions, or the smallest unit or portion of the single integrated component. For example, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0068] Certain embodiments described herein may be implemented as software (e.g., program 140) comprising instructions readable by a machine (e.g., a computer) stored in a machine-readable storage medium (e.g., internal memory 136 or external memory 138). The machine may call the instructions stored in the storage medium and may operate according to the called instructions, and may include an electronic device (e.g., electronic device 101) according to the disclosed embodiments. When the instructions are executed by a processor (e.g., processor 120) under the control of a processor, the processor may directly execute the function corresponding to the instruction or use other components under the control of the processor to execute the function corresponding to the instruction. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" merely means that the storage medium is a tangible device and does not include signals, but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0069] According to an embodiment, the method according to some embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be released in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be downloaded via an application store (e.g., the Play Store). TM ) The computer program product is published online. If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a storage medium (such as a memory of a manufacturer's server, an application store's server, or a forwarding server).

[0070] According to certain embodiments, each component (e.g., module or program) in the above-mentioned components may include a single entity or multiple entities. One or more subcomponents in the above-mentioned subcomponents may be omitted, or one or more other subcomponents may be added. Alternatively or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, according to certain embodiments, the integrated component may still perform the one or more functions of each component in the multiple components in the same or similar manner as a corresponding component in the multiple components performed one or more functions before integration. According to certain embodiments, the operations performed by a module, program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more operations in the operations may be run or omitted in a different order, or one or more other operations may be added.

[0071] According to an embodiment, the processor 120 may control the communication module 190 to transmit an image (preview image) obtained by the camera 180 to an external electronic device (e.g., electronic device 102, electronic device 104, or server 108) for translation services. For example, the processor 120 may determine whether to transmit the image obtained by the camera 180 based on at least one of the quality of the image obtained by the camera 180 or the motion information obtained by the sensor module 176. For example, when the clarity (e.g., blur) of the image obtained by the camera 180 is less than or equal to a reference value, the processor 120 may determine that there are restrictions on the translation of the image obtained by the camera 180. In other words, when the clarity of the image obtained by the camera 180 is less than or equal to the reference value, the processor 120 may determine not to transmit the image obtained by the camera 180 to the external electronic device. For example, when the number of feature points in the image obtained by the camera 180 is less than a reference number, or when the feature points are distributed at a distance longer than a reference distance, the processor 120 may determine that there are restrictions on the translation of the image obtained by the camera 180. The reference distance may include the maximum distance between feature points that can form text. For example, when the movement of the electronic device 100 falls outside the reference range based on the movement information obtained by the sensor module 176, the processor 120 may determine that there is a restriction on the translation of the image obtained by the camera 180. For example, when it is determined that the image obtained by the camera 180 is to be sent to an external electronic device, the processor 120 may control the communication module 190 to send at least one partial image and the overall image corresponding to the image to the external electronic device. For example, the processor 120 may extract at least one partial image based on at least one of the history of using the translation service or the distribution of feature points of the image. The history of using the translation service is at least one text region probability model corresponding to the history of using the translation service, which may include area information about the display device 160 on which the text of the external object has been captured and displayed for the translation service in the electronic device 101. For example, the processor 120 may control the communication module 190 to send at least one partial image and the overall image corresponding to the image to different external electronic devices.

[0072] According to an embodiment, the processor 120 can calibrate the display area of ​​text translated into different languages ​​received from an external electronic device (e.g., electronic device 102, electronic device 104, or server 108) through the communication module 190. For example, the processor 120 can receive text included in an image (partial image or entire image) sent to the external electronic device and regional information of the text (e.g., the position, size, and length of the text) through the communication module 190. The processor 120 can set a region of interest (ROI) corresponding to the text in the image sent to the external electronic device based on the regional information of the text. The processor 120 can set pixels with the same brightness or color attributes in the region of interest as a text candidate region. The processor 120 can set the text display area by removing outliers that are not suitable for the features of the text in the text candidate region. For example, the outlier can include at least one pixel located on the boundary of the region of interest in the text candidate region. For example, the text can include text recognized by an optical character recognition method or text translated by a translation engine.

[0073] According to an embodiment, the processor 120 can determine the additional attributes of the text through the display area of ​​the text to display the text received from the external electronic device and translated into different languages ​​on the corresponding area. For example, the additional attributes of the text may include at least one of the color of the text, the background color, or the size or font of the text. For example, the processor 120 can identify the brightness or color attributes of the pixels included in the display area of ​​the text and can distinguish the text area from the background area. The processor 120 can set at least one of the color, size, or font of the text translated into different languages ​​based on the size of the text area or at least one of the color attributes of the pixels included in the text area. The processor 120 can set the color (background color) of the area (background) other than the text in the display area of ​​the text based on the color attributes of the pixels included in the background area. For example, the color of the area other than the text can be set based on the average value of the color attributes of the pixels included in the background area. For example, the color of the area other than the text can be set based on the color attribute with the highest distribution among the color attributes of the pixels included in the background area.

[0074] According to an embodiment, the processor 120 may calibrate the display position of text translated into a different language and received from an external electronic device (e.g., electronic device 102, electronic device 104, or server 108) based on the motion information of the electronic device 101 detected by the sensor module 176. For example, the processor 120 may continuously collect motion information of the electronic device 101 through the sensor module 176 from the time the image (preview image) is obtained by the camera 180. The processor 120 may detect a difference between the image sent to the external electronic device and the image received from the external electronic device showing the text translated into a different language based on the motion information of the electronic device 101. The processor 120 may calibrate the display position of the text translated into a different language based on the difference between the images.

[0075] According to an embodiment, processor 120 may refine the results of a translation service based on text translated into different languages, corresponding to multiple images (e.g., partial images or the entire image) received from an external electronic device (e.g., electronic device 102, electronic device 104, or server 108) via communication module 190. For example, upon receiving text translated into a different language corresponding to one of the images sent to the external electronic device, processor 120 may determine whether text corresponding to the other image exists in an area for displaying text translated into a different language. If text corresponding to the other image does not exist in the corresponding area, processor 120 may render the text translated into a different language received from the external electronic device for display in the corresponding area. If text corresponding to the other image exists in the corresponding area, processor 120 may select text with a relatively high reliability from the text corresponding to the other image and the text translated into a different language received from the external electronic device as the text to be displayed in the corresponding area. For example, the reliability of the text may be calculated based on at least one of the position of the text in the image or the accuracy of the text.

[0076] Figure 3 FIG3 is a flowchart 300 for providing a translation service by using an external electronic device in an electronic device according to some embodiments of the present disclosure. In the following description, the electronic device may include Figure 1 electronic device 101 or at least a portion of the electronic device 101 (eg, processor 120).

[0077] refer to Figure 3 In operation 301, the electronic device (eg, the processor 120) may display a video signal on a display device (eg, Figure 1 The display device 160) displays the image data transmitted by the camera (eg, Figure 1The first image may include a first image including at least one external object obtained by the camera 180. For example, the first image may include a preview image obtained by the camera 180. One of the external objects may carry text.

[0078] In operation 303, the electronic device (e.g., processor 120) may transmit at least one partial image of the first image to an external electronic device. For example, processor 120 may determine whether to transmit the preview image based on at least one of the clarity of the preview image (e.g., blur), the number of feature points in the preview image, or motion information obtained by sensor module 176. If it is determined that the preview image is to be transmitted to the external electronic device, processor 120 may control transmission of at least one partial image and the entire image of the preview image to the external electronic device via communication module 190. For example, the partial image may be extracted from the preview image based on at least one of a history of using a translation service (e.g., a text region probability model) or a distribution of feature points in the image. For example, processor 120 may control transmission of at least one partial image and the entire image of the preview image to different external electronic devices via communication module 190. For example, processor 120 may control transmission of at least one image (e.g., at least one of the partial image and the entire image) and language information for translating text included in the image to the external electronic device via communication module 190. In some embodiments, the partial image may be extracted to extract text as one of the objects.

[0079] In operation 305, the electronic device (e.g., the processor 120) may recognize the movement of the electronic device or an external object included in the first image. For example, the processor 120 may collect, through the sensor module 176, movement information of the electronic device that occurs when transmitting an image for translating text included in the image and receiving text corresponding to the image.

[0080] In operation 307, the electronic device (e.g., processor 120) may determine whether text (text information) corresponding to the partial image sent to the external electronic device is received. For example, the text corresponding to the partial image may include text in a second language (e.g., Korean) translated from a text in a first language (e.g., English) extracted from the image. For example, the processor 120 may receive text corresponding to the partial image and regional information of the text in the corresponding image (e.g., the position, size, and length of the text). However, the present disclosure is not limited to language translation. In some embodiments, the received text may be a synonym of the text. In other embodiments, the received text may be an update of the text. For example, if the text is information that changes over time, the received text may include an update of the information.

[0081] When text corresponding to the partial image transmitted to the external electronic device is not received (No in operation 307 ), in operation 305 , the electronic device (eg, processor 120 ) may continuously recognize movement of the electronic device or an external object included in the first image.

[0082] When the text corresponding to the partial image transmitted to the external electronic device is received ("YES" in operation 307), in operation 309, the electronic device (e.g., the processor 120) may compensate the position of the text corresponding to the partial image based on the motion information of the electronic device or the external object included in the first image, and may display the text on the display device (e.g., Figure 1 The text is displayed on the display device 160).

[0083] For example, the processor 120 may detect a position change of the first image or the external object included in the first image displayed on the display device 160 between the time when the first image was obtained and the current time based on the motion information of the electronic device or the external object included in the first image. The processor 120 may compensate for the position of the text corresponding to the partial image based on the position change of the first image or the external object included in the first image, and may control the display of the text through the display device 160 to cover at least one external object.

[0084] Detection of a change in position of an external image from a first image to a second image can be determined in various ways. In some embodiments, the camera may encode the first image and the second image using MPEG. Motion information can be determined by examining motion vectors in the first image, the second image, and each image between the first and second images. In one embodiment, the electronic device may create motion vectors by encoding the first image as an I-picture according to the MPEG standard and encoding the second image as a P-picture that relies on data from the first image. The motion vectors of an approaching object can be used to determine motion.

[0085] Figure 4 FIG4 is a flowchart 400 for selectively sending an image for a translation service to an external electronic device in an electronic device according to some embodiments of the present disclosure. Figure 3 In operation 303, at least one partial image of the first image obtained by the camera 180 is transmitted to the external electronic device. In the following description, the electronic device may include Figure 1 The first electronic device 101 or at least a portion of the electronic device 101 (eg, the processor 120).

[0086] refer to Figure 4 , when viewed through a camera (e.g., Figure 1The first image (eg, preview image) obtained by the camera 180) is displayed on the display device (eg, Figure 1 When the display device 160 is displayed (for example, Figure 3 In operation 401, the electronic device (e.g., the processor 120) may identify the quality of the first image. For example, the quality of the first image may include at least one of the clarity of the first image (e.g., blur state), the number of feature points included in the first image, or the distribution of the feature points included in the first image.

[0087] In operation 403, the electronic device (e.g., the processor 120) may identify the motion of the electronic device or at least one external object included in the first image. For example, the processor 120 may collect motion information of the electronic device 101 or at least one external object included in the first image from the time the first image is obtained by the camera 180. For example, the motion of the electronic device 101 may be obtained by the sensor module 176 (e.g., an acceleration sensor, a gravity sensor).

[0088] In operation 405, the electronic device (e.g., processor 120) may determine whether to request translation of the first image based on the quality of the first image and the motion information of the electronic device or at least one external object included in the first image. For example, when the resolution (e.g., blur state) of the first image is higher than or equal to a reference value (i.e., good), the processor 120 may determine to provide a translation service for the first image. As described above, although the present embodiment uses translation, the present disclosure is not limited to translation. For example, when the number of feature points of the first image is greater than a reference number or the feature points are concentrated in a certain area, the processor 120 may determine to provide a translation service for the first image. For example, when the motion of the electronic device 101 obtained by the sensor module 176 falls within a reference range, the processor 120 may determine to provide a translation service for the first image. For example, when the motion of the preview image obtained by the camera 180 falls within a reference range, the processor 120 may determine to provide a translation service for the first image. For example, the motion of the preview image may include the motion of at least one object included in the preview image. The processor 120 may detect the motion of the object included in the preview image by comparing consecutive preview images obtained by the camera 180 (e.g., pixels forming the image).

[0089] When it is determined that translation of the first image is not requested (e.g., "No" in operation 405), the electronic device (e.g., processor 120) may impose restrictions on the transmission of the first image. For example, when it is determined that translation of the first image is not requested, processor 120 may control the display device 160 to display a guidance message indicating that there are restrictions on the translation service. For example, the guidance message may include the reason why the translation service is restricted.

[0090] When it is determined that translation of the first image is requested (e.g., "Yes" in operation 405), the electronic device (e.g., processor 120) may transmit at least one partial image and the entire image of the first image to at least one external electronic device. For example, processor 120 may extract at least one partial image from the first image based on at least one of a history of using a translation service (e.g., a text region probability model) or a distribution of first feature points of the first image. Processor 120 may transmit the at least one partial image and the entire image extracted from the first image to at least one external electronic device via communication module 190.

[0091] Figure 5 is a flowchart 500 for extracting a partial image of an image obtained by a camera in an electronic device, according to certain embodiments of the present disclosure. Figure 6 FIG. 6 shows a configuration 600 of an image obtained by a camera according to some embodiments of the present disclosure. The following description may be about Figure 4 In the following description, the electronic device may include an operation of transmitting the image to the external electronic device in operation 407. Figure 1 The electronic device 101 or at least a portion of the electronic device 101 (eg, the processor 120) may be configured to capture text 610. The electronic device may capture a picture 620 of the text-bearing object and extract text 660, 662.

[0092] refer to Figure 5 , when provided by a camera (e.g. Figure 1 When the translation service of the first image (eg, preview image) obtained by the camera 180) is performed (eg, Figure 4 In operation 501, the electronic device (e.g., processor 120) may identify a probability model of at least one text region corresponding to the history of using the translation service. For example, the probability model of the text region corresponding to the history of using the translation service may include information about the region of the display 160 on which the text of the external object has been captured and displayed for the translation service. The probability model of the text region corresponding to the history of using the translation service may be improved as the translation service is provided and may include at least one region.

[0093] In operation 503, the electronic device (eg, the processor 120) may identify the image data transmitted by the camera (eg, Figure 1A probability model of a text region corresponding to image analysis information of a first image obtained by a camera 180 (using a user's computer). For example, the probability model of a text region corresponding to the image analysis information of the first image may include at least one piece of region information set based on the distribution of feature points extracted from the first image. For example, the probability model of a text region corresponding to the image analysis information of the first image may include information of at least one region in which a plurality of feature points greater than or equal to a reference number are concentrated within the first image.

[0094] In operation 505, the electronic device (e.g., the processor 120) may set at least one text region of the first image. The at least one text region of the first image may be based on a probability model of the text region. The probability model may correspond to at least one of a history of using a translation service or image analysis information of the first image. For example, the processor 120 may control the display device 160 to display a preview image 620 of the external object 610 obtained by the camera 180, such as Figure 6 When the probability model of the text area corresponding to the history of using the translation service is arranged at the center of the display device 160, the processor 120 may set at least a portion of the center of the preview image 620 as the text area of ​​the first image. The processor 120 may set at least a portion of the center of the preview image 620 as the text area of ​​the first image, with the feature points of the text "apple" and "banana" in the preview image 620 being concentrated at the center of the preview image 620.

[0095] The processor 120 may set an area corresponding to at least one of a plurality of probability models of text areas corresponding to a history of using a translation service as a text area of ​​the first image, the at least one probability model overlapping with an area on which feature points are at least partially concentrated. For example, the text area of ​​the first image may include at least some areas estimated to have text within the first image. For example, the preview image 620 may include an image 630 of an external object 610, a language 650 of text included in the image, and a language 652 for translation of the text. The display device 160 may display a service list 640 related to the preview image 620 on at least some areas, such as Figure 6 The service list 640 may include a menu 642 for selecting a translation service (eg, "Text") .

[0096] In operation 507, the electronic device (eg, the processor 120) may extract at least one partial image corresponding to at least one text region of the first image. For example, the processor 120 may extract at least one partial image corresponding to the at least one text region of the first image. Figure 6The processor 120 may generate partial images 660 and 662 corresponding to corresponding text regions of the preview image 620. For example, the processor 120 may generate partial images 660 and 662 corresponding to corresponding text regions that are not continuous within the preview image 620. In some embodiments, the partial images 660 and 662 may be selected to primarily contain the text and surrounding pixels of the image. For example, OCR may be used to recognize text in an image, and the partial images 660 and 662 may be selected to extract the portion of the image that primarily includes the text and surrounding pixels.

[0097] In some cases, a partial image (e.g., partial image 662) can properly isolate text. For example, partial image 662 includes the text "apple" and a portion of the text "banana," with the bottom portion of "banana" cut off.

[0098] In operation 509, the electronic device (eg, the processor 120) may transmit at least one partial image and the entire image of the first image to at least one external electronic device. For example, the processor 120 may control the transmission of the first image from the communication module 190 to the external electronic device. Figure 6 The partial images 660 and 662 extracted from the preview image 620 are transmitted to at least one external electronic device. For example, the processor 120 may control the transmission of the first partial image 660, the second partial image 662, and the entire image 620 to different external electronic devices through the communication module 190.

[0099] Figure 7 FIG7 is a flowchart 700 for displaying, in an electronic device, translated text (or other corresponding text) that may be received from an external electronic device, according to certain embodiments of the present disclosure. Figure 8A A screen 800 is shown for a translation service of an image obtained through a camera according to certain embodiments of the present disclosure. Figure 8B A screen 800 is shown for a translation service of an image obtained through a camera according to certain embodiments of the present disclosure. Figure 8C A screen 800 is shown for a translation service of an image obtained through a camera according to certain embodiments of the present disclosure. Figure 8D A screen 800 illustrating a translation service of an image obtained through a camera, wherein an object has moved, according to certain embodiments of the present disclosure. Figure 8E A screen 800 is shown for a translation service of an image obtained through a camera according to certain embodiments of the present disclosure. Figure 8F A screen 800 is shown for a translation service for an image obtained through a camera according to some embodiments of the present disclosure. The following description may be about displaying Figure 3 In the following description, the electronic device may include: Figure 1electronic device 101 or at least a portion of the electronic device 101 (eg, processor 120).

[0100] refer to Figure 7 , when a text corresponding to an image is received from an external electronic device (eg, Figure 3 In operation 307 of the embodiment of the present invention, in operation 701, the electronic device (e.g., the processor 120) can calibrate the text received from the external electronic device. For example, when receiving text corresponding to an image from the external electronic device, the processor 120 can identify the regional information of the text extracted in the external electronic device (e.g., the position, length, and size of the text). The processor 120 can improve the regional information of the text based on the text display area estimated based on the image. For example, the image can include any one of a plurality of images (at least one partial image and an overall image) related to a first image, which is sent to the external electronic device for the translation service of the first image.

[0101] In operation 703, the electronic device (e.g., the processor 120) may determine whether there is text corresponding to another image received from the external electronic device. For example, the processor 120 may determine whether text corresponding to another image among the at least one partial image and the entire image sent to the external electronic device has been received previously. For example, the other image may include another image different from an image among a plurality of images (the at least one partial image and the entire image) related to the first image sent to the external electronic device for the translation service of the first image.

[0102] When there is text corresponding to another image received from an external electronic device at a previous time (e.g., "yes" in operation 703), in operation 705, the electronic device (e.g., processor 120) can improve the result of the translation service received from the external electronic device at a previous time based on the text corresponding to the image. For example, when there is text corresponding to another image received at a previous time at the same position as the position of the text corresponding to the image received from the external electronic device, the processor 120 can select the text to be displayed at the corresponding position based on the reliability of each text. For example, the reliability of the text can be determined based on the position of the text in the image (partial image or entire image) sent to the external electronic device to extract the text or the accuracy of the text. For example, the accuracy of the text can indicate the accuracy of whether the text received from the external electronic device can be interpreted as text in a specific language. For example, when there is a difference between the position of the text corresponding to the image received from the external electronic device and the text corresponding to the other image received at a previous time, the processor 120 can add the text corresponding to the image to the result of the translation service received from the external electronic device.

[0103] When there is no text corresponding to another image received from an external electronic device at a previous time (e.g., "No" in operation 703), or when the result of the translation service is perfected (e.g., operation 705), in operation 707, the electronic device (e.g., processor 120) can calibrate the display position of the text received from the external electronic device based on the motion information of the electronic device or the external object included in the first image. For example, the processor 120 can detect the difference between the first image sent to the external electronic device for the translation service and the second image obtained by the camera 180 when displaying the text received from the external electronic device based on the motion information of the electronic device or the external object included in the first image. The processor 120 can calibrate the display position of the text received from the external electronic device based on the difference between the first image and the second image. For example, the motion information of the electronic device or the external object included in the first image can be continuously collected from the time when the first image for the translation service is obtained.

[0104] In some embodiments, the first image may include a live image provided by the camera and displayed on the display when the camera has not yet captured an image. The second image may include a live image provided by the camera when the corresponding text is received. During the time between sending the text to the external device and receiving the corresponding text, the object may have moved or the camera may have moved. Therefore, the object carrying the text will be in a different position in the second image. Therefore, in order to place the received corresponding text on the object carrying the text, the motion is compensated from the original position of the text in the first image.

[0105] In operation 709, the electronic device (e.g., processor 120) may determine a text color and a background color for displaying text received from the external electronic device based on color information of a text display area for displaying text received from the external electronic device within the first image.

[0106] For example, it is noted that in the second image, the color of the pixels forming the object bearing the image may have changed. For example, if the object has moved, the lighting may have changed, resulting in a different color of the pixels.

[0107] In operation 711, the electronic device (e.g., processor 120) may overlay text received from the external electronic device on at least a portion of the preview image based on the calibrated display position, text color, and background color of the text. Figure 6When the second partial image 662 corresponds to the preview image 810 and is translated into a text in a different language, the processor 120 can control the display device 160 to display the text translated into a different language in the area corresponding to the corresponding text in the preview image 810. (Korean for apple)" and " (Butter in Korean) 820", such as Figure 8A When the second time is received with Figure 6 When the first partial image 660 corresponds to the first partial image 660 and is translated into text in a different language, the processor 120 may compare the reliability of the text corresponding to the first partial image 660 and the reliability of the text corresponding to the second partial image 662, and may select the text corresponding to the first partial image 660 (" ” (banana in Korean). For example, when the text corresponding to the first partial image 660 is selected, the processor 120 may control the display of the text translated into a different language on the area corresponding to the corresponding text in the preview image 810 through the display device 160, “ ” (Korean for apple) and “ ” (Korean for banana) 830, such as Figure 8B When the third time is received Figure 6 When the overall image 620 corresponds to the preview image 810 and is translated into a text in a different language, the processor 120 can control the display device 160 to display the text translated into a different language on the area corresponding to the corresponding text in the preview image 810. ” (Apple in Korean), “ ” (Korean for banana) and “ ” (Korean carrot) 840, such as Figure 8C For example, Figure 8D As shown, when the position of the external object included in the first image obtained by the camera 180 changes (850), the processor 120 may control the display device 160 to display the text translated into a different language at a display position calibrated based on the motion information of the electronic device 101 or the external object included in the first image, “ ” (Apple in Korean), “ ” (Korean for banana) and “ ” (Korean carrot) 860, such as Figure 8E shown.

[0108] According to an embodiment, the electronic device 101 (eg, the processor 120) may set a language for translating the text extracted from the first image. For example, when a language for translating the text is detected, Figure 6When input (eg, touch input) is performed on the language 652 of the text displayed on the preview image 620, the processor 120 may control the display of the available language list 870 for translation through the display device 160, such as Figure 8F As shown. Processor 120 can set the language selected from available language list 870 as the language used to translate the text extracted from the first image. As mentioned above, the present disclosure is not limited to language translation. In some embodiments, instead of listing languages ​​870, the user can select whether the corresponding text is a synonym, updated information, or any other type of different correspondence.

[0109] According to an embodiment, the electronic device 101 (e.g., the processor 120) can recognize text included in an image through a first external electronic device (e.g., a first server) and can translate the text included in the image into text in a different language through a second external electronic device (e.g., a second server). For example, the processor 120 can control the transmission of at least a partial image and the entire image of a preview image to the first external electronic device via the communication module 190. Upon receiving text extracted from the image and corresponding to a first language from the first external electronic device, the processor 120 can control the transmission of the text corresponding to the first language to the second external electronic device via the communication module 190. As in operations 705 to 711, the processor 120 can control the display of the text received from the second external electronic device and translated into a second language via the display device 160 by compensating for differences in the motion of external objects included in the first image. For example, when the text corresponding to the first language is transmitted to the second external electronic device and translated into text corresponding to the second language, the processor 120 can calibrate the text corresponding to the first language (operation 701). As mentioned above, the present disclosure is not limited to language translation and may include various other relationships, such as synonyms and updates, to name a few.

[0110] Figure 9 FIG9 is a flowchart 900 for setting a display position of a translated text or a corresponding text in an electronic device according to certain embodiments of the present disclosure. Figure 10A shows a configuration 1000 for setting a display position of a translated text in an external electronic device according to certain embodiments of the present disclosure, Figure 10B shows a configuration 1000 for setting a display position of a translated text in an external electronic device according to certain embodiments of the present disclosure, Figure 10C FIG1000 shows a configuration 1000 for setting a display position of a translated text in an external electronic device according to some embodiments of the present disclosure. Figure 7 In operation 701, the electronic device may include an operation of calibrating a text received from an external electronic device. Figure 1electronic device 101 or at least a portion of the electronic device 101 (eg, processor 120).

[0111] refer to Figure 9 , when a text corresponding to an image is received from an external electronic device (e.g., Figure 3 ), in operation 901, the electronic device (e.g., the processor 120) may set at least a portion of the image corresponding to the text received from the external electronic device as a region of interest. For example, the processor 120 may identify region information of the text received from the external electronic device through the communication module 190. For example, the region information of the text may include at least one of the position, size, and length of the text extracted from the image. The processor 120 may set at least a portion of the image corresponding to the text received from the external electronic device as a region of interest based on the region information of the text received from the external electronic device. For example, the processor 120 may set at least a portion of the local image 1010 as a region of interest 1020, as shown in FIG. Figure 10A For example, the region of interest may be set to be larger or smaller than region information of text in the image received from the external electronic device.

[0112] In operation 903, the electronic device (eg, the processor 120) may identify candidate regions for displaying text within the region of interest. Figure 10B As shown, the processor 120 can set the pixels with the same brightness or color attribute in the region of interest 1020 within the partial image 1010 as text candidate regions 1030 and 1032. For example, the processor 120 can set the text candidate regions 1030 and 1032 by an algorithm, which is used to Figure 10B Regions where changes in the binary coding threshold are stable (eg, maximally stable extremal regions (MSERs)) are found among the surrounding relatively brighter or darker pixels within the region of interest 1020 .

[0113] In operation 905, the electronic device (e.g., the processor 120) may remove surrounding pixels that are too far away from the text candidate region. For example, the processor 120 may remove outliers that are unnecessary for surrounding text based on at least one of the position, size, or length of the text candidate region. Figure 10B As shown, the processor 120 may identify the second text candidate region 1032 located on the boundary of the region of interest 1020 as an outlier and may remove it from the text candidate regions.

[0114] In operation 907, the electronic device (e.g., the processor 120) may set the text candidate region from which the outliers are removed as a text display region for displaying text within the image. For example, when the second text candidate region 1032 located on the boundary of the region of interest 1020 is identified as an outlier and removed, the processor 120 may set the first text candidate region 1030 as the text display region 1040 within the partial image 1010, as shown in FIG. Figure 10C For example, the text display area may indicate an area where text extracted from an image in a first language is translated into a second language and displayed, or where the text is replaced in other ways.

[0115] Figure 11 FIG1 is a flowchart 1100 for displaying a translated text corresponding to a partial image in an electronic device according to some embodiments of the present disclosure. The following description may be about Figure 7 In the following description, the electronic device may include an operation of improving the result of the translation service in operations 703 and 705. Figure 1 electronic device 101 or at least a portion of the electronic device 101 (eg, processor 120).

[0116] refer to Figure 11 When calibrating text corresponding to an image (e.g., a partial image or an entire image) received from an external electronic device, the electronic device (e.g., the processor 120) may detect the reliability of the corresponding text in operation 1101. For example, the processor 120 may detect the reliability of the corresponding text based on at least one of the position of the text received from the external electronic device in the image (e.g., a partial image or an entire image) or the accuracy of the text. For example, Figure 6 In the case of the first partial image 660, since the text "banana" in the first partial image 660 is closer to the center of the first partial image 660, the processor 120 can determine that the text has a relatively high reliability. Figure 6 In the case of a second partial image 662, processor 120 can detect the reliability of each of the text "apple" and "banana" based on the position of the text in the second partial image 662. In this case, based on the positions of the text "apple" and "banana" in the second partial image 662, it can be determined that the reliability of the text "apple" is higher than the reliability of the text "banana". For example, the accuracy of the text can be determined based on whether the text is mapped to text in a specific language.

[0117] In operation 1103, the electronic device (e.g., the processor 120) may determine whether another text received from the external electronic device at a previous time exists in an area for displaying text received from the external electronic device in the first image (preview image). For example, the processor 120 may determine whether text corresponding to another image (e.g., a partial image or an entire image) received from the external electronic device at a previous time exists. When text corresponding to another image received from the external electronic device at a previous time exists, the processor 120 may determine whether the display area of ​​the text corresponding to the other image in the first image (preview image) overlaps at least a portion of the display area of ​​the text whose reliability was detected in operation 1101. For example, when there is no text corresponding to another image received from the external electronic device at a previous time, or when the display area of ​​the text of the other image does not overlap at least a portion of the display area of ​​the text in the first image, the processor 120 may determine that there is no other text in the area of ​​the first image for displaying the text received from the external electronic device. For example, when the display area of ​​text in the first image corresponding to another image received from an external electronic device at a previous time overlaps at least a portion of the display area of ​​the text, the processor 120 can determine that another text exists in the area in the first image for displaying the text received from the external electronic device.

[0118] When there is no text corresponding to another image in the area for displaying text received from the external electronic device in the first image (preview image) (e.g., "No" in operation 1103), in operation 1105, the electronic device (e.g., processor 120) may display the text received from the external electronic device on the area for displaying text received from the external electronic device in the overall image. For example, the processor 120 may render an image to display the text received from the external electronic device and translated into a different language on the text display area in the preview image set through operations 901 to 907. For example, when a text corresponding to another image is received at the first time, Figure 6 When the text corresponding to the second partial image 662 in the preview image 810 (the text translated into a different language) is displayed, the processor 120 may render the preview image to display the text translated into a different language on the display area of ​​the text corresponding to the second partial image 662 in the preview image 810, “ ” (Korean for apple) and “ ” (Butter in Korean) 820, such as Figure 8A shown.

[0119] When text corresponding to another image exists in the area for displaying text received from the external electronic device in the first image (preview image) (e.g., "yes" in operation 1103), the electronic device (e.g., processor 120) may determine whether the reliability of the text received from the external electronic device is higher than the reliability of the other text in operation 1107. For example, when a text corresponding to another image is received at the second time Figure 6 When the text corresponding to the first partial image 660 (the text translated into a different language) is displayed, the processor 120 may compare the text corresponding to the first partial image 660 (" ” (banana in Korean)) and the text corresponding to the second partial image 662 (“ ” (Korean for butter). For example, since the text corresponding to the first partial image 660 (“ ” (Korean for banana)) is located at the image center of the first partial image 660, so the text may have a relatively higher height than the text corresponding to the second partial image 662 (“ ” (Korean for butter)) is reliable, and the text corresponding to the second partial image 662 (“ ” (butter in Korean)) is located on the boundary of the second partial image 662.

[0120] When the reliability of the text received from the external electronic device is higher than the reliability of the text corresponding to the other image (e.g., "Yes" in operation 1107), the electronic device (e.g., processor 120) may complete the text of the text display area with the text received from the external electronic device in operation 1109. For example, when it is determined that the text corresponding to the first partial image 660 (" When the reliability of the ” (Korean banana)) is high, the processor 120 can render the preview image to Figure 8A The text “ "(Butter in Korean) perfect as Figure 8B The text shown is " ” (banana in Korean), and the complete text is displayed.

[0121] When the reliability of the text received from the external electronic device is lower than or equal to the reliability of another text (e.g., "No" in operation 1107), the electronic device (e.g., processor 120) may maintain the text corresponding to the other image displayed on the text display area of ​​the first image (preview image).

[0122] According to an embodiment, when the text of the first display area received at the first time is the same as the text of the first display area received at the second time, the electronic device 101 (e.g., the processor 120) may maintain the text displayed on the first display area of ​​the preview image. For example, when the text of the first display area received at the first time is the same as the text of the first display area received at the second time, operations 1107 to 1109 may be omitted.

[0123] According to an embodiment, the electronic device 101 (e.g., the processor 120) may continuously collect motion information of the electronic device 101 or an external object included in the first image from the time the first image is obtained by the camera 180. The electronic device 101 may end the translation service before the translation service is completed based on the motion information of the electronic device 101 or the external object included in the first image. For example, when the motion of the electronic device 101 or the external object included in the first image falls outside the reference range, the processor 120 may determine that it is impossible to display the text translated into a different language. Therefore, the processor 120 may stop the translation service.

[0124] According to certain embodiments, an electronic device and an operating method thereof transmit an image obtained by a camera to an external electronic device (e.g., a server), calibrate the display position of the translated text received from the external electronic device, or compensate for the difference between the image sent to the external electronic device (e.g., a server) for a translation service and the image displayed on the display, and display the translated text. Thus, the text translated by the external electronic device can be smoothly displayed on the image obtained by the camera.

[0125] According to certain embodiments, an electronic device and operating method thereof selectively transmits an image for a translation service to an external electronic device (e.g., a server) based on at least one of the quality of an image obtained by a camera (e.g., the distribution of feature points) or motion information of the electronic device. Thus, the number of translations performed using the external electronic device can be reduced, thereby reducing the consumption of network resources.

[0126] The electronic device and operating method according to certain embodiments transmits at least one partial image and an entire image corresponding to an image obtained by a camera to an external electronic device (e.g., a server). This reduces delays in translation services caused by computations in the external electronic device and translation delays in the network.

[0127] Certain embodiments of the present disclosure may be implemented by a computer program including a computer program stored on a machine (eg, computer) readable storage medium (eg, Figure 1The machine is a device that is capable of retrieving stored instructions from a storage medium and operating according to the retrieved instructions, and may include electronic devices 100, 102, 104 or server 108. When the instructions are executed by a processor (e.g., processor 120), the processor may directly perform the function corresponding to the instruction, or may use other elements under the control of the processor to perform the function corresponding to the instruction. The instruction may include code generated or run by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" refers only to the fact that the storage medium is tangible and does not include signals, regardless of whether the data is stored semi-permanently or temporarily in the storage medium.

[0128] The methods according to certain embodiments disclosed herein may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. For example, the computer program product may be a downloadable application or computer program traded between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be downloaded via an app store (e.g., PlayStore). TM If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as a memory of a relay server, a server of an application store, or a manufacturer's server.

[0129] According to each element (for example, module or program) of certain embodiments, can comprise single entity or multiple entities, and in certain embodiments, can omit some sub-elements in above-mentioned element, or can add other sub-elements.Alternatively or additionally, some elements (for example, module or program) can be integrated in single element, and integrated element can still be carried out the function performed by each corresponding element in the same or similar manner before being integrated with corresponding element.According to the operation performed by module, programming module or other elements of certain embodiments, can be run sequentially, in parallel, repeatedly or in a heuristic manner.At least some operations can be run according to another order, can be omitted, or can also include other operations.

[0130] The present disclosure has been described with reference to various exemplary embodiments thereof. Those skilled in the art will appreciate that the present disclosure may be implemented in a modified form without departing from the essential characteristics of the present disclosure. Therefore, the disclosed embodiments should be considered from a descriptive perspective rather than a restrictive perspective. The scope of the present disclosure is defined not by the detailed description but by the appended claims, and all differences within the scope are to be understood as included in the present disclosure.

Claims

1. An electronic device comprising: camera; monitor; transceiver; at least one processor; as well as a memory storing instructions that, when executed by the at least one processor, cause the electronic device to: displaying, by the display, a first image obtained by using the camera and including one or more external objects; During at least a portion of the time the first image is displayed, identifying at least one external object of the one or more external objects that corresponds to text; Determining that a translation service is to be provided when at least one of the following is satisfied: based on motion information of the electronic device or at least one external object included in the first image, a motion of the electronic device is within a reference range, or quality of the first image meets requirements; and When it is determined that a translation service is to be provided, transmitting a partial image of the first image corresponding to the at least one external object to an external electronic device through the transceiver; receiving text corresponding to the text from the external electronic device through the transceiver; identifying a motion of the electronic device or the at least one external object generated during a time period of transmitting the partial image and receiving a corresponding text from the external electronic device; as well as When a second image including the at least one external object is displayed through the display, the corresponding text is displayed on the at least one external object by compensating for the motion.

2. The electronic device according to claim 1, wherein The corresponding text is from a first language recognized based on image recognition corresponding to the partial image, or is text from a second language different from the first language translated based on the text corresponding to the first language, and When the instructions are executed by the at least one processor, the electronic device displays text in the first language or text in the second language on the partial image via the display based on the text.

3. The electronic device according to claim 1, wherein When the instructions are executed by the at least one processor, the electronic device: receiving region information corresponding to the text from the external electronic device, and setting a region of interest in the first image based on the received region information; detecting a candidate region based at least on a comparison between pixels included in the region of interest or a region adjacent to the region of interest, the candidate region including pixels similar in at least one attribute of brightness or color; as well as The region information is calibrated based on the candidate region.

4. The electronic device according to claim 3, wherein: When the instructions are executed by the at least one processor, the electronic device: determining additional attributes related to corresponding text based on at least some areas of the first image corresponding to the calibrated area information; and displaying text information on the at least one external object based on the additional attribute, and The additional attributes include at least one of the color, size or font of the text or the background color.

5. The electronic device according to claim 1, wherein When the instructions are executed by the at least one processor, the electronic device: When a plurality of corresponding texts are received, comparing the reliability among the plurality of corresponding texts, and Any one of the plurality of corresponding texts selected based on a result of the comparison is displayed on the at least one external object. The electronic device according to claim 5 , wherein: When executed by the at least one processor, the instructions cause the electronic device to detect the reliability of the plurality of corresponding texts based on positions of the plurality of corresponding texts in the partial image.

7. A method for an electronic device, comprising: displaying, through a display, a first image obtained by using a camera of the electronic device; Determining that a translation service is to be provided when at least one of the following is satisfied: based on motion information of the electronic device or at least one external object included in the first image, a motion of the electronic device is within a reference range, or quality of the first image meets a requirement; When it is determined that a translation service is to be provided, transmitting at least one partial image and the entire image of the first image to an external electronic device via a transceiver of the electronic device; When text information corresponding to the partial image or the entire image is received from the external electronic device through the transceiver, completing an area for displaying the text information based on the partial image or the entire image; calibrating a position of the text information based on motion information of the electronic device or at least one external object included in the first image; as well as The text information is displayed on the second image based on the improved area information and position information.

8. The method of claim 7, wherein: The text information includes text corresponding to a first language recognized based on image recognition corresponding to the at least one partial image, or text corresponding to a second language different from the first language translated based on the text corresponding to the first language.

9. The method according to claim 7, further comprising: When it is determined that the translation service is to be provided, the at least one partial image and the entire image of the first image are transmitted to the external electronic device through the transceiver.

10. The method according to claim 7, further comprising: At least one partial image is extracted from the first image based on at least one of a history of using a translation service or distribution information of feature points included in the first image.

11. The method according to claim 7, further comprising: setting a region of interest in the partial image based on the region information of the text information received from the external electronic device; detecting a candidate region in the region of interest, the candidate region comprising at least one pixel similar in at least one attribute of brightness or color; as well as The region information of the text information is improved based on the candidate region.

12. The method according to claim 11, further comprising: determining additional attributes related to the text information based on at least some areas of the first image corresponding to the refined area information of the text information; as well as displaying the text information to overlap at least a portion of the second image based on the additional attribute, The additional attributes include at least one of the color, size or font of the text or the background color.

13. The method according to claim 7, further comprising: When receiving a plurality of text information corresponding to at least some areas of the first image, comparing the reliability of the plurality of text information; as well as Based on the comparison result, any one piece of text information is selected from the plurality of text information as the text information corresponding to the at least some areas.

14. The method according to claim 13, further comprising: The reliability of the text information is detected based on the position of the text information in the partial image.

Images