Method, system, and computer program for extracting and providing text color and background color in an image

Abstract

To provide functions for detecting and translating text in an image.SOLUTION: A method for extracting and providing a text color and a background color in an image is provided, the method comprising: detecting a first area containing text in a given image; extracting a representative text color representing the text and a representative background color representing background of the first area from the first area; and overlaying a second area containing a translation result of the text on the given image and applying the representative text color and the representative background color to a text color and a background color of the second area.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The following description relates to techniques that provide text detection and translation capabilities within an image. [Background technology]

[0002] In general, character recognition technology separates characters contained in an image captured by a camera from the background, extracts their features, and then compares them with pre-stored character feature patterns to provide the character with the most similar pattern as the recognition result.

[0003] Such general character recognition techniques have a high character recognition rate for recognizing a limited range of characters and numbers, such as vehicle license plates and postal codes, and are in practical use.

[0004] On the other hand, although there is a wide variety of languages ​​in the world, there are limitations to understanding the languages ​​of other countries, so research into automatic translation systems that translate incomprehensible languages ​​into understandable languages ​​is becoming more active.

[0005] For example, Patent Document 1 (published on January 30, 2020) discloses a technology that provides a translation service for images written in a foreign language. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Korean Patent Publication No. 10-2020-0009205 Summary of the Invention [Problem to be solved by the invention]

[0007] When providing text detection and translation functions for an image, the translation result area is provided by overlaying it on the corresponding image.

[0008] The character color and background color are extracted from the character detection area in the image, and the extracted character color and background color are applied to the character and background of the translation result area. [Means for solving the problem]

[0009] A method executed by a computer system including at least one processor configured to execute computer readable instructions contained in a memory, the method including the steps of: detecting a first region including a character in a given image by the at least one processor; extracting a character representative color representing the character in the first region and a background representative color representing a background of the first region by the at least one processor; and overlaying a second region including a translation result for the character on top of the image by the at least one processor, applying the character representative color and the background representative color as the character color and background color of the second region.

[0010] According to one aspect, the detecting step may detect the first region using an optical character reader (OCR).

[0011] According to another aspect, the extracting step may include extracting a histogram for each pixel of the first region; extracting a character base color and a background base color based on a color distribution of the first region; selecting the character representative color based on a distance value between the histogram for each pixel of the first region and the character base color; and selecting the background representative color based on a distance value between the histogram for each pixel of the first region and the background base color.

[0012] According to another aspect, the extracting step may further include transforming the first region into a certain form using an image perspective transform algorithm.

[0013] According to another aspect, the extracting step may further include sorting the histograms in order of magnitude.

[0014] According to another aspect, extracting the histogram may include extracting a 3D histogram based on a combination of color channels for each pixel in the first region.

[0015] According to another aspect, extracting the histogram may include quantizing all pixel values ​​in the first region to a particular multiple.

[0016] According to another aspect, the step of extracting the character base color and the background base color may include the steps of clustering a color distribution of the first region into two colors using a K-means algorithm, adding up the number of pixels for each of the two colors for a frame region of the first region, and selecting one of the two colors having a greater number of pixels as the background base color and selecting the remaining color as the character base color.

[0017] According to another aspect, the step of selecting the character representative color and the background representative color may include a step of selecting a color of a pixel having a distance value from the character primary color equal to or less than a threshold as the character representative color, and a step of selecting a color of a pixel having a distance value from the background primary color equal to or less than the threshold as the background representative color.

[0018] According to another aspect, selecting the representative color of the character and the representative color of the background may include increasing the threshold if there are no pixels below the threshold.

[0019] According to another aspect, the method may further include the at least one processor extracting a representative color of an image object and applying the representative color as a color constituting a UI associated with the image object.

[0020] According to yet another aspect, the method may further include the at least one processor extracting a representative color of the icon and applying it as a background theme color.

[0021] According to yet another aspect, the method may further include the at least one processor extracting a representative color from each of a previous frame and a current frame of the video, and detecting a scene change based on a difference between the representative color of the previous frame and the representative color of the current frame.

[0022] A computer program is provided, recorded on a non-transitory computer readable recording medium, for causing the computer system to execute the method.

[0023] A computer system is provided, the computer system including at least one processor implemented to execute computer-readable instructions contained in a memory, the at least one processor including a character detection unit that detects a first area including characters within a given image, a color extraction unit that extracts a character representative color that represents the character in the first area and a background representative color that represents a background of the first area, and a translation result providing unit that overlays a second area including a translation result for the character on the image, applying the character representative color and the background representative color as the character color and background color of the second area. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 illustrates an example of a network environment in accordance with an embodiment of the present invention. [Diagram 2] 2 is a block diagram illustrating the internal configuration of the electronic device and the server according to the embodiment of the present invention. FIG. [Diagram 3] FIG. 2 illustrates example components that may be included in a server processor in one embodiment of the present invention. [Figure 4]4 is a flow chart illustrating an example method that may be performed by a server in one embodiment of the present invention. [Diagram 5] FIG. 1 illustrates an example usage scenario for image text detection and translation functionality in accordance with an embodiment of the present invention. [Figure 6] 11A to 11C are diagrams illustrating a specific example of a color extraction process in an embodiment of the present invention. [Figure 7] 4 is an exemplary diagram illustrating a histogram extraction process according to an embodiment of the present invention; FIG. [Figure 8] 4 is an exemplary diagram illustrating a basic color selection process according to an embodiment of the present invention; [Figure 9] 4 is an exemplary diagram illustrating a basic color selection process according to an embodiment of the present invention; [Figure 10] 4 is an exemplary diagram illustrating a basic color selection process according to an embodiment of the present invention; [Figure 11] 4 is an exemplary diagram illustrating a histogram sorting process according to an embodiment of the present invention; FIG. [Figure 12] 5 is an exemplary diagram illustrating a representative color selection process according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0026] SUMMARY OF THE DISCLOSURE Embodiments of the present invention relate to techniques for providing text detection and translation functionality within an image.

[0027] An embodiment including those specifically disclosed in this specification provides a text detection and translation function for an image by overlaying a translation result area consisting of the text color and background color extracted in the text detection area onto the corresponding image, thereby applying the color most similar to that of the original image, thereby minimizing the sense of incongruity in the translation result area.

[0028] Fig. 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment in Fig. 1 shows an example including a plurality of electronic devices 110, 120, 130, 140, a plurality of servers 150, 160, and a network 170. Fig. 1 is merely an example for explaining the invention, and the number of electronic devices and the number of servers are not limited to those shown in Fig. 1.

[0029] The electronic devices 110, 120, 130, and 140 may be fixed terminals or mobile terminals realized by computer systems. Examples of the electronic devices 110, 120, 130, and 140 include smartphones, mobile phones, navigation systems, personal computers (PCs), notebook PCs, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), tablets, game consoles, wearable devices, Internet of Things (IoT) devices, virtual reality (VR) devices, and augmented reality (AR) devices. As an example, FIG. 1 shows a smartphone as an example of the electronic device 110, but in the embodiment of the present invention, the electronic device 110 may mean one of various physical computer systems that can communicate with other electronic devices 120, 130, and 140 and / or servers 150 and 160 via a network 170 using a substantially wireless or wired communication method.

[0030] The communication method is not limited, and may include not only a communication method using a communication network (such as a mobile communication network, a wired Internet, a wireless Internet, a broadcast network, a satellite network, etc.) that the network 170 can include, but also short-range wireless communication between devices. For example, the network 170 may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, etc. Furthermore, the network 170 may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc.

[0031] Each of the servers 150 and 160 may be realized by one or more computer devices that communicate with the multiple electronic devices 110, 120, 130, and 140 via the network 170 to provide instructions, codes, files, content, services, and the like. For example, the server 150 may be a system that provides a first service to the multiple electronic devices 110, 120, 130, and 140 connected via the network 170, and the server 160 may be a system that provides a second service to the multiple electronic devices 110, 120, 130, and 140 connected via the network 170. As a more specific example, the server 150 may provide a service (e.g., an image-based translation service, etc.) targeted by an application, which is a computer program installed and executed in the multiple electronic devices 110, 120, 130, and 140, as a first service to the multiple electronic devices 110, 120, 130, and 140. As another example, the server 160 may provide, as a second service, a service of distributing files for installing and executing the above-mentioned application to the multiple electronic devices 110, 120, 130, and 140.

[0032] Fig. 2 is a block diagram for explaining the internal configuration of an electronic device and a server in an embodiment of the present invention. In Fig. 2, the internal configuration of electronic device 110 and the internal configuration of server 150 are explained as examples of electronic devices. In addition, other electronic devices 120, 130, 140 and server 160 may have the same or similar internal configuration as electronic device 110 or server 150 described above.

[0033] The electronic device 110 and the server 150 may include memories 211 and 221, processors 212 and 222, communication modules 213 and 223, and input / output interfaces 214 and 224. The memories 211 and 221 may be non-transitory computer-readable recording media and may include persistent mass storage devices such as random access memory (RAM), read only memory (ROM), disk drives, solid state drives (SSD), flash memories, and the like. Here, persistent mass storage devices such as ROM, SSD, flash memories, and disk drives may be included in the electronic device 110 and the server 150 as persistent storage devices separate from the memories 211 and 221. In addition, the memories 211 and 221 may store an operating system and at least one program code (for example, a browser installed and executed in the electronic device 110, or a code for an application installed in the electronic device 110 to provide a specific service, etc.). Such software components may be loaded from a computer-readable recording medium separate from the memories 211 and 221. Such another computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory 211, 221 through a communication module 213, 223 that is not a computer-readable recording medium. For example, at least one program may be loaded into the memory 211, 221 based on a computer program (for example, the above-mentioned application) that is installed by a file provided via the network 170 by a developer or a file distribution system that distributes an installation file of the application (for example, the above-mentioned server 160).

[0034] The processors 212, 222 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processors 212, 222 by the memory 211, 221 or the communication modules 213, 223. For example, the processors 212, 222 may be configured to execute instructions received according to program code stored in a storage device such as the memory 211, 221.

[0035] The communication modules 213 and 223 may provide a function for the electronic device 110 and the server 150 to communicate with each other via the network 170, or may provide a function for the electronic device 110 and / or the server 150 to communicate with other electronic devices (for example, the electronic device 120) or other servers (for example, the server 160). For example, a request generated by the processor 212 of the electronic device 110 according to a program code recorded in a recording device such as the memory 211 may be transmitted to the server 150 via the network 170 under the control of the communication module 213. Conversely, a control signal, an instruction, a content, a file, etc. provided under the control of the processor 222 of the server 150 may be received by the electronic device 110 through the communication module 213 of the electronic device 110 via the communication module 223 and the network 170. For example, control signals, instructions, content, files, etc. from the server 150 received through the communication module 213 may be transmitted to the processor 212 or memory 211, and the content, files, etc. may be recorded on a recording medium (the above-mentioned permanent recording device) that the electronic device 110 may further include.

[0036] The input / output interface 214 may be a means for interfacing with the input / output device 215. For example, the input device may include a device such as a keyboard, a mouse, a microphone, or a camera, and the output device may include a device such as a display, a speaker, or a haptic feedback device. As another example, the input / output interface 214 may be a means for interfacing with a device in which functions for input and output are integrated into one, such as a touch screen. The input / output device 215 may be configured as one device together with the electronic device 110. Also, the input / output interface 224 of the server 150 may be a means for interfacing with a device for input or output (not shown) that may be connected to the server 150 or included in the server 150. As a more specific example, when the processor 212 of the electronic device 110 processes the instructions of the computer program loaded in the memory 211, a service screen or content configured using data provided by the server 150 or the electronic device 120 may be displayed on a display through the input / output interface 214.

[0037] Also, in other embodiments, the electronic device 110 and the server 150 may include more components than those in FIG. 2. However, it is not necessary to clearly show most of the components of the prior art. For example, the electronic device 110 may be realized to include at least some of the above-mentioned input / output devices 215, and may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, a database, etc. As a more specific example, if the electronic device 110 is a smartphone, various components such as an acceleration sensor, a gyro sensor, a camera module, various physical buttons, a button using a touch panel, an input / output port, a vibrator for vibration, etc., which are generally included in a smartphone, may be realized to be further included in the electronic device 110.

[0038] Specific embodiments of a method and system for extracting and providing text and background colors within an image are described below.

[0039] FIG. 3 is a block diagram illustrating example components that may be included in a server processor in one embodiment of the present invention, and FIG. 4 is a flowchart illustrating an example method that may be performed by a server in one embodiment of the present invention.

[0040] The server 150 according to the present embodiment serves as a platform for providing an image-based translation service. The server 150 may be configured with a translation result providing system implemented by a computer. The server 150 provides an image-based translation service for a plurality of electronic devices 110, 120, 130, and 140, which are clients, and may provide text detection and translation functions for an image by connecting to a dedicated application installed on the electronic devices 110, 120, 130, and 140 or a web / mobile site related to the server 150. In particular, the server 150 may provide a translation result area consisting of a text color and a background color extracted from a text detection area in an image by overlaying the corresponding image.

[0041] The processor 222 of the server 150 may include a character detection unit 310, a color extraction unit 320, and a translation result providing unit 330 as components for performing the translation result providing method according to Fig. 4, as shown in Fig. 3. Depending on the embodiment, the components of the processor 222 may be selectively included or excluded from the processor 222. Also, depending on the embodiment, the components of the processor 222 may be separated or combined to express the functions of the processor 222.

[0042] Such processor 222 and components of processor 222 may control server 150 to execute steps S410 to S430 included in the translation result providing method of Fig. 4. For example, processor 222 and components of processor 222 may be realized to execute instructions according to the code of an operating system and the code of at least one program included in memory 221.

[0043] Here, the components of the processor 222 may represent different functions executed by the processor 222 according to instructions provided by the program code recorded in the server 150. For example, the character detection unit 310 may be used as a functional representation of the processor 222 that controls the server 150 according to the above-mentioned instructions so that the server 150 detects characters from an image.

[0044] The processor 222 may read necessary instructions from the memory 221, which has been loaded with instructions related to the control of the server 150. In this case, the read instructions may include instructions for controlling the processor 222 to execute steps S410 to S430 described below. Steps S410 to S430 described below may be executed in an order different from that shown in FIG. 4, and some of steps S410 to S430 may be omitted or additional steps may be included.

[0045] In the embodiment described below, the translation result providing method of FIG. 4 is described as being executed on the server 150 side, but this is not limited thereto, and at least a part of steps S410 to S430 included in the translation result providing method may also be executed on the electronic device 110.

[0046] Referring to FIG. 4, in step S410, the character detection unit 310 may detect an area corresponding to characters (hereinafter, referred to as a "character detection area") from within an image uploaded by the electronic device 110. The processor 222 provides an image-based character translation function and may detect a character portion to be translated from within an image using a character recognition technology such as an optical character reader (OCR). The character detection unit 310 may specify a character portion in an image as a rectangular frame area, and may output character frame information including coordinate values ​​on the image for the character detection area. For example, the character frame information may include a top left coordinate, a top right coordinate, a bottom left coordinate, and a bottom right coordinate.

[0047] As another example, when the above-described character detection function is executed in electronic device 110, character detection unit 310 may receive character box information as a character detection result from electronic device 110 together with the image.

[0048] In step S420, the color extraction unit 320 may extract a color representing the character (hereinafter referred to as a "character representative color") and a color representing the background (hereinafter referred to as a "background representative color") from the character detection area. The color extraction unit 320 may derive optimal character representative colors and background representative colors through a series of image processes for a specific area in an image, i.e., the character detection area. The color extraction unit 320 may receive an image and coordinates of the character detection area in the image, and extract the character representative color and background representative color while investigating the area of ​​the corresponding coordinates as a color extraction target area. A specific color extraction process will be described in detail below.

[0049] In step S430, the translation result providing unit 330 may overlay an area including a translation result for the characters in the image (hereinafter, referred to as a "translation result area") on the corresponding image and provide it. The translation result providing unit 330 may include a translation engine and may translate the characters detected from the image into a predetermined translation language using the translation engine. In this case, the translation engine may provide a machine translation based on statistics or a machine translation based on a neural network, and may be formed by learning using a machine learning method such as deep learning. In some embodiments, the translation may be performed by another server (for example, the server 160) that can be linked with the server 150. In this case, a request for translation of the characters detected from the image may be transmitted to the server 160, and the server 160 may perform the translation using the translation engine included in the server 160 and provide the translation result to the translation result providing unit 330. The translation result providing unit 330 may overlay the translation result area on the original image, so that the user of the electronic device 110 can check the image including the translation result area.

[0050] In particular, when overlaying and displaying the translation result area on the image, the translation result providing unit 330 may apply the character representative color and background representative color extracted in step S420 to the character color and background color of the translation result area. The translation result providing unit 330 may use the character representative color and background representative color of the character detection area in the image as the character color and background color of the translation result area to be overlaid on the image.

[0051] 5, the color extraction unit 320 may receive an image file for the image 500 and character frame information including coordinate values ​​of the character detection area 501, and may then search the character detection area 501 in the image 500 based on the character frame information and extract a character representative color and a background representative color through a series of image processing. As a result, the translation result providing unit 330 may display the character color and background color of the translation result area 502 overlaid on the image 500 as the character representative color and background representative color extracted from the character detection area 501. The translation result area 502 overlaid on the image 500 may be configured to have the same position and size as the character detection area 501, as well as the character and background colors.

[0052] Therefore, the translation result providing unit 330 can minimize the sense of incongruity in the translation result area 502 by applying the colors (text and background) most similar to those of the original image 500 to the translation result area 502 .

[0053] As another example, when the above-mentioned translation result providing function is executed in the electronic device 110, the electronic device 110 receives a representative character color and a representative background color along with the translation result from the server 150 and overlays the translation result area on the image, and the representative character color and the representative background color may be applied to and displayed in the translation result area.

[0054] FIG. 6 is a diagram showing a specific example of a color extraction process in one embodiment of the present invention.

[0055] 6, an image file 601 and character frame information 602 including the coordinate values ​​of the character detection region 501 may be input to the color extraction unit 320 for the image 500.

[0056] The color extraction unit 320 searches for a region for color extraction from the image file 601, and may search for the character detection region 501 based on the character box information 602 (S61). The image file 601 and the character box information 602 input by the user are input to the color extraction unit 320, and the character detection region 501 corresponding to the character box information 602 may be set as a region of interest requiring color extraction. At this time, the color extraction unit 320 may convert the character detection region 501 into a rectangular shape using an image perspective transform algorithm. The perspective transform algorithm corresponds to a computer vision algorithm that converts a region with a tilt or twist into a desired shape (e.g., a rectangular shape).

[0057] The color extraction unit 320 may extract a 3D histogram for the character detection region 501 (S62). A conventional 2D histogram is not suitable for grasping the distribution of color combinations because calculations are performed for each color component (R, G, B), but a 3D histogram is a combination of colors that is treated as a single element, and the number of such combinations is calculated. For example, as shown in FIG. 7, the color extraction unit 320 may extract a histogram for color combinations for all pixels in the character detection region 501.

[0058] In some embodiments, the color extraction unit 320 may quantize all pixel values ​​to a specific multiple to reduce the amount of calculation required to extract a 3D histogram. For example, when obtaining a histogram for a pixel with R:123, G:234, and B:5, R:123, G:234, and B:5 may be changed to R:124, G:236, and B:8, which are multiples of 4. This can reduce the amount of calculation and time required to extract a histogram when quantization is performed.

[0059] Referring again to FIG. 6, the color extraction unit 320 may extract two predominant basic colors from all colors distributed within the text detection region 501 (S63). As an example, referring to FIG. 8, the color extraction unit 320 may cluster the color distribution of the original image corresponding to the text detection region 501 into two colors using a K-mean algorithm. As a result of the clustering, an image of the same size as the original image of the text detection region 501 is generated, and the image includes one type of label value. When clustering into two colors, a label of 1 or 2 is included.

[0060] The color extraction unit 320 may select one of the two basic colors as a background basic color and the remaining one as a character basic color. As an example, referring to FIG. 9, the color extraction unit 320 searches the frame regions 91 on four sides constituting the character detection region 501 and adds up the number of pixels labeled 1 and the number of pixels labeled 2. In this case, it is possible to search the frame region 91 having a size of one pixel as well as a frame region 91 having a predetermined size, i.e., a size of two or more pixels. In addition to searching the frame regions 91 on four sides constituting the character detection region 501, it is also possible to search some of the frame regions (e.g., upper left, lower left, upper right, lower right, etc.) among the four sides. The color extraction unit 320 may select the color of the label having the greater number of added pixels as the background basic color and the remaining one as the character basic color. The reason why the frame region 91 of the character detection region 501 is searched to select the background basic color is that it is predicted that the number of pixels corresponding to the background is greater than the number of pixels corresponding to characters in the frame region 91. In addition to the method of selecting the color most widely distributed in the frame region 91 as the background basic color, it is also possible to select the color of a pixel at a specific position determined in advance in the frame region 91 as the background basic color. For example, as shown in FIG. 10, the color of the first pixel 102 of the left frame of the frame region 91 may be selected as the background basic color. Also, the color of the first pixel of the right frame, the last pixel of the left frame, the last pixel of the right frame, the center pixel of the upper frame, the center pixel of the lower frame, etc. may be selected as the background basic color.

[0061] 6 again, the color extraction unit 320 may sort the histograms extracted in step 62 (S64). As shown in FIG 11, the color extraction unit 320 may sort the 3D histograms in order of size, and at this time, the sorted histograms may be used as a text color histogram and a background color histogram.

[0062] 6 again, the color extraction unit 320 may select a background representative color and a character representative color based on the basic colors extracted in step S63 and the histograms sorted in step S64 (S65). The color extraction unit 320 may obtain a color distance value using the histograms sorted in step S64 and the basic colors extracted in step S63, and may select the best color based on the color distance value. As an example, the Euclidean method may be used as a method for obtaining the color distance, and the distance between the color combination of the histogram and the basic colors may be obtained by the Euclidean method.

[0063] As shown in FIG. 12, the color extraction unit 320 calculates the distance between the color combination of the histogram and the basic color for each of the text basic color and the background basic color in the sorting order of the histogram. The color extraction unit 320 may select a pixel color 121 having the shortest distance from the text basic color or a distance equal to or less than a predetermined threshold (e.g., 40) as the text representative color, and select a pixel color 122 having the shortest distance from the background basic color or a distance equal to or less than the threshold as the background representative color. The representative color has the characteristic of showing a large color distribution in the image and having a color similar to the basic color. If no pixel having a distance equal to or less than the threshold is found, the threshold is increased, and the above process is repeated.

[0064] Therefore, the color extraction unit 320 can obtain the optimal character representative color and background representative color for the image by using the histogram and the dominant basic color for the original image of the character detection area.

[0065] The color extraction method according to the present invention can be used in various environments other than an environment in which text in a given image is translated and provided. As an example, text in a real-time video being shot by the electronic device 110 may be translated, and a suitable color may be searched for and overlaid on the translation result. As another example, a representative color of an image object may be searched for and used as a color constituting a UI (user interface) related to the object. For example, a representative color of an icon displayed on the screen of the electronic device 110 may be searched for and used as a background theme color. In this case, a color similar to the outline area of ​​the icon and a color not similar to the outline area of ​​the icon may be extracted based on two representative colors of the icon. As another example, the color extraction method according to the present invention may be used for detecting a scene change in a video. For example, a representative color may be extracted from each of a previous frame and a current frame, and it may be determined that a scene has changed if the difference between the representative colors of the frames is equal to or greater than a certain size. Furthermore, a time point determined as a scene change by the method of extracting the representative color may be set as an insertion position of additional content (e.g., advertisement, etc.).

[0066] In this way, according to an embodiment of the present invention, when providing a translation result for characters in an image by overlaying the corresponding image, a representative character color and a representative background color are extracted from the character detection area, and the extracted representative character color and representative background color are applied to the characters and background of the translation result area, thereby minimizing the sense of incongruity of the translation result area on the image.

[0067] The above-mentioned devices may be realized by hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described in the embodiments may be realized using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications that run on the OS. The processing device may also respond to the execution of the software to access, record, manipulate, process, and generate data. For convenience of understanding, a single processing device may be described as being used, but one skilled in the art will understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0068] The software may include computer programs, codes, instructions, or a combination of one or more of these, and may configure a processing device to operate as desired or instruct the processing device, either independently or collectively. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed and stored or executed in a distributed manner on computer systems connected by a network. The software and data may be stored on one or more computer-readable storage media.

[0069] The method according to the embodiment may be realized in the form of program instructions executable by various computer means and recorded on a computer-readable medium. Here, the medium may be one that continuously records a computer-executable program or one that temporarily records the program for execution or download. The medium may be one of various recording or storage means in the form of a single or multiple hardware devices combined, and may not be limited to a medium directly connected to a certain computer system, but may be one that is distributed over a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROMs, RAMs, flash memories, etc., configured to record program instructions. Other examples of the medium include recording media or storage media managed by application stores that distribute applications, or sites, servers, etc. that supply or distribute various other software.

[0070] Although the embodiments have been described based on the limited embodiments and drawings as above, those skilled in the art will be able to make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from that described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a manner different from that described, or may be counterbalanced or replaced by other components or equivalents, and still achieve suitable results.

[0071] Therefore, different embodiments are within the scope of the appended claims provided they are equivalent to the claims. [Explanation of symbols]

[0072] 222: Processor 310: Character detection unit 320: Color extraction section 330: Translation result providing department

Claims

1. 1. A method executed by a computer system, comprising: The computer system includes at least one processor configured to execute computer-readable instructions contained in a memory; The method comprises: detecting, by the at least one processor, a first region in a given image that includes text; extracting, by the at least one processor, a character representative color representative of the character in the first region and a background representative color representative of a background of the first region; and the at least one processor overlays a second region including a translation result for the character on the image, and at this time, applies the character representative color and the background representative color as a character color and a background color of the second region. Including, The extracting step comprises: extracting a histogram for each pixel of the first region; extracting a character basic color based on the color distribution of the first region; and selecting the character representative color based on a distance value between the histogram and the character basic color for each pixel in the first region; A method comprising:

2. The detecting step includes: The method of claim 1 , further comprising detecting the first region using OCR.

3. The step of extracting the character representative color representing the character and the background representative color representing the background of the first region in the first region comprises: extracting a background base color based on the color distribution of the first region; and selecting the background representative color based on a distance value between the histogram and the background basic color for each pixel in the first region; The method of claim 1 , comprising:

4. The step of extracting the character representative color representing the character and the background representative color representing the background of the first region in the first region comprises: converting the first region into a certain shape using an image perspective transformation algorithm; The method of claim 1 further comprising:

5. The step of extracting the character representative color representing the character and the background representative color representing the background of the first region in the first region comprises: Sorting the histograms in order of magnitude. The method of claim 1 further comprising:

6. The step of extracting the histogram comprises: extracting a 3D histogram according to a combination of color components for each pixel in the first region; The method of claim 1 , comprising:

7. The step of extracting the histogram comprises: quantizing all pixel values ​​of the first region to a particular multiple; The method of claim 1 , comprising:

8. The steps of extracting the character basic color and the background basic color include: clustering the color distribution of the first region into two colors using a K-means algorithm; summing up the number of pixels for each of the two colors in a frame region of the first region; and selecting one of the two colors having a larger number of pixels as the background base color and selecting the remaining color as the character base color. The method of claim 3 , comprising:

9. The steps of selecting a character representative color and selecting a background representative color include: selecting a color of a pixel having a distance value between the color of the pixel and the basic character color that is equal to or smaller than a threshold value as the representative color of the character; and selecting a color of a pixel whose distance value from the background basic color is equal to or smaller than the threshold value as the background representative color; The method of claim 3 , comprising:

10. The steps of selecting a character representative color and selecting a background representative color include: increasing the threshold if there are no pixels below the threshold.

10. The method of claim 9, comprising:

11. The method comprises: The at least one processor extracts a representative color of an image object and applies the representative color to a UI associated with the image object. The method of claim 1 further comprising:

12. The method comprises: The at least one processor extracting a representative color of the icon and applying it as a background theme color. The method of claim 1 further comprising:

13. The method comprises: the at least one processor extracting a representative color from each of a previous frame and a current frame of a moving image, and detecting a scene change based on a difference between the representative color of the previous frame and the representative color of the current frame. The method of claim 1 further comprising:

14. A computer program product adapted to cause said computer system to carry out the method according to any one of claims 1 to 13.

15. 1. A computer system comprising: At least one processor configured to execute computer readable instructions contained in the memory. Including, The at least one processor a character detector for detecting a first region containing characters within a given image; a color extraction unit that extracts a character representative color that represents the character in the first region and a background representative color that represents a background of the first region; and a translation result providing unit that overlays a second area including a translation result for the character on the image, and provides the second area by applying the representative character color and the representative background color as a character color and a background color of the second area; Including, The color extraction unit Extracting a histogram for each pixel in the first region; Extracting a basic character color based on the color distribution of the first region; selecting the character representative color based on a distance value between the histogram and the character basic color for each pixel in the first region; Computer system.

16. The color extraction unit Extracting a background base color based on the color distribution of the first region; The background representative color is selected based on a distance value between the histogram and the background basic color for each pixel in the first region.

16. The computer system of claim 15.

17. The color extraction unit A 3D histogram of a combination of color components is extracted for each pixel in the first region.

16. The computer system of claim 15.

18. The color extraction unit quantizing all pixel values ​​of said first region to a particular multiple; 16. The computer system of claim 15.

19. The color extraction unit clustering the color distribution of the first region into two colors using a K-means algorithm; summing up the number of pixels for each of the two colors in a frame area of ​​the first region; The color having the greater number of pixels among the two colors is selected as the background base color, and the remaining color is selected as the character base color.

17. The computer system of claim 16.

20. The color extraction unit a color of a pixel whose distance value from the basic character color is equal to or smaller than a threshold value is selected as the representative character color; A color of a pixel whose distance value from the background basic color is equal to or less than the threshold value is selected as the background representative color.

17. The computer system of claim 16.

Images