Electronic device for extracting word included in image and methods thereof

Abstract

An electronic device is disclosed. The present device: divides characters included in an input image into at least one box by grouping the characters on the basis of characteristics of each character; configures a plurality of character strings by connecting adjacent characters included in the at least one box in a plurality of directions; identifies a use frequency of each of the plurality of character strings on the basis of a word database including words used by a plurality of users; and extracts one of the plurality of character strings according to the use frequency. Accordingly, the character included in the image can be accurately detected.

Description

Electronic devices and methods for extracting words contained in images

[0001] The present invention relates to an electronic device and method for extracting words contained in an image.

[0002] Previously, users had to manually input characters to perform Internet searches or translations, but with the advancement of OCR technology, electronic devices can now directly recognize characters through photos or live views and use them as linguistic information.

[0003] Accuracy is crucial when extracting text from images and using it for linguistic purposes like search or translation. While it was possible to detect text within images in the past, this process involved grouping adjacent characters together and extracting them as words without considering their linguistic meaning. This led to problems such as inaccurately extracting the words users were looking for or errors occurring.

[0004] Accordingly, the need for a method to more accurately extract text contained in an image has arisen.

[0005] According to at least one embodiment of the present disclosure, an electronic device includes a memory and a processor for storing an input image.

[0006] The processor groups characters included in the input image based on the characteristics of each character and divides them into at least one box, connects adjacent characters included in the at least one box in multiple directions to form a plurality of strings, and identifies the frequency of use of each of the plurality of strings based on a word database including words used by a plurality of users, and extracts one of the plurality of strings based on the frequency of use.

[0007] In addition, a method for extracting words using an electronic device includes a step of grouping characters included in an input image based on the characteristics of each character and dividing them into at least one box, a step of connecting adjacent characters included in at least one box in multiple directions to form a plurality of character strings, a step of identifying the frequency of use of each of the plurality of character strings based on a word database including words used by a plurality of users, and a step of extracting one of the plurality of character strings based on the frequency of use.

[0008] In addition, in a non-transitory computer-readable medium storing one or more instructions executed by a control unit of an electronic device to cause the electronic device to perform an operation, the operation is a computer-readable medium including a step of grouping characters included in an input image based on characteristics of each character and dividing the characters into at least one box, a step of connecting adjacent characters included in at least one box in a plurality of directions to form a plurality of character strings, a step of identifying a frequency of use of each of the plurality of character strings based on a word database including words used by a plurality of users, and a step of extracting one of the plurality of character strings based on the frequency of use.

[0009] FIG. 1 is a diagram for explaining a word extraction operation of an electronic device according to at least one embodiment of the present disclosure.

[0010] FIG. 2 is a block diagram illustrating a configuration of an electronic device according to at least one embodiment of the present disclosure.

[0011] FIGS. 3 and 4 are drawings for explaining examples of cases where characters included in an image according to at least one embodiment of the present disclosure are incorrectly detected.

[0012] FIGS. 5 to 9 are drawings for explaining a method of extracting words from an image according to at least one embodiment of the present disclosure.

[0013] FIGS. 10 and 11 are drawings for explaining the results of correcting a word included in an image that is incorrectly detected according to at least one embodiment of the present disclosure.

[0014] FIG. 12 is a block diagram showing a configuration when an electronic device according to at least one embodiment of the present disclosure is a smartphone.

[0015] FIG. 13 is a flowchart illustrating a method for an electronic device to extract words included in an image according to at least one embodiment of the present disclosure.

[0016] FIG. 14 is a flowchart illustrating a method for extracting characters included in an image into words based on their frequency of use according to at least one embodiment of the present disclosure.

[0017] The terms used in the various embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should be defined based on the meaning of the terms and the overall content of this disclosure, rather than simply their names.

[0018] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0019] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0020] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0021] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0022] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this disclosure, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0023] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0024] In this disclosure, the term user may refer to a person using an electronic device or a device used by the person.

[0025] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0026] FIG. 1 is a diagram for explaining a word extraction operation of an electronic device according to at least one embodiment of the present disclosure.

[0027] According to FIG. 1, the electronic device (100) can recognize characters included in an image and extract linguistically meaningful words using the location and language information of each character.

[0028] In FIG. 1, the image (10) may be input in various ways. For example, the image (10) may be an image previously stored in the memory of the electronic device (100), may be obtained by taking a picture with a built-in camera or an external camera, may be obtained by downloading an image from a server, or may be obtained by receiving an image from an external device. In addition, the image may be a live view image corresponding to light incident through the camera lens and focused on the image sensor, as well as a photographed image captured by the camera.

[0029] In addition, the characters included in the image may be characters of various languages, and may include not only characters but also symbols or numbers. In FIG. 1, the electronic device (100) is illustrated as a smartphone, but is not limited thereto, and the electronic device (100) may be implemented as various types of electronic devices such as a server device, a PC, a laptop PC, a tablet PC, a set-top box, a TV, a kiosk, and other home appliances. When the electronic device (100) is implemented as a server device or a set-top box, an external camera or an external display device may be connected and used.

[0030] When an electronic device (100) receives an image (10), it groups characters included in the input image based on the characteristics of each character and divides them into at least one box. In addition, the electronic device (100) connects adjacent characters included in at least one box in multiple directions to form a plurality of character strings, identifies the frequency of use of each of the plurality of character strings based on a word database including words used by a plurality of users, and extracts one of the plurality of character strings based on the frequency of use.

[0031] In the present disclosure, a box is a virtual area for specifying a portion containing characters or words extracted by an electronic device. Alternatively, it may be described as a block or an area, but in the present disclosure, it is described as a box to help with more intuitive understanding.

[0032] In addition, the word database can be a data set containing various words. The word database may be otherwise referred to as a word set, library, word group, or word collection, but is described as a word database in the present disclosure. The word database may be pre-stored in the memory of the electronic device (100) or may be stored on an external server. The external server may be a workstation, a cloud, a data drive, a data station, etc. For example, in the case of a Social Network Service (SNS) server, a word database can be built by collecting words uploaded by various users on a social network. Other web servers can also extract words contained in posts uploaded to web pages and store them in the form of a database.

[0033] When an image (10) of a signboard such as FIG. 1 is input into an electronic device (100), the electronic device (100) distinguishes characters included in the image (10) by character unit. In the example of FIG. 1, the characters “24-hour” can be distinguished into “2,” “4,” and “hour.” FIG. 1 illustrates an image (10) including characters in Korean. In the Korean language, consonants and vowels combine to form a syllable, and at least one syllable can be combined to form a word. In this case, the aforementioned character unit may be a syllable unit. On the other hand, in the case of English, at least one alphabet listed in one direction can form a word. Therefore, in the case of English, the electronic device (100) can distinguish characters by alphabet unit. In the case of Chinese, characters can be distinguished by Hanja unit, and in the case of Japanese, characters, Hiragana, or Katakana unit. The following description will be based on Korean.

[0034] After distinguishing the character units, the electronic device (100) groups characters with similar characteristics by considering characteristics such as the size of the distinguished characters, the distance between characters, the font, etc. In the present disclosure, grouping may be an operation of specifying characters included in the same box.

[0035] In the case of Fig. 1, since the fonts of each word included in the sign image (10) are all the same, the electronic device (100) performs grouping based on the font size and distance. As a result, the electronic device (100) forms a total of three groups by distinguishing “2, 4, si”, “so, mang, ma, t”, and “bok, juk, jab, hwa, eol, eum”. In the present disclosure, a group may be an area composed of at least one box. That is, in Fig. 1, “2, 4, si” and “so, mang, ma, t” are each identified by one box (11, 12), and the “bok, juk, jab, hwa, eol, eum” group is distinguished by a total of three boxes (13, 14, 15).

[0036] Furthermore, to determine whether the characters within each box can be connected to form a word, the electronic device (100) compares the characteristics of the formed boxes. The three groups formed in FIG. 1 are determined to be groups with no possibility of connection because their characteristics, such as size and direction, differ from each other. Therefore, adjacent characters within a box are connected to form a string. A string can be a value obtained by arranging multiple characters consecutively.

[0037] At this time, the “2,4,si” and “so,mang,ma,t” written in the signboard image (10) are less likely to be incorrectly extracted as the strings are connected only horizontally, but “bok,juk,jap,hwa,eol,eum” can be read horizontally or vertically, so there is a possibility that they will be read as “bokjap-eol, jukhwa-eum” rather than as normal words with meanings such as “firecracker,” “general merchandise,” and “ice.”

[0038] At this time, in order to extract accurate words, the electronic device (100) identifies the frequency of use of each of a plurality of strings based on a word database containing words used by a plurality of users. Since “bokjab-eol, jukhwa-eum” are non-existent words, the above-mentioned words will not be searched in the word database, and the electronic device (100) can accurately extract the characters included in the sign image (10) as the words “24 hours” (11), “SOMANG mart” (12), “firecracker” (13), “general merchandise” (14), and “ice” (15).

[0039] The process of grouping characters, comparison between boxes, and direction of connecting words in the operation of the electronic device (100) described above will be described in detail later based on other drawings.

[0040] FIG. 2 is a block diagram illustrating a configuration of an electronic device according to at least one embodiment of the present disclosure.

[0041] According to FIG. 2, the electronic device (100) includes a memory (110) and a processor (120).

[0042] The memory (110) is a configuration for storing various software, commands, control codes, and data required for the operation of the electronic device (100). The memory (110) may be implemented as at least one of various memories, such as DRAM (dynamic RAM), SRAM (static RAM), SDRAM (synchronous dynamic RAM), OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory, hard drive, or solid state drive (SSD).

[0043] According to at least one embodiment of the present disclosure, the memory (110) may store an input image or a word database containing words used by multiple users. Furthermore, the processor (120) may store software modules for performing operations. Specifically, the memory (110) may store various software modules, such as a module for finding a character area on the screen, a module for recognizing characters in the character area, and a module for re-aligning characters to the correct units and directions by considering the linguistic directionality and area of ​​the recognized characters.

[0044] The processor (120) is configured to control the overall operation of the electronic device (100).

[0045] The processor (120) may include one or more of a digital signal processor (DSP), a microprocessor, a central processing unit (CPU), a microcontroller unit (MCU), a microprocessor (MPU), a controller, an application processor (AP), a communication processor (CP), an ARM processor, or an artificial intelligence (AI) processor, or may be defined by the terms thereof. In addition, the processor (120) may be implemented as a system on chip (SoC) or large scale integration (LSI) having a processing algorithm built in, or may be implemented in the form of a field programmable gate array (FPGA). The processor (120) may perform various functions by executing computer executable instructions stored in the memory (110). The processor (120) may be composed of one or more processors.

[0046] The processor (120) performs various operations using the memory (110).

[0047] Specifically, the processor (120) can store an image input into the memory (110) when the image is transmitted from an external device via wired or wireless communication, or when the image is input through a camera built into or connected to the electronic device (100). Alternatively, there may be an image already stored in the memory (110) in advance.

[0048] The processor (120) uses a character recognition module to identify characters contained in an image stored in the memory (110) on a character-by-character basis. The processor (120) identifies the characteristics of each identified character. The characteristics may include various information such as the size, color, font, and location of the character. When characters with the same or similar characteristics are located within a certain distance range, the processor (120) can group them and distinguish them into at least one box.

[0049] In addition, the processor (120) can form a plurality of strings by connecting adjacent characters among the characters included in each box in a plurality of directions. The plurality of directions may be, but are not limited to, horizontal, vertical, and diagonal directions, and may include directions in which characters are connected at various angles. For example, while words are generally written horizontally in most countries, some countries may write them vertically. Alternatively, words may be written diagonally or in a direction that is curved to have an arbitrary curvature, such as on billboards or other postings. If a plurality of characters are sequentially and side by side in these various directions, the processor (120) can connect them to form a single string.

[0050] The processor (120) can search for a constructed string in a word database and identify the frequency of use of the string. If the frequency of use is 0 or a very low number, the processor (120) can identify the string as meaningless. Conversely, if the frequency of use is above a threshold frequency, the processor (120) can recognize the string as constituting a meaningful word within the image. The recognized word can be utilized in various ways, which will be described in detail in the following section.

[0051] Meanwhile, while the processor (120) has been described above as directly detecting boxes from an input image, the grouping method can be implemented in various ways. For example, the processor (120) can group using a bitmap clustering method.

[0052] Specifically, the processor (120) can first detect an area containing characters within an input image. The operation of detecting an area containing characters is performed by a module that searches for character areas. A character area may be a partial image that includes an object estimated to be a character.

[0053] The method of detecting a text area can be performed according to various algorithms. For example, the processor (120) identifies the pixel value of each pixel in the input image and detects an area where pixel values ​​in the same or similar ranges are continuously arranged. The processor (120) determines whether the pixel value and the shape of the area in the detected area correspond to a character, and can detect the surrounding area including the character as a text area. The text area may be the character itself, but is not limited thereto, and the background area surrounding the character may also be detected as a text area. That is, the processor (120) can detect the text area by cropping the portion containing the character from the entire input image. When the text area is detected, the processor (120) can also determine the rotation angle (box direction) of the detected text area. When users take a picture using a camera, they generally take the picture while holding the camera vertically with respect to the floor, but depending on the situation, they may take the picture by turning the camera horizontally or changing the vertical position. The image captured in this state may be rotated at various angles, such as 90 degrees, 180 degrees, and 270 degrees, compared to a normal captured image. The processor (120) can identify the rotation angle of the image by detecting at least one character in the image and then comparing the shapes of the phonemes (e.g., ㄱ, ㄴ, ㅏ, ㅑ, etc.) constituting the character with the shapes of normal phonemes. In the case of alphabets, the rotation angle can be identified by comparing the shape of each alphabet with a reference shape.

[0054] Alternatively, the processor (120) may recognize the rotation direction and rotation angle of the image based on metadata added to the image. That is, if the device that captured the image includes a geomagnetic sensor, a gyro sensor, an azimuth sensor, etc., the device may detect pitch angle, roll angle, yaw angle, etc. based on the output values ​​of each sensor at the time of capture, and then include the detected pitch angle, roll angle, yaw angle, etc. in the metadata for the corresponding photo. The electronic device (100) may recognize the rotation direction and rotation angle of the image based on the pitch angle, roll angle, yaw angle, etc. included in the metadata. When template matching or pattern recognition technology is used for character recognition, rotated characters may not match general templates or patterns, thereby reducing the accuracy of the character recognition function of the processor (120). Taking this into account, the processor (120) may determine the angle and direction by which the detected character area is rotated to determine whether a word including characters is written horizontally or vertically.

[0055] Upon detecting a character region, the processor (120) can identify a box direction based on the rotation angle of the character region and a confidence score for the comparison results between characters. The box direction may be the arrangement direction of a box that groups characters included in the character region. Specifically, the box direction may be a horizontal direction, a vertical direction, a diagonal direction, or other curved directions.

[0056] Additionally, the processor (120) can identify the center position of each individual character to extract characters within a box. Specifically, the processor (120) can identify the total area occupied by a syllable based on the sizes and mutual distances of the phonemes constituting the syllable, and then identify the portion corresponding to the center within that total area as the center position. The processor (120) can distinguish characters based on the center position of each individual character.

[0057] After the character classification process, the processor (120) identifies the characteristics of each character through a module that analyzes the characteristics of each individual character. Specifically, the font characteristics, color, size, etc. of the character can be identified. The processor (120) compares the similarity of the identified characteristics and groups characters that are adjacent to each other and within a similar range into a single box.

[0058] Meanwhile, when there are multiple characters in an image, multiple boxes may be identified through grouping. When the characters included in the input image are divided into multiple boxes, the processor (120) identifies the possibility of connection between the characters included in each of the multiple boxes based on the positions and distances between the multiple boxes. Factors that can be considered at this time may include, as described above, the size of the boxes, the linear distance between the boxes, and the area of ​​the overlapping area (Projection IOU) when the boxes are moved to overlap each other.

[0059] The processor (120) determines that two boxes are related when the sizes of the boxes are similar to each other, the distance between the boxes is close, and the area of ​​the overlapping area between the boxes is large. In addition, the position, size, and order of the characters contained within the boxes may also serve as criteria for the determination. The reason for determining whether there is a relationship between boxes is because, in some cases, the connection between characters is not properly established during the grouping process following the step of recognizing individual characters.

[0060] Another embodiment of determining the possibility of connection between characters is a method of directly comparing the coordinates or sizes of individual characters without determining the relationship between boxes. However, this specification focuses on a method of determining the possibility of connection between characters by comparing the relationship between boxes.

[0061] If two boxes are related, then even if the identified characters are grouped into different groups, it is possible to connect at least one word from each group to form a word. Conversely, if there is no connection between two boxes, characters in different boxes cannot be connected to each other, and connections exist only between characters within the boxes.

[0062] Once the correlation between boxes is determined, the processor (120) constructs multiple strings of characters that can be connected. Characters can be connected in multiple directions to construct strings.

[0063] The processor (120) can construct strings in various ways, as described above. When multiple strings are constructed, the processor (120) identifies the frequency of use of each string based on a word database, and extracts the string if the frequency of use exceeds a threshold. The extracted string can be identified as a word contained within the corresponding image.

[0064] As described above, the word database may be data collected from words actually used, and may be stored directly in the electronic device (100) or in an external device. Since the processor (120) extracts words from data from multiple users, more accurate results can be obtained.

[0065] Specifically, the processor (120) can configure a first string by connecting adjacent characters included in at least one box horizontally, a second string by connecting them vertically, and a third string by connecting them diagonally. The processor (120) identifies the usage frequency of each configured string based on a word database, and if the identified usage frequency exceeds a threshold, it can be determined to be a linguistically meaningful word and extract the string as a word.

[0066] Meanwhile, a single string may contain multiple words. That is, if multiple words are written sequentially, without spacing between them, at regular intervals, the words can be recognized as a single string.

[0067] If at least one of the first to third strings contains multiple words, the processor (120) can detect each of the multiple words contained in the strings. The processor (120) can identify the frequency of use of each detected word based on the aforementioned word database, and extract at least one word whose identified frequency of use exceeds a threshold. A detailed example of this will be described later.

[0068] FIGS. 3 and 4 are drawings for explaining examples of cases where characters included in an image according to at least one embodiment of the present disclosure are incorrectly detected.

[0069] According to Fig. 3, the words included in the first signboard image (30) were divided and detected.

[0070] The words included in the first signboard image (30) are “HyunSuMak; banner” and “SilSa; actual image printing.” However, according to conventional technology, the distinction between words is not made accurately, so there is a risk that the characters will be detected as “HyunSu” (31), “Mak” (32), “Sil” (33), and “Sa” (34).

[0071] Also, according to Fig. 4, the word included in the second sign image (40) was detected as being divided in the wrong direction.

[0072] The words included in the second signboard image (40) are “MaeMae (deal)” and “Weolsae (monthly rent)”. According to conventional technology, there is a risk of extracting meaningless words such as “Maeweol (41)” and “MaeSae (42)” by misjudging words written horizontally as vertically.

[0073] For example, in an image in English, if the word “Sale” is written horizontally and the word “Rent” is written horizontally below the word “Sale,” there is a risk that the four words “SR,” “AE,” “Ln,” and “ET” will be recognized instead of the words “Sale” and “Rent.”

[0074] These problems arise because conventional techniques determine character groups and orientations solely based on their location, such as whether the characters are adjacent to each other, without considering the user's word usage frequency or the linguistic information of the characters. Various embodiments of the present disclosure can minimize this risk.

[0075] In the following, a detailed example of extracting words by considering both linguistic information and positional information of characters based on Figures 5 to 9 is described.

[0076] FIGS. 5 to 9 are diagrams for explaining a method of extracting words from an image according to various embodiments of the present disclosure.

[0077] Figure 5 is an image containing various words with different writing directions and fonts.

[0078] Before extracting words, the processor (120) detects character regions using a module that finds character regions. Accordingly, the processor (120) recognizes all characters included in the image and separates character portions from non-character portions.

[0079] In Figure 5, “Sodaemi (51)”, “Hyung (52)”, “HyungGyeonYong (53)”, “MeongMeong (55)”, “Meong-meong (56)”, “Gawuicutjeonmun (54)”, “#Miyong (57)”, “#Hotel (58)”, and “#LolE (59)” are shown as character regions extracted. The character region detection step is a step that detects only the region where characters exist before extracting words that have linguistic meaning, as described above. Therefore, in this step, even if characters are grouped into the same region, they may not have linguistic meaning. For example, among the character regions detected in Figure 5, “Hotel (58)” has the linguistic meaning of “hotel” meaning a lodging establishment, but “Sodaemi (51)” has no linguistic meaning.

[0080] After detecting the character area, the processor (120) recognizes and reassembles individual characters to extract meaningful words. In this case, the processor (120) can identify the rotation angle (box direction) and center position of the character area. Since this has been described above, redundant description is omitted. The processor (120) divides the character area into character units based on the center position of each character. The reason for identifying the center position of each character is to accurately divide the strings arranged at regular intervals within the character area into each character.

[0081] Figure 6 shows the state in which characters within a character area are divided into character units.

[0082] According to FIG. 6, the processor (120) determines that the character areas including “Sodaemi (51)” and “HyungGyeonYong (53)” are arranged vertically, divides the character areas horizontally (51-1, 53-1), and then recognizes the individual characters “So,” “Dae,” “Mi,” “Hyung,” “Hyung,” “Gyeon,” and “Yong” (51-1, 5253-1). Since the remaining character areas are arranged horizontally, the processor (120) divides the character areas vertically to recognize the individual characters (54-1, 55-1, 56-1, 57-1).

[0083] After individual character recognition, the processor (120) determines whether at least one of the font and size among each character identified as a character unit through bitmap clustering described above in FIG. 2 is included in a similar range and whether the characters are arranged adjacently in the horizontal, vertical, or diagonal direction, and groups characters with similar characteristics.

[0084] FIG. 7 is a diagram illustrating a grouping of adjacent character areas, each having a similar font and size, and separated by a box. Characters (51-1) included in the first character area, the individual character “Hyung,” and characters (53-1) included in the second character area are adjacent to each other, and have similar fonts and character sizes. In addition, characters (54-1) included in the third character area and individual characters included in the above-described character areas are also located at an adjacent distance, and have similar fonts and character sizes, so they can be separated into one group (hereinafter referred to as the first group (701)). On the other hand, characters included in the first group (701) are not adjacent to characters included in other character areas located on the right, and since their fonts and character sizes are clearly different, they cannot be grouped together.

[0085] As described above, the characters (55-1) included in the 4th character area, the characters (56-1) included in the 5th character area, and the characters (57-1) included in the 6th character area are not located in adjacent positions and have differences in font and character size. On the other hand, the characters (55-1) included in the 4th character area have similar font and character size and are adjacent, so they can be grouped together. Therefore, the characters included in the 4th character area are not grouped together with other character areas, but become a group themselves (hereinafter referred to as the second group (702)). Other character areas not described above also become a group themselves for the same reason. In Fig. 7, the processor (120) can set a total of 4 boxes. In the following description, a box composed of characters (56-1) included in the fifth character area (56) is described as a third box (703), and a box composed of characters (57-1) included in the sixth character area (57, 58, 59) is described as a fourth box (704).

[0086] Once grouping is complete, the processor (120) identifies the possibility of connection between characters contained in each of the plurality of boxes based on the positions and distances between the plurality of boxes. In other words, it connects characters that have the potential to form a meaningful word. To determine this, it is first determined whether there is a connection between the boxes that distinguish the groups of characters. As described above in FIG. 2, if both boxes are connected, there is a possibility that the identified characters can be connected to form a word even if they are grouped into different groups. On the other hand, if there is no connection between the two boxes, the characters in different boxes cannot be connected to each other, and the possibility of connection exists only between the characters within the boxes.

[0087] Since the factors determining the possibility of connection have been described above, duplicate description is omitted.

[0088] According to FIG. 8, the first box (701) and the second box (702) have a large difference in box size, are not adjacent to each other, and the distance between the boxes is too far for the characters to be connected to form a word, so the two boxes are not related. Therefore, there is no room for the characters in the first box (701) and the characters in the second box (702) to be connected to each other. Using this judgment method, it can be determined that the first box (701) is not related to the third box (703). Although the fourth box (704) is adjacent to the first box (701), the connection direction of the characters in each box is not consistent and unnatural for forming a word, so the two boxes can be determined to be not related. Therefore, the first box (701) can only be connected to the characters within the box. Through the above-described judgment method, it is possible to determine that the second box (702) to the fourth box (704) are also not related to other boxes, and that only the characters within each box can be connected.

[0089] When the correlation between boxes is determined, the electronic device (100) connects characters to form a string that is a candidate for a word. Based on the character “Dae” in the first box, there are two strings that can be formed: “SoDaeMiGa” in the vertical direction and “Daehyung-gyeon” in the horizontal direction. In this way, various strings can be formed depending on the direction in which words are connected. On the other hand, there are cases where the character connection direction is formed in only one direction. This corresponds to cases where strings can be formed only in the horizontal direction, such as the second to fourth boxes. In this case, one string is formed.

[0090] After composing a string, the processor (120) identifies the frequency of use of each string based on a word database, and then extracts a word by judging it as a word with linguistic meaning if the frequency of use exceeds a threshold.

[0091] The above-mentioned “SoDaeMiGa” is not searched in the word database because it is a non-existent word, but “Daehyung-gyeon” means a big-sized dog, so the number of times the word “Daehyung-gyeon” is used in the word database will exceed a threshold. Finally, the processor (120) can extract the word “Daehyung-gyeon.”

[0092] Also, as shown in FIG. 9, there may be cases where multiple words with linguistic meaning are included in a string. That is, “#Beauty#Hotel#Play” in FIG. 9 is a string that includes three words: “Beauty,” “Hotel,” and “Play.” In this case, as described above, words that are used in excess of a threshold can be extracted through a word database. The processor (120) determines the number of cases in which adjacent letters can be combined to form words in the string “#Beauty#Hotel#Play” in FIG. 9, and based on the word database, divides the string into “#Beauty (Miyeong; #grooming shop),” “#Hotel (Hotel; #hotel),” and “#Play (NolE; #play),” thereby extracting words with meaning.

[0093] FIGS. 10 and 11 are drawings for explaining the results of correcting a word included in an image that is incorrectly detected according to at least one embodiment of the present disclosure.

[0094] According to FIG. 10, the words included in the sign image (1000) are originally “MaeMae (deal) (1003)”, “JuTeok (house) (1004)”, and “Togi (land) (1005)”, but if the direction of progression of the words is detected by grouping them differently from their linguistic meaning, words without meaning, such as “MaeJuTo (1001)” and “MaeTaekJi (MaeTaekJi; 1002)”, may be identified. According to various embodiments of the present disclosure, such errors can be resolved by determining the frequency of use of a string composed according to the direction of connection of words.

[0095] Image (1100) of Fig. 11 illustrates a case where an error occurred because the word progression direction was correctly determined, but the word units were not properly grouped. In the area (1103) where the error occurred, the words "HakGyoCheYook (school physical education) (1104)" and "Julnumgi (jump rope) (1105)" should have been extracted, but an error may occur because the word connection was not properly made. According to various embodiments of the present disclosure, such an error can be resolved through a grouping operation and a method for determining the correlation between boxes.

[0096] In this way, the word extraction method described above based on FIGS. 2 to 9 can more accurately extract words included in an image because it considers not only the location of characters but also linguistic characteristics.

[0097] FIG. 12 is a block diagram showing a configuration when an electronic device according to at least one embodiment of the present disclosure is a smartphone.

[0098] According to FIG. 12, a smartphone (1200) includes a memory (1230), a communication unit (1220), a camera (1240), a display (1250), and a processor (1210).

[0099] The memory (1230) is configured to store various software, commands, control codes, and data required for the operation of the smartphone (1200). Specific examples of the memory are described in Fig. 2, so they are omitted.

[0100] The communication unit (1220) is configured to perform communication with at least one external device.

[0101] For example, a smartphone (1200) can communicate with other external devices through a communication unit (1220). Specifically, the communication unit (1220) can communicate with external devices that store a word database.

[0102] The processor (1210) can transmit a plurality of strings configured in the manner described in the various embodiments described above to an external device through the communication unit (1220), and receive information on the usage frequency of each of the plurality of strings from the external device.

[0103] Specifically, the processor (1210) can communicate with a web portal server through a communication unit (1220) to download information from a database of words actually used by users and store or periodically update the information in a memory (1230), and can extract words by downloading images existing on a web server.

[0104] The communication unit (1220) can transmit and receive various signals and data with external devices through various wired and wireless communication methods such as Bluetooth, AP-based Wi-Fi (Wireless LAN network), Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc.

[0105] The camera (1240) is a hardware device built into the smartphone device (1200) that provides photo and video recording functions.

[0106] When implemented in the form of a smartphone including a camera (1240) as in Fig. 12, the smartphone (1200) can identify characters included in a photo taken by the camera (1240) or a live view image corresponding to light incident through the camera in the above-described manner.

[0107] The display (1250) is configured to display various screens under the control of the processor (1210). The display (1250) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (1250) may also include a driving circuit, a backlight unit, etc., which may be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc.

[0108] The processor (1210) is a configuration for controlling the overall operation of the smartphone device (1200). The types and specific examples of the processor (1210) have been described above, and redundant description is omitted.

[0109] In FIG. 12, the processor (1210) stores a photographed image captured by a camera (1240) in the memory (1230) of a smartphone device (1200), or receives an image containing a word using a live view image corresponding to light incident through the camera.

[0110] The process of the processor (1210) receiving an image and extracting words has been described above, so redundant description is omitted.

[0111] The processor (1210) can use words extracted from an image in various ways. For example, the processor (1210) can control the display (1250) to search for a word extracted from an image as a keyword on the web and display the search results.

[0112] Alternatively, the processor (1210) may control the display (1250) to display foreign language translation results for words extracted from the image together with the image. In this case, the arrangement direction of the translation results may be determined in accordance with the arrangement direction of the words on the image as described above. That is, for a word (12) written vertically as in FIG. 1, the processor (1210) may control the display (1250) to display the foreign language translation results vertically on one side of the word (12).

[0113] For this operation, the processor (1210) controls the display (1250) to search for information corresponding to the extracted string (word) from at least one external device connected through the communication unit (1220) and display the search result.

[0114] FIG. 13 is a flowchart illustrating a method for an electronic device to extract words included in an image according to at least one embodiment of the present disclosure.

[0115] According to FIG. 13, when an image is input, the electronic device groups characters included in the input image based on the characteristics of each character and divides them into at least one box (S1310). When the characters are grouped and divided into boxes, adjacent characters included in the boxes are connected in multiple directions to form multiple character strings (S1320). In addition, the electronic device can identify the frequency of use of each of the multiple character strings based on a word database containing words used by multiple users (S1330), and extract one of the multiple character strings based on the frequency of use (S1340).

[0116] Specific examples of the operation of an electronic device to extract words contained in an image have been described above, so redundant description is omitted.

[0117] The control method of FIG. 13 can be performed by an electronic device (100) having the configuration described in FIG. 2 or FIG. 12, but is not necessarily limited thereto, and can also be performed by a device having a different configuration.

[0118] FIG. 14 is a flowchart illustrating a method for extracting characters included in an image into words based on their frequency of use according to at least one embodiment of the present disclosure.

[0119] According to FIG. 14, when characters included in an image are divided into a plurality of boxes (S1410), the electronic device identifies the possibility of connection between characters included in each of the plurality of boxes based on the positions and distances between the plurality of boxes (S1420).

[0120] Additionally, the electronic device constructs a string by connecting characters that are likely to be connected (S1430) and identifies the frequency of use of the string based on a word database (S1440). The electronic device determines whether the frequency of use of the string exceeds a threshold (S1450) and, if so, extracts the string (S1460).

[0121] Specific examples of the operation of extracting characters into words based on their frequency of use have been described above, so redundant descriptions are omitted.

[0122] The control method of FIG. 14 can be performed by an electronic device (100) having the configuration described in FIG. 2 and FIG. 12, but is not necessarily limited thereto, and can also be performed by a device having a different configuration.

[0123] The programs or instructions for performing the various information processing methods described above may be provided stored on a non-transitory, readable medium. The non-transitory, readable medium may be loaded and used in a device capable of recalling the instructions stored in the storage medium and performing operations according to the recalled instructions. Accordingly, when the program or instructions stored on the non-transitory, readable medium are executed by a processor, the processor may directly, or under the control of the processor, utilize other components to perform the operations described in the various embodiments described above.

[0124] A non-transitory computer-readable medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0125] Instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not contain signals and is tangible, but does not distinguish between whether data is stored semi-permanently or temporarily on the storage medium.

[0126] Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be implemented as a product distributed online through an application store, in addition to the non-transitory readable recording medium described above. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0127] Accordingly, in a non-transitory computer-readable medium storing one or more instructions executed by a processor of an electronic device to cause the electronic device to perform an operation, the operation of the medium is:

[0128] It includes a step of grouping characters included in an input image based on the characteristics of each character and dividing them into at least one box, a step of configuring a plurality of strings by connecting adjacent characters included in at least one box in a plurality of directions, a step of identifying the frequency of use of each of the plurality of strings based on a word database including words used by a plurality of users, and a step of extracting one of the plurality of strings based on the frequency of use.

[0129] While the present invention has been described with reference to the attached drawings, the scope of the present invention is determined by the claims described below and should not be construed as being limited to the aforementioned embodiments and / or drawings. Furthermore, it should be clearly understood that improvements, modifications, and variations apparent to those skilled in the art, as defined in the claims, are also included within the scope of the present invention.

Claims

1. In electronic devices, Memory for storing the input image; and Processor; including; The above processor, Group the characters included in the input image based on the characteristics of each character and separate them into at least one box, Constructing a plurality of strings by connecting adjacent characters included in at least one of the above boxes in multiple directions, An electronic device that identifies the frequency of use of each of a plurality of strings based on a word database containing words used by a plurality of users, and extracts one of the plurality of strings based on the frequency of use.

2. In paragraph 1, The above processor, When the characters included in the input image are divided into multiple boxes, the possibility of connection between the characters included in each of the multiple boxes is identified based on the positions and distances between the multiple boxes, An electronic device that connects characters with the possibility of connection to form a string, identifies the frequency of use of the string based on the word database, and extracts the string when the frequency of use exceeds a threshold.

3. In paragraph 2, An electronic device in which the processor configures a first string by connecting adjacent characters included in at least one box horizontally, a second string by connecting them vertically, and a third string by connecting them diagonally, identifies the frequency of use of each string based on the word database, and extracts a string whose identified frequency of use exceeds the threshold.

4. In paragraph 3, The above processor, An electronic device that, if at least one of the first to third strings contains multiple words, identifies the frequency of use for each word based on the word database and extracts at least one word whose identified frequency of use exceeds the threshold.

5. In paragraph 2, The above processor, Identify the characters contained in the input image on a character-by-character basis, An electronic device that groups a plurality of characters, each of which has a similar range of font, size, and color, and is arranged adjacently in a horizontal, vertical, or diagonal direction, into one box.

6. In paragraph 5, camera; display; and Including the Department of Communications, The above input image is, Includes a photographed image captured by the camera or a live view image corresponding to light incident through the camera, The above processor, An electronic device that controls the display to search for information corresponding to the extracted string from at least one external device connected through the communication unit and display the search result.

7. In paragraph 1, It further includes a communication unit for communicating with an external device in which the above word database is stored, The above processor, An electronic device that transmits the plurality of strings to the external device through the communication unit and receives information about the frequency of use of each of the plurality of strings from the external device.

8. In a word extraction method using an electronic device, A step of grouping characters included in an input image based on the characteristics of each character and dividing them into at least one box; A step of forming a plurality of strings by connecting adjacent characters included in at least one box in multiple directions; A word extraction method comprising: a step of identifying the frequency of use of each of a plurality of strings based on a word database containing words used by a plurality of users; and a step of extracting one of the plurality of strings based on the frequency of use.

9. In paragraph 8, When characters included in the input image are divided into a plurality of boxes, a step of identifying a possibility of connection between characters included in each of the plurality of boxes based on the positions and distances between the plurality of boxes; A word extraction method, further comprising: a step of concatenating characters with the possibility of connection to form a string; a step of identifying the frequency of use of the string based on the word database; and a step of extracting the string if the frequency of use exceeds a threshold.

10. In paragraph 9, The step of composing the above multiple strings is: A word extraction method comprising a first string horizontally connected with adjacent characters included in at least one box, a second string vertically connected with adjacent characters, and a third string diagonally connected with adjacent characters included in at least one box.

11. In paragraph 10, If at least one of the first to third strings contains multiple words, a step of identifying the frequency of use of each word based on the word database; and A word extraction method, further comprising the step of extracting at least one word whose identified usage frequency exceeds the threshold.

12. In paragraph 8, The step of grouping characters included in the input image based on the characteristics of each character and dividing them into at least one box is as follows: A step of identifying characters included in the input image on a character-by-character basis; and A word extraction method, comprising: a step of grouping a plurality of characters, each of which has at least one of a font, a size, and a color within a similar range and is arranged adjacently in a horizontal, vertical, or diagonal direction, into one box.

13. In paragraph 12, It further includes a step of searching for information corresponding to the extracted string from an external device and displaying the search result. The above input image is, A word extraction method comprising a captured photographic image or a live view image corresponding to incident light.

14. In paragraph 8, A word extraction method further comprising the step of transmitting the plurality of strings to an external device in which the word database is stored, and receiving information on the frequency of use of each of the plurality of strings from the external device.

15. In a non-transitory computer-readable medium storing one or more instructions that are executed by an electronic device to perform a word extraction method, The above word extraction method is, A step of grouping characters included in an input image based on the characteristics of each character and dividing them into at least one box; A step of forming a plurality of strings by connecting adjacent characters included in at least one box in multiple directions; A step of identifying the frequency of use of each of the plurality of strings based on a word database containing words used by a plurality of users; and a step of extracting one of the plurality of strings based on the frequency of use. Computer readable recording medium.

Images