Electronic device and control method thereof

Abstract

An electronic device is disclosed. The electronic device includes: a storage unit for storing a training model of a multidimensional long short-term memory (MDLSTM); and a processor for acquiring an image containing at least one of handwritten text and printed text, identifying each text line region in the image through image processing, and identifying text contained in each identified text line region based on the training model.

Description

Technical Field

[0001] The present disclosure relates to an electronic device for recognizing text and a control method thereof. More specifically, the device relates to a technology for recognizing text based on an artificial neural network. Background Art

[0002] There has been technology in the related art for recognizing text that is not directly typed on an electronic device.

[0003] However, the related art has problems in correctly recognizing handwritten text or tilted text.

[0004] Text recognition technology based on Convolutional Neural Network (CNN), an artificial intelligence technology, still has the problem of being unable to correctly recognize overlapping characters or handwritten text.

[0005] Therefore, there is a need for technology that can accurately and quickly recognize various types of text including handwritten text. Summary of the invention

[0006] Technical issues

[0007] The present disclosure is based on the above needs and will provide an electronic device and a control method thereof that can accurately and quickly recognize text contained in an image based on Long-Short Term Memory (LSTM), a recurrent neural network (RNN).

[0008] Technical Solution

[0009] According to one embodiment of the present disclosure, an electronic device includes: a memory configured to store a training model of a multi-dimensional long short-term memory (MDLSTM); and a processor configured to: acquire an image containing at least one of handwritten text and printed text, identify each text line region in the image through image processing, and identify the text contained in each identified text line region based on the training model.

[0010] The processor may be configured to binarize a text region in the image and perform image processing by performing correction processing on the binarized text region.

[0011] The correction process may include performing at least one of de-skewing, height normalization, and skeletonization on the text region.

[0012] The processor may be configured to identify a region containing the identified text line as a plurality of vertical blocks and calculate a probability of a character estimated from each vertical block based on pixel values ​​of each vertical block.

[0013] The processor can be configured to combine the probability of existence of the first character in the first vertical block, the probability of non-existence of the first character in the first vertical block, the probability of existence of the second character in the second vertical block, and the probability of non-existence of the second character in the second vertical block to calculate multiple probabilities in which at least one of the first character and the second character is included in the text line area.

[0014] The processor may be configured to apply token passing decoding and a preset language model to the plurality of calculated probabilities and obtain text recognized in the text line region.

[0015] The device may further include a camera and a display, wherein the image containing at least one of handwritten text and printed text refers to an image obtained by the camera or a handwritten image input on the display.

[0016] The processor may be configured to identify a function related to the identified text and execute the identified function.

[0017] A method for controlling an electronic device, wherein the electronic device stores a training model of a multidimensional long short-term memory (MDLSTM), the method comprising: acquiring an image containing at least one of handwritten text and printed text; identifying each text line region in the image through image processing and identifying the text contained in each identified text line region based on the training model.

[0018] Identifying each text line region may include binarizing the text region in the image and performing image processing by performing correction processing on the binarized text region.

[0019] The correction process may include performing at least one of de-skewing, height normalization, and skeletonization on the text region.

[0020] The method may further include: identifying a region including the identified text line as a plurality of vertical blocks; and calculating a probability of a character estimated from each vertical block based on a pixel value of each vertical block.

[0021] Calculating the probability may include: combining the existence probability of the first character in the first vertical block, the non-existence probability of the first character in the first vertical block, the existence probability of the second character in the second vertical block, and the non-existence probability of the second character in the second vertical block to calculate multiple probabilities in which at least one of the first character and the second character is included in the text line area.

[0022] Recognizing the text may include applying token passing decoding and a preset language model to the plurality of calculated probabilities and obtaining text recognized in the text line region.

[0023] The method may further include identifying a function related to the identified text and executing the identified function.

[0024] Beneficial technical effects

[0025] As described above, according to various embodiments of the present disclosure, printed text, handwritten text, and text included in an image captured by a camera can be quickly and accurately recognized through text recognition technology via LSTM, thereby providing convenience for users. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a view showing an electronic device for inputting an image including text according to an embodiment of the present disclosure;

[0027] Figure 2A and Figure 2B is a block diagram showing a configuration of an electronic device according to an embodiment of the present disclosure;

[0028] Figure 3 is a view showing image processing according to an embodiment of the present disclosure;

[0029] Figure 4 is a view showing an operation of recognizing each text line region according to an embodiment of the present disclosure;

[0030] Figure 5 is a view showing an operation of calculating a probability of text estimated in each vertical block according to an embodiment of the present disclosure;

[0031] Figure 6 is a view illustrating a process of recognizing text included in an image according to an embodiment of the present disclosure;

[0032] Figure 7 is a view showing an operation of recognizing a handwritten text according to an embodiment of the present disclosure;

[0033] Figure 8 is a view in the electronic device that identifies a function related to the identified text and shows execution of the identified function; and

[0034] Fig. 9 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] Below, exemplary embodiments will be described in detail with reference to the accompanying drawings. When describing exemplary embodiments, if the detailed description of related known functions or components makes the description of the subject unclear, the detailed description may be omitted. In addition, exemplary embodiments may be changed in various forms, so the technical scope is not limited to the following exemplary embodiments. On the contrary, these exemplary embodiments are provided to make this disclosure thorough and complete.

[0036] Unless otherwise expressly stated, the term "comprising" will be understood to imply the inclusion of components but not the exclusion of any other components. In addition, various elements and regions in the drawings are schematically drawn. Therefore, the technical concept is not limited by the relative sizes or intervals drawn in the drawings.

[0037] Furthermore, the expression “at least one of A, B, and C” may mean “A,” “B,” “C,” “A and B,” “A and C,” “B and C,” or “A, B, and C.”

[0038] Figure 1 is a view illustrating an electronic device in which an image including text is input according to an embodiment of the present disclosure.

[0039] The electronic device 100 may be implemented as a smart phone capable of recognizing text from an image containing text, but is not limited thereto, and may also be implemented as a laptop computer, a tablet computer, a PC, a camera, and the like.

[0040] The text included in the image may be at least one of handwritten text and printed text. The handwritten text may be text typed by handwriting, and the printed text may be text typed by printing.

[0041] According to an embodiment of the present disclosure, the electronic device 100 can quickly and correctly recognize text included in an image based on long short-term memory (LSTM), a type of recurrent neural network (RNN). Hereinafter, various embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings.

[0042] RNN refers to Recurrent Neural Network and is a deep learning model for learning data that changes over time, such as time series data. LSTM is a type of RNN that solves the vanishing gradient problem of RNN by adding cell states to the hidden layer stage of RNN. The vanishing gradient problem is a problem in which the learning ability decreases as the gradient gradually decreases during backpropagation when the distance between the relevant information and the point at which it is used moves away from the RNN.

[0043] Figure 2A and Figure 2B is a block diagram showing a configuration of an electronic device according to an embodiment of the present disclosure.

[0044] Reference Figure 2A, the electronic device 100 includes a memory 110 and a processor 120 .

[0045] The memory 110 may store various data, programs or applications for driving and controlling the electronic device 100. The memory 110 may store a control program for controlling the electronic device 100 and the processor 120, an application originally provided by a manufacturer or downloaded from the outside, a database or related data.

[0046] In particular, the memory 110 may store a training model of a multidimensional long short-term memory (MDLSTM). In addition, the memory 110 may store an image captured by a camera or an image captured on a screen of the electronic device 100. Since the MDLSTM method is a prior art, a detailed description is omitted.

[0047] The processor 120 may control the overall operation of the electronic device 100 .

[0048] According to one embodiment, the processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital signals, but is not limited thereto, and may include one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a communication processor (CP), and an ARM processor, or may be defined as a corresponding term. Moreover, the processor 120 may be implemented by a system on chip (SoC), a large-scale integration (LSI), or a field programmable gate array (FPGA) with a built-in processing algorithm.

[0049] The processor 120 may acquire an image including at least one of handwritten text and printed text.

[0050] The image including at least one of handwritten text and printed text may be an image acquired by a camera or an image including text input on a display. Alternatively, the image may be a captured image of a screen on a display.

[0051] For example, the processor 120 may recognize text included in an image captured by a camera and text input by a hand touch or a pen touch on a display. Also, the processor 120 may recognize text included in an image of a screen displayed on a capture display.

[0052] The processor 120 may identify each text line region in the image through image processing.

[0053] According to one embodiment, the processor 120 may binarize the text region in the image, and may perform image processing by performing correction processing on the binarized text region.

[0054] Binarization may be an operation of distinguishing a text region from the rest of the region. In other words, the processor 120 may perform a binarization operation of identifying a text region in an image and distinguishing the text from the rest of the region.

[0055] The correction process may include performing at least one of de-skewing, height normalization, and skeletonization on the text region.

[0056] For example, the processor 120 may remove the tilt so that the slope of a text line tilted 30 degrees from the horizontal plane is 0 degrees, and perform correction to uniformly normalize each text height.

[0057] Skeletonization is an operation in which pixels forming text are densely composed of pixels of the same color. For example, when the pixels forming the consonant "B" are black, all pixels forming the "B" may not be composed of black when the "B" is enlarged. For example, a "B" written with a pencil at low pressure is dark, so that not all pixels forming the "B" are composed of black. For clarity, the processor 120 can correct that all colors of the pixels forming the "B" are black. Through the skeletonization process, the processor 120 can clearly express and recognize the corresponding text. For example, the processor 120 can provide pixels forming the text as black pixels and pixels indicating the remaining area as white pixels through skeletonization.

[0058] The processor 120 may identify each text line region in the text region corrected by the above-described image processing.

[0059] When a text line included in an image on which image processing has been performed is plural, the processor 120 may individually recognize each text line region to improve a text recognition rate.

[0060] For example, the processor 120 may calculate the average value of the standardized text height through image processing, and apply the calculated horizontal line of the average height and the horizontal lines spaced at equal intervals in the vertical direction based on the horizontal line of the average height to the text line area to determine the text line. In other words, the processor 120 may apply three lines to the text line area and determine the text spanning the three lines as one text line. This will be described later in Figure 4 Described in detail in.

[0061] The processor 120 may identify the area including the identified text line as a plurality of vertical blocks, and calculate the probability of the text estimated in each vertical block based on the pixel value of each vertical block. The processor 120 may calculate the probability of the character estimated in each vertical block based on Multi-Dimensional Multi-Directional Long Short-Term Memory (MDMDLSTM). "Multi-directional" may mean scanning the image input to the electronic device 100 in various directions, so the processor 120 may improve the recognition rate of the text included in the image.

[0062] For example, 26 vertical blocks may be applied to a region including the handwritten text "The cat" so that the processor 120 may calculate the probability of the estimated character in each vertical block based on the pixel values ​​on each vertical block.

[0063] The processor 120 may combine the existence probability of the first character in the first vertical block, the non-existence probability of the first character in the first vertical block, the existence probability of the second character in the second vertical block, and the non-existence probability of the second character in the second vertical block to calculate multiple probabilities in which at least one of the first and second characters is included in the text line area.

[0064] The processor 120 may apply token passing decoding and a preset language model to the plurality of calculated probabilities to obtain text recognized in the text line region.

[0065] The processor 120 can identify the character with the highest probability among the characters estimated in each vertical block based on token passing decoding. When the highest probability in a specific vertical block is that the character does not exist in the corresponding block, the processor 120 can recognize that there is no text in the corresponding block. Therefore, the processor 120 can identify the character estimated with the highest probability for each block.

[0066] For example, when the probability of not containing any characters in the first vertical block is the highest, the processor 120 may recognize that there are no characters in the first vertical block, and when the letter "T" has the highest probability of existence in the third vertical block, the processor 120 may recognize that "T" exists in the third vertical block. This will be described in the following Figure 5 Described in more detail in .

[0067] The processor 120 may combine candidate texts estimated to have the highest probability in each vertical block to provide candidate texts. For example, the processor 120 may provide "The eat" as a candidate text.

[0068] The processor 120 may modify the provided candidate text based on a preset language model. For example, the processor 120 may modify "The eat" to "The cat" based on the language model.

[0069] Token passing decoding can obtain text in a text line area based on pixel values ​​on each vertical block using a preset language model and the Viterbi algorithm.

[0070] The preset language model may be a data - a language actually used by a user (e.g., a sentence, a phrase, etc.) - modeled. Using this language model, the most appropriate text may be identified as the next word of the input text based on the sequentially input text. According to an embodiment of the present disclosure, the preset language model may be an N-Grams language model, but is not limited thereto. Since the above-mentioned Viterbi algorithm and N-Grams language model are prior arts, detailed descriptions are omitted.

[0071] When both handwritten text and printed text exist in the image, the processor 120 can recognize both the handwritten text and the printed text. In addition, when texts in different languages ​​are included in the image, the processor 120 can recognize the text included in the image by recognizing the language of each text.

[0072] The processor 120 may provide the recognized text through the above process. For example, the processor 120 may provide the recognized text on a display or recognize the meaning of the recognized text to perform an operation corresponding thereto.

[0073] In other words, the processor 120 may recognize a function related to the recognized text and execute the recognized function.

[0074] For example, when the recognized text is "wake-up at 6am", the processor 120 may set an alarm at 6 a.m. Alternatively, when the recognized text is "Meeting with AAA at 2:00 on September 25", the processor 120 may register a schedule to meet with AAA at 2:00 on September 25.

[0075] In addition, when the recognized text is “shopping list: egg, milk, apple, bread”, the processor 120 may recognize it as a shopping list and store “shopping list: egg, milk, apple, bread” in the notepad.

[0076] Figure 2A and Figure 2B is a block diagram showing a configuration of an electronic device according to an embodiment of the present disclosure.

[0077] according to Figure 2B , the electronic device 100 includes a memory 110, a processor 120, a communicator 130, a camera 140, and a display 150. Figure 2A The components overlap Figure 2B The components will not be described in detail below.

[0078] The memory 110 may be implemented as an internal memory such as a ROM, a RAM, etc. included in the processor 120 or may be implemented as a memory separate from the processor 120. The memory 110 may be implemented in the form of a memory embedded in the electronic device 100 or may be implemented in the form of a memory detachable from the electronic device 100 according to data storage usage. For example, data for driving the electronic device 100 may be stored in the memory embedded in the electronic device 100, and data for expanding the electronic device 100 may be separated from the electronic device 100. Meanwhile, the memory embedded in the electronic device 100 may be implemented in the form of a nonvolatile memory, a volatile memory, a hard disk drive (HDD), a solid state drive (SSD), etc.

[0079] The processor 120 may include, for example, a CPU 121, a ROM (or non-volatile memory 143) in which a control program for controlling the electronic device 100 is stored, and a RAM (or volatile memory) for storing data input from outside the electronic device 100 or for serving as a storage area corresponding to various operations performed in the electronic device 100.

[0080] When a predetermined event occurs, the processor 120 may execute an operating system (OS), programs, and various applications stored in the memory 110. The processor 120 may include a single core, a dual core, a triple core, a quad core, and multiple cores.

[0081] The CPU 121 may access the memory 110 to perform a boot sequence using an operating system (OS) stored in the memory 110. In addition, the CPU 143 may perform various operations using various programs, contents, data, etc. stored in the memory 110.

[0082] The communicator 130 can communicate with an external device. The text included in the image can be recognized by the processor 120 by receiving the image containing the text from the external electronic device via the communicator 130. In addition, the text included in the image input on the electronic device 100 can be recognized and the text or information related to the text recognized as the external electronic device can be provided via the communicator 130.

[0083] For example, the communicator 130 may perform wireless communication with an external electronic device through various communication schemes such as Bluetooth (BT), Wireless Fidelity (WI-FI), Zigbee, infrared (IR), serial interface, universal serial bus (USB), near field communication (NFC), etc.

[0084] Specifically, when a preset event occurs, the communicator 130 can operate in an interoperable state by performing communication according to a predefined communication method. Here, "interoperability" may represent each state in which communication is available, including, for example, operations of initializing communication between the electronic device 100 and an external electronic device, operations of forming a network, operations of performing device pairing, and the like. For example, device identification information of an external device may be provided to the electronic device 100. As a result, a pairing process between the two devices may be performed. For example, when a preset event occurs in the electronic device 100 or an external electronic device, the interoperable state may be implemented by searching for surrounding devices and performing pairing with the searched devices via the Digital Living Network Alliance (DLNA).

[0085] The camera 140 may photograph and acquire an image including at least one of handwritten text and printed text.

[0086] The display 150 may be implemented with various types of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a liquid crystal on silicon (LCoS), a digital light processing (DLP), a quantum dot (QD) display panel, and the like.

[0087] The display 150 can be implemented in the form of a touch screen, which forms a mutual layer structure with a touch pad. The display 150 can also be used as a user interface (not shown) in addition to an output device. Here, the touch screen can not only detect the position and area of ​​the touch input, but also detect the pressure of the touch input.

[0088] Therefore, the processor 120 may recognize text input on the display 150 through a hand touch or a pen touch.

[0089] Figure 3 is a view illustrating image processing according to an embodiment of the present disclosure.

[0090] Reference Figure 3 , the electronic device 100 may perform image processing to more clearly express the text included in the image.

[0091] The original image 310 may be input to the electronic device 100 by various methods. For example, the text may be photographed by the camera 140 or may be input by hand touch or pen touch on the display 150 of the electronic device 100 or an original image including text may be received from another electronic device.

[0092] The electronic device 100 may binarize the text region in the image, and perform image processing by performing correction processing on the binarized text region.

[0093] Binarization may be an operation to distinguish a text region from the rest of the region. In other words, the electronic device 100 may identify a text region in an image and perform a binarization operation to distinguish the text from the rest of the region.

[0094] The correction process may include at least one of gradient removal 330 , height normalization 340 , and skeletonization 350 for the text region.

[0095] For example, it may remove the gradient so that a text line tilted 30 degrees from the horizontal plane has a gradient of 0 degrees, and perform a correction for normalization of the height of each text.

[0096] Skeletonization is an operation in which pixels forming text are densely composed of pixels of the same color. For example, when the pixels forming the consonant "B" are black, when "B" is enlarged, not all pixels forming "B" may be composed of black. For example, "B" written with a pencil at low pressure is dark, so that not all pixels forming "B" are composed of black. In order to make "B" clear, the electronic device 100 can process all color corrections of the pixels forming "B" to black. Through the skeletonization work, the electronic device can clearly express and recognize the corresponding text. For example, the electronic device 100 can provide the pixels forming the text with black pixels and provide the pixels indicating the remaining area with white pixels through skeletonization.

[0097] Figure 4 is a view illustrating an operation of recognizing each text line region according to an embodiment of the present disclosure.

[0098] When a text line included in the image 410 on which image processing has been performed is a plurality of lines, the electronic device 100 may individually recognize each text line region to improve a text recognition rate.

[0099] For example, the electronic device 100 may calculate the average value of the standardized text height through image processing, and apply the calculated horizontal line of the average height and the horizontal lines spaced at equal intervals in the vertical direction based on the horizontal line of the average height to the text line area to identify the text line. In other words, the electronic device 100 may apply three lines to the text line area and identify the text spanning the three lines as one text line.

[0100] For example, according to Figure 4, the electronic device 100 may recognize “pleasure and pain will”, “give you a complete”, and “account to the system” as separate text lines.

[0101] The electronic device 100 may individually recognize only the first text line 430. When each of the text line regions is not individually recognized, the text recognition rate of the electronic device 100 may be reduced.

[0102] For example, when each of the text line regions is not recognized separately, the second text line may interfere with text recognition of the first text line, and thus the electronic device 100 may erroneously recognize the first text line.

[0103] Figure 5 is a view illustrating an operation of calculating a probability of text estimated in each vertical block according to an embodiment of the present disclosure.

[0104] according to Figure 5 , the text area 510 may be text on which image processing has been performed.

[0105] The electronic device 100 may identify a region including the recognized text line as a plurality of vertical blocks, and calculate a probability of text estimated in each vertical block based on a pixel value of each vertical block.

[0106] according to Figure 5 , 26 vertical blocks are applied to the area including the handwritten text "cat", so that the electronic device 100 can calculate the probability of the text estimated in each vertical block based on the pixel value on each vertical block.

[0107] The electronic device 100 can combine the existence probability of the first character in the first vertical block, the non-existence probability of the first character in the first vertical block, the existence probability of the second character in the second vertical block, and the non-existence probability of the second character in the second vertical block to calculate multiple probabilities in which at least one of the first character and the second character is included in the text line area.

[0108] For example, when the probability that the first vertical block does not include a character is the highest, the electronic device 100 can recognize that there is no text in the first vertical block, and when the probability that the letter "T" exists in the third vertical block is the highest, the processor 120 can recognize that "T" exists in the third vertical block. The electronic device 100 can identify and combine characters that exist or do not exist with the highest probability in each vertical block.

[0109] The electronic device 100 may identify a plurality of probabilities according to 520 for each vertical block in order to identify a character having the highest probability in each vertical block.

[0110] The electronic device 100 may apply token passing decoding and a preset language model to the plurality of calculated probabilities to obtain text recognized in the text line region.

[0111] The electronic device 100 can identify the character with the highest probability among the characters estimated in each vertical block based on token passing decoding. When the highest probability in a specific vertical block is that there is no character in the corresponding block, the electronic device 100 can recognize that there is no text in the corresponding block. Therefore, the electronic device 100 can identify the character estimated to have the highest probability for each block. The electronic device 100 can provide candidate text by combining the characters with the highest probability.

[0112] The electronic device 100 may modify the provided candidate text based on a preset language model. For example, when the candidate text recognized based on the estimated character with the highest probability for each block is "The eat", the electronic device 100 may modify "The eat" to "The cat" based on the language model.

[0113] Figure 6 is a view illustrating a process of recognizing a text included in an image according to an embodiment of the present disclosure.

[0114] When an original image is input to the electronic device 100 , the electronic device 100 may perform an image processing operation.

[0115] Based on the MDLSTM of the electronic device 100 , the probability of the estimated character in each vertical block may be calculated based on the pixel values ​​applied to each vertical block of the text area.

[0116] according to Figure 6 The sub-sample layer, feed-forward, MDLSTM and CTC layer are various types of layers used to form a neural network structure. The Conectionist Temporal Classification (CTC) layer can be a cost function used only for training a neural network.

[0117] The electronic device 100 may repeat the subsample layer, feedforward, and MDLSTM twice in the order of subsample layer 1, feedforward 1, MDLSTM 1, subsample layer 2, feedforward 2, and MDLSTM 2. Through this repetition, overload on the electronic device 100 may be prevented.

[0118] The electronic device 100 can calculate the probability of each text included in the original image through this repetition. The probability can refer to Figure 5 The probability calculated in each vertical block of .

[0119] The electronic device 100 may identify a character estimated to have the highest probability for each vertical block based on token passing decoding. The electronic device 100 may provide a candidate text by combining characters estimated to have the highest probability.

[0120] The electronic device 100 may modify the provided candidate text based on a preset language model. For example, when the candidate text recognized based on the character with the highest estimated probability for each block is "The eat", the electronic device 100 may modify "The eat" to "The cat" based on the language model.

[0121] Token passing decoding can use a preset language model and the Viterbi algorithm to obtain the recognized text in the text line area.

[0122] The preset language model may be data - the language actually used by the user (e.g., sentences, phrases, etc.) - modeled. Using the language model, the most appropriate text may be identified as the next word of the input text based on the sequentially input text. According to an embodiment of the present disclosure, the preset language model may be an N-Grams language model, but is not limited thereto. Since the above-mentioned Viterbi algorithm and N-Grams language model are prior arts, detailed descriptions are omitted.

[0123] The electronic device 100 may recognize the text modified by the language model.

[0124] Figure 7 is a view illustrating an operation of recognizing a handwritten text according to an embodiment of the present disclosure.

[0125] Figure 7 A shows at least one of handwritten text and printed text included in an image photographed by the camera 140 input to the electronic device 100, an image input by a hand touch or a pen touch on the display 150, or an image in which an image output on the display 150 is captured.

[0126] The electronic device 100 may recognize each text line region through image processing including correction processing, and recognize text included in each text line region based on a training model.

[0127] Figure 7 B shows recognizing text included in an image input on the electronic device 100. The electronic device 100 can recognize the input text and provide the text as Figure 7 B shows the typed text form.

[0128] Each step of recognizing a text has been described above, so a detailed description thereof will be omitted.

[0129] Figure 8It is a view in which a function related to the recognized text is recognized in the electronic device and the recognized function is executed.

[0130] according to Figure 8 A. When at least one of a handwritten text and a printed text is input to the electronic device 100, the electronic device 100 may recognize the text based on the above operation. The electronic device 100 may recognize a function related to the recognized text.

[0131] according to Figure 8 B. When the recognized text is "Meet with Mr. Brown at 10pm tomorrow", the electronic device 100 may set a reminder titled "Meet with Mr. Brown at 10pm". In addition, the electronic device 100 may register a schedule of "Meet with Mr. Brown" at 10pm on the date corresponding to tomorrow on the calendar.

[0132] Fig. 9 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

[0133] according to Fig. 9 , the electronic device may acquire an image including at least one of a handwritten text and a printed text (S910).

[0134] The electronic device may recognize each text line region in the image through image processing ( S920 ).

[0135] The electronic device may binarize a text region in an image and perform image processing through correction processing for the binarized text region.

[0136] The correction process may include performing at least one of skew removal, height normalization, and skeletonization on the text region.

[0137] The electronic device may identify a region including the recognized text line as a plurality of vertical blocks, and calculate a probability of a character estimated in each vertical block based on a pixel value of each vertical block.

[0138] For example, the electronic device may combine the existence probability of the first character in the first vertical block, the non-existence probability of the first character in the first vertical block, the existence probability of the second character in the second vertical block, and the non-existence probability of the second character in the second vertical block to calculate multiple probabilities in which at least one of the first character and the second character is included in the text line area.

[0139] The electronic device may recognize text included in each of the recognized text line regions based on the training model (S930).

[0140] The electronic device 120 may apply token passing decoding and a preset language model to the plurality of calculated probabilities to obtain text recognized in the text line region.

[0141] The electronic device may recognize a function related to the recognized text and execute the recognized function.

[0142] For example, when the recognized text is "wake-up at 6am", the electronic device may set a reminder at 6 a.m. Alternatively, when the recognized text is "Meeting with AAA at 2:00 on September 25", the processor 120 may register a schedule on the calendar to meet with AAA at 2:00 on September 25.

[0143] In addition, when the recognized text is "shopping list: egg, milk, apple, bread", the electronic device can recognize it as a shopping list and store it as a handwritten text in the notepad or change it into a printed text form.

[0144] The detailed operation of each step has been described above, so its detailed description will be omitted.

[0145] At least some of the methods according to the various embodiments of the present disclosure described above may be implemented in the form of an application that can be installed in an existing electronic device.

[0146] Furthermore, the method according to the above-described exemplary embodiments may be implemented by upgrading software or hardware of an existing electronic device.

[0147] The above-described exemplary embodiments may be performed by an embedded server in the electronic device or by an external server outside the electronic device.

[0148] The various exemplary embodiments described above may be embodied in a recording medium, which may be read by a computer or a device similar to a computer using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. In software configuration, the various embodiments (such as processes and functions) described in the specification may be embodied as separate software modules. The software modules may respectively perform one or more functions and operations described in the specification.

[0149] The method of controlling a display device according to various exemplary embodiments may be stored on a non-transitory computer-readable medium. When executed by a processor to implement a specific device, these computer instructions stored in the non-transitory computer-readable medium may allow processing operations according to the above-described various embodiments.

[0150] The method of controlling a display device according to various exemplary embodiments may be stored on a non-transitory computer-readable medium. When executed by a processor to implement a specific device, these computer instructions stored in the non-transitory computer-readable medium may allow processing operations according to the above-described various embodiments.

[0151] When executed by a processor of a first electronic device, the operation of a non-temporary computer-readable medium storing computer instructions for causing the first electronic device to perform operations may include: obtaining first error-related information for a target time interval and second error-related information for a reference time interval from error-related information of a second electronic device, the reference time interval including a target time interval and a time interval different from the target time interval; obtaining frequency information of each number of errors for each target time interval based on the first error-related information; and comparing the frequency information of each number of errors for the target time interval with the frequency information of each number of errors for each reference time interval to identify an error level for the target time interval.

[0152] A non-transitory computer-readable recording medium refers to a medium that stores data and can be read by a device. For example, a non-transitory computer-readable medium may be a CD, a DVD, a hard disk, a Blu-ray disc, a USB, a memory card, a ROM, etc.

[0153] The foregoing exemplary embodiments and advantages are merely exemplary and should not be construed as limiting the present invention. The teachings may be easily applied to other types of devices. Moreover, the description of the exemplary embodiments is intended to be illustrative rather than limiting the scope of the claims, and many alternatives, modifications, and variations will be apparent to those skilled in the art.

Claims

1. An electronic device, comprising: Memory, used to store the training model of multidimensional long short-term memory MDLSTM; and a processor configured to: acquire an image containing at least one of handwritten text and printed text, identify each text line region in the image through image processing, and identify text included in each identified text line region based on the training model, The processor is further configured to, for each text line region: Calculating the average value of the normalized text height of the text line area by image processing, applying a first horizontal line of the calculated average value of the normalized text height and two second horizontal lines spaced apart from the first horizontal line at equal intervals in two opposite directions of the first horizontal line to the text line area, and Identify the text that spans the first horizontal line and two second horizontal lines as one text line, The plurality of text line regions in the image include a plurality of text lines, and The processor is further configured to individually identify each text line in the multiple text lines, and identify text in the image that spans a first horizontal line and two second horizontal lines as one text line in the multiple text lines.

2. The device according to claim 1, wherein: The processor is configured to binarize a text region in the image and perform the image processing through correction processing for the binarized text region.

3. The device as claimed in claim 2, wherein: The correction process includes performing at least one of de-skewing, height normalization and skeletonization on the text region.

4. The device according to claim 1, wherein: The processor is configured to identify a region containing the identified text line as a plurality of vertical blocks, and calculate a probability of a character estimated from each vertical block based on a pixel value of each vertical block.

5. The device according to claim 4, wherein: The processor is configured to combine the probability of existence of a first character in a first vertical block, the probability of non-existence of the first character in the first vertical block, the probability of existence of a second character in a second vertical block, and the probability of non-existence of the second character in the second vertical block to calculate multiple probabilities that at least one of the first character and the second character is included in the text line area.

6. The device according to claim 5, wherein: The processor is configured to apply token passing decoding and a preset language model to a plurality of calculated probabilities and obtain text recognized in the text line region.

7. The apparatus of claim 1, further comprising: camera; and monitor, The image containing at least one of written text and printed text refers to an image obtained by the camera or a handwritten image input on the display.

8. The device of claim 1, wherein: The processor is configured to identify a function associated with the identified text and execute the identified function.

9. A method for controlling an electronic device storing a training model of a multi-dimensional long short-term memory (MDLSTM), the method comprising: acquiring an image including at least one of handwritten text and printed text; identifying each text line region in the image by image processing; and Recognize the text contained in each recognized text line region based on the training model, Wherein, identifying each text line region includes, for each text line region, Calculating the average value of the normalized text height of the text line area by image processing, applying a first horizontal line of the calculated average value of the normalized text height and two second horizontal lines spaced apart from the first horizontal line at equal intervals in two opposite directions of the first horizontal line to the text line area, and The text spanning the first horizontal line and two second horizontal lines is determined as one text line. The plurality of text line regions in the image include a plurality of text lines. wherein identifying each text line region comprises: individually identifying each of the plurality of text lines by applying a first horizontal line and two second horizontal lines to each of the plurality of text line regions in the image, and The text in the image that spans the first horizontal line and the two second horizontal lines is recognized as one text line among the multiple text lines.

10. The method of claim 9, wherein: Identifying each text line region includes binarizing the text region in the image and performing the image processing by performing correction processing on the binarized text region.

11. The method of claim 10, wherein: The correction process includes performing at least one of de-skewing, height normalization and skeletonization on the text region.

12. The method of claim 9, further comprising: identifying a region containing the identified text line as a plurality of vertical blocks; And the probability of the character estimated from each vertical block is calculated based on the pixel values ​​of each vertical block.

13. The method of claim 12, wherein: Calculating the probability includes: combining the existence probability of the first character in the first vertical block, the non-existence probability of the first character in the first vertical block, the existence probability of the second character in the second vertical block, and the non-existence probability of the second character in the second vertical block to calculate multiple probabilities that at least one of the first character and the second character is included in the text line area.

14. The method of claim 13, wherein: Recognizing the text includes applying token passing decoding and a preset language model to the plurality of calculated probabilities and acquiring text recognized in the text line region.

15. The method of claim 9, further comprising: A function associated with the recognized text is identified and the identified function is performed.

Images