Pointing-based information provision method and system

Abstract

To provide an information providing method and a system based on pointing that provides information on a word corresponding to finger coordinates.SOLUTION: An information providing method comprises steps of: determining, in a process of recognizing a character included in an offline bulletin object of a user to output as voice, user-specified coordinates on an imaged image of the offline bulletin object in response to a trigger generated by user input; determining a word corresponding to the determined user-specified coordinates from among characters included on the image; and providing additional information for the determined word.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The following description relates to a pointing-based information provision method and system. [Background technology]

[0002] There are devices and / or services that recognize characters in offline postings such as books, synthesize the recognized characters as voice, and output the voice from a speaker to read the offline postings aloud. When an unknown word appears in the text of the offline posting, the word may be one that the user does not understand or one whose more accurate meaning he or she would like to know. When using an artificial intelligence speaker, the user may directly ask about the meaning of the unknown word while speaking it out loud. Alternatively, the user may directly look up the unknown word using another device or a dictionary. In this case, there is a possibility that an error may occur during the speaking or typing process.

[0003] In addition, there is a conventional technology that highlights the word area and provides the meaning in a dictionary when an unknown word is selected using a finger or a specific pointing device. In this case, it is well known to detect fingertips or fingertip points, and it is also well known to capture specific symbols while recognizing hand gestures on a mobile device. In addition, there is also a technology that adjusts a device far away from the fingertips by using a multi-angle camera and gaze angle.

[0004] However, conventional techniques for obtaining finger coordinates from captured images have problems such as slow speed and high error rates when multiple fingers appear in the image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Korean Patent Publication No. 10-2020-0049435 Summary of the Invention [Problem to be solved by the invention]

[0006] To provide an information providing method and system capable of providing information on a word corresponding to a finger coordinate by providing the finger coordinate in a character recognition engine using a trigger for obtaining the finger coordinate in the process of recognizing characters contained in an offline posting for reading out the offline posting of a user.

[0007] To provide an information providing method and system capable of improving user convenience by enabling a user to set a start position so that reading of an offline notice begins from a desired part based on a word corresponding to finger coordinates.

[0008] To provide an information providing method and system capable of providing a function of repeatedly reading out a sentence including a word corresponding to a finger coordinate a plurality of times. [Means for solving the problem]

[0009] The present invention provides an information provision method for a computer device including at least one processor, the information provision method including a step of determining user-specified coordinates on an image of an offline posting in response to a trigger generated by a user input during a process in which the at least one processor recognizes characters contained in an offline posting and outputs the characters by voice, a step of the at least one processor determining a word corresponding to the determined user-specified coordinates from among the characters contained in the image, and a step of the at least one processor providing additional information of the determined word.

[0010] According to one aspect, the step of determining the user-specified coordinates may be characterized in that a center coordinate of a fingernail recognized on the image is determined as the user-specified coordinates.

[0011] According to another aspect, the step of determining the user-specified coordinates may include determining coordinates of a pointing tool recognized on the image as the user-specified coordinates.

[0012] According to another aspect, characters contained on the image may be recognized by an OCR (Optical Character Reader) in units of boxes containing at least one character, and the step of determining the word may be characterized by selecting a word contained in a box that is closest to the user-specified coordinates as the word corresponding to the user-specified coordinates.

[0013] According to another aspect, the distance may include a distance between the user-specified coordinate and a bottom line of four lines forming the box, or a distance between the user-specified coordinate and a midpoint of the bottom line.

[0014] According to another aspect, determining the word may include extracting the word from the box by natural language processing of characters contained in the box.

[0015] According to another aspect, the step of providing the additional information may include the steps of receiving additional information for the determined word from a server providing at least one of an online dictionary service and an online translation service, converting the received additional information into speech, and outputting the converted speech.

[0016] According to another aspect, determining the user-specified coordinates may include generating the trigger by recognizing a preset intention from a user's utterance.

[0017] According to another aspect, the step of determining the user-specified coordinates may include a step of inputting an image corresponding to the trigger and determining one finger from a plurality of fingers included in the image corresponding to the trigger by a machine learning module trained to receive an image and determine one finger from a plurality of fingers included in the image, and a step of determining finger coordinates of the determined finger as the user-specified coordinates.

[0018] According to another aspect, the step of determining the word may be characterized by recognizing the word corresponding to the user-specified coordinates from a previous image taken of the offline posting when the word corresponding to the user-specified coordinates cannot be recognized because at least a portion of the word is obscured by a finger or a pointing tool.

[0019] According to another aspect, the information providing method may further include a step of the at least one processor specifying the position of the determined word as a starting position for reading the offline posting, and a step of the at least one processor outputting the recognized characters from the starting position aloud.

[0020] According to yet another aspect, the information providing method may further include a step of the at least one processor recognizing a sentence including the determined word, and a step of the at least one processor outputting the recognized sentence aloud by repeating it a number of times.

[0021] A computer program is provided that is recorded on a computer-readable recording medium and that, when combined with a computer device, causes the computer device to execute the method.

[0022] A computer-readable recording medium is provided on which a program for causing a computer device to execute the method is recorded.

[0023] A computer device is provided, comprising at least one processor implemented to execute computer-readable instructions, wherein the at least one processor, in a process of recognizing characters contained in an offline posting and outputting the characters by voice, responds to a trigger generated by a user input to determine user-specified coordinates on an image of the offline posting, determines a word corresponding to the determined user-specified coordinates from among the characters contained on the image, and provides additional information for the determined word. Effect of the Invention

[0024] In the process of recognizing characters contained in an offline posting in order to read out the user's offline posting, a trigger for obtaining finger coordinates is used to provide finger coordinates to a character recognition engine, thereby providing word information corresponding to the finger coordinates.

[0025] By making it possible to set the start position so that reading of an offline message begins from a portion desired by the user based on the word corresponding to the finger coordinates, it is possible to improve user convenience.

[0026] It is possible to provide a function that can read out a sentence including a word corresponding to the finger coordinates repeatedly multiple times. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 illustrates an example of a network environment in accordance with an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a computing device in accordance with an embodiment of the present invention. [Diagram 3] FIG. 1 is a diagram illustrating an example of an information providing system according to an embodiment of the present invention. [Figure 4] FIG. 13 is a diagram illustrating an example of a process for providing information on a pointed-to word in accordance with an embodiment of the present invention. [Diagram 5]FIG. 13 is a diagram illustrating an example of a process for providing information on a pointed-to word in accordance with an embodiment of the present invention. [Figure 6] FIG. 13 is a diagram illustrating an example of a process for providing information on a pointed-to word in accordance with an embodiment of the present invention. [Figure 7] 1 is a flowchart illustrating an example of a method for providing information in accordance with an embodiment of the present invention. [Figure 8] 1 is an image showing an example of one embodiment of the present invention where one finger is pointing and a word is clearly recognizable. [Figure 9] 1 is an image illustrating an example of multiple fingers pointing and clearly recognizable words in accordance with one embodiment of the present invention. [Figure 10] 1 is an image illustrating an example of one finger pointing, occluding letters but allowing word recognition in one embodiment of the present invention. [Figure 11] 1 is an image illustrating an example of a case in which multiple fingers are pointing and the text is obscured in accordance with an embodiment of the present invention. [Figure 12] 11A and 11B are diagrams illustrating an example of a process for setting a start position in an embodiment of the present invention. [Figure 13] 11A and 11B are diagrams illustrating an example of a process for setting a repetition area in an embodiment of the present invention. [Figure 14] FIG. 13 is a diagram showing another example of setting a repetition region in an embodiment of the present invention. [Figure 15] FIG. 13 is a diagram showing another example of setting a repetition region in an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0029] The information provision system according to the embodiment of the present invention may be realized by at least one computer device, and the information provision method according to the embodiment of the present invention may be executed by at least one computer device that realizes the information provision system. A computer program according to an embodiment of the present invention may be installed and executed in the computer device, and the computer device may execute the information provision method according to the embodiment of the present invention according to the control of the computer program being executed. The above-mentioned computer program may be stored in a computer-readable recording medium in combination with the computer device to cause the computer device to execute the information provision method.

[0030] FIG. 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment in FIG. 1 shows an example including a plurality of electronic devices 110, 120, 130, 140, a plurality of servers 150, 160, and a network 170. FIG. 1 is merely an example for explaining the invention, and the number of electronic devices and the number of servers are not limited to those shown in FIG. 1. In addition, the network environment in FIG. 1 is merely an example for explaining one of the environments applicable to this embodiment, and the environment applicable to this embodiment is not limited to the network environment in FIG. 1.

[0031] The electronic devices 110, 120, 130, and 140 may be fixed terminals or mobile terminals realized by computer devices. Examples of the electronic devices 110, 120, 130, and 140 include smartphones, mobile phones, navigation systems, personal computers (PCs), notebook PCs, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), and tablets. As an example, FIG. 1 shows a smartphone as an example of the electronic device 110, but in the embodiment of the present invention, the electronic device 110 may mean one of various physical computer devices that can communicate with other electronic devices 120, 130, and 140 and / or servers 150 and 160 via a network 170 using a substantially wireless or wired communication method.

[0032] The communication method is not limited, and may include not only a communication method using a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network 170 can include, but also short-range wireless communication between devices. For example, the network 170 may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. Furthermore, the network 170 may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or hierarchical network, and the like.

[0033] Each of the servers 150, 160 may be realized by one or more computing devices that communicate with the multiple electronic devices 110, 120, 130, 140 via the network 170 to provide instructions, codes, files, content, services, etc. For example, the server 150 may be a system that provides services (such as a content provision service, a group calling service (or an audio conference service), a messaging service, an email service, a social networking service, a map service, a translation service, a financial service, a payment service, a search service, etc.) to the multiple electronic devices 110, 120, 130, 140 connected via the network 170.

[0034] 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention. Each of the electronic devices 110, 120, 130, and 140 and the servers 150 and 160 described above may be realized by a computer device 200 shown in FIG.

[0035] Such a computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input / output interface 240, as shown in FIG. 2. The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a permanent mass storage device such as a ROM or a disk drive may be included in the computer device 200 as a separate permanent storage device separate from the memory 210. The memory 210 may also store an operating system and at least one program code. Such software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory 210 through a communication interface 230 that is not a computer-readable recording medium. For example, the software components may be loaded into the memory 210 of the computing device 200 based on a computer program that is installed by a file received over a network.

[0036] The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication interface 230. For example, the processor 220 may be configured to execute instructions received according to program code stored in a storage device such as the memory 210.

[0037] The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (for example, the above-mentioned recording device) via the network 170. For example, requests, instructions, data, files, etc. generated by the processor 220 of the computer device 200 according to program codes recorded in a recording device such as the memory 210 may be transmitted to other devices via the network 170 under the control of the communication interface 230. Conversely, signals, instructions, data, files, etc. from other devices may be received by the computer device 200 via the communication interface 230 of the computer device 200 via the network 170. The signals, instructions, data, etc. received via the communication interface 230 may be transmitted to the processor 220 or the memory 210, and the files, etc. may be recorded on a recording medium (the above-mentioned permanent recording device) that the computer device 200 may further include.

[0038] The input / output interface 240 may be a means for interfacing with the input / output device 250. For example, the input device may include a device such as a microphone, a keyboard, or a mouse, and the output device may include a device such as a display, a speaker, etc. As another example, the input / output interface 240 may be a means for interfacing with a device in which functions for input and output are integrated into one, such as a touch screen. The input / output device 250 may be configured as one device together with the computer device 200.

[0039] Also, in other embodiments, the computing device 200 may include fewer or more components than those of Figure 2. However, most of the prior art components need not be explicitly shown in the figures. For example, the computing device 200 may be implemented to include at least some of the input / output devices 250 described above, and may further include other components such as a transceiver, a database, etc.

[0040] Fig. 3 is a diagram showing an example of an information providing system according to an embodiment of the present invention. Fig. 3 shows an information providing device 300, a user 310, an offline posting 320, and a server 330. Although Fig. 3 shows one server 330, there may be multiple servers for each service.

[0041] The information providing device 300 may be a physical electronic device that recognizes characters included in the offline posting 320 of the user 310, converts the recognized characters into voice, and outputs the voice to read the offline posting 320 to the user 310. The information providing device 300 may be realized by the computer device 200 described with reference to FIG. 2 as an example, and may include a camera 301 for recognizing characters included in the offline posting 320, a speaker 302 for outputting voice, and, depending on the embodiment, a microphone 303 for receiving a command based on the user 310's voice. The camera 301, the speaker 302, the microphone 303, and the like may be included in the input / output device 250 described with reference to FIG. 2. Depending on the embodiment, the information providing device 300 may be configured as a dedicated device for reading the offline posting 320. As an example, the information providing device 300 may be a device created in the form of a light or an artificial intelligence speaker.

[0042] Here, the offline notice 320 is not limited to a book, but may be any offline notice including characters such as a magazine or an advertisement paper.

[0043] The information providing device 300 may use an OCR (Optical Character Reader) technology to recognize characters. As an example, the information providing device 300 may include an OCR engine 304 that recognizes characters from an image input to a camera. Since the OCR technology is a well-known technology, a detailed description thereof will be omitted. However, in an embodiment of the present invention, the OCR engine 304 may not only recognize characters but also further recognize and provide finger coordinates in response to a specific trigger.

[0044] At this time, the OCR engine 304 may recognize the fingernails and extract the center coordinates of the nails on the image as finger coordinates. In addition, in order to resolve various positions of the offline posting 320 and the fingers, machine learning trained using a large number of learning images may be utilized. As an example, the machine learning module may be trained to determine one finger from among multiple fingers by using a large number of learning images including information on which finger is correct on an image including multiple fingers. In this case, the OCR engine 304 may input an image for calculating finger coordinates to the trained machine learning module, and if a specific finger is determined by the trained machine learning module, calculate and provide finger coordinates for the determined finger. Although an example of using the center coordinates of the nail has been described above, the present invention is not limited thereto. As an example, the finger coordinates may include the end portion coordinates of the finger.

[0045] Meanwhile, the trigger may be realized in various ways. As an example, the trigger may be generated based on the voice utterance of the user 310. When the user 310 places his / her finger under a specific word in the offline posting 320 and utters an utterance for the trigger (for example, "Hei, what does this word mean?" (here, "Hei" means an example of an utterance that is preset to activate the artificial intelligence speaker of the information providing device 300, and may vary depending on the settings), the information providing device 300 may generate a trigger by recognizing the utterance of the user 310 with the microphone 303. In this case, the OCR engine 304 may recognize and provide the finger coordinates in response to the generated trigger. As another example, the trigger may be , may be generated by a specific button input provided by the information providing device 300. Any method capable of recognizing the intention of the user 310 who wishes to be provided with additional information on a specific word may be used as an event for generating a trigger, without being limited thereto. For example, the information providing device 300 may periodically capture an image of the offline posting 320 to recognize that a page of the offline posting 320 is being turned. In this case, the information providing device 300 may generate a trigger when a specific pointing tool or mark is recognized from the captured image.

[0046] In the process of reading the offline posting 320, the OCR engine 304 may provide a character recognition result for at least a part of the image of the offline posting 320 input to the camera 301. If a trigger occurs in this process, the OCR engine 304 may provide the recognized finger coordinates together with the character recognition result of the image related to the trigger. In this case, the information providing device 300 may identify a word corresponding to the provided finger coordinates and provide additional information of the identified word. The additional information may be generated based on information stored in a local storage location of the information providing device 300, but may preferably be generated based on information obtained from a server 330 connected via the Internet, etc. As an example, the server 330 may be a server providing an online dictionary service or a server providing an online translation service. In this case, the information providing device 300 may obtain information on the dictionary meaning of the word or translation information of the word from the server 330, and may generate and provide additional information to be provided to the user 310 based on the obtained information.

[0047] As an example, the information providing device 300 may provide the additional information to the user 310 by converting the additional information into voice and then outputting the converted voice from the speaker 302. The voice conversion of the additional information may utilize a well-known TTS (Text To Speech) technology.

[0048] Meanwhile, depending on the embodiment, character recognition and finger coordinate provision of the OCR engine 304, recognition of the user's utterance, and / or voice conversion of additional information may be processed by a service provided by the server 330. As an example, the information providing device 300 may transmit at least a part of an image input to the camera 302 and a trigger to the server 330, and the server 330 may recognize characters included in the image and generate finger coordinates. In this case, the information providing device 300 may receive and utilize the character recognition result and finger coordinates from the server 330. In the same manner, recognition of the user's utterance and voice conversion of additional information may be processed by the server 330. In other words, in this specification, an expression that the information providing device 300 processes a specific operation (for example, recognition of the user's utterance) does not exclude the information providing device 300 processing the specific operation through the server 330.

[0049] Meanwhile, the OCR engine 304 sets and provides a box for each text unit recognized as a character recognition result. At this time, if the OCR engine 304 provides the character recognition result and the finger coordinates, the information providing device 300 may determine the word of the box having the closest distance to the finger coordinates as the word intended by the user 310. At this time, the information providing device 300 may measure the distance between a specific position on the box and the finger coordinates. As an example, the information providing device 300 may measure the distance between the midpoint of the bottom line of the box and the finger coordinates. As another example, the information providing device 300 may measure the distance between the finger coordinates and the bottom line of the box. A method for measuring the distance between points or the distance between a point and a line is well known, and therefore a detailed description thereof will be omitted.

[0050] Meanwhile, the OCR engine 304 sets a box for each text unit recognized as a character recognition result and provides it. At this time, since the box unit is not necessarily a word unit, the information providing device 300 may search and recognize words in word units based on the proofreading result of natural language processing. Meanwhile, when one box includes multiple words, the word closest to the finger coordinates may be selected from the multiple recognized words.

[0051] Also, in the image corresponding to the trigger, at least a part of the word to be recognized may be hidden by the finger of the user 310. In such a case, it becomes difficult for the information providing device 300 to obtain a word that provides additional information to the user 310. To solve this, the information providing device 300 may recognize a word corresponding to the finger from a previous image. As an example, when a word on a box corresponding to the finger coordinates cannot be recognized in a state where the finger coordinates are obtained, the information providing device 300 may attempt to recognize a word from a box corresponding to the finger coordinates on the previous image.

[0052] 4 to 6 are diagrams showing an example of a process for providing information on a word pointed to by a finger in one embodiment of the present invention.

[0053] FIG. 4 shows an example of an image 400 captured by the information providing device 300 with the camera 302 when a user (for example, user 310 in FIG. 3) points at a particular word on an offline posting 410 with his / her finger and speaks something like, "Hey, what does this word mean?"

[0054] 5 illustrates an example of a process in which the information providing device 300 determines finger coordinates in the image 400 using the OCR engine 304. Here, the finger coordinates may be coordinates on the image 400 or may be center coordinates of the nail, but are not limited thereto.

[0055] 6 shows an example of a process in which the information providing device 300 determines the word closest to the finger coordinates based on the character recognition result provided by the OCR engine 304 and the finger coordinates. In this embodiment, the word "meet" is determined as the word closest to the finger coordinates. As described above, the information providing device 300 may select a specific box based on the distance between the center position (position on the image 400) of the bottom line of the box and the finger coordinates, and may determine the word included in the selected box as the word corresponding to the finger coordinates. However, as described above, the position of the box is not limited to the center position of the bottom line.

[0056] Once the word intended by the user is determined, the information providing device 300 may search for the dictionary meaning or translation result of the word determined by the server 330 to generate additional information of the determined word, and may convert the generated additional information into audio and provide it to the user.

[0057] FIG. 7 is a flowchart showing an example of an information providing method according to an embodiment of the present invention. The information providing method according to this embodiment may be executed by a computer device 200. In this case, the processor 220 of the computer device 200 may be realized to execute control instructions according to the operating system code and at least one computer program code included in the memory 210. Here, the processor 220 may control the computer device 200 so that the computer device 200 executes steps 710 to 730 included in the method of FIG. 7 according to the control instructions provided by the code recorded in the computer device 200.

[0058] In step 710, the computer device 200 may determine finger coordinates on an image of a captured offline posting in response to a trigger generated by a user input during the process of recognizing characters included in the offline posting and outputting them as voice. As an example, the computer device 200 may determine the center coordinates of a fingernail recognized on the image as the finger coordinates. However, this is merely an example, and it will be easily understood that various embodiments are possible, such as using the end of a finger as the finger coordinate.

[0059] Meanwhile, the computing device 200 may generate a trigger by recognizing a preset intent based on the user's utterance. Although the above describes an example of using a specific utterance such as "Hey, what does this word mean?", the trigger may be generated by other expressions of the same intent (for example, "Hey, what does this word mean?"). Determining the intent of an expression is a well-known technique.

[0060] Also, multiple fingers may be recognized from the image. In this case, fingers far from the area of ​​the offline posting or objects that are not fingers (for example, toes) may be excluded from the recognition. Also, fingers located at a certain distance or more from the text included in the offline posting may be excluded from the recognition. When multiple fingers are recognized as being located within a certain distance from the text included in the offline posting, the OCR engine may output the coordinates of each of the multiple recognized fingers. In this case, the computer device 200 may determine coordinates that are appropriate for the user's intention from the multiple coordinates output by the OCR engine based on the distance between the coordinates and the text.

[0061] Meanwhile, the computer device 200 may input an image corresponding to a trigger by a machine learning module trained to receive an image and determine one of a plurality of fingers included in the image, and may determine finger coordinates of the determined finger. Such a machine learning module may be used to determine the finger intended by the user when there are a plurality of fingers on one image. According to an embodiment, the OCR engine may determine finger coordinates of each of the recognized fingers and then select one finger coordinate from among the finger coordinates using the machine learning module. In this case, the machine learning module may be trained to output one finger coordinate from among the plurality of finger coordinates in a learning image including an image, a plurality of finger coordinates, and a correct answer finger coordinate.

[0062] In step 720, the computer device 200 may determine a word corresponding to the determined finger coordinates from among characters included in the image. As an example, as described above, characters included in the image may be recognized by OCR in units of boxes including at least one character. In this case, the computer device 200 may select a word included in a box having a closest distance to the finger coordinates as a word corresponding to the finger coordinates. Here, the distance may include a distance between the finger coordinates and a bottom line of four lines forming the box, or a distance between the finger coordinates and a midpoint of the bottom line. In addition, the computer device 200 may extract a word from the box by natural language processing for the characters included in the box. This may be used when the box does not separate characters in units of words.

[0063] Furthermore, when the computer device 200 cannot recognize a word corresponding to the finger coordinates because at least a part of the word is obscured by the finger, the computer device 200 may recognize the word corresponding to the finger coordinates from a previous image captured of the offline posting.

[0064] In step 730, the computer device 200 may provide additional information of the determined word. As an example, the computer device 200 may receive the additional information of the determined word from a server that provides at least one of an online dictionary service and an online translation service. At this time, the computer device 200 may convert the received additional information into voice and provide the additional information to the user by outputting the converted voice. As described above, converting the additional information into voice may be based on TTS technology, and the voice may be output from a speaker included in the computer device 200 or connected to the computer device 200. In addition, according to an embodiment, the computer device 200 may generate and provide the additional information using information stored in a local storage location of the computer device 200 without going through a server.

[0065] According to an embodiment, the computer device 200 may designate the position of the word determined in step 720 as a starting position for reading the offline posting, and output the recognized characters by voice from the starting position. In other words, the computer device 200 may start reading the offline posting from the word pointed to by the user's finger. This embodiment will be described in more detail with reference to FIG. 12.

[0066] In another embodiment, the computer device 200 may recognize a sentence including the word determined in step 720 and output the recognized sentence aloud multiple times. In other words, the computer device 200 may read out a sentence including the word pointed to by the user with a finger multiple times. This embodiment will be described in more detail with reference to FIG. 13.

[0067] 8 is an image showing an example of a case where one finger is pointing and a word can be clearly recognized in an embodiment of the present invention. In FIG. 8, one finger is pointing at the character "young" and the OCR engine 304 can clearly recognize the character "young". In this case, the OCR engine 304 may provide an OCR recognition result and finger coordinates for the character "young" as shown in Table 1 below.

[0068] [Table 1]

[0069] In Table 1, "boundingBox" indicates the coordinates of the four corners of a box on an image, "confidence" indicates the confidence of the character recognized corresponding to the box, "isVertical" indicates whether the recognized character is vertical, and "text" indicates the character recognized corresponding to the box. "Group" may be a criterion for bundling the results from one recognition into one group, and "subGroup" may be a clustered value based on sorting and positional distance within the entire recognition result, which may be used to determine the accuracy of the corresponding area. "Fingertips" indicates the finger coordinates of the fingers on the image, and "succeeded" indicates whether the recognition of the finger coordinates was successful. In this case, the information providing device 300 may calculate the distance between the finger coordinates [940,600] and the box coordinates [897,588], for example. The information providing device 300 may also calculate the distance between the finger coordinates and other recognized boxes, and the box with the closest distance may be selected.

[0070] Table 2 below shows an example of the overall character recognition results provided by the OCR engine 304 for the image of FIG.

[0071] [Table 2]

[0072] Figure 9 is an image showing an example of a case where multiple fingers are pointing and a word is clearly recognizable in accordance with one embodiment of the present invention. The image points to TIFF0007626566000003.tif28127, but other fingers are also shown as examples when present on offline postings.

[0073] In this case, the following Table 3 shows the characters provided by the OCR engine 304 in the image of FIG. This shows the recognition result and finger coordinates for TIFF0007626566000004.tif20144.

[0074] [Table 3]

[0075] As described above, when multiple finger coordinates are recognized, one finger may be determined using machine learning, etc. Alternatively, finger coordinates that are a certain distance or more away may be excluded in advance based on the position of the recognized box.

[0076] Figure 10 is an image showing an example of a case where one finger is pointing and obscuring the letters but still allowing word recognition in one embodiment of the present invention. In Figure 10, one finger is obscuring part of the letters "faster!" but still allowing word recognition.

[0077] At this time, the following Table 4 shows the recognition result and finger coordinates of the character "itfaster" provided by the OCR engine 304 in the image of FIG.

[0078] [Table 4]

[0079] Meanwhile, in the example of Fig. 10, the characters "it faster" are erroneously recognized as "itfaster", but this can be separated by techniques such as natural language processing. In this case, as described above, of the two words "it" and "faster", the word "faster" that is closest to the finger coordinates may be selected and used to provide additional information.

[0080] Figure 11 is an image showing an example of a case where multiple fingers are pointing and obscuring characters in one embodiment of the present invention. Figure 11 shows an example of a case where four fingers are recognized, but one of the fingers obscures part of the characters, making it difficult to recognize the word.

[0081] At this time, the following Table 5 shows the character recognition result and finger coordinates provided by the OCR engine 304 in the image of FIG.

[0082] [Table 5]

[0083] In this case, the word the user intends is TIFF0007626566000008.tif21139, but in Table 5 1 shows an example in which the word TIFF0007626566000009.tif30127 is recognized. In this case, as described above, the information providing device 300 may select one of the four finger coordinates by using machine learning. As described above, finger coordinates that are a certain distance or more away from the character to be recognized may be removed in advance. Also, if the character is hidden by the finger, the information providing device 300 may re-recognize the character by using a previous image.

[0084] According to an embodiment, the information providing device 300 may re-perform the recognition by requesting the user to re-perform pointing or adjust the position of the offline posting.

[0085] In addition, according to an embodiment, the information providing device 300 may provide a function of setting a starting position using finger coordinates so as to start reading from a specific area of ​​the offline posting in the process of recognizing characters included in the offline posting and outputting them as voice.

[0086] FIG. 12 is a diagram showing an example of a process of setting a start position in an embodiment of the present invention. FIG. 12 shows an example of an image 1200 captured by the information providing device 300 using the camera 302 when a user (for example, the user 310 in FIG. 3) utters "Hey, read from here" while pointing at a specific word on an offline posting 1210. As described above, the information providing device 300 may extract finger coordinates, determine the finger coordinates on the image 1200 capturing the offline posting 1210, and determine a word (the word "My" in the embodiment of FIG. 12) corresponding to the determined finger coordinates from among the characters included on the image 1200. In this case, the information providing device 300 may start reading from the determined word "My" in response to the user's utterance "Hey, read from here". In other words, the position of the word "My" may be set as the reading start position. As described above, the reading by the information providing device 300 may be a process of recognizing characters included in an offline posting and outputting the characters as voice. In this case, in the embodiment of Fig. 12, the information providing device 300 may output voice corresponding to "My name is Gil-dong Hong. What's your name." from the word "My" at the starting position.

[0087] If the word corresponding to the finger coordinates is "Gil-dong," the information providing device 300 will output a voice corresponding to "Gil-dong Hong. What's your name." from the word "Gil-dong" which is the starting position.

[0088] In this manner, according to this embodiment, it is possible to read out text not only from the beginning of an offline posting, but also from a start position that can be easily and conveniently specified by the user.

[0089] In another embodiment, the information providing device 300 may utilize the finger coordinates to identify a particular area that the user wishes to have read out repeatedly. In other words, the user can directly specify a particular area that the user wishes to have read out repeatedly using the finger coordinates.

[0090] FIG. 13 is a diagram illustrating an example of a process of setting a repetition area in an embodiment of the present invention. FIG. 13 illustrates an example of an image 1300 captured by the information providing device 300 with the camera 302 when a user (for example, the user 310 in FIG. 3) utters "Hey, read this sentence three times" while pointing at a specific word on an offline posting 1310. In this case, the information providing device 300 may extract finger coordinates and determine the finger coordinates on the image 1300 capturing the offline posting 1310. Also, the information providing device 300 may determine a word (the word "meet" in the embodiment of FIG. 13) corresponding to the finger coordinates from among the characters included on the image 1300. In this case, the information providing device 300 may recognize the sentence "Nice to meet you." including the determined word "meet" as a response to the user's utterance "Hey, read this sentence three times" and may output a voice corresponding to the recognized sentence "Nice to meet you." three times.

[0091] Thus, according to the embodiment of FIG. 13, the information providing device 300 is capable of repeatedly reading out a sentence containing a word specified by the user multiple times, thereby making it possible to add a variety of learning functions.

[0092] In some embodiments, a user may say something like "Hey, read from here to here three times" while changing finger positions. In this case, the information providing device may recognize the specific portion that the user wants to have read aloud repeatedly by utilizing a first finger coordinate corresponding to the first "here" in the user's utterance and then a second finger coordinate corresponding to the second "here" in the user's utterance.

[0093] 14 and 15 are diagrams showing another example of setting a repetition area in an embodiment of the present invention. FIG. 14 and FIG. 15 show an example in which a user (for example, the user 310 in FIG. 3) utters "Hei, read from here to here three times" while changing the position of his / her finger on the offline posting 1410. In this case, the information providing device 300 may determine the first finger coordinate in the first image 1400 corresponding to the time when the first "here" is uttered, and may determine the second finger coordinate in the second image 1500 corresponding to the time when the second "here" is uttered. In some embodiments, an image may be captured after all of the user's utterances are analyzed. In this case, the first finger coordinate and the second finger coordinate may be determined based on two finger coordinates recognized in one image. In this case, which of the first finger coordinate and the second finger coordinate is first may be determined by analyzing the text of the first and second finger coordinates. In another embodiment, the user's utterance may be input twice. For example, a first utterance "Hei, from here" and a second utterance "Hei, read up to here three times" may be used to determine two utterances in total, and a first finger coordinate and a second finger coordinate may be determined from images captured in association with each of the two utterances. In addition, the information providing device 300 may recognize a specific portion of text that the user wishes to have read aloud, [Nice to meet you. My name is Gil-dong Hong. What's your name], by determining the words [Nice, name] corresponding to each of the first and second finger coordinates. In this case, the information providing device 300 may output a voice corresponding to the recognized specific portion of text, [Nice to meet you. My name is Gil-dong Hong. What's your name], three times repeatedly.

[0094] In the present embodiment, an example of using the utterance "here" has been described, but it is also possible to define and use utterances for specifying the start and end of a desired part individually. For example, specific terms such as "start" and "end" such as "Hey, read from the start to the end three times" may be predefined and used as terms for recognizing a specific part of text.

[0095] Meanwhile, the "finger" for recognizing the coordinates may be replaced with a pointing tool such as a pen. Such a pointing tool may be a dedicated tool provided for recognizing the coordinates, a specific mark such as a QR code, or a tool arbitrarily used by the user. The mark can be immediately recognized at the camera preview stage, and can also be used as a trigger when necessary. In this case, the above-mentioned finger coordinates may mean coordinates that recognize the position of a specific pointing tool or marker on an image. For example, when a ballpoint pen is used as a pointing tool, the information providing device 300 may recognize and use the coordinates on the image for the tip of the ballpoint pen on the image. The dedicated tool may include a pattern or expression that is preset so that the information providing device 300 can easily recognize the coordinates on the image. In this case, the information providing device 300 may recognize and use the coordinates of a position where a pattern or expression that is preset on the image exists as the finger coordinates. For this reason, the term "finger coordinates" may be used in an expanded form to "user-designated coordinates" that are coordinates for a position that the user wishes to designate.

[0096] Also, in the above-described embodiment, when multiple fingers are detected on an image, one finger is determined using machine learning or the like, or finger coordinates that are a certain distance or more are excluded in advance based on the position of a recognized box. Meanwhile, in some embodiments, when multiple finger coordinates (user-specified coordinates) are detected, the information providing device 300 may assign a priority to each of the multiple finger coordinates. As an example, the priority may be determined according to the method of reading aloud a book. When the method of reading a book proceeds from top to bottom and then from left to right, the information providing device 300 may set priorities for the multiple finger coordinates such that finger coordinates located at the top of the book have a higher priority, and finger coordinates located at the left side of the book have a higher priority if the height is the same / similar. Thereafter, the information providing device 300 may provide additional information for each word in order according to the set priority. However, even in this case, finger coordinates that are a certain distance or more may be excluded in advance based on the position of a recognized box. Alternatively, at least two finger coordinates may be used simultaneously to simultaneously specify the start position and end position for reading a sentence.

[0097] As described above, according to the embodiment of the present invention, in the process of recognizing characters included in an offline posting to read out a user's offline posting, the character recognition engine provides finger coordinates based on a trigger for obtaining finger coordinates, thereby providing information on a word corresponding to the finger coordinates. Also, the start position can be set so that the reading out of the offline posting can start from a part desired by the user based on the word corresponding to the finger coordinates, thereby improving user convenience. Furthermore, a function can be provided that allows a sentence including a word corresponding to the finger coordinates to be read out repeatedly multiple times.

[0098] The above-described systems or devices may be realized by hardware components or a combination of hardware and software components. For example, the devices and components described in the embodiments may be realized using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications that run on the OS. The processing device may also respond to the execution of the software to access, store, manipulate, process, and generate data. For convenience of understanding, a single processing device may be described as being used, but those skilled in the art will understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0099] The software may include computer programs, codes, instructions, or a combination of one or more of these, and may configure or instruct the processing device to operate as desired, either independently or collectively. The software and / or data may be embodied in any type of machine, component, physical device, virtual device, computer storage medium, or device to be interpreted on the processing device or to provide instructions or data to the processing device. The software may be distributed and stored and executed in a distributed manner on computer systems connected by a network. The software and data may be stored in one or more computer-readable recording media.

[0100] The method according to the embodiment may be realized in the form of program instructions executable by various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The medium may be one that continuously records a computer-executable program, or one that temporarily records the program for execution or download. The medium may be one that is a recording or storage means in the form of a single or multiple hardware devices, and is not limited to a medium that is directly connected to a certain computer system, but may be one that is distributed over a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROMs, RAMs, flash memories, and the like, which are configured to record program instructions. Other examples of the medium include recording media or storage media managed by application stores that distribute applications, or sites, servers, and the like that supply or distribute various other software. Examples of the program instructions include not only machine language codes such as those generated by compilers, but also high-level language codes that are executed by a computer using an interpreter or the like.

[0101] Although the embodiments have been described based on the limited embodiments and drawings as above, those skilled in the art will be able to make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from that described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a manner different from that described, or may be counterbalanced or replaced by other components or equivalents, and still achieve suitable results.

[0102] Therefore, different embodiments are within the scope of the appended claims provided they are equivalent to the claims. [Explanation of symbols]

[0103] 300: Information provision device 301: Camera 302: Speaker 303:Mike 304: OCR engine 310:User 320: Offline postings 330: Server

Claims

1. 1. A method for providing information in a computer device including at least one processor, comprising: determining, by the at least one processor, a user-specified coordinate on a captured image of an offline posting in response to a trigger generated by a user input, the user input including a user speech; determining, by the at least one processor, a word from among characters contained on the image that corresponds to the determined user-specified coordinates; and said at least one processor providing additional information for said determined words. Including, The step of determining the user-specified coordinates comprises: A method for providing information, comprising determining a central coordinate point of a fingernail recognized on the image as the user-specified coordinate.

2. The step of determining the user-specified coordinates comprises:

2. The information providing method according to claim 1, further comprising determining coordinates of a pointing tool recognized on the image as the user-specified coordinates.

3. The characters included in the image are recognized by an OCR (Optical Character Reader) in units of boxes including at least one character, The step of determining words comprises:

3. The information providing method according to claim 1, further comprising the step of selecting a word contained in a box having a closest distance to the user-specified coordinates as the word corresponding to the user-specified coordinates.

4. The information providing method according to claim 3 , characterized in that the distance includes a distance between the user-specified coordinates and a bottom line of the four lines forming the box, or a distance between the user-specified coordinates and a midpoint of the bottom line.

5. The step of determining words comprises: extracting the word from the box by natural language processing the characters included in the box. The information providing method according to claim 3, further comprising:

6. The step of providing additional information comprises: receiving additional information for the determined words from a server providing at least one of an online dictionary service and an online translation service; converting the received additional information into audio; and outputting the converted voice The information providing method according to any one of claims 1 to 5, comprising:

7. The step of determining the user-specified coordinates comprises: generating the trigger by recognizing a preset intention from a user's utterance; The information providing method according to any one of claims 1 to 6, comprising:

8. At least one processor implemented to execute computer readable instructions Including, The at least one processor: In response to a trigger generated by a user input including a user utterance, a user-specified coordinate is determined on a captured image of the offline posting; determining a word from among characters contained on the image that corresponds to the determined user-specified coordinates; and providing additional information about the determined words; determining said user specified coordinates, by said at least one processor; determining a central coordinate point for the fingernails recognized on the image as the user-specified coordinate; A computer device comprising:

9. The characters included in the image are recognized by an OCR (Optical Character Reader) in units of boxes including at least one character, by said at least one processor to determine said words; selecting a word contained in a box that is closest to the user-specified coordinates as the word corresponding to the user-specified coordinates; 9. The computer device of claim 8.

10. by said at least one processor to provide said additional information; receiving additional information for the determined words from a server providing at least one of an online dictionary service and an online translation service; converting the received additional information into audio; outputting the converted audio; 10. Computer device according to claim 8 or 9, characterized in that

11. The at least one processor: designating the determined word position as a starting position for reading the offline posting; outputting the recognized characters from said starting position by voice; A computer device according to any one of claims 8 to 10, characterized in that

12. The at least one processor: Recognizing sentences containing the determined words; Repeating the recognized sentence a number of times and outputting it as voice. A computer device according to any one of claims 8 to 11, characterized in that

Images