Method and device for augmented reality

Abstract

The present disclosure provides an augmented reality system (200), wherein the augmented reality system includes an input unit (204), a text recognition unit (206), a natural language processing unit (208), a positioning unit (210), and an output unit (212). The input unit (204) captures an image. The text recognition unit (206) recognizes information on a surface depicted in the image and generates input data based on the information. The natural language processing unit (208) determines the context of the input data and generates at least one piece of auxiliary information based on the context. The positioning unit (210) determines one or more spatial attributes based on the image and generates positioning information based on the spatial attributes. The output unit (212) displays the auxiliary information based on the positioning information.

Description

Technical Field

[0001] The present disclosure generally relates to augmented reality, and more particularly but not exclusively, to devices and methods for assisting a user when writing in an augmented reality system. Background Art

[0002] Writing, as an art, has been cherished by humans since 3200 BC. The tools used for writing have evolved significantly since prehistoric times. Modern writing tools such as pens, paper, and pencils are widely used by people today. However, the writing experience remains the same. When writing with a pen and paper, special functions such as spell correction and word prediction, which are commonly used when typing on a keyboard, are not available. The availability of these features has led most users to prefer typing over writing.

[0003] However, the integration of styluses and pens with smartphones and tablets has made writing popular again. Additionally, in major regions of the world, writing remains the preferred mode of communication for schoolwork and exams. Several conventional techniques have been tried to provide the above-mentioned special functions to users when writing with a pen and paper.

[0004] One conventional technique is a smart pen that detects grammar and spelling errors and provides an alert to the user accordingly. The smart pen uses motion detection to identify the characters and words written by the user. However, the alert is provided through vibration, which is not intuitive for the user. The smart pen uses expensive hardware and is therefore not economically viable for a large number of people. The smart pen checks only one error at a time. Therefore, when there are a large number of errors in the writing, the smart pen may miss one or more errors. Additionally, the smart pen does not provide a visual indication of the error, making it extremely difficult for the user to identify the error.

[0005] Another smart pen allows the user to write words in large handwriting and check the spelling of the words. The smart pen has a built-in display that is operated either by buttons or by voice recognition. When the user speaks a word into the microphone of the smart pen, the word is displayed on the display of the smart pen. The smart pen also displays the meaning and spelling of the handwritten word on the display. However, the smart pen does not recommend or suggest words. The smart pen depends on the correct pronunciation of the user. The smart pen does not notify the user in case of an error. The smart pen interrupts the natural flow of writing.

[0006] Another smart pen digitizes handwritten content. The smart pen records the handwritten content and uploads the record to a computer or a smart phone. Thereafter, the smart pen synchronizes it with the recorded audio. This allows the user to replay portions of the record by tapping on the handwritten text that they were writing while the record was being made. However, the smart pen is not a viable solution because it does not work on ordinary surfaces and requires special paper for writing. The smart pen only digitizes handwritten content. The smart pen is bulky and weighs five times more than an ordinary ballpoint pen. The smart pen is expensive compared to an ordinary ballpoint pen.

[0007] There is no conventional technology that provides writing assistance when writing with an ordinary pen on ordinary paper.

[0008] Therefore, there is a need for a system that provides assistance to the user during writing. Summary of the Invention

[0009] Technical Problem

[0010] One aspect of the present disclosure is to provide an apparatus and method for assisting a user during writing in an augmented reality system.

[0011] Technical Solution

[0012] The present disclosure provides concepts related to systems and methods for augmented reality. The present disclosure is neither intended to identify the essential features of the present disclosure nor intended to determine or limit the scope of the present disclosure.

[0013] In an embodiment of the present disclosure, an augmented reality system is provided. The augmented reality system includes an input unit, a text recognition unit, a natural language processing unit, a positioning unit, and an output unit. The input unit is configured to capture an image. The text recognition unit is configured to recognize information on a surface depicted in the image. The text recognition unit generates input data based on the information. The natural language processing unit is configured to determine the context of the input data. The natural language processing unit generates at least one piece of assistance information based on the context. The positioning unit is configured to determine one or more spatial attributes based on the image. The positioning unit generates positioning information based on the spatial attributes. The output unit is configured to display the assistance information based on the positioning information.

[0014] In another embodiment of the present disclosure, an augmented reality server is provided. The augmented reality server includes an input unit, a text recognition unit, a natural language processing unit, a positioning unit, and an output unit. The input unit is configured to receive an image. The text recognition unit is configured to recognize information on a surface depicted in the image. The text recognition unit generates input data based on the information. The natural language processing unit is configured to determine the context of the input data. The natural language processing unit generates at least one piece of auxiliary information based on the context. The positioning unit is configured to determine one or more spatial attributes based on the image. The positioning unit generates positioning information based on the spatial attributes. The output unit is configured to send the auxiliary information and the positioning information.

[0015] In another embodiment of the present disclosure, an augmented reality method is provided. The augmented reality method is implemented in an augmented reality system. The augmented reality method includes capturing an image and recognizing information on a surface depicted in the image. The method further includes determining the context of the input data and generating at least one piece of auxiliary information based on the context. Thereafter, one or more spatial attributes are determined based on the image. Positioning information is generated based on the spatial attributes. The auxiliary information is displayed based on the positioning information.

[0016] In an exemplary embodiment, the spatial attribute includes at least one of the following: an angle between the augmented reality system and the surface, a distance between the augmented reality system and the surface, and an obstacle in the field of view between the augmented reality system and the surface.

[0017] In another exemplary embodiment, the positioning unit is further configured to determine a position based on the distance, the angle, and a set plane. The positioning unit determines a style based on at least one of the following: the auxiliary information, the distance, the angle, the background of the information, and a predefined style preference. The positioning unit further generates positioning information indicating at least one of the plane, the position, and the style.

[0018] In another exemplary embodiment, the positioning unit is further configured to compare the angle with a predetermined threshold angle. If the angle is less than the threshold angle, the positioning unit sets a two-dimensional plane, and if the angle is not less than the threshold angle, the positioning unit sets a three-dimensional plane.

[0019] In another exemplary embodiment, the natural language processing unit is further configured to determine the type of the auxiliary information based on the context of the input data. The natural language processing unit determines the content of the auxiliary information based on the context of the input data. In addition, the natural language processing unit generates the auxiliary information. The auxiliary information includes the type and the content.

[0020] In another exemplary embodiment, the type of the auxiliary information includes one or more of the following items: text, audio, video, image, and animation.

[0021] In another exemplary embodiment, the content of the auxiliary information includes one or more of the following items: context-related information, an explanation of the information, text correction, text prediction, grammar errors, syntactic errors, and an indication of plagiarism.

[0022] In an exemplary embodiment, the augmented reality system is at least one of a head-mounted device and a hand-held device.

[0023] In an exemplary embodiment, the surface includes a non-digital writing surface.

[0024] In another exemplary embodiment, the surface includes a digital display of a user device.

[0025] In another exemplary embodiment, the augmented reality system further includes a communication unit that communicates with the user device. The communication unit is configured to: receive a user input from the user device. The communication unit sends the auxiliary information to the user device based on the user input.

[0026] In another exemplary embodiment, the handwriting reconstruction unit is configured to detect a handwriting style of the information and reconstruct the handwriting style.

[0027] In another exemplary embodiment, the style includes one or more of the following items: size, font, and visual effects.

[0028] In another exemplary embodiment, the auxiliary information is displayed in the form of the handwriting style.

[0029] In another exemplary embodiment, the augmented reality system further includes a user interaction unit. The user interaction unit is configured to detect one or more user gestures indicating an interaction with the auxiliary information or with the information on the surface depicted in the image. In addition, the user interaction unit causes updated auxiliary information to be displayed based on the user gestures.

[0030] In another exemplary embodiment, the information is at least one of the following: handwritten information, printed information, electronically displayed information, and virtual projection information.

[0031] In another exemplary embodiment, the input unit captures a plurality of images in real time.

[0032] In another exemplary embodiment, the positioning unit dynamically updates positioning information based on real-time images.

[0033] Beneficial effects

[0034] Various embodiments of the present disclosure provide an effect of improving the convenience of a user who takes notes through an electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The detailed description is described with reference to the accompanying drawings. The same reference numerals are used throughout the drawings to refer to the same features and modules.

[0036] Figure 1 is a schematic block diagram of an augmented reality (AR) system according to an embodiment of the present disclosure.

[0037] Figure 2 is a schematic block diagram of an AR system according to an embodiment of the present disclosure.

[0038] Figure 3 is a schematic block diagram of an AR server according to an embodiment of the present disclosure.

[0039] Figure 4 is a schematic block diagram of a natural language processing unit according to an embodiment of the present disclosure.

[0040] Figure 5 is a schematic block diagram of a positioning unit according to an embodiment of the present disclosure.

[0041] Figure 6 is a flowchart showing a method of error correction according to an embodiment of the present disclosure.

[0042] Figure 7 is a flowchart showing a method of word prediction according to an embodiment of the present disclosure.

[0043] Figure 8 is a flowchart showing a method of word recommendation according to an embodiment of the present disclosure.

[0044] Figure 9 is a flowchart showing an AR method according to an embodiment of the present disclosure.

[0045] Figure 10 is a graphical representation showing the extraction of map points according to an embodiment of the present disclosure.

[0046] Figure 11 is a graphical representation showing a technique of error highlighting and correction display according to an embodiment of the present disclosure.

[0047] Figures 12A to 12C is a graphical representation showing a change in the style of auxiliary information according to an embodiment of the present disclosure.

[0048] Figure 13 is a graphical representation showing a technique of context word prediction according to an embodiment of the present disclosure.

[0049] Figure 14is a graphical representation showing a technique for displaying word prediction in 3D format according to an embodiment of the present disclosure.

[0050] Figure 15 is a graphical representation showing a technique for highlighting different errors and displaying corresponding corrections in the user's writing style according to an embodiment of the present disclosure.

[0051] Figure 16 is a graphical representation showing a technique for similarity checking according to an embodiment of the present disclosure.

[0052] Figure 17 is a graphical representation showing a technique for checking for duplicate information according to an embodiment of the present disclosure.

[0053] Figure 18 is a graphical representation showing a technique for providing writing assistance to a user via virtual gridlines and margins according to an embodiment of the present disclosure.

[0054] Figure 19 is a graphical representation showing a technique for displaying readability metrics to a user according to an embodiment of the present disclosure.

[0055] Figure 20 is a graphical representation showing a technique for simultaneously displaying word recommendations and error corrections to a user on a digital surface according to an embodiment of the present disclosure.

[0056] Figure 21 is a graphical representation showing interactive assistance information according to an embodiment of the present disclosure.

[0057] Figures 22A to 22B is a graphical representation showing a technique for displaying three-dimensional (3D) assistance information according to an embodiment of the present disclosure.

[0058] Figures 23A to 23B is a graphical representation showing a technique for displaying interactive assistance information according to an embodiment of the present disclosure.

[0059] Those skilled in the art will understand that any block diagram herein represents a conceptual diagram of an exemplary system implementing the principles of the present disclosure.

[0060] Similarly, it will be understood that any flowchart, flow diagram, etc. represents various processes that can be substantially represented in a computer-readable medium and thus executed by a computer or a processor, whether or not such a computer or processor is explicitly shown. Detailed Description

[0061] In the following description, for purposes of explanation, specific details are set forth in order to provide an understanding of the present disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure may be practiced without these specific details. One of ordinary skill in the art will recognize that embodiments of the present disclosure, some of which are described below, may be incorporated into multiple systems.

[0062] In addition, the connections between components and / or modules in the figures are not intended to be limited to direct connections. Instead, these components and modules may be modified, reformatted, or otherwise changed by intermediate components and modules.

[0063] References to "one embodiment" or "an embodiment" in the present disclosure mean that a particular feature, structure, characteristic, or function described in connection with the embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" that appears in various places in the specification does not necessarily refer to the same embodiment.

[0064] Various embodiments of the present disclosure provide a system and method for augmented reality.

[0065] In an embodiment of the present disclosure, an augmented reality system is provided. The augmented reality system includes an input unit, a text recognition unit, a natural language processing unit, a positioning unit, and an output unit. The input unit is configured to capture an image. The text recognition unit is configured to recognize information on a surface depicted in the image. The text recognition unit generates input data based on the information. The natural language processing unit is configured to determine the context of the input data. The natural language processing unit generates at least one piece of auxiliary information based on the context. The positioning unit is configured to determine one or more spatial attributes based on the image. The positioning unit generates positioning information based on the spatial attributes. The output unit is configured to display the auxiliary information based on the positioning information.

[0066] In another embodiment of the present disclosure, an augmented reality server is provided. The augmented reality server includes an input unit, a text recognition unit, a natural language processing unit, a positioning unit, and an output unit. The input unit is configured to receive an image. The text recognition unit is configured to recognize information on a surface depicted in the image. The text recognition unit generates input data based on the information. The natural language processing unit is configured to determine the context of the input data. The natural language processing unit generates at least one piece of auxiliary information based on the context. The positioning unit is configured to determine one or more spatial attributes based on the image. The positioning unit generates positioning information based on the spatial attributes. The output unit is configured to send the auxiliary information and the positioning information.

[0067] In another embodiment of the present disclosure, an augmented reality method is provided. The augmented reality method is implemented in an augmented reality system. The augmented reality method includes capturing an image and identifying information on a surface depicted in the image. The method further includes determining the context of the input data and generating at least one piece of auxiliary information based on the context. Thereafter, one or more spatial attributes are determined based on the image. Positioning information is generated based on the spatial attributes. The auxiliary information is displayed based on the positioning information.

[0068] The spatial attributes include at least one of the following: the angle between the augmented reality system and the surface, the distance between the augmented reality system and the surface, and an obstacle in the field of view between the augmented reality system and the surface.

[0069] The positioning unit is further configured to determine a position based on the distance, the angle, and a set plane. The positioning unit determines a style based on at least one of the following: the auxiliary information, the distance, the angle, the background of the information, and a predefined style preference. The positioning unit also generates positioning information indicating at least one of the plane, the position, and the style. The positioning unit is further configured to compare the angle with a predetermined threshold angle. If the angle is less than the threshold angle, the positioning unit sets a two-dimensional plane, and if the angle is not less than the threshold angle, the positioning unit sets a three-dimensional plane.

[0070] The natural language processing unit is further configured to determine the type of the auxiliary information based on the context of the input data. The natural language processing unit determines the content of the auxiliary information based on the context of the input data. In addition, the natural language processing unit generates the auxiliary information. The auxiliary information includes the type and the content.

[0071] The type of the auxiliary information includes one or more of the following: text, audio, video, image, and animation. The content of the auxiliary information includes one or more of the following: context-related information, an explanation of the information, text correction, text prediction, grammar errors, syntactic errors, and an indication of plagiarism.

[0072] The augmented reality system is at least one of a head-mounted device and a hand-held device. In an example, the surface includes a non-digital writing surface. In another example, the surface includes a digital display of a user device.

[0073] The augmented reality system further includes a communication unit that communicates with a user device. The communication unit is configured to: receive a user input from the user device. The communication unit sends the auxiliary information to the user device based on the user input.

[0074] The handwriting reconstruction unit is configured to detect the handwriting style of the information and reconstruct the handwriting style. The style includes one or more of the following items: size, font, and visual effect. In an example, the auxiliary information is displayed in the form of the handwriting style.

[0075] The augmented reality system further includes a user interaction unit. The user interaction unit is configured to detect one or more user gestures indicating an interaction with the auxiliary information or with the information on the surface depicted in the image. In addition, the user interaction unit prompts the display of updated auxiliary information based on the user gestures.

[0076] In an example, the information is at least one of the following: handwritten information, printed information, electronically displayed information, and virtual projection information.

[0077] The input unit captures multiple images in real time. The positioning unit dynamically updates the positioning information based on the real-time images.

[0078] Now referring to Figure 1 , a schematic block diagram of an augmented reality (AR) system (100) according to an embodiment of the present disclosure is shown. The AR system (100) includes an application processing unit (102), a communication module (104), an input device (106), a display (108), an interface (110), a plurality of sensors (112), a memory (114), an audio module (116), a power management module (118), a battery (120), a subscriber identity module (SIM) card (122), a speaker (124), a receiver (126), headphones (128), a microphone (130), a camera module (132), an indicator (134), and a motor (136).

[0079] The communication module (104) includes a cellular module (138), a Wi-Fi module (140), a Bluetooth (BT) module (142), a global navigation satellite system (GNSS) module (144), a near field communication (NFC) module (146), and a radio frequency (RF) module (148).

[0080] The input device (106) includes a touch panel (150), a pen sensor (152), keys (154), and a gesture input (156).

[0081] The display (108) includes a panel (158) and a projector (160).

[0082] The interface (110) includes a high definition multimedia interface (HDMI) (164), an optical interface (166), and a universal serial bus (USB) (168).

[0083] The sensor (112) includes a gesture sensor (112a), a gyro sensor (112b), an atmospheric pressure sensor (112c), a magnetic sensor (112d), a grip sensor (112e), an acceleration sensor (112f), a proximity sensor (112g), an RGB sensor (112h), a light sensor (112i), a biometric sensor (112j), a temperature / humidity sensor (112k), and a UV sensor (112l).

[0084] The memory (114) includes an internal memory (170) and an external memory (172).

[0085] The application processing unit (102) includes a text recognition unit (174), a natural language processing unit (176), a positioning unit (178), and a handwriting reconstruction unit (162). In an example, the positioning unit (178) is implemented as a simultaneous localization and mapping (SLAM) module.

[0086] Now referring to Figure 2 , a schematic block diagram of an augmented reality (AR) system (200) according to an embodiment of the present disclosure is shown. The AR system (200) includes a processing unit (202), an input unit (204), a text recognition unit (206), a natural language processing unit (208), a positioning unit (210), an output unit (212), a communication unit (214), a user interaction unit (216), and a handwriting reconstruction unit (218).

[0087] The AR system (200) may be in the form of a head-mounted device such as glasses or a hand-held device such as a smart phone. In an example, when reading text information written or printed on any digital or non-digital surface, or when writing text information on a surface, the user wears the AR system (200) in the form of smart glasses. In another example, the user uses the AR system (200) in the form of a smart phone to view the information presented on a surface.

[0088] Examples of the AR system (200) include devices for receiving live video of the real world or physical environment. The AR system (200) includes an electronic device capable of supporting an AR display, including but not limited to a personal computer, a mobile phone, an electronic tablet, a game console, a media player, etc. In some embodiments, the AR system (200) may be an electronic tablet or an electronic smart phone having a touch-sensitive surface.

[0089] The input unit (204) includes a sensor for dynamically capturing an image of a surface in real time. In an example, the input unit (204) is a camera connected to the AR system (200). Optionally, the input unit (204) can capture a series of images or real-time video. The input unit (204) sends the image to the text recognition unit (206).

[0090] The text recognition unit (206) receives the image from the input unit (204). The text recognition unit (206) processes the image to identify the surface and the information presented on the surface. The information can be printed, written, projected, engraved, or drawn on the surface. The information can be handwritten or hand-drawn. The information can exist on a passive surface (such as paper) or an active surface (such as a digital display). The text recognition unit (206) generates input data based on the information. For example, the text recognition unit (206) recognizes the words presented on the surface and uses a character recognition method to generate a text string indicating the words presented on the surface as input data. In an example, the input data is in text or string format. The text recognition unit (206) sends the input data to the natural language processing unit (208).

[0091] The natural language processing unit (208) receives the input data and determines the context of the input data. In an example, when the input information contains words related to a single topic (such as, "pollution"), the natural language processing unit (208) determines that the context of the input information may be a note or article about the "pollution" topic. The natural language processing unit (208) generates one or more pieces of auxiliary information based on the context. The auxiliary information includes the type of the auxiliary information and the content of the auxiliary information. The natural language processing unit (208) determines the type and content of the auxiliary information based on the context of the input data. The type of the auxiliary information includes text, audio, video, image, and animation. Those of ordinary skill in the art will understand that the auxiliary information is not limited to the above types and may also include other data formats. The content of the auxiliary information includes, but is not limited to, context-related information, interpretation of information, text correction, text prediction, grammar errors, syntactic errors, similarity indication, text projection, and additional information related to the context. For example, in the above example, the natural language processing unit (208) determines the type of the auxiliary information as text and determines the content of the auxiliary information as the dictionary meaning of the word "pollution". The natural language processing unit (208) sends the auxiliary information to the output unit (212).

[0092] The positioning unit (210) receives an image from the input unit (204) and determines one or more spatial attributes based on the image. Examples of spatial attributes include the angle between the augmented reality system and the surface, the distance between the augmented reality system and the surface, and obstacles in the field of view between the augmented reality system and the surface. The positioning unit (210) generates positioning information based on the spatial attributes. The positioning information indicates the position, plane, and style for displaying the auxiliary information.

[0093] In an exemplary embodiment, the positioning unit (210) compares the angle with a predetermined threshold angle. If the angle is less than the threshold angle, the positioning unit (210) sets a two-dimensional plane. If the angle is not less than the threshold angle, the positioning unit (210) sets a three-dimensional plane. For example, when the positioning unit (210) sets a three-dimensional plane, the auxiliary information is displayed in a three-dimensional (3D) manner, and when the positioning unit (210) sets a two-dimensional plane, the auxiliary information is displayed in a two-dimensional (2D) manner.

[0094] The positioning unit (210) determines the position based on the distance, angle, and plane. The positioning unit (210) determines the style based on at least one of the following: the auxiliary information, the distance, the angle, the background of the information, and a predefined style preference. The style includes the size, font, and visual effects of the auxiliary information. Thereafter, the positioning unit (210) generates positioning information indicating the plane, position, and style. The positioning unit (210) sends the positioning information to the output unit (212).

[0095] The output unit (212) receives the positioning information from the positioning unit (210) and the auxiliary information from the natural language processing unit (208). Based on the positioning information, the output unit (212) displays the auxiliary information to the user. In an exemplary embodiment, the output unit (212) virtually displays the auxiliary information by projecting the information to the user via smart glasses.

[0096] In an embodiment, the surface is a digital display, such as a touch screen display of a user device. The communication unit (214) is configured to communicate wirelessly with the user device. The communication unit (214) receives a user input from the user device and sends the auxiliary information to the user based on the user input. In an example, the user device may communicatively transmit a user input indicating to provide further information related to the displayed auxiliary information to the AR system (200). In another example, the user device may communicatively transmit a user input indicating to turn on or off the display of the auxiliary information to the AR system (200).

[0097] The user interaction unit (216) detects one or more gestures made by the user as an interaction with the auxiliary information or with the information presented on the surface. The user interaction unit (216) updates the auxiliary information based on the detected gesture and displays the updated auxiliary information to the user.

[0098] The handwriting reconstruction unit detects the user's handwriting and reconstructs the user's handwriting. Thereafter, the AR system (200) displays auxiliary information in the user's handwriting.

[0099] The AR system (200) operates in real time. The input unit (204) captures images in real time. The positioning unit (210) dynamically updates the positioning information based on the real-time images. Accordingly, the auxiliary information is dynamically positioned. In an example, the position of the auxiliary information is synchronized with the movement or change of the user's position or perspective.

[0100] Now referring to Figure 3 , a schematic block diagram of an AR server (304) according to another embodiment of the present disclosure is shown. The AR server (304) communicates with a user device (302). The user device (302) includes an input unit (306), an output unit (308), a communication unit (310), and a processing unit (312). The AR server (304) includes a natural language processing unit (312), a user interaction unit (314), a text recognition unit (316), a positioning unit (318), an input / output unit (320), a processing unit (322), and a handwriting reconstruction unit (324).

[0101] The AR server (304) communicates with the user device (302) via a wired, wireless, or cellular communication network (such as but not limited to Wi-Fi, Bluetooth, and Long Term Evolution (LTE)). Examples of the user device (302) include smart glasses. The user device (302) can be used when the user writes information on a surface.

[0102] The input unit (306) of the user device includes a sensor for dynamically capturing an image of the surface in real time. In an example, the input unit (306) is a camera connected to the user device (302). Optionally, the input unit (306) can capture a series of images or real-time video. The user device (302) sends the image to the AR server (304) via the communication unit (310).

[0103] The AR server (304) receives the image via the input / output unit (320). The text recognition unit (316) processes the image to identify the surface and the information presented on the surface. The text recognition unit (316) generates input data based on the information. For example, the text recognition unit (316) identifies the words presented on the surface and uses a character recognition method to generate a text string indicating the words presented on the surface as input data. In an example, the input data is in text or string format. The text recognition unit (316) sends the input data to the natural language processing unit (312).

[0104] The natural language processing unit (312) receives input data and determines the context of the input data. The natural language processing unit (312) generates one or more pieces of auxiliary information based on the context. The auxiliary information includes the type of the auxiliary information and the content of the auxiliary information. The natural language processing unit (312) determines the type and content of the auxiliary information based on the context of the input data. The natural language processing unit (312) sends the auxiliary information to the input / output unit (320).

[0105] The positioning unit (318) determines spatial attributes based on an image. The positioning unit (318) generates positioning information based on the spatial attributes. The positioning information indicates the position, plane, and style for displaying the auxiliary information. In an embodiment, the positioning unit (318) is structurally and functionally similar to the positioning unit (210).

[0106] The input / output unit (320) receives the positioning information from the positioning unit (318) and the auxiliary information from the natural language processing unit (312). The input / output unit (320) sends the positioning information and the auxiliary information to the user device (302).

[0107] The user device (302) receives the positioning information and the auxiliary information. The user device (302) displays the auxiliary information to the user through the output unit (308) based on the received positioning information.

[0108] Now referring to Figure 4 , a schematic block diagram of the natural language processing units (208, 312) according to an embodiment of the present disclosure is shown. The natural language processing units (208, 312) include a memory (402), a language detection unit (404), a context detection unit (406), a database updater unit (408), an error detection and correction unit (410), a word prediction unit (412), and a word recommendation unit (414). The memory (402) stores a dictionary (416) and grammar rules (418).

[0109] The language detection unit (404) receives input data and detects the language of the input data. In an example, the language detection unit (404) determines the vocabulary of the input data and the language characteristics of the input data, compares them with the dictionary (416) and grammar rules (418) stored in the memory (402), and determines the language based on the comparison.

[0110] The context detection unit (406) determines the context of the input data. Examples of the context of the input data include, but are not limited to, concepts, events, or statements related to the input data. In an example, the context detection unit (406) determines a similarity metric and PoS (part-of-speech) tags to determine the context of the input data. The context detection unit (406) can be used to provide personalized recommendations to the user.

[0111] An error detection and correction unit (410) processes input data to determine errors or mistakes in the input data, such as grammar errors, punctuation errors, style errors, and spelling errors. Thereafter, the error detection and correction unit (410) determines one or more corrections for the identified errors or mistakes. In an example, the error detection and correction unit (410) also provides a user with virtual margins or virtual gridlines for providing writing assistance. In an example, the assistance information is in the form of virtual margins or virtual gridlines.

[0112] A word prediction unit (412) predicts the word that the user is writing on a surface. In an example, the word prediction unit (412) predicts the word based on the user's writing history and / or the context of the input data.

[0113] A word recommendation unit (414) recommends words based on the input data. The recommended word can be the next word in the sentence that the user is writing. The word recommendation unit (414) can recommend words based on the user's writing history and / or the context of the input data.

[0114] A database updater unit (408) updates a dictionary (416) stored in a memory (402). In an example, the database updater unit (408) updates the dictionary (416) by querying a remote server for vocabulary updates. In an alternative example, the remote server pushes vocabulary updates to the database updater unit (408). The database updater unit (408) can also update the dictionary (416) with user-defined words.

[0115] In an embodiment of the present disclosure, a natural language processing unit (208) provides the following corrections: spelling correction, grammar correction, and style correction. For spelling correction, the natural language processing unit (208) searches for the extracted word in the dictionary (416) and accordingly highlights the spelling error and the correct spelling through an output unit (212). For grammar correction, the natural language processing unit (208) checks whether the extracted complete sentence follows grammar rules (418). If the natural language processing unit (208) determines that there are any grammar errors in the input data, the errors and corrections are highlighted through the output unit (212). For style correction, the natural language processing unit (208) checks for style errors present in the input data. Examples of style errors include switching between tenses in a sentence, equal sentence lengths (monotony), etc. The natural language processing unit (208) also detects whether the user needs assistance in determining outer margins or gridlines to align words more with each other.

[0116] In another embodiment of the present disclosure, the natural language processing unit (208) performs word prediction. In word prediction, the natural language processing unit (208) detects incomplete words and predicts the complete words that the user is writing. Here, known words are stored in a database within the memory (402) in the form of a tree data structure (e.g., a search tree, an ordered tree data structure for storing dynamic sets or associative arrays where the keys are typically strings). The position of a node in the tree defines the corresponding keyword. When the user writes a character, words corresponding to the letter are searched for in the database. Since there may be multiple words corresponding to the written letter, the context of the input data that has been written is used to reduce the search space and filter out words that are not relevant to the user. Sometimes, the user may spell a word incorrectly, which may result in an incorrect word prediction or even no prediction at all. In such cases, fuzzy matching / search is used. Fuzzy matching / search works with a match that may be less than 100% perfect when a correspondence is found between a portion of the input data and an entry in the previously translated database. A threshold percentage value is set, and when an exact match is not found, the natural language processing unit (208) searches for matches that exceed the threshold percentage value.

[0117] In another embodiment of the present disclosure, the natural language processing unit (208) provides word recommendations to the user to intelligently recommend the next word that can be written in a sentence. Here, words are recommended based on the context of the input data determined by the context detection unit (406). Based on the context of the input data, the natural language processing unit (208) searches the database for relevant words that follow the grammar of the language. The natural language processing unit (208) may use models such as an N-Gram model, a Unigram model / finite state automaton (FSA), or a neural language model to provide word recommendations.

[0118] The N-Gram model attempts to guess the next word in a sentence based on the (n - 1) previous words in the sentence. This model guesses the probability of a given word without any context and the probability of a word given the most recent (n - 1) words. Bigram and trigram models represent the n-gram models with n = 2 and n = 3 respectively.

[0119]

[0120] The Unigram model / finite state automaton (FSA) is a special case of the N-Gram model, where n = 1. The unigram model used in information retrieval can be considered a combination of several FSAs. Here, the probability of each word depends on the probability of that word by itself in the document.

[0121]

[0122] Neural language models are also known as continuous space language models. Neural language models use continuous representations or embeddings of words to make predictions using neural networks. The neural network represents words in a distributed manner as a non-linear combination of weights in the neural network and is trained to predict the lexical probability distribution given some language context. The neural network architecture can be feed-forward or recursive.

[0123]

[0124] Now referring to Figure 5 , a schematic block diagram of a positioning unit (210, 318) according to an embodiment of the present disclosure is shown. The positioning unit (210, 318) includes a tracking unit (502), a local map construction unit (504), a loop detection unit (506), a place recognition unit (510), and a map (512). The map (512) is stored in a memory (not shown). In an exemplary embodiment, the positioning unit (210, 318) is a Simultaneous Localization and Mapping (SLAM) module. The tracking unit (502) includes a feature extraction unit (514), a writing surface recognition unit (516), a pose prediction or relocalization unit (518), a viewing angle calculation unit (520), a viewing distance calculation unit (522), a local map tracking unit (524), and a new key frame determination unit (528). The local map construction unit (504) includes a key frame insertion unit (530), a map point culling unit (532), a new point creation unit (534), and a local Bundle Adjustment (BA) unit (536). The map (512) includes one or more key frames (552), one or more map points (554), a covisibility graph (556), and a spanning tree (558).

[0125] In an embodiment, the positioning unit (210, 318) has three main parallel threads: a tracking thread, a local map construction thread, and a loop closure thread.

[0126] The tracking thread is executed in the tracking unit (502). The tracking unit (502) utilizes the camera of the positioning input unit (204) for each frame. It detects the writing surface, continuously tracks the distance and angle between the device and the writing surface, and determines the insertion of a new key frame. The position recognition unit (510) is used to perform global relocalization of the camera and the writing surface in case of loss of tracking due to some sudden movement. After an initial estimate of the camera pose and feature matching, the covisibility graph (556) of the key frames (552) is used to retrieve the local visible map. Then, a match with the local map points (554) is searched through reprojection to optimize the camera pose.

[0127] The feature extraction unit (514) extracts FAST corners (1000 - 2000) from each captured frame according to its resolution. As the resolution increases, more corners are required. In the example, for an image with a resolution of 640×480, it is suitable to extract 1000 corners.

[0128] The local map tracking unit (524) tracks the local map after estimating the camera pose and the set of initial feature matches. The local map contains the set of key frames that share map points (554) with the current frame, and the set of neighbors of the key frames in the covisibility graph (556). The local map also has a reference key frame that shares most of the map points (554) with the current frame.

[0129] The new key frame determination unit (528) determines whether the current frame is generated as a new key frame. If the current frame has more than 50 map points and less than 90% of the map points of the tracking reference frame, the current frame is calculated as a new key frame.

[0130] The local map construction thread is executed in the local map construction unit (504). The local map construction unit (504) processes new key frames (552) and optimizes the map points to achieve the best reconstruction in the environment of the camera pose. It attempts to triangulate new points by matching the features in the new key frame with the features of the key frames (552) existing in the covisibility graph (556). Then, it applies a point culling strategy to the extracted map points to retain only high-quality points.

[0131] The key frame insertion unit 530 adds nodes for each key frame in the covisibility graph 556 and updates its boundaries with other nodes according to the map points shared with other key frames 552. Thereafter, the spanning tree 558 is updated by linking the inserted key frame with the key frame having the most common points.

[0132] The map point culling unit (532) ensures that the frame is trackable and not triangulated incorrectly. For a point to be considered trackable, it must be present in a quarter of the frames where it is predicted to be visible. The map point culling unit also detects and removes redundant key frames. If 90% of the map points of a key frame already exist in at least three other key frames, it is considered redundant.

[0133] The new point creation unit (534) creates new map points (554) from the connected key frames in the covisibility graph (556). A match is searched for each unmatched feature in the key frame, and the matches that do not conform to the epipolar constraint are discarded.

[0134] The local BA unit (536) optimizes all the map points seen through the currently processed key frame and the key frames (552) connected to it.

[0135] The loop closure thread is executed in the loop detection unit (506). The loop detection unit searches for loops in each new key frame. If a loop is detected, both sides of the loop are aligned, and duplicate points are fused. The loop is verified using the similarity transformation from the current key frame and the loop key frame.

[0136] Now refer to Figure 6 , a flowchart depicting a method of error correction is shown according to an embodiment of the present disclosure. The following explains with reference to Figure 2 the AR system (200) of Figure 6 the flowchart.

[0137] In step 602, the input unit (204) dynamically and real-time captures an image.

[0138] In step 604, the processing unit (202) processes the image to determine whether the image is clear.

[0139] If in step 604 the processing unit (202) determines that the image is clear, then step 606 is executed.

[0140] In step 606, the text recognition unit (206) extracts the handwriting information of the input data from the image.

[0141] If in step 604 the processing unit (202) determines that the image is not clear, then step 602 is executed.

[0142] In step 608, the natural language processing unit (208) determines whether there are any errors in the input data.

[0143] If in step 608 the text natural language processing unit (208) determines that there are errors in the input data, then step 610 is executed.

[0144] In step 610, the output unit (212) highlights the errors in the image. The highlighting is virtually displayed to the user on the image.

[0145] In step 612, the output unit (212) displays the error correction of the input data. The correction is virtually displayed to the user on the image.

[0146] Now refer to Figure 7 , a flowchart depicting a method of word prediction is shown according to an embodiment of the present disclosure. The following explains with reference to Figure 2 the AR system (200) of Figure 7 the flowchart.

[0147] In step 702, the input unit (204) dynamically and real-time captures an image.

[0148] In step 704, the processing unit (202) processes the image to determine whether the image is clear.

[0149] If in step 704 the processing unit (202) determines that the image is clear, then step 706 is executed.

[0150] In step 706, the text recognition unit (206) extracts the handwriting information of the input data from the image.

[0151] If in step 704 the processing unit (202) determines that the image is not clear, then step 702 is executed.

[0152] In step 708, the natural language processing unit (208) determines whether there are any incomplete words in the input data.

[0153] If in step 708 the natural language processing unit (208) determines that there are incomplete words in the input data, then step 710 is executed.

[0154] In step 710, the natural language processing unit (208) predicts the incomplete words.

[0155] In step 712, the output unit (212) virtually displays the word prediction to the user on the image.

[0156] Now refer to Figure 8 , a flowchart depicting a method of word recommendation according to an embodiment of the present disclosure is shown. The following refers to Figure 2 of the AR system (200) to explain Figure 8 the flowchart.

[0157] In step 802, the input unit (204) dynamically and real-time captures an image.

[0158] In step 804, the processing unit (202) processes the image to determine whether the image is clear.

[0159] If in step 804 the processing unit (202) determines that the image is clear, then step 806 is executed.

[0160] In step 806, the text recognition unit (206) extracts the handwriting information of the input data from the image.

[0161] If in step 804 the processing unit (202) determines that the image is not clear, then step 802 is executed.

[0162] In step 808, the natural language processing unit (208) determines whether there are any incomplete sentences in the input data.

[0163] If in step 808 the natural language processing unit (208) determines that there are incomplete sentences in the input data, then step 810 is executed.

[0164] In step 810, the natural language processing unit (208) determines word recommendations for an incomplete sentence.

[0165] In step 812, the output unit (212) virtually displays the word recommendations to the user on the image.

[0166] Now refer to Figure 9 , a flowchart depicting an AR method is shown according to an embodiment of the present disclosure. The following refers to Figure 2 the AR system (200) of Figure 9 to explain the flowchart.

[0167] In step 902, the input unit (204) captures an image.

[0168] In step 904, the text recognition unit (206) recognizes information on the surface depicted in the image.

[0169] In step 906, the text recognition unit (206) generates input data based on the information.

[0170] In step 908, the natural language processing unit (208) determines the context of the input data.

[0171] In step 910, the natural language processing unit (208) generates at least one piece of auxiliary information based on the context.

[0172] In step 912, the positioning unit (210) determines one or more spatial attributes based on the image.

[0173] In step 914, the positioning unit (210) generates positioning information based on the spatial attributes.

[0174] In step 916, the output unit (212) displays the auxiliary information based on the positioning information.

[0175] Now refer to Figure 10 , the extraction of map points is depicted according to an embodiment of the present disclosure.

[0176] The positioning unit (210) extracts features from an image in the user's field of view. These features are called map points (554) and are used to track the camera pose. After an initial pose estimate, a constant velocity model is used to predict the camera pose and perform a guided search for the map points (554) that the user is observing.

[0177] Now refer to Figure 11 , the techniques of error highlighting and correction display are depicted according to an embodiment of the present disclosure.

[0178] The auxiliary information can be displayed in many ways, such as error detection, error correction, word prediction, or word recommendation. The user can be notified of errors in various ways, for example, underlining the error, changing the color of the error, striking out the error, circling the error, and notifying the error by voice. The display of error correction, word prediction, and word recommendation can be by overlaying the auxiliary information on the writing surface, non-overlay display of the auxiliary information, notifying the user by voice, and displaying on other user devices (such as smart phones, wearable devices, etc.).

[0179] Now refer to Figures 12A to 12C , according to an embodiment of the present disclosure, depicts changing the style of the auxiliary information according to the viewing angle.

[0180] In the example depicted in FIGS. 12A to Figure 12C , the user (1202) has written the information "There is a Mestake" on the surface (1204). This information contains a spelling error, that is, the user (1202) misspelled "Mistake" as "Mestake".

[0181] The AR system (200) captures an image of the information written by the user (1202) through the input unit (204). The text recognition unit (206) processes the image to recognize the text written by the user (1202). The natural language processing unit (208) recognizes the context of the text written by the user (1202) and generates appropriate auxiliary information (1206). In this example, the auxiliary information is in the form of error correction, that is, "Mistake". The positioning unit (210) determines the angle formed by the user (1202) and the surface (1204). The positioning unit (210) compares the angle with a threshold angle and determines the positioning information of the auxiliary information (1206). The auxiliary information (1206) is displayed according to the position and orientation calculated by the positioning unit (210). The positioning unit (210) continuously monitors the field of view of the user (1202) and updates the position and orientation of the displayed auxiliary information (1206) accordingly.

[0182] In Figure 12A , the angle between the user (1202) and the surface (1204) is 60°. The positioning unit (210) determines whether the angle is greater than or less than the threshold angle and accordingly sets the plane for displaying the auxiliary information (1206). Here, the positioning unit (210) sets a 3D plane, and therefore, the auxiliary information (1206) is displayed to the user (1202) in 3D format and in an inclined manner.

[0183] In Figure 12BIn this case, the angle between the user (1202) and the surface (1204) is 30°. Here, the positioning unit (210) sets a 3D plane, and thus, the auxiliary information (1206) is displayed to the user (1202) in 3D format and in an upright manner.

[0184] In Figure 12C this case, the angle between the user (1202) and the surface (1204) is 90°. The positioning unit (210) determines whether the angle is greater than or less than a threshold angle, and thus sets a plane (1206) for displaying the auxiliary information. Here, the positioning unit (210) sets a 2D plane, and thus, the auxiliary information (1206) is displayed to the user (1202) in 2D format.

[0185] Now refer to Figure 13 which depicts a technique for context word prediction according to an embodiment of the present disclosure.

[0186] Here, the AR system (200) detects an incomplete word starting with "conta". The AR system (200) searches a database and infers that the word the user is trying to write is "contamination". The AR system (200) also calculates visual aspects such as the size and position of the displayed word prediction. Then the predicted word is displayed by the output unit (212) such that the user does not encounter problems when accessing the result prompt.

[0187] Now refer to Figure 14 which depicts a technique for displaying word prediction in 3D format according to an embodiment of the present disclosure.

[0188] The AR system (200) can also utilize the z-axis or a 3D plane to display word prediction.

[0189] Now refer to Figure 15 which depicts a technique for highlighting different errors and displaying corresponding corrections in the user's writing style according to an embodiment of the present disclosure.

[0190] Now refer to Figure 16 which depicts a technique for similarity check according to an embodiment of the present disclosure.

[0191] The AR system (200) extracts input data from an image of the user's field of view. The input data is compared with information available on the Internet and other available information sources to obtain similarity. The similarity between the text written by the user and the available information is displayed to the user in the form of a similarity percentage.

[0192] Now refer to Figure 17 which depicts a technique for checking for duplicate information according to an embodiment of the present disclosure.

[0193] The AR system (200) detects one or more repeated words / sentences in the user's input data and highlights the repeated information to the user.

[0194] Now referring to Figure 18 ,techniques for providing writing assistance to a user via virtual gridlines and margins are depicted according to an embodiment of the present disclosure.

[0195] The AR system (200) captures an image of information written by a user (not shown) via an input unit (204). A text recognition unit (206) processes the image to recognize the text written by the user. A natural language processing unit (208) determines that the user is practicing writing English letters. The natural language processing unit (208) determines that the assistance information to be displayed to the user should be in the form of gridlines (such as virtual boundaries or virtual margins) to assist the user in writing the letters.

[0196] The natural language processing unit (208) also determines an error in the letters written by the user. Here, the natural language processing unit (208) determines that the letter "E" written by the user is misaligned or crosses a virtual boundary.

[0197] The natural language processing unit (208) also predicts the next letter in a series of letters, i.e., the letter "H". The predicted letter is displayed to the user as assistance information. Here, the AR system (200) virtually projects the letter "H". The user can trace the projected letter "H" to write on a surface.

[0198] Now referring to Figure 19 ,techniques for displaying a readability metric to a user are depicted according to an embodiment of the present disclosure.

[0199] The AR system (200) captures an image of information written by a user (not shown) via an input unit (204). A text recognition unit (206) processes the image to recognize the text written by the user. A processing unit (202) determines the readability degree of the user's handwritten text and generates assistance information indicating the readability percentage of the user's handwriting. The assistance information is displayed to the user in the form of a readability percentage.

[0200] In addition, the processing unit (202) determines whether the written word is legible. The AR system (200) highlights the illegible words.

[0201] Now referring to Figure 20 and Figure 21 ,techniques for simultaneously displaying word recommendations and error correction to a user are depicted according to an embodiment of the present disclosure.

[0202] In Figure 21Among them, the surface for displaying information is a digital screen of a user device (not shown), such as a smart phone screen. The AR system (200) communicates with the user device. The AR system (200) projects interactive auxiliary information on the digital screen of the user device. The user device receives user input from the user and sends the user input to the AR system (200). The AR system (200) receives the user input from the user in the device.

[0203] As Figure 21 depicted in, the user input indicates whether the user wishes to automatically correct an error. If the user input is affirmative, the AR system (200) automatically corrects the error. If the user input is negative, the AR system (200) does not correct the error in the text typed or written by the user. Therefore, the AR system (200) dynamically modifies the auxiliary information based on the user input in response to the displayed interactive auxiliary information.

[0204] Now refer to Figures 22A to 22B , a technique for displaying 3D auxiliary information is depicted according to an embodiment of the present disclosure.

[0205] The AR system (200) determines the context of the text being written by the user. In an example, the context of the text is specifically determined according to the sentence or word that the user is writing. Therefore, the auxiliary content is dynamically determined according to the word or sentence written by the user.

[0206] In Figure 22A the example depicted in, the user is writing information about a football game and the dimensions of a football field. Therefore, the AR system (200) determines the context as "football field". Therefore, the AR system (200) presents a visual schematic with a 3D model of a football field (with players on the football field). The 3D football field depicts details such as the dimensions of the football field (100 yards × 130 yards) and corner kicks.

[0207] In Figure 22B another example depicted in, the user is writing an article about natural disasters. When the user writes the word "hurricane", the AR system (200) displays a 3D hurricane with a high speed such as 119 km / h, where the 3D hurricane is a natural disaster. When the user writes "earthquakes", the AR system (200) displays an earthquake caused by lava and gas emissions ejected from a volcano, and also displays an image of the earth's crust that causes the earthquake.

[0208] As Figures 22A to 22B depicted in, the 3D auxiliary is accompanied by text explanatory words such as the speed of the hurricane, lava, gas emissions, the earth's crust, etc. This helps the user's quick and easy understanding.

[0209] Now refer to Figures 23A to 23B, embodiments in accordance with the present disclosure depict techniques for displaying interactive assistive information.

[0210] When the user starts writing, the AR system (200) determines the context of the text written by the user. Here, the AR system (200) determines the context as "soccer" or "FIFA". Thereafter, the AR system (200) fetches data related to soccer from the Internet and presents it to the user. The AR system (200) classifies the assistive information into different categories or sub-topics to make it more convenient for the user. When the user selects a sub-topic, the assistive information related to the sub-topic is displayed. When the user selects "FIFA World Cup 2018", the assistive information related to the World Cup is displayed and corresponding words are recommended to the user.

[0211] It should be noted that the description only shows the principles of the present disclosure. Therefore, it will be understood that those skilled in the art will be able to design various arrangements that, although not explicitly described herein, implement the principles of the present disclosure.

[0212] In addition, all the examples described herein are mainly and explicitly intended for illustrative purposes only to help the reader understand the principles and concepts of the present invention, where the principles and concepts of the present invention are contributed by the inventor for the further development of the art and are to be construed as not limited to these specifically recited examples and circumstances.

[0213] In addition, all statements in this document that describe the principles, aspects, and embodiments of the present invention, as well as its specific examples, are intended to cover their equivalents.

Claims

1. An electronic device in an augmented reality system, the electronic device comprising: An inputter; An outputter; At least one processor; A memory storing instructions which, when executed by the at least one processor, cause the electronic device to perform the following operations: Control the inputter to capture an image, Recognize text information on the surface depicted in the image from the image, Generate input data in text or string format based on the text information; Determine the context of the input data, the context including concepts, events or statements related to the input data, and Generate at least one piece of auxiliary information providing writing assistance based on the context; Determine one or more spatial attributes based on the image, the one or more spatial attributes including the angle between the augmented reality system and the surface depicted in the image, Set the type of plane for the at least one piece of auxiliary information based on the comparison result of the angle with a threshold angle, Generate positioning information for positioning the at least one piece of auxiliary information relative to the surface depicted in the image based on the one or more spatial attributes; And Control the outputter to display the at least one piece of auxiliary information based on the positioning information and the set type of plane.

2. The electronic device according to claim 1, wherein, The one or more spatial attributes further include at least one of the following items: The distance between the augmented reality system and the surface depicted in the image, and Obstacles in the field of view between the augmented reality system and the surface depicted in the image.

3. The electronic device according to claim 2, wherein, When executed by the at least one processor, the instructions further cause the electronic device to perform the following operations: Determine a position based on the distance, the angle and the set type of plane, Determine a style based on at least one of the following: the at least one piece of auxiliary information, the distance, the angle, the background of the text information and predefined style preferences, and Generate the positioning information indicating at least one of the set type of plane, the position and the style.

4. The electronic device according to claim 1, wherein When executed by the at least one processor, the instructions further cause the electronic device to perform the following operations: Determine the type of the at least one piece of auxiliary information based on the context of the input data, Determine the content of the at least one piece of auxiliary information based on the context of the input data, and Generate the at least one piece of auxiliary information including the type and the content.

5. The electronic device according to claim 1, wherein, The augmented reality system includes at least one of a head-mounted device and a handheld device.

6. The electronic device according to claim 1, wherein, The surface depicted in the image includes a non-digital writing surface.

7. The electronic device according to claim 1, wherein, The surface depicted in the image includes a digital display of a user device.

8. The electronic device according to claim 1, wherein, When executed by the at least one processor, the instructions further cause the electronic device to perform the following operations: Detect the handwriting style of the text information, and Reconstruct the handwriting style.

9. The electronic device according to claim 1, wherein, The augmented reality system includes: a user gesture detector configured to: Detect one or more user gestures indicating an interaction with the at least one piece of auxiliary information or with the text information on the surface depicted in the image, and Display updated auxiliary information based on the one or more user gestures.

10. The electronic device according to claim 1, wherein, The text information includes at least one of the following items: handwritten information, printed information, electronically displayed information, and virtual projection information.

11. A method performed by an electronic device in an augmented reality system, the method comprising: Capturing an image; Identifying text information on a surface depicted in the image; Generating input data in text or string format based on the text information; Determining a context of the input data, the context including concepts, events, or statements related to the input data; Generating at least one piece of auxiliary information that provides writing assistance based on the context; Determining one or more spatial attributes based on the image, the one or more spatial attributes including an angle between the augmented reality system and the surface depicted in the image; Setting a type of a plane for the at least one piece of auxiliary information based on a comparison result of the angle with a threshold angle; Generating positioning information for positioning the at least one piece of auxiliary information relative to the surface depicted in the image based on the one or more spatial attributes; and Displaying the at least one piece of auxiliary information based on the positioning information and the set type of the plane.

12. The method according to claim 11, wherein, The spatial attributes further include at least one of the following items: A distance between the augmented reality system and the surface depicted in the image, and An obstacle in a field of view between the augmented reality system and the surface depicted in the image.

13. The method according to claim 11, further comprising: Determining a type of the at least one piece of auxiliary information based on the context of the input data; Determining content of the at least one piece of auxiliary information based on the context of the input data; And Generating the at least one piece of auxiliary information including the type and the content.

14. The method according to claim 11, further comprising: Detecting a handwriting style of the text information; Reconstructing the handwriting style; And Displaying the at least one piece of auxiliary information in the form of the handwriting style.

15. The method according to claim 11, wherein The text information includes at least one of the following items: handwritten information, printed information, electronically displayed information, and virtual projection information.

Images