Smart glasses and navigation method using same

Abstract

The navigation method using smart glasses, according to an embodiment of the present invention, comprises the operations of: communicatively connecting to an external terminal device on which a navigation application is running; transmitting a request for capturing a screen displayed on the external terminal device; extracting direction indication information from the screen that has been captured in response to the request; and converting the direction indication information into a format suitable for the smart glasses. In addition, the smart glasses display the converted direction indication information at a location selected depending on a reality view. As such, the smart glasses, without having a separate GPS mounted thereon, can effectively utilize a navigation function performed by the external terminal device, simply by interworking therewith.

Description

Smart glasses and navigation method using the same

[0001] The present invention relates to smart glasses and a navigation method using the same, and more specifically, to smart glasses that are linked with a navigation application running on an external terminal device and a navigation method using the same.

[0002] Smart glasses are configured to allow users to view image information by linking with external devices or by projecting image information received through other communication networks onto the user's wearing location.

[0003] Wearable devices like smart glasses typically don't incorporate GPS due to their lightweight design. Therefore, to perform navigation functions, smart glasses often interface with external devices to receive and output location and navigation information.

[0004] However, since smart glasses are worn by users like eyeglasses, it's difficult to directly apply information displayed on other external devices to smart glasses. Specifically, smart glasses must display AR images on top of the real-world view, and information like navigation must be provided in the appropriate location and format, taking into account the wearer's movements. Furthermore, compatibility between smart glasses and other external devices must be considered.

[0005] In this regard, Korean Patent No. 10-1873805 (hereinafter referred to as "Prior Document 1") discloses a technology in which smart glasses receive smart information from a smart device such as a watch or terminal and then convert it into an image. However, because Prior Document 1 transmits the smart information from the smart device directly to the smart glasses, the smart glasses must then convert it back into an image, and compatibility is limited depending on the conditions for conversion.

[0006] In addition, Korean Patent No. 10-2150074 (hereinafter referred to as "Prior Document 2") discloses a technology for providing POI information of a point of interest (POI) based on a point of interest input from a handheld terminal, and controlling the movement of navigation arrow information on the handheld terminal using a motion sensor accelerometer of the AR glasses. However, in Prior Document 2, there is a limitation in that the AR glasses do not have a navigation function and are used only as a device that performs an auxiliary function of the handheld terminal.

[0007] Accordingly, the present invention aims to solve the above-mentioned problems and other problems.

[0008] According to some embodiments of the present invention, the purpose is to provide smart glasses and a navigation method using the smart glasses that can perform a navigation function by only linking with an external terminal device without wearing GPS on the smart glasses.

[0009] In addition, according to some embodiments of the present invention, the purpose is to provide smart glasses and a navigation method using the smart glasses that can provide intuitive navigation information without generating separate navigation information in the smart glasses or performing image conversion work compatible with a linked external terminal device.

[0010] In addition, according to some embodiments of the present invention, the purpose is to provide smart glasses and a navigation method using the smart glasses, which can effectively extract navigation information from an external terminal device linked with the smart glasses and output the same in an appropriate form at an appropriate location on the smart glasses.

[0011] To this end, a navigation method using smart glasses according to an embodiment of the present invention discloses technologies for converting direction indication information provided from a navigation application running on an external terminal device into a form suitable and intuitive for smart glasses and providing the information through smart glasses.

[0012] Specifically, a navigation method using smart glasses according to an embodiment of the present invention may be implemented by including the following steps. The navigation method may include: a step of connecting the smart glasses to an external terminal device on which a navigation application is executed so as to be able to communicate; a step of transmitting a request to capture a screen displayed on the external terminal device and extracting direction indication information (TBT) from the captured screen executed according to the request; a step of converting the extracted direction indication information into a format suitable for the smart glasses; and a step of displaying the direction indication information at a selected location according to a reality view of the smart glasses.

[0013] According to an embodiment, the step of extracting the direction indication information (TBT) may be a step of capturing a screen displayed on an external terminal device frame by frame upon request, and extracting information of the same area including the direction indication information from the captured screen.

[0014] According to an embodiment, the navigation method may include a step of detecting movement according to the direction indication information based on vision recognition of the smart glasses; and a step of requesting an update of the next direction indication information from the external terminal device in response to the detection.

[0015] According to an embodiment, the step of converting into a format suitable for the smart glasses may further include the step of chroma-key processing a portion of the same region excluding direction indication information and information related thereto; and the step of removing the chroma-key processed portion and receiving information converted into the suitable format in the smart glasses.

[0016] According to an embodiment, the step of rendering at the selected location includes the step of rendering a first AR graphic object corresponding to the converted direction indication information at the selected location according to the reality view of the smart glasses, wherein at least one of the selected location and the first AR graphic object may vary depending on the route guidance mode for each means of transportation.

[0017] According to an embodiment, the navigation method may further include a step of rendering a second AR graphic object on the detected road sign information based on the detected road sign information matching the direction indication information in the real view of the smart glasses.

[0018] According to an embodiment, the step of rendering at the selected location may include the step of setting a customized location in the reality view of the smart glasses based on an input; and the step of rendering the direction indication information fixedly at the customized location.

[0019] According to an embodiment, the step of rendering at the selected location may include a step of rendering a graphic object corresponding to the direction indication information while avoiding an object in the reality view of the smart glasses; and a step of changing the graphic object according to a change in the reality view of the smart glasses according to a movement of the wearer's gaze.

[0020] According to an embodiment, the step of rendering at the selected location may include a step of adjusting the transparency of the rendered graphic object based on vision recognition of the smart glasses and analysis of wearer behavior based on the wearer's gaze movement.

[0021] In addition, smart glasses according to an embodiment of the present invention are implemented including a communication unit that communicates to link with a third-party navigation application; a processor that transmits a request to capture a screen of the linked navigation application, receives direction indication information extracted from the captured screen according to the request, and converts it into a format suitable for the smart glasses. At this time, the processor can render an AR image corresponding to the direction indication information at a location selected according to the reality view of the smart glasses.

[0022] According to an embodiment, the smart glasses may further include a vision recognition unit that detects movement based on the direction indication information. In this case, the processor may, in response to detection by the vision recognition unit, transmit an update request for the next direction indication information to the third-party navigation application.

[0023] According to an embodiment, the processor may extract direction information and text included in the direction indication information to generate guidance information, control the generated guidance information to be converted into voice in a selected language, and transmit the voice in the selected language to a speaker to be output.

[0024] The effects of the smart glasses according to the present invention and the navigation method using the same are described as follows.

[0025] According to the smart glasses and the navigation method using the same according to an embodiment of the present invention, the navigation function executed on the external terminal device can be effectively utilized by simply linking with the external terminal device without wearing a separate GPS on the smart glasses.

[0026] Furthermore, according to the smart glasses and navigation method using the same according to an embodiment of the present invention, intuitive navigation information can be provided without generating separate navigation information in the smart glasses or converting images to be compatible with external terminal devices linked to them. In particular, navigation information can be extracted in a quick and efficient manner without performing machine learning on the external terminal device linked to the smart glasses, and the information converted to be suitable for the smart glasses can be directly provided.

[0027] In addition, according to the smart glasses and the navigation method using the same according to an embodiment of the present invention, when a user wearing the smart glasses drives and uses a navigation function, the user can concentrate on driving without having to move his / her gaze, and can expand the user experience of the navigation function through the lightweight smart glasses while continuing to use the navigation application of the terminal device that he / she is accustomed to using.

[0028] FIG. 1 is a block diagram for explaining detailed configurations related to the operation of the present invention in smart glasses and an external terminal device linked thereto according to an embodiment of the present invention.

[0029] FIG. 2 is an exemplary flowchart for explaining a navigation method using smart glasses according to an embodiment of the present invention.

[0030] FIG. 3 is a drawing specifically explaining extracting navigation information from an external terminal device and displaying it on smart glasses according to an embodiment of the present invention.

[0031] FIG. 4 is an exemplary drawing for explaining a method of extracting direction indication information from an external terminal device according to an embodiment of the present invention.

[0032] FIG. 5 is a drawing for explaining changing the display of navigation information on smart glasses according to a route guidance mode, according to an embodiment of the present invention.

[0033] FIG. 6 is a drawing for explaining displaying an AR image on road sign information matching navigation information according to an embodiment of the present invention.

[0034] FIG. 7 is a drawing for explaining displaying navigation information in a customized location of user settings according to an embodiment of the present invention.

[0035] FIG. 8 is a drawing for explaining a method of changing the display of navigation information according to the head movement of a wearer according to an embodiment of the present invention.

[0036] FIG. 9 is a diagram for explaining a method of changing the display of navigation information based on analysis of a wearer's behavior, according to an embodiment of the present invention.

[0037] FIG. 10 is a drawing for explaining a method of providing voice guidance for navigation information displayed on smart glasses according to a set language, according to an embodiment of the present invention.

[0038] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0039] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0040] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0041] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0042] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0043] Meanwhile, the "smart glasses" disclosed herein are configured to allow the user to view image information by linking with external devices or projecting image information received through other communication networks onto the user's wearing position. Furthermore, the "smart glasses" can render AR images onto the real-world view, allowing the user to experience the received image information in the form of augmented reality (AR) or mixed reality (MR).

[0044] The "external terminal device" disclosed herein can communicate with the smart glasses and may include various types of fixed terminals or mobile terminals capable of executing navigation applications. For example, the external terminal device may include a smartphone, a smart watch, a navigator installed in a vehicle, etc.

[0045] Embodiments of the present invention disclose a method for effectively performing a navigation function by simply linking with an external terminal device without wearing a separate GPS on smart glasses.

[0046] FIG. 1 is a block diagram for explaining the detailed configuration related to the operation of the present invention in a smart glass (100) and an external terminal device (200) linked thereto according to an embodiment of the present invention.

[0047] Referring to FIG. 1, smart glasses (100) are linked to an external terminal device (200) and are capable of mutual communication. The smart glasses (100) can receive navigation information, including turn-by-turn (TBT) information, from the linked external terminal device (200) in a format suitable for output. In addition, the smart glasses (100) can transmit navigation information transmission requests and update requests to the linked external terminal device (200).

[0048] Smart glasses (100) may include AR connectivity services (130), AR applications (110), and AR navigation (120).

[0049] The AR connectivity service (130) provides a connection function for communicating with an external terminal device (200), and supports, for example, a wireless communication network.

[0050] To this end, the AR connectivity service (130) may include modules such as a Wi-Fi connector, a connection manager, a BLE server, Wi-Fi (Direct), and Bluetooth. However, the present invention is not limited thereto, and may include various wireless communication networks that can be implemented in embodiments of the present invention.

[0051] Here, the BLE server transmits the BLE name and UUID (unique address) to the external terminal device (200). The BLE client, i.e., the external terminal device (200), can connect using this information (i.e., the BLE name and UUID (unique address)), and to prevent other BLE clients from connecting thereafter, the BLE name, etc. are no longer transmitted externally, and a list of services and characteristic information of the corresponding services are transmitted.

[0052] Smart glasses (100) can receive navigation information, including direction display information (TBT), from an external terminal device (200) using an AR connectivity service (130) and transmit a request signal for updates, etc.

[0053] The AR application (110) and AR navigation (120) provide companion services and AR overlay functions to output navigation information, including received direction indication information, in an appropriate location and format. For example, the smart glasses (100) can render the received direction indication information (TBT) into an AR image suitable for the actual view via the AR application (110) and AR navigation (120).

[0054] In the external terminal device (200), a third-party navigation application that is linked to the smart glasses (100), for example, a navigation application, is executed.

[0055] To this end, FIG. 1 shows that smart glasses (100) are linked to an external terminal device (200) through an AR connected framework (250). In other words, the smart glasses (100) are connected to a navigation application of an external terminal device through the AR connected framework and can receive direction information, etc. from it.

[0056] Meanwhile, in FIG. 1, the AR connected framework (250) is depicted as being mounted within an external terminal device (200), but is not limited thereto. For example, the AR connected framework (250) may also be implemented as a separate service platform that is not mounted within the external terminal device (200) but is connected to and operates with smart glasses (100).

[0057] In other words, the AR connected framework (250) may be implemented in an external terminal device (200) as shown in FIG. 1, in smart glasses (100), or through a separate service platform that communicates with the external terminal device / smart glasses.

[0058] The AR connected framework (250) may include an AR navigation SDK (252) and an AR connected SDK (254). Here, the SDK (Software Development Kit, SDK) generally refers to a tool that enables the creation of applications for a specific software framework, hardware platform, computer system, game console, operating system, etc., and is comprised of data / commands.

[0059] The AR navigation SDK (252) performs the function of capturing the execution screen of the navigation application (210) of the external terminal device (200), extracting direction indication information (TBT), and converting the extracted direction indication information (TBT) into an appropriate format.

[0060] Specifically, the AR navigation SDK (252) is connected to a navigation application (210) that provides TBT information output (211) and a navigation view (212) for communication. The AR navigation SDK (252) provides a function for capturing the execution screen of the navigation application (210), extracting direction indication information (TBT) from the captured screen, and converting the extracted direction indication information to fit the smart glasses (100).

[0061] Here, the function of converting the direction indication information to fit the smart glasses (100) may include, for example, functions of information separation of the extracted direction indication information, image processing, and conversion into an AR object.

[0062] The AR Connect SDK (254) provides a connection function with smart glasses (100). Specifically, the AR Connect SDK (254) may include an AR data transmission unit for transmitting navigation information including converted direction display information (TBT) to the smart glasses, an AR connection manager, and an AR BLE server.

[0063] In embodiments of the present invention, the smart glasses (100) can be linked with a third-party navigation application of an external terminal device (200). The smart glasses (100) transmit a request to capture the execution screen of the linked third-party navigation application, and, in response to the request, receive direction indication information extracted from the captured screen and convert it into a format suitable for the smart glasses.

[0064] Additionally, the smart glasses (100) render an AR image corresponding to the converted direction indication information at a selected location according to the real view. To this end, the smart glasses (100) may include an AR engine or renderer, or may be implemented to generate an AR image through an AR application (110).

[0065] FIG. 2 is an exemplary flowchart for explaining a navigation method using smart glasses according to an embodiment of the present invention.

[0066] Referring to FIG. 2, the smart glasses (100) according to an embodiment of the present invention are connected to an external terminal device (200) in which a navigation application is executed (S10).

[0067] Here, there is no limitation on the type of the navigation application. That is, it is not limited to a navigation application compatible with smart glasses (100).

[0068] Additionally, in some embodiments, when wearing of the smart glasses (100) is detected and a navigation application is executed on the external terminal device (200), the smart glasses (100) may be directly connected to the navigation application of the external terminal device (200).

[0069] In another example, when a navigation application is executed on an external terminal device (200), the smart glasses (100) may ask for connection with the application, and in response to this, the navigation application of the external terminal device (200) may be connected.

[0070] When the smart glasses (100) and the external terminal device (200) are connected in this way, the smart glasses (100) transmit a request to capture the screen displayed on the external terminal device (200), and extracts direction indication information (TBT) from the captured screen generated in response to the request (S20).

[0071] At this time, the request may be transmitted directly to the navigation application of the external terminal device (200) or may be transmitted indirectly to the navigation application through a service platform linked with the navigation application of the external terminal device (200).

[0072] In addition, the generated capture screen may be a screen captured while the navigation execution screen is displayed across the entire display of the external terminal device, or a screen captured while it is overlaid on another screen or displayed together with another application.

[0073] Additionally, extracting the direction display information (TBT) may mean extracting the same area including the direction information (TBT) included in the execution screen of the navigation application.

[0074] Alternatively, extracting the direction display information (TBT) may mean extracting the direction display information (TBT) and other navigation information together. In this case, the other navigation information may be a type of navigation information that is associated with the direction display information (TBT) and is preset as a set value / default value.

[0075] Next, the smart glasses (100) convert the extracted direction indication information (TBT) into a format suitable for the smart glasses (S30). That is, the smart glasses (100) do not receive the extracted direction indication information (TBT) as is, but request that it be converted into a format suitable for output, and then receive the direction indication information (TBT) converted according to the request, thereby avoiding performing additional processing and image conversion processes.

[0076] According to an embodiment, converting into a format suitable for output from smart glasses (100) may include image separation, image chroma key processing, etc., and may further include transmitting the processed information to the smart glasses.

[0077] In this way, converting to a format suitable for output from smart glasses (100) is to minimize obstruction of the field of view due to the characteristics of the glasses and to provide necessary information in a layout that is easy for the user to view.

[0078] Next, the smart glasses (100) display direction information at a selected location according to the reality view (S40). At this time, the direction information may be output in the form of an AR or MR image in the reality view.

[0079] Here, the location selected based on the real-world view may refer to a location that minimizes occlusion of objects detected in the real-world view. Furthermore, the location selected based on the real-world view may include a customized location determined based on behavioral analysis, user information, and configuration information of the user wearing the smart glasses.

[0080] Additionally, the direction indication information provided (output) through smart glasses can be changed or an update request can be generated based on movement information detected through the sensors of the smart glasses.

[0081] For example, if the real view changes according to the position or movement of the head of a user wearing smart glasses or the movement of a vehicle, the AR image corresponding to the direction information may change accordingly or the output location may change.

[0082] Also, for example, when direction indication information is provided from smart glasses and a matching or different direction change is detected through the sensors of the smart glasses (e.g., accelerometer, geomagnetic sensor, etc.), a request for an update of the direction indication information can be transmitted to a linked external terminal device.

[0083] In this way, according to an embodiment of the present invention, navigation information can be provided through the smart glasses being worn without generating separate navigation information in the smart glasses or performing image conversion work compatible with a linked external terminal device.

[0084] FIG. 3 is a drawing showing a specific example corresponding to the flowchart described in FIG. 2 above, according to an embodiment of the present invention.

[0085] Referring to FIG. 3, the smart glasses (100) communicate with a third-party navigation application installed in an external terminal device (200) through a communication unit (130).

[0086] At this time, the communication unit (130) may refer to the AR connectivity service described in FIG. 1, and may include modules such as a Wi-Fi connector, connection manager, BLE server, Wi-Fi (Direct), and Bluetooth for wireless communication.

[0087] The communication unit (130) receives navigation information including direction indication information (TBT) from a linked external terminal device (200). In addition, the communication unit (130) transmits a transmission request and / or an update request of navigation information including direction indication information (TBT) to the linked external terminal device (200).

[0088] On the display of the external terminal device (200), a first navigation screen (310) in which the screen of the navigation application being executed is displayed in full screen may be displayed, or a second navigation screen (320) in which the screen is displayed as a partial screen overlaid on another screen (e.g., a home screen) may be displayed.

[0089] At this time, both the first navigation screen (310) and the second navigation screen (320) include direction display information (TBT) (311, 321). For example, the first navigation screen (310) may include direction display information (TBT) (311) as a part thereof, and the second navigation screen (320) may display direction display information (TBT) (321) in its entirety.

[0090] When the navigation application of the smart glasses (100) and the external terminal device are connected, a request for extraction and transmission of direction indication information (TBT) (311, 321) is transmitted from the smart glasses (100) to the external terminal device (200) through the communication unit (130).

[0091] This request is transmitted to the external terminal device (200) or the navigation application of the external terminal device (200) via the SDK installed in the external terminal device (200). In another embodiment, the request may be transmitted to the navigation application of the external terminal device (200) via a separate service platform (not shown).

[0092] In accordance with the above request, a screen capture (screen capture) operation is performed (330) for the first navigation screen (310) or the second navigation screen (320) displayed on the external terminal device (200). The captured screen may be stored in the external terminal device (200) or a designated storage device, and may be deleted after the navigation information including the direction display information (TBT) (311, 321) is extracted, converted, or transmitted.

[0093] According to the above request, an operation of extracting direction display information (TBT) (311, 321) from the captured screen is performed by an external terminal device (200) or the navigation application of the external terminal device (200) itself or a separate service platform (340).

[0094] Meanwhile, the extraction of direction indicator information (TBT) (311, 321) was accomplished using segmentation techniques rather than machine learning methods for speed and simplicity. A detailed description of the extraction of direction indicator information using segmentation techniques from captured screens will be provided below with reference to Figure 4.

[0095] Continuing, after the extraction operation (340) according to the above request, an operation of converting the extracted information into a format suitable for outputting to smart glasses (100) is performed (350).

[0096] Converting into a format suitable for output on smart glasses (100) involves processing the image so that it is positioned so that it is easily visible and minimizes any obstruction to the view of a user wearing smart glasses (100). Specifically, the conversion operation (350) includes separating the extracted information, performing image processing (chroma keying), and setting the layout.

[0097] Direction indication information (TBT) (355) converted according to the above conversion operation (350) is transmitted to smart glasses (360) (360).

[0098] Meanwhile, the conversion operation (350) is performed before the transmission operation (360) of the direction indication information (TBT) (311, 321) based on the direction indication information (TBT) (311, 321) extracted from the external terminal device (200). Accordingly, since the smart glasses (100) do not need to perform the task of converting this back into a suitable image, compatibility with the navigation application of the external terminal device (200) is not an issue.

[0099] When converted direction display information (TBT) (355) is received through the communication unit (130) of the smart glasses (100), the processor (150) projects / renders it at an appropriate location according to the reality view. To this end, the processor (150) is implemented to include an AR engine / renderer, etc. for rendering an AR image matching the converted direction display information (TBT) (355).

[0100] For example, as illustrated in FIG. 3, for example, the direction indication information after the first distance and the next direction indication information after the first distance are distinguished, and AR direction information (371) laid out vertically can be displayed as an overlay on the real view (370) seen through the glasses. In addition, the AR direction information (371) is displayed in an area that can minimize object occlusion of the real view, for example, in the upper right corner.

[0101] In some embodiments, the smart glasses (100) may transmit a request for transmission / update of the next direction indication information to the external terminal device (200) when the set condition is satisfied.

[0102] Here, the set conditions may include detecting a movement that matches or does not match the currently output direction indication information. In this case, the detection of the movement is not based on GPS location information, but rather is performed through a vision recognition unit, such as a camera (140) of smart glasses (100), so that it can be immediately detected that the user has moved in a direction different from the navigation information.

[0103] According to an embodiment, the smart glasses (100) can detect movement according to a change in the reality view based on an image acquired through a vision recognition unit, for example, a camera (140). In another example, the smart glasses (100) can detect movement that satisfies the above-described conditions through other sensors, for example, an accelerometer, an inclinometer, etc.

[0104] In such cases, the smart glasses (100) can determine whether a movement that matches or does not match the currently output direction indication information has been detected. Based on such determination, the smart glasses (100) can transmit a signal requesting an update to the next direction indication information to an external terminal device (200) or through a separate service platform.

[0105] Below, Figure 4 illustrates exemplary drawings illustrating a method for extracting direction indicator information from an external terminal device using segmentation techniques. While direction indicator information (TBT) can be extracted through machine learning of captured screens, extracting direction indicator information (TBT) using segmentation techniques is faster and simpler.

[0106] Meanwhile, FIG. 4 illustrates the first navigation screen (310, FIG. 3) in which the screen of a running navigation application is displayed on the entire display of an external terminal device (200). However, it goes without saying that the same can be applied to the second navigation screen (320) displayed as a partial screen in FIG. 3.

[0107] Based on a request from the smart glasses (100), the external terminal device (200) captures the screen and extracts direction indication information (TBT) from the captured screen. The process of extracting direction indication information (TBT) is specifically as follows.

[0108] First, the screen (current screen) of the navigation application is captured frame by frame, and then the captured screen is divided into appropriate segments to search for direction display information (TBT) (a).

[0109] Here, the direction indicator information (TBT) refers to information about the next direction of movement from the current location when following a driving route to a set destination. For example, the direction indicator information (TBT) may include one or more of an arrow image indicating the next direction of movement (e.g., an arrow image), a distance to the next direction of movement (movement distance), an image indicating the direction of movement after the next direction of movement (e.g., an arrow image), and destination information.

[0110] Additionally, the number of appropriate segments can be determined in proportion to the size of the captured screen. Furthermore, the number of appropriate segments can vary depending on whether the navigation application's execution screen corresponding to the captured screen is full-screen or partial-screen.

[0111] Next, if the recognizable direction indicator information (TBT) (entire) is not searched within a specific segment, the segment size is expanded, re-segmented, and the process of re-searching the direction indicator information (TBT) is repeated (b).

[0112] Specifically, the captured screen is divided into segments of an appropriate size according to the display size (hereinafter, referred to as "the first set of segments") to search for direction information. If direction information is not searched in the first set of segments, the size of the segments is increased (hereinafter, referred to as "the second set of segments") to search for direction information again. Here, the second set of segments is smaller in size and larger in number than the first number of segments.

[0113] If a recognizable direction indicator (TBT) is found within a specific segment, the found direction indicator (TBT) is recognized and the search is terminated (c). At this time, the found direction indicator (TBT) may be one or more.

[0114] For example, in (c) of FIG. 4, only the present direction indication information (TBT) (411) included in the first segment may be searched. Alternatively, the present direction indication information (TBT) (411) included in the first segment and the next direction indication information (TBT) (420) included in the second segment may be searched together.

[0115] In another embodiment, the number of searched turn-by-turn information (TBT) can be pre-determined based on user settings. In this case, if only one TBT is searched based on user settings, only the current turn-by-turn information (TBT) (411) is searched.

[0116] (c) After this, the direction indication information (TBT) is extracted by expanding the range of the same area based on the searched direction indication information (TBT).

[0117] At this time, the range of the same area is determined based on the color, pattern, etc. of the searched direction indicator information (TBT). For example, in (c) of Fig. 4, the area where the extracted present direction indicator information (TBT) (411) is displayed is expanded to a range that can be viewed as the range of the same area with the same color and area boundary, and the direction indicator information (TBT) (412) is extracted.

[0118] Direction indication information (TBT) (412) includes an arrow image (e.g., an arrow image) indicating the direction to move and the distance to reach that direction of movement (movement distance).

[0119] In some embodiments, the direction indicator information (TBT) (412) may be extended to next direction indicator information (TBT) (420). In such cases, the extracted direction indicator information includes an arrow image (e.g., an arrow image) indicating the direction to move and the distance to the next direction to move (movement distance), as well as an image indicating the next direction to move and the associated distance to move. In addition, although not illustrated, the direction indicator information (TBT) (412) may additionally include destination information.

[0120] The initial start of direction indication information extraction is performed at the request of the smart glasses (100), and subsequent extraction cycles may be performed at set intervals. For example, the set cycle may be 30 frames per second (fps).

[0121] In another embodiment, if there is no change in direction after the direction indication information (TBT), even if the direction indication information is extracted in the set cycle, transmission and update to the smart glasses (100) may not be performed.

[0122] Additionally, in some embodiments, the direction indication information (TBT) may be updated to be extracted and transmitted to the smart glasses (100) only when there is a change in direction after the direction indication information (TBT).

[0123] In addition, according to some embodiments, direction indication information (TBT) may be displayed on the smart glasses (100), and a signal may be generated to request extraction and transmission of new direction indication information (TBT) to an external terminal device (200) based on a change in direction and movement detected through a vision recognition unit (e.g., a camera, etc.) of the smart glasses (100).

[0124] Additionally, according to some embodiments, some data of the direction indicator information (TBT) transmitted to the smart glasses (100), for example, the distance to the direction of movement (movement distance), may be independently updated based on movement detection through the vision recognition unit (e.g., camera, etc.) of the smart glasses (100). This can be updated more accurately and quickly because it reflects the displacement according to the movement based on the initial movement distance value rather than based on GPS location information.

[0125] Meanwhile, if there are multiple pieces of direction indication information extracted through (a) to (d) of FIG. 4, the direction information is excluded from the searched specific segment after separation and chroma-keyed. That is, the background of the direction information is removed. Then, the chroma-keyed portion is removed, and the information converted into a suitable format is transmitted to the smart glasses (100). Thereafter, the transmitted information is projected / output at a designated location or at a location that does not obscure objects in the real view.

[0126] Meanwhile, an SDK (Software Development Kit) such as a software framework or hardware platform that provides a linkage function with smart glasses can be installed within the existing navigation application installed on an external terminal device.

[0127] In this case, since the navigation application's execution screen is captured and the direction information is extracted from the captured screen is performed within the application itself, the navigation application can directly link with smart glasses. This means that the screen capture and direction information extraction processes can be omitted on the external terminal device.

[0128] For example, smart glasses can receive navigation information, including turn-by-turn direction information (TBT), from an external terminal device via a BLE server through a Software Development Kit (SDK) embedded in a specific navigation application installed on the external terminal device. At this time, the navigation information, including turn-by-turn direction information (TBT), is converted into direction information and distance information separately so that it can be displayed on the smart glasses, and appropriate processing is performed via wireless communication technologies such as BLE. Thereafter, the navigation information, including turn-by-turn direction information (TBT), can be rendered onto a real-world view through an AR navigation service platform embedded in the smart glasses, thereby providing the necessary navigation information.

[0129] Below, various embodiments of a method for receiving navigation information including converted direction display information (TBT) and displaying it on smart glasses (100) are described.

[0130] Assuming that the embodiments applied herein are not in conflict, the converted direction indicator information (TBT) is displayed in an area (e.g., the upper right area) where occlusion of an object is minimized in the view of reality seen through smart glasses (100). In addition, the converted direction indicator information (TBT) may be temporarily or for a predetermined period of time stopped from being displayed when occlusion of a designated object occurs.

[0131] FIG. 5 is a drawing showing a method of displaying navigation information on smart glasses by changing the display according to the route guidance mode.

[0132] Figure 6 is a drawing showing a method of displaying an AR image on road sign information matching navigation information.

[0133] And, Fig. 7 is a drawing showing a method of displaying navigation information including direction indication information in a customized location of user settings.

[0134] The smart glasses (100) can render a first AR graphic object corresponding to the converted direction indication information at a location selected according to the reality view. At this time, at least one of the location selected according to the reality view and the first AR graphic object may vary depending on the route guidance mode for each means of transportation set in the smart glasses (100).

[0135] Here, the above-mentioned route guidance mode means an operation mode that changes the display method and / or display location of direction information according to the properties and settings matching the means of transportation (e.g., vehicle, bicycle, walking, etc.) of the user wearing the smart glasses (100).

[0136] Alternatively, the above-described route guidance mode may refer to an operation mode that changes the display method and / or display location of direction information according to the characteristics of the user wearing the smart glasses (100) (e.g., user's age, whether the user is a guest, etc.) and surrounding environment information (e.g., night, day, road congestion, etc.).

[0137] The above-described route guidance mode may include, for example, a vehicle route guidance mode, a bicycle route guidance mode, and a walking route guidance mode. The smart glasses (100) can change and output route guidance information on the smart glasses (100) according to the type of route guidance provided by the navigation system for each means of transportation, such as a car, bicycle, or walking guidance mode.

[0138] In this regard, referring to FIG. 5, the direction indication information (501) projected through smart glasses (501) may have a display method, type (range), and / or location that may be changed depending on whether it is, for example, a vehicle guidance mode (510), a bicycle guidance mode (520), or a walking guidance mode (530).

[0139] For example, in the vehicle guidance mode (510), instrument panel information, vehicle status information (e.g., fuel, etc.), and (in the case of a taxi or shared vehicle) estimated fare information may be displayed as or in conjunction with direction display information (501). For this purpose, the smart glasses (501) may additionally communicate with the system of the vehicle in which the user is riding.

[0140] In addition, for example, in the bicycle route guidance mode (520), the user's calorie information and heart rate information may be displayed as or in conjunction with the direction display information (501). In addition, in the bicycle route guidance mode (520), the UI display position of the direction display information and the like may be adjusted upward. To this end, the smart glasses (501) are additionally implemented to use the sensor values ​​of a sensor provided in the frame, etc., or to receive sensing values ​​from a sensor attached to the handlebar of the bicycle.

[0141] Additionally, for example, in the walking guidance mode (530), the user's step count information, heart rate information, public transportation boarding location information, and related AR highlights may be displayed as or in conjunction with the direction display information (501). To this end, the smart glasses (501) may communicate with an additional external device, such as a handheld device, or receive POI information related to the surrounding map information through the smart glasses (501).

[0142] Continuing with reference to FIG. 6, the smart glasses (100) can display / render an AR image on the information of the detected road sign in response to the detection of a road sign that matches navigation information including direction information.

[0143] Specifically, the smart glasses (100) can display a first AR graphic object (e.g., 'arrow image and distance information') corresponding to the converted direction indication information at a location selected according to the reality view (e.g., upper right area), and can render a second AR graphic object for a highlight effect (e.g., 'overlapping a highlight image over the matching text of a road sign') on a road sign matching the converted direction indication information.

[0144] For example, in FIG. 6, direction indication information (601) extracted from a navigation application of an external terminal device (200) is received by smart glasses (100), and projected as a first AR graphic object (602) (e.g., “Osan IC left 30m”) on a certain area of ​​a view of reality seen through the user’s two eyes, and a second AR graphic object (603) can be displayed as a highlight on the text of an actual road sign (e.g., a milestone) matching this.

[0145] Accordingly, a driver can receive more clear and diverse route guidance information through smart glasses (100). In particular, when a user wearing smart glasses (100) is driving on a new route, it helps prevent them from taking the wrong route.

[0146] Continuing, the smart glasses (100) can display navigation information including direction information fixedly at a user-specified location in the view of reality.

[0147] That is, the smart glasses (100) can display virtual navigation information at a location desired by the user based on user settings.

[0148] For example, referring to FIG. 7, the wearer (10) of the smart glasses (100) can set an area (702) of the windshield (50) inside the vehicle as a fixed display location for navigation information. Alternatively, in FIG. 7, the wearer (10) of the smart glasses (100) can set a location (701) where navigation is typically placed (i.e., this may be the case where there is no navigation installed in the vehicle) as a fixed display location for navigation information.

[0149] To this end, the smart glasses (100) can track the set fixed display location based on the feature points of the image for the set fixed display location and can project navigation information fixedly on the corresponding fixed display location.

[0150] Accordingly, navigation information can be expected to be output at a fixed display location even when the wearer's (10) head or gaze shifts, as shown in FIG. 7. This can be particularly useful when a separate navigation system is not installed within the vehicle. Furthermore, virtual navigation can be displayed at a customized location desired by the wearer (10) of the smart glasses (100) anywhere within the vehicle.

[0151] Below, we will look at examples of methods for changing the display of navigation information based on the wearer's head movement or based on behavioral analysis estimated through the wearer's gaze movement, etc.

[0152] First, FIG. 8 relates to a method for changing the display of navigation information according to the head movement of the wearer of smart glasses (100).

[0153] Smart glasses (100) perform a navigation function by displaying / rendering navigation information including received direction information in the form of an AR graphic object while avoiding objects in the real view.

[0154] Meanwhile, the reality view seen through the smart glasses (100) changes depending on the head movement of the wearer of the smart glasses (100), and in response to this change in the reality view, navigation information can be changed and displayed according to the head movement of the wearer.

[0155] To this end, the smart glasses (100) can change and display navigation information based on the sensing value of a sensor (e.g., an accelerometer) that detects the wearer's head movement and an image acquired through a vision recognition unit (e.g., a camera) that recognizes a real view.

[0156] For example, in FIG. 8, when navigation information (801) including direction indication information is projected through smart glasses (100), if the wearer of the smart glasses (100) moves his / her head in the left direction (L), the length of the arrow of the navigation information (801) pointing in the right direction may be changed to be displayed longer in the right direction. At this time, the projection of the direction indication information received from the external terminal device (200) may be maintained, and an AR image that emphasizes only the length of the arrow in the right direction may be overlapped.

[0157] Alternatively, for example, if the wearer of the smart glasses (100) moves his / her head in the right direction (L), the length of the arrow of the navigation information (801) pointing in the right direction may be displayed shorter than before. Accordingly, the location of the (next) direction to which the wearer of the smart glasses (100) should move can be more intuitively determined based on the current direction of the wearer of the smart glasses (100).

[0158] Meanwhile, in an embodiment, smart glasses (100) can display navigation information including direction information with emphasis or limit the display thereof based on behavioral analysis based on tracking of the wearer's gaze movements.

[0159] To this end, the smart glasses (100) can track the wearer's gaze movements using an IR sensor or a gaze tracking camera to track the wearer's gaze movements and transmit the tracking results to a processor (150) to perform behavior analysis of the wearer.

[0160] At this time, if the behavior analysis result by the processor (150) determines that the wearer is focusing on the front while driving (e.g., driving a car / bicycle, etc.), the direction indication information (e.g., "280m to the right toward the district office") shown on the left side of Fig. 9 may be changed to an AR image object (902) that is highlighted. Alternatively, for example, the direction indication information (e.g., "280m to the right toward the district office") may be increased in size or displayed in a highlight color.

[0161] On the other hand, if the behavior analysis result by the processor (150) determines that the wearer has moved his / her gaze to check surrounding objects (e.g., other vehicles / bicycles or pedestrians) or to check mirrors / rearview mirrors located on the right / left while driving (e.g., driving a car / bicycle, etc.), the direction indication information shown on the left side of Fig. 9 (e.g., "280m to the right toward the district office") may be temporarily not displayed or changed to an AR image object with increased transparency. By limiting the display of information in this way, the wearer can focus on driving.

[0162] Meanwhile, although not shown, in addition to tracking the gaze movement of the wearer of the smart glasses (100), additional information related to driving (e.g., whether the wearer holds the steering wheel for a certain period of time) can be collected to limit the display of navigation information including direction information or increase transparency to prevent it from interfering with driving.

[0163] FIG. 10 is a drawing for explaining a method of providing voice guidance for navigation information displayed on smart glasses according to a set language, according to an embodiment of the present invention.

[0164] According to an embodiment, the smart glasses (100) can receive voice guidance information provided by a navigation application from an external terminal device (200). To this end, the smart glasses (100) may include a voice output means (not shown), such as a speaker. Additionally, the smart glasses (100) may be linked with an external device or service platform that provides a translation function to support various languages.

[0165] Alternatively, in some embodiments, the voice guidance for directional information may be translated into one or more languages ​​and provided through the processor (150) of the smart glasses (100).

[0166] Specifically, the processor (150) of the smart glasses (100) can extract direction information and text included in the direction indication information displayed through the smart glasses (100) and generate guidance information based on the extracted information. The processor (150) can convert the generated guidance information into voice in a language selected through user input / setting, and can control the output of voice guidance in the selected language through a voice output means (not shown).

[0167] Referring to FIG. 10, while the smart glasses (100) display direction indication information (1001) received from an external terminal device (200) in a real view, the processor (150) performs analysis of the displayed route guidance information (1011). At this time, the analysis of the displayed route guidance information includes determining whether the direction indication information (1001) includes elements that can generate voice guidance information, such as direction information.

[0168] Next, the processor (150) of the smart glasses (100) extracts movement information and text information (1012) from the direction display information (1001). For example, in FIG. 10, 'right direction' is extracted as movement information, and 'Yangcheon-gu direction' and '300m' are extracted as text information, respectively.

[0169] The processor (150) of the smart glasses (100) translates and arranges the extracted information into a selected language, generates voice guidance information, and converts it into voice data (1013). This is then transmitted to the voice output means of the smart glasses (100).

[0170] The voice output means of the smart glasses (100) outputs voice in the selected language (1014). For example, if "Korean" is selected as the language in FIG. 10, a voice such as "After 300 meters, turn right toward Yangcheon-gu" (a) may be output. If "English" is selected as the language in FIG. 10, a voice such as "After 300 meters, turn right to Yangcheon-gu" (b) may be output.

[0171] Meanwhile, some of the processes (1011 to 1013) by the processor (150) described above (e.g., translation-related) may be performed by an external device or a separate service platform that provides a translation function.

[0172] Meanwhile, the navigation method using the smart glasses described above may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program instructions, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0173] According to the smart glasses and navigation method using the same according to the embodiment of the present invention described above, navigation functions executed on external terminal devices can be effectively utilized by simply linking with the smart glasses without having to wear a separate GPS. Furthermore, intuitive navigation information can be provided without having to generate separate navigation information on the smart glasses or convert images to be compatible with the linked external terminal devices. In particular, navigation information can be extracted in a fast and efficient manner without performing machine learning on the external terminal device linked to the smart glasses, and the information converted to be suitable for the smart glasses can be directly provided. Furthermore, when a user wearing smart glasses uses the navigation function while driving, he or she can concentrate on driving without having to shift his or her gaze, and can expand the user experience of the navigation function through the lightweight smart glasses while continuing to use the navigation application on the terminal device he or she is familiar with.

[0174] Further scope of the applicability of the present invention will become apparent from the detailed description below. However, since various modifications and variations within the spirit and scope of the present invention will become apparent to those skilled in the art, it should be understood that the detailed description and specific examples, such as preferred embodiments of the present invention, are given by way of example only.

[0175] The features, structures, effects, etc. described in the embodiments above are included in at least one embodiment of the present invention, and are not necessarily limited to just one embodiment. Furthermore, the features, structures, effects, etc. exemplified in each embodiment can be combined or modified in other embodiments by those skilled in the art to which the embodiments pertain. Therefore, the contents related to such combinations and modifications should be construed as falling within the scope of the present invention.

[0176] In addition, although the above description focuses on examples, these are merely examples and do not limit the present invention. Those skilled in the art to which the present invention pertains will appreciate that various modifications and applications not exemplified above are possible without departing from the essential characteristics of the present invention. For example, each component specifically shown in the examples can be modified and implemented. In addition, differences related to such modifications and applications should be interpreted as being included within the scope of the present invention defined in the appended claims.

Claims

1. A step of connecting to an external terminal device on which a navigation application is running so that communication can be made; A step of transmitting a request to capture a screen displayed on the external terminal device and extracting direction indication information from the captured screen according to the request; A step of converting the extracted direction indication information into a format suitable for smart glasses; and Including a step of displaying the direction indication information at a selected location according to the reality view of the smart glasses. Navigation method using smart glasses.

2. In paragraph 1, The steps for extracting the direction indicator information (TBT) are: A step of capturing the displayed screen frame by frame according to the above request and extracting information of the same area including direction indication information from the captured screen. Navigation method using smart glasses 3. In paragraph 1, A step of detecting movement according to the direction indication information based on vision recognition of smart glasses; In response to the above detection, comprising a step of requesting an update of the next direction indication information to the external terminal device, Navigation method using smart glasses 4. In paragraph 2, Steps to convert to a format suitable for smart glasses are: A step of chroma keying a portion of the same area except for direction indication information and related information; and A step of removing the chroma key processed portion and receiving information converted into the appropriate format from the smart glasses, Navigation method using smart glasses.

5. In paragraph 1, The step of rendering at the selected location is: A step of rendering a first AR graphic object corresponding to the converted direction indication information at a selected location according to the reality view of the smart glasses is included. At least one of the above-mentioned selected location and the above-mentioned first AR graphic object varies depending on the route guidance mode for each means of transportation. Navigation method using smart glasses.

6. In paragraph 5, A step of rendering a second AR graphic object on the detected road sign information based on the road sign information matching the direction indication information detected in the reality view of the smart glasses, Navigation method using smart glasses.

7. In paragraph 1, The step of rendering at the selected location is: A step of setting a customized position in the reality view of the smart glasses based on the input; and Including a step of rendering the direction indication information fixedly at the above customizing location, Navigation method using smart glasses.

8. In paragraph 1, The step of rendering at the selected location is: A step of rendering a graphic object corresponding to the direction indication information while avoiding an object in the reality view of the smart glasses; and A step of changing the graphic object according to a change in the reality view of the smart glasses according to the movement of the wearer's gaze, Navigation method using smart glasses.

9. In paragraph 1, The step of rendering at the selected location is: A step of adjusting the transparency of the rendered graphic object based on the vision recognition of the smart glasses and the wearer's behavior analysis based on the wearer's gaze movement. Navigation method using smart glasses.

10. A communication unit that communicates to link with a third-party navigation application; A processor is included that transmits a request to capture the screen of the linked navigation application, receives direction indication information extracted from the captured screen according to the request, and converts it into a format suitable for smart glasses. The above processor, Rendering an AR image corresponding to the above directional information at a selected location according to the reality view of the smart glasses. Smart glasses.

11. In paragraph 10, It further includes a vision recognition unit that detects movement according to the above direction indication information, The above processor, In response to detection by the above vision recognition unit, a request for updating the next direction indication information is sent to the third party navigation application. Smart glasses.

12. In paragraph 10, The above processor, Render an AR image corresponding to the above directional information at a selected location according to the reality view of the smart glasses, Characterized in that at least one of the above-mentioned selected location and the above-mentioned AR image varies depending on the route guidance mode for each means of transportation. Smart glasses.

13. In paragraph 10, The above processor, Setting a customized position of the reality view of the smart glasses based on the input, and rendering an AR image corresponding to the direction indication information fixedly at the set customized position. Smart glasses.

14. In Article 11 The above processor, Detecting an object in the real view of the smart glasses through the vision recognition unit, avoiding the detected object, and rendering an AR image corresponding to the direction indication information, The AR image changes according to the change in the real view due to the wearer's gaze movement. Smart glasses.

15. In paragraph 10, The above processor, A method characterized in that the direction information and text included in the direction indication information are extracted to generate guidance information, the generated guidance information is controlled to be converted into a voice in a selected language, and the voice in the selected language is transmitted to a speaker to be output. Smart glasses.

Images