Wearable device and control method thereof
By introducing focus adjustment elements into the laser projector of the wearable device and dynamically adjusting the laser focus and scanning direction, the problems of visual radiation adjustment conflicts and lack of depth in augmented reality glasses are solved, achieving a more natural augmented reality experience.
Patent Information
- Application Number
- CN201980044635.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-08-21
- Filing Date
- 2019-01-03
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2039-01-03
AI Technical Summary
Vision radiation regulation conflicts in existing augmented reality glasses may lead to eye fatigue and motion diseases, and laser scanning projectors are difficult to express the depth sense of virtual images, resulting in the problem of all distances of virtual images utilizing the same depth.
The focus adjustment element is introduced in the laser projector of the wearable device, identifying the user's gaze and object position through the sensor and camera, and dynamically adjusting the focus and scanning direction of the laser to provide augmented reality information corresponding to the object on the display.
It effectively solves the problems of visual radiation regulation conflict and lack of depth, reduces eye fatigue and vertigo, and provides a more natural augmented reality experience.
Smart Images

Figure CN112384883B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a wearable device and a control method thereof, and in particular to a wearable device that displays AR (augmented reality) information of an object on a display of the wearable device and a control method thereof. Background Art
[0002] In order to realize augmented reality glasses, waveguides and displays can be used. Waveguides can be divided into reflective type and diffraction type, and liquid crystal display (LCD), liquid crystal on silicon (LCoS), digital light processing (DLP), and laser scanning projector (LSP) types can be used as displays.
[0003] In the method for outputting a virtual image, a vergence-accommodation conflict may occur in any type of display, and thus, the sense of reality may be destroyed, and this may cause eye fatigue or motion sickness (eg, vertigo).
[0004] Meanwhile, since the LSP type uses line light, depth (distance or sense of depth) may not be expressed in a virtual image, thereby causing a problem that all distances of the virtual image are expressed with the same depth. Summary of the invention
[0005] Technical issues
[0006] The present disclosure is proposed to solve the above-mentioned problems, and an object of the present disclosure is to provide a wearable device and a control method thereof, which displays AR information corresponding to an object by adding a focus adjustment element to a laser projector.
[0007] Technical Solution
[0008] According to aspects of the present disclosure, a wearable device is provided, comprising: a camera; a sensor; a display; a laser projector; and a processor, configured to: identify a user's gaze based on sensing data obtained by the sensor, identify position information of at least one object included in an image obtained by the camera, and control the laser projector to provide augmented reality (AR) information related to the object on the display based on the identified user's gaze and the position information of the object, wherein the laser projector comprises: a light emitting element configured to emit laser light; a focus adjustment element configured to adjust the focus of the emitted laser light; and a scanning mirror configured to control a scanning direction of light with its focus adjusted, and wherein the processor is configured to: control a driving state of the focus adjustment element based on the identified user's gaze and the position information of the object.
[0009] The processor may be configured to identify whether the position of the object corresponds to the gaze of the user by comparing the gaze of the user with the position information of the object, and control a driving state of the focus adjustment element based on the identified result.
[0010] The processor can be configured to: provide AR information related to the object on the display by controlling the focal length of the focus adjustment element to a first focal length based on the position of the object corresponding to the user's gaze; and provide AR information related to the object on the display by controlling the focal length of the focus adjustment element to a second focal length based on the position of the object not corresponding to the user's gaze.
[0011] The first focal length may be a unique focal length of the focus adjustment element, and the second focal length may be different from the unique focal length.
[0012] The processor can be configured to: identify position information of each object among multiple objects included in the obtained image; control the focal length of the focus adjustment element to become the first focal length based on providing first AR information related to a first object corresponding to the user's gaze among the multiple objects; and control the focal length of the focus adjustment element to become the second focal length based on providing second AR information related to a second object not corresponding to the user's gaze among the multiple objects.
[0013] The processor may be configured to: based on the second AR information being provided, control the focal length of the focus adjustment element to become a second focal length according to the depth information of the second object.
[0014] The processor may be configured to: obtain a difference between a depth value included in the depth information of the second object and a depth value corresponding to the recognized gaze of the user; and control the focal length of the focus adjustment element to become a second focal length based on the obtained difference in depth values.
[0015] The processor may be configured to control the scanning mirror by determining at least one of a scanning start position, a scanning direction, and a scanning path based on position information of each of the multiple objects based on the fact that the acquired image includes multiple objects.
[0016] The processor may be configured to determine at least one of a scanning start position, a scanning direction, and a scanning path based on position information of each of the plurality of objects, so that the focal length of the focus adjustment element changes in stages.
[0017] The processor may be configured to determine at least one of a scanning start position, a scanning direction, and a scanning path based on position information of each of the plurality of objects to minimize a rotation operation of the scanning mirror.
[0018] According to another aspect of the present disclosure, a method for controlling a wearable device including a laser projector is provided, the laser projector including: a light emitting element configured to emit laser light; a focus adjustment element configured to adjust the focus of the emitted laser light; and a scanning mirror configured to control a scanning direction of light with the focus adjusted, the method including: identifying a user's gaze based on sensing data obtained by a sensor; identifying position information of at least one object included in an image obtained by a camera; controlling a driving state of the focus adjustment element based on the identified user's gaze and the position information of the object; and providing augmented reality (AR) information related to the object on a display based on the identified user's gaze and the position information of the object.
[0019] Controlling the driving state of the focus adjustment element may include identifying whether the position of the object corresponds to the user's gaze by comparing the user's gaze with position information of the object, and controlling the driving state of the focus adjustment element based on the identified result.
[0020] Providing AR information on a display may include: providing AR information related to the object on the display by controlling the focal length of a focus adjustment element to a first focal length based on the position of the object corresponding to the user's gaze; and providing AR information related to the object on the display by controlling the focal length of the focus adjustment element to a second focal length based on the position of the object not corresponding to the user's gaze.
[0021] The first focal length may be a unique focal length of the focus adjustment element, and the second focal length may be different from the unique focal length.
[0022] The method for controlling a wearable device may further include: identifying position information of each object among multiple objects included in the obtained image, wherein controlling the driving state of the focus adjustment element may include: based on providing first AR information related to a first object corresponding to the user's gaze among the multiple objects, controlling the focal length of the focus adjustment element to become a first focal length; and based on providing second AR information related to a second object not corresponding to the user's gaze among the multiple objects, controlling the focal length of the focus adjustment element to become a second focal length.
[0023] Controlling the driving state of the focus adjustment element may include: based on providing the second AR information, controlling the focal length of the focus adjustment element to become a second focal length according to the depth information of the second object.
[0024] Controlling the driving state of the focus adjustment element may include: obtaining the difference between the depth value included in the depth information of the second object and the depth value corresponding to the identified user's gaze; and based on the obtained difference in depth values, controlling the focal length of the focus adjustment element to become a second focal length.
[0025] Controlling the driving state of the focus adjustment element may include: based on the obtained image including multiple objects, controlling the scanning mirror by determining at least one of a scanning start position, a scanning direction, and a scanning path based on position information of each of the multiple objects.
[0026] Controlling the scanning mirror may include: determining at least one of a scanning start position, a scanning direction, and a scanning path based on position information of each object of the plurality of objects, so that a focal length of the focus adjustment element is changed in stages.
[0027] Controlling the scanning mirror may include determining at least one of a scanning start position, a scanning direction, and a scanning path based on position information of each of the plurality of objects to minimize a rotation operation of the scanning mirror. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1a and Figure 1b is a diagram showing the clarity of an object according to a user's gaze;
[0029] Figure 2a and Figure 2b is a diagram showing AR information according to the gaze of a user wearing a wearable device;
[0030] Figure 3 is a diagram showing an embodiment in which AR information is displayed when a focus adjustment element is not provided;
[0031] Figure 4 is a diagram showing an embodiment of pre-processing AR information when a focus adjustment element is set;
[0032] Figure 5 is a block diagram illustrating a wearable device according to an embodiment of the present disclosure;
[0033] Figure 6 is a diagram showing an embodiment in which AR information is displayed when a focus adjustment element is provided;
[0034] Figure 7 and Figure 8 A diagram showing a situation where one pixel of a virtual image is formed on the retina;
[0035] Fig. 9 and Fig.10 is a diagram showing the movement path of the laser light reflected on the scanning mirror;
[0036] Figures 11 to 14 is a diagram showing an AR information display method according to each embodiment;
[0037] Figures 15 to 18 is a diagram showing the amount of change of the focus adjustment element according to each embodiment; and
[0038] Fig.19 is a flowchart illustrating a method for controlling a wearable device according to an embodiment. DETAILED DESCRIPTION
[0039] The present disclosure will be described in detail after briefly explaining the manner in which the specification and the drawings are described.
[0040] The functions in the embodiments of the present disclosure have been considered, and the terms used in the specification and claims are selected as general terms as much as possible. However, these terms may change according to the intentions of those skilled in the art, precedents, technical interpretations, the emergence of new technologies, etc. In addition, there are also terms arbitrarily selected by the applicant. If there is no definition of a specific term, such a term may be interpreted as the meaning defined in this specification and may be interpreted based on the general content of the specification and common knowledge in the technical field.
[0041] The same reference numerals or symbols in the drawings in this specification represent parts or components that perform substantially the same functions. For ease of description and understanding, the same reference numerals will be used to describe in different embodiments. That is, although components with the same reference numerals are shown in multiple drawings, the multiple drawings do not illustrate one embodiment.
[0042] In addition, ordinals such as "first" or "second" may be used to distinguish components in the specification and claims. Such ordinals are used to distinguish identical or similar components, and the terms should not be interpreted restrictively due to the use of ordinals. These terms are only used to distinguish elements from each other. For example, for components using such ordinals, the order of use or arrangement should not be interpreted restrictively using their numbers. If necessary, the ordinals can be interchanged.
[0043] Unless otherwise specifically defined, singular expressions may encompass plural expressions. It is to be understood that terms such as "including" or "consisting of" are used herein to indicate the presence of a characteristic, quantity, step, operation, element, component, or combination thereof, without excluding the possibility of the presence or addition of one or more other characteristics, quantities, steps, operations, elements, components, or combinations thereof.
[0044] These embodiments of the present disclosure may be variously modified and include various embodiments, and specific embodiments will be shown in the drawings and described in detail in the specification. However, it should be understood that this does not limit the scope of the specific embodiments, and includes all modifications, equivalents and / or alternative forms included in the disclosed spirit and technical scope. When describing the present disclosure, the detailed description of the prior art may be omitted when it is determined that the detailed description may unnecessarily obscure the main points of the present disclosure.
[0045] Terms such as "module", "unit" or "component" in the present disclosure are used to refer to components that perform at least one function or operation, and such components can be implemented as hardware, software, or a combination of hardware and software. In addition, except when each of multiple "modules", "units", "components", etc. needs to be implemented in a single specific hardware, these components can be integrated in at least one module or chip and can be implemented in at least one processor.
[0046] In addition, in the embodiments of the present disclosure, in addition to direct connection, the connection between a specific component and another component may include an indirect connection via another medium. When describing that a specific component includes another specific component, unless otherwise mentioned, it implies that another component may also be included, rather than excluding another component.
[0047] Figure 1a a and b are graphs showing the clarity of an object according to which the user is gazing.
[0048] Figure 1a The case where the user fixes the user's eyes on the vehicle 10 on the left side in front is shown. When the user fixes the user's eyes on the vehicle 10 as an object located on the left side, the user may vaguely recognize objects other than the portion where the user's eyes are fixed. For example, the user may vaguely recognize a building 11 other than the left portion where the user's eyes are fixed. For the sake of convenience of description, in this specification, an object clearly recognized by the user may be displayed with a solid line, and an object vaguely recognized may be displayed with a dotted line. For example, the vehicle 10 where the user's eyes are fixed is displayed with a solid line, and the building 11 located on the portion where the user's eyes are not fixed is displayed with a dotted line.
[0049] Figure 1b is a diagram showing a case where the user's eyes are fixed on a building 11 on the front right. When the user fixes the user's eyes on the building 11 as an object located on the right, the user may vaguely recognize objects other than the portion where the user's eyes are fixed. For example, the vehicle 10 may be vaguely recognized.
[0050] Figure 2a and Figure 2b is a diagram showing AR information according to the gaze of a user wearing a wearable device.
[0051] The wearable device can identify an object within a user-recognizable range and display AR information related to the object on a display included in the wearable device. For example, the wearable device can display AR information 20 of vehicle 10, and the AR information 20 of vehicle 10 can include: name, manufacturing company, and distance from the wearable device to the identified object.
[0052] Figure 2a An embodiment is shown in which the user focuses the user's eyes on the vehicle 10 on the left after wearing the wearable device. The user can recognize the vehicle 10 object and the building 11 object in front. The wearable device can display AR information about the recognized vehicle 10 object and the building 11 object. Figure 2a , the wearable device can display AR information including at least one of the type (Vehicle), the manufacturing company (Company: Samsung), and the distance between the wearable device and the vehicle 10 object (Distance: 10m). In addition, the wearable device can display AR information including at least one of the type (Building), the building name (Name: abc), and the distance between the wearable device and the building 11 object (100m). The AR information provided here can be the type of information set by the user through a menu, etc., information provided by the currently operated application, information according to the type of the object, etc., but is not limited thereto.
[0053] Meanwhile, the wearable device can display various information in addition to the AR information of the object. For example, the wearable device can display information related to the navigation function. Figure 2a and Figure 2b , the wearable device can display AR information related to the navigation function in front of the user (Go 10m (forward 10m)), and the information related to the navigation function can refer to the direction and distance mark for reaching the destination. Figure 2a and Figure 2b Only "Go 10m" is shown, but the information related to the navigation function may include: information showing various directions and distances, such as "left 15m", "right 20m", "turn back 30m", and the like.
[0054] Figure 2a An embodiment is shown in which the wearable device displays AR information when the user focuses his eyes on the vehicle 10 on the left after wearing the wearable device.
[0055] Figure 2bAn embodiment is shown in which the wearable device displays AR information when the user focuses his eyes on the building 11 on the right after wearing the wearable device.
[0056] The user can clearly identify the object that the user's eyes are looking at. However, the user may vaguely identify the object that the user's eyes are not looking at. The wearable device can display AR information regardless of which object the user is looking at. Figure 2a and Figure 2b In a corresponding embodiment, the wearable device can clearly display the AR information corresponding to the vehicle 10 and the building 11. In other words, the wearable device can display the AR information with the same clarity regardless of the position where the user is looking.
[0057] Figure 3 is a diagram showing an embodiment in which AR information is displayed when a focus adjustment element is not provided.
[0058] The wearable device may include a laser scanning projector (LSP) or laser projector 150, which will be referred to later. Figure 5 The laser projector 150 may include a light emitting element 151, a focus adjustment element 152, and a scanning mirror 153. The light emitting element 151 here may be an element that emits red, green, and blue lasers and may be referred to as an RGB laser.
[0059] refer to Figure 3 , the wearable device can display AR information on the display using a laser projector 150 without a focus adjustment element 152.
[0060] Reference numeral 310 may represent an image seen through a transparent display. Image 320 may be an AR information image displayed by a wearable device on a transparent display. Image 320 may include: AR information 20 and 21 or additional information 22 corresponding to each object. The laser projector 150 may use a light emitting element 151 to display image 320 on a display. The wearable device may display image 320 on a transparent display using a laser projector 150, and the image that the wearable device ultimately displays to the user may be image 330. Image 330 may be an image obtained by combining an image seen through a transparent display with an image displayed with AR information.
[0061] Meanwhile, the distances of the vehicle 10 object and the building 11 object recognized by the user from the wearable device may be different from each other. However, the wearable device may display AR information with the same clarity regardless of the position where the user is looking, and therefore, the user may be confused about the distance or sense of depth. For example, the objects recognized by the user may be at different distances from the wearable device. However, if the wearable device displays the AR information corresponding to each object with the same clarity regardless of the position where the user is looking, the user may be confused about the distance or sense of depth of each object and the AR information corresponding to each object.
[0062] To solve such a problem, the wearable device may perform a pre-processing operation on an image regarding AR information corresponding to each object.
[0063] Figure 4 is a diagram illustrating an embodiment in which the wearable device 100 pre-processes AR information when the laser projector 150 does not include a focus adjustment element.
[0064] refer to Figure 4 , the wearable device can identify the position where the user is looking, and perform a pre-processing operation on some AR information of the AR information image by considering the position where the user is looking.
[0065] For example, the wearable device can distinguish between an object at a position where the user is looking and an object not at a position where the user is looking. The wearable device can clearly display AR information corresponding to the object at the position where the user is looking, and can fuzzily display AR information corresponding to the object not at the position where the user is looking.
[0066] In other words, the wearable device may perform a preprocessing operation on the AR information in advance, and the wearable device may obtain in advance an image 420 in which the clarity (clear or blurred state) of the AR information corresponding to each object is determined by considering the position at which the user is looking. The preprocessing operation may refer to an operation suitable for changing the clarity of the AR information in advance, and the wearable device may output an image in which the clarity of some AR information is changed via the laser projector 150. Here, the laser projector 150 may include a focus adjustment element.
[0067] Reference numeral 410 may denote an image seen through a transparent display. Reference numeral 420 may denote an AR information image displayed by a wearable device on a transparent display. Image 420 may include: AR information 20 and 21 corresponding to each object and additional information 22. The laser projector 150 may use a light emitting element 151 to display image 420 on a display. The wearable device may display image 420 on a transparent display using the laser projector 150, and the image that the wearable device ultimately displays to the user may be image 430. Image 430 may be an image obtained by combining an image seen through a transparent display with an image displayed with AR information.
[0068] Figure 5 is a block diagram illustrating a wearable device according to an embodiment of the present disclosure.
[0069] refer to Figure 5 , the wearable device 100 may include: a camera 110 , a sensor 120 , a display 130 , a processor 140 , and a laser scanning projector (LSP) or a laser projector 150 .
[0070] The camera 110 may be a device used by the wearable device 100 to identify an object in front of the user, and the camera 110 may capture an image for identifying an object and an environment in front. The wearable device 100 may detect the characteristics of an object located in front of the user using the camera 110. In addition, the wearable device 100 may detect how the objects identified by the camera 110 are distributed in space (depth map) and what each object is (object recognition). In addition, the wearable device 100 may determine which information to display at the corresponding position by using the measured information.
[0071] The sensor 120 can be used as a device for recognizing the user's gaze. For example, the sensor 120 can detect the movement of the user's pupil to recognize where the user is looking. Here, the sensor 120 and the camera 110 are shown separately, but the wearable device 100 may include a camera for recognizing a front object in addition to the camera 110, and other cameras that can be attached to the sensor 120.
[0072] For the display 130 , a transparent display may be used so that the user can see the front. The user may look forward through the display 130 , and AR information corresponding to a specific object may be displayed on a portion of the display 130 .
[0073] The processor 140 may perform overall control operations of the wearable device. Specifically, the processor may use the image information captured by the camera 110 to identify an object and obtain AR information corresponding to the object. In addition, the processor may control the laser projector 150 to display AR information on the display 140.
[0074] The processor may be implemented as a digital signal processor (DSP) or a microprocessor that processes digital signals. However, the present disclosure is not limited thereto, and the processor may include one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP) or a communication processor (CP), and an ARM processor, or may be defined as a corresponding term. In addition, the processor 140 may be implemented as a system on chip (SoC) or a large-scale integration (LSI) including a processing algorithm, or may be implemented in the form of a field programmable gate array (FPGA).
[0075] The processor 140 may recognize the gaze of the user based on the sensing data obtained by the sensor 120, and the processor 140 may recognize information about at least one object included in the image obtained by the camera 110. The information about the object may include object information such as the type, name, shape, color, etc. of the object, and may include position information about where the object is located.
[0076] The processor 140 may obtain augmented reality (AR) information related to the object based on the recognized user's gaze and the information of the object. In order to obtain the information of the object, the user may analyze the image captured by the camera 110. The object may be identified in the image captured by the camera, and AR information corresponding to the corresponding object may be obtained. As a method for obtaining AR information, the processor 140 may use an internal storage device. The information of the object may be obtained in advance by using a database stored in the internal storage device.
[0077] Meanwhile, in actual implementation, the processor 140 may use an external server to obtain AR information corresponding to the object. The processor 140 sends the information of the object to the external server and receives the corresponding AR information from the external server. Here, the processor 140 may further include a communicator (not shown) for sending and receiving data.
[0078] At the same time, the processor 140 may control the laser projector 150 to provide the obtained AR information on the display 130 .
[0079] The laser projector 150 may include a light emitting element 151 that emits laser light, a focus adjusting element 152 that adjusts the focus of the emitted laser light, and a scanning mirror 153 that controls a scanning direction of the light whose focus is adjusted.
[0080] refer to Figure 5 The laser projector 150 may be a laser scanning projector (LSP), and may include: a light emitting element 151 , a focus adjustment element 152 , and a scanning mirror 153 .
[0081] The laser projector 150 may be a device for outputting a laser light source, and may use an LSP method.
[0082] The light emitting element 151 may mainly output light sources having three colors of red, green, and blue, and may output light having different colors by additionally combining a white light source with the three light sources.
[0083] The focus adjustment element 152 may be an element through which the light output by the light emitting element 151 is transmitted and which may refract the light. When the focus adjustment element 152 is controlled, the refraction degree of the transmitted light may be changed. When the refraction degree of the light is changed, the focus of the virtual image may be changed, and thus, the user may recognize the object clearly or vaguely.
[0084] The focus adjustment element 152 may be formed of a liquid lens, a liquid crystal lens, etc. The liquid lens may refer to a lens whose refractive index is adjusted by physically changing the thickness or shape of the lens, and the liquid crystal lens may refer to a lens whose refractive index is adjusted by changing the chemical configuration or arrangement of molecules.
[0085] The processor 140 may control the driving state of the focus adjustment element 152 based on the recognized user's gaze and the position information of the object.
[0086] Here, the processor 140 may identify whether the position of the object corresponds to the user's gaze by comparing the user's gaze with the position information of the object. The processor 140 may control the driving state of the focus adjustment element 152 based on the identified result.
[0087] Specifically, when the position of the object corresponds to the user's gaze, the processor 140 can control the focal length of the focus adjustment element 152 to become a first focal length, and can provide AR information related to the object on the display 130. Here, the focal length can be the distance between the focus and the reference point at the intersection after the light is transmitted through the focus adjustment element 152. The reference point may refer to the vertex of the focus adjustment element 152. The focal length may vary according to the refractive index of the focus adjustment element 152. For example, when the refractive index of the focus adjustment element 152 changes, the focal length may also change. The first focal length may correspond to a pre-stored distance in a state where the focus adjustment element 152 has a specific refractive index.
[0088] When the processor 140 changes the refractive index by adjusting the focus adjustment element 152, the focal length may be changed. When the focal length changes, the clarity of the object may be different. The difference in clarity may imply a change in the clarity of the image formed on the user's retina. Specifically, when the light output by the laser projector 150 is the same, the clarity of the image recognized by the user may change according to the change in the refractive index of the focus adjustment element 152.
[0089] Meanwhile, when the position of the object does not correspond to the user's gaze, the processor 140 may control the focal length of the focus adjustment element 152 to become a second focal length, and may provide AR information related to the object on the display 130 .
[0090] Here, the first focal length may refer to a unique focal length of the focus adjustment element 152, and the second focal length may be different from the unique focal length.
[0091] At the same time, the processor 140 can identify the position information of each of the multiple objects included in the obtained image. When providing the first AR information related to the first object corresponding to the user's gaze among the multiple objects, the processor 140 can control the focal length of the focus adjustment element 152 to become the first focal length.
[0092] When providing the second AR information about the second object that does not correspond to the user's gaze among the plurality of objects, the processor 140 may control the focal length of the focus adjustment element 152 to become the second focal length.
[0093] In addition, when the second AR information is provided, the processor 140 may control the focal length of the focus adjustment element 152 to become the second focal length based on the depth information of the second object.
[0094] In addition, the processor 140 may obtain a difference between a depth value included in the depth information of the second object and a depth value corresponding to the recognized gaze of the user, and control the focal length of the focus adjustment element 152 to become a second focal length based on the obtained difference in depth values. The depth value here may refer to an index representing a sense of distance of an object. The depth value may use the distance from the object to the wearable device 100 as a reference.
[0095] In addition, when the obtained image includes a plurality of objects, the processor 140 may control the scanning mirror 153 by determining at least one of a scanning starting position, a scanning direction, and a scanning path based on the position information of each of the plurality of objects. The scanning mirror 153 may reflect the light (light source) output by the light emitting element 151 to project the light to a specific position. Therefore, the processor 140 may reflect the light (light source) output by the light emitting element 151 to project the light to a specific position by adjusting the starting position, direction, path, etc. of the scanning mirror 153.
[0096] In addition, the processor 140 may determine at least one of a scanning start position, a scanning direction, and a scanning path based on the position information of each object among the plurality of objects, thereby changing the focal length of the focus adjustment element 152 in stages.
[0097] In addition, the processor 140 may determine at least one of a scanning start position, a scanning direction, and a scanning path based on the position information of each of the plurality of objects, thereby minimizing the rotation operation of the scanning mirror 153. Figures 15 to 18 This is described in detail.
[0098] When using the laser projector 150, even if the focus adjustment element 152 is not provided, the AR information may be blurred in advance and then displayed on the display. In such a case, a similar effect may be exhibited. However, the pixels that are actually formed on the retina through the lens correspond to extremely small laser dots and therefore do not correspond to an image that is optically blurred. Therefore, the user may feel awkward due to the difference between the clarity information of the image sent to the brain and the clarity of the image actually formed on the retina.
[0099] However, since the wearable device 100 according to an embodiment of the present disclosure outputs an image via the focus adjustment element 152, the size of the light corresponding to one pixel distributed (or focused or formed) to the retina can be optically the same as when the object actually appears blurred. Therefore, the user can clearly recognize the AR information. In addition, the AR information other than the area the user is looking at is blurred, which is conducive to protecting the user's retina.
[0100] Therefore, when the wearable device does not include the focus adjustment element 152, the user may feel awkward about the depth (distance or sense of depth) of the virtual image or AR information. However, when using the wearable device 100 according to an embodiment of the present disclosure, a natural augmented reality environment can be experienced by reflecting the depth (distance or sense of depth) of the AR information.
[0101] Figure 6 is a diagram showing an embodiment in which AR information is displayed when a focus adjustment element is provided.
[0102] Reference has been made to Figures 1 to Figure 4 In the above description, the wearable device does not include the focus adjustment element 152, but with reference to Figure 6 , the laser projector 150 of the wearable device may further include a focus adjustment element 152 .
[0103] Reference numeral 610 may denote an image seen through a transparent display. Reference numeral 620 may denote an AR information image displayed by a wearable device on a transparent display. Image 620 may include: AR information 20 and 21 or additional information 22 corresponding to each object. The laser projector 150 may use a light emitting element 151 to display image 620 on a display. The wearable device may display image 620 on a transparent display using the laser projector 150, and the image that the wearable device ultimately displays to the user may be image 630. Image 630 may be an image obtained by combining an image seen through a transparent display with an image displayed with AR information.
[0104] Here, the AR information included in the image 620 may be information corresponding to an image that has not been subjected to a pre-processing operation.
[0105] When the laser projector 150 emits the image 620 without a pre-processing operation through the light emitting element 151 , the wearable device may control the laser emitted through the light emitting element 151 to pass through the focus adjustment element 152 .
[0106] Here, the wearable device can recognize the position of each object and the position where the user is looking. In addition, the wearable device can control the focus adjustment element 152 by considering the position of the object and the position where the user is looking.
[0107] In addition, the wearable device can distinguish between an object at a position where the user is looking and an object not at a position where the user is looking. The wearable device can control the focus adjustment element 152 to clearly display the AR information corresponding to the object at the position where the user is looking, and can control the focus adjustment element 152 to fuzzily display the AR information corresponding to the object not at the position where the user is looking.
[0108] There are various methods for controlling the focus adjustment element 152 by the wearable device. The wearable device can adjust the clarity of the image by changing the refractive index of the focus adjustment element 152. In the focus adjustment element 152 according to the embodiment of the present disclosure, the refractive index can be changed when the thickness of the lens changes. However, in the focus adjustment element 152 according to the embodiment of the present disclosure, although the thickness of the lens is not changed, the refractive index can be changed by changing the material and internal components of the lens, and the clarity of the image can be adjusted according to the change in the refractive index.
[0109] Therefore, the wearable device may transmit the image 620 that has not been subjected to the pre-processing operation through the focus adjustment element 152 and allow the user to recognize each of the AR information corresponding to each object through different focuses from each other.
[0110] The user can see an image displayed with AR information through the transparent display, wherein the AR information is obtained by considering both the position of the object and the position of the user's gaze. The final image 630 displayed by the transparent display can be displayed with appropriate AR information corresponding to each object, and can be specifically an image obtained by reflecting the distance and depth sense. Therefore, the user can be less confused about the distance and depth sense.
[0111] At the same time, in reference Figure 6 In the description of FIG. 1 , it is described that the focus adjustment element 152 is disposed between the light emitting element 151 and the scanning mirror 153 , but according to another embodiment, the focus adjustment element 152 may exist at various positions.
[0112] Figure 7 and Figure 8 A diagram showing how one pixel of a virtual image is formed on the retina.
[0113] Figure 7 is a diagram illustrating an embodiment in which the wearable device 100 pre-processes AR information when the laser projector 150 does not include a focus adjustment element.
[0114] Reference numeral 710 may represent an image seen through a transparent display. Image 720 may be an AR information image displayed by a wearable device on a transparent display. Image 720 may include: AR information 20 and 21 or additional information 22 corresponding to each object. The laser projector 150 may use a light emitting element 151 to display image 720 on a display. The wearable device may display image 720 on a transparent display using the laser projector 150, and the image that the wearable device ultimately displays to the user may be image 730. Image 730 may be an image obtained by combining an image seen through a transparent display with an image displayed with AR information.
[0115] Here, the AR information included in the image 720 may be information corresponding to the image after the pre-processing operation. Figure 4 The processing operation is described in detail.
[0116] The wearable device 100 may pre-process the AR information and output an image obtained by changing the clarity of some AR information via the laser projector 150. In this case, the size of the light corresponding to the image output by the laser projector 150 and distributed (or focused or formed) to the user's retina may be the same. Specifically, when the laser projector 150 does not include a focus adjustment element, the size of the light corresponding to one pixel of the virtual image 720 including the AR information focused (or formed or distributed) on the retina may be the same. This may be a result caused by the properties of the laser. When the user recognizes the image output by the laser, the user can recognize an image with the same clarity regardless of the distance information.
[0117] Meanwhile, even if AR information is preprocessed according to distance information of an object, a user may recognize the AR information awkwardly. An image output via a laser may unnaturally express a sense of distance, and a user may feel eye fatigue or dizziness.
[0118] To solve such a problem, the wearable device 100 may further include a focus adjustment element 152 in the laser projector 150 .
[0119] Figure 8 2 is an example in which the wearable device 100 displays AR information when the focus adjustment element 152 is further included.
[0120] Reference numeral 810 may represent an image seen through a transparent display. Image 820 may be an AR information image displayed by a wearable device on a transparent display. Image 820 may include: AR information 20 and 21 corresponding to each object or additional information 22. The laser projector 150 may use a light emitting element 151 to display image 820 on a display. The wearable device may display image 820 on a transparent display using a laser projector 150, and the image that the wearable device ultimately displays to the user may be image 830. Image 830 may be an image obtained by combining an image seen through a transparent display with an image displayed with AR information.
[0121] At the same time, Figure 7 Differently, portions 21 and 22 corresponding to some AR information in image 820 may not be subjected to the pre-processing operation.
[0122] In the wearable device 100, the focus adjustment element 152 may be disposed between the light emitting element 151 and the scanning mirror 153, and an image may be output.
[0123] In this case, the size of light corresponding to the image distributed to the user's retina output by the laser projector 150 may vary according to the focus adjustment element 152. The wearable device 100 may adjust the focus adjustment element 152 after determining whether the AR information of the object is clearly displayed, and in this case, the size of the image formed on the user's retina may vary according to the refractive index of the focus adjustment element.
[0124] For example, AR information of an object corresponding to the position where the user is looking can be output via the laser projector 150, and the graphic pixels of the AR information as a virtual image can form a small image on the user's retina. Here, the pixels of the AR information 22 at a distance of about 10m from the user's gaze can be formed as a slightly larger image on the user's retina. In addition, the pixels of the AR information 21 at a distance of about 90m from the user's gaze can be formed as a very large image on the user's retina.
[0125] When comparing the sizes, the sizes of the images may become larger in the order of AR information 20, 22, and 21. The larger the size of the image formed on the retina, the more blurred the corresponding portion that the user can recognize. Therefore, the user can recognize AR information 20 most clearly, recognize AR information 22 with moderate blurriness, and recognize AR information 21 most blurrily.
[0126] The wearable device 100 can adjust the focus adjustment element 152 based on the position information of the object or the distance information to the wearable device 100, thereby changing the size of the image formed on the retina. Since the user recognizes the size of the image differently, the user can naturally recognize the AR information displayed on the transparent display. Figure 8 A user of the wearable device 100 of an embodiment may experience less eye fatigue and dizziness.
[0127] Fig. 9 and Fig.10 It is a diagram showing the movement path of the laser light reflected by the scanning mirror.
[0128] The laser projector 150 may include a scanning mirror 153, and the wearable device 100 may project light output from the light emitting element 151 to various locations through the scanning mirror 153. The scanning mirror 153 may be used to reflect light, and the wearable device 100 may project light to a desired location by adjusting the angle of the scanning mirror 153.
[0129] The wearable device 100 may control the scanning mirror 153 to project light through a desired moving path.
[0130] Fig. 91 is a diagram showing an embodiment of a movement path of the wearable device 100 by controlling the scanning mirror 153 to project light. The images 910, 920, and 930 here can be respectively Figure 6 The images 610 , 620 and 630 correspond to each other.
[0131] The wearable device 100 can control the scanning mirror 153 of the laser projector 150 to display AR information and move the scanning mirror 153 to an arbitrary position of the display. The image 930 can be an image obtained by combining the image 920 including the AR information with the image 910 seen through the transparent display. The wearable device 100 can output the AR information 20, 21, and 22 using the laser projector 150 to synthesize the image 920. In this case, the laser projector 150 can control the scanning mirror 153 to project light to all positions.
[0132] Fig. 9 The light is shown to be projected to all positions, but this is only for explaining the positions to which the light can be projected by the scanning mirror 153, and actually the light may be projected only to the area where the AR information 20, 21, and 22 corresponding to the image 920 is displayed. For example, the light emitting element 151 emits light to the area where the AR information 20, 21, and 22 corresponding to the image 920 is displayed, but the light emitting element 151 may not emit light to the area where the AR information 20, 21, and 22 corresponding to the image 920 is not displayed.
[0133] At the same time, according to another embodiment, the wearable device 100 can emit light corresponding to visible light to an area where AR information 20, 21 and 22 corresponding to the image 920 is displayed, and can emit light corresponding to invisible light that cannot be recognized by the human eye to an area where AR information 20, 21 and 22 corresponding to the image 920 is not displayed.
[0134] at the same time, Fig.10 is a diagram showing another embodiment of the movement path of the wearable device 100 by controlling the scanning mirror 153 to project light. The images 1010, 1020 and 1030 here can be respectively Figure 6 The images 610 , 620 and 630 correspond to each other.
[0135] refer to Fig.10 , the wearable device 100 may not move the scanning mirror 153 to all possible positions. The wearable device 100 may move the scanning mirror 153 only to the area where the AR information 20, 21, and 22 are displayed in the image 1020. Fig.10 The illustrated movement path may be a minimum movement path for passing through only the area in which the AR information 20 , 21 , and 22 are displayed in the image 1020 .
[0136] Fig. 9 and Fig.10 The illustrated moving path of the scanning mirror 153 is merely a description of some of the various moving paths, and various methods not shown may also be applied in actual implementation.
[0137] At the same time, Figure 3 , Figure 4 , Figure 6 , Figure 7 , Figure 8 , Fig. 9 and Fig.10 In the embodiment, an image seen through a transparent display and an image showing AR information are described separately, but in actual implementation, the image can be realized by a method for combining a captured image with AR information. Specifically, the wearable device 100 can capture a scene in front of a user, store it as an image, and newly generate an image related to AR information. The wearable device 100 can generate an image by combining an image obtained by capturing a scene in front of the user and an image related to AR information. The wearable device 100 can display a combined image on a display.
[0138] Figures 11 to 14 is a diagram showing an AR information display method according to each embodiment.
[0139] refer to Fig.11 , a plurality of objects 30 to 33 may be located in front of the user, and the user may place the gaze 50 on any one of the plurality of objects. Fig.11 In FIG. 3 , it is assumed that the objects Sign 30 , bear 31 , vehicle 32 and building 33 are located in front of the user. It is assumed that the user's gaze is located on Sign 30 . Fig.11 The assumption can be Figures 12 to 18 Same as in.
[0140] Fig.12 is a diagram illustrating an embodiment in which the wearable device 100 displays AR information when the laser projector 150 does not include the focus adjustment element 152 .
[0141] Each AR information corresponding to each object may include a type (category), a name, and distance information from the wearable device 100 .
[0142] The AR information 40 corresponding to the Sign 30 object may include: Sign, stop, and 20m. In other words, regarding the AR information 40, the type may be stored as Sign, the name may be stored as Stop, and the distance information from the wearable device 100 to the object may be stored as 20m. When the wearable device 100 recognizes the Sign 30 object, the wearable device 100 may obtain the AR information 40 corresponding to the Sign 30 object, and in this case, the AR information 40 may be Sign, stop, and 20m.
[0143] The AR information 41 corresponding to the bear 31 object may include animal, bear, and 30m. In other words, regarding the AR information 41, the type may be stored as animal, the name may be stored as bear, and the distance information from the wearable device 100 to the object may be stored as 30m. When the wearable device 100 recognizes the bear 31 object, the wearable device 100 may obtain the AR information 41 corresponding to the bear 31 object, and in this case, the AR information 40 may be animal, bear, and 30m.
[0144] The AR information 42 corresponding to the vehicle 32 object may include: vehicle, Samsung, and 10m. In other words, regarding the AR information 42, the type may be stored as vehicle, the name may be stored as Samsung, and the distance information from the wearable device 100 to the object may be stored as 10m. When the wearable device 100 recognizes the vehicle 32 object, the wearable device 100 may obtain the AR information 42 corresponding to the vehicle 32 object, and in this case, the AR information 42 may be vehicle, Samsung, and 10m.
[0145] The AR information 43 corresponding to the building 33 object may include: building, abc, and 100m. In other words, regarding the AR information 43, the type may be stored as building, the name may be stored as abc, and the distance information from the wearable device 100 to the object may be stored as 100m. When the wearable device 100 recognizes the building 33 object, the wearable device 100 may obtain the AR information 43 corresponding to the building 33 object, and in this case, the AR information 43 may be building, abc, and 100m.
[0146] refer to Fig.12, when the focus adjustment element 152 is not set, the wearable device 100 can clearly display all AR information 40, 41, 42 and 43 corresponding to each object, respectively, regardless of the stored distance information.
[0147] However, if the wearable device 100 clearly displays all AR information 40, 41, 42, and 43, the user may feel eye fatigue and confusion due to the difference in the sense of distance between the object that the user is not looking at and the corresponding AR information. Specifically, if the user looks at the Sign at a distance of 20m from the wearable device 100, the user may vaguely see the building 33 at a distance of 100m from the wearable device 100. If the building 33 is vaguely seen while the AR information 43 corresponding to the building 33 is clearly seen, the user may feel eye fatigue and may be confused by the sense of distance.
[0148] Therefore, the wearable device 100 can solve the above problem by using the focus adjustment element 152.
[0149] Fig.13 and Fig.14 is an embodiment showing that the wearable device 100 displays AR information using the focus adjustment element 152 .
[0150] refer to Fig.13 and Fig.14 The wearable device 100 can use the focus adjustment element 152 to change the size of the image formed on the user's retina and the clarity of the image. The user can see the AR information clearly or blurrily according to the change of the focus adjustment element 152.
[0151] Fig.13 is a diagram illustrating an embodiment of displaying AR information through two definition levels using distance information from the wearable device 100 to an object.
[0152] refer to Fig.13 (a), the user's gaze can be located on the Sign object, and the wearable device 100 can control the focus adjustment element 152 to clearly display the AR information 40 corresponding to the Sign.
[0153] The wearable device 100 may control the focus adjustment element 152 to blurrily display the AR information 41, 42, and 43 corresponding to the object that the user is not looking at.
[0154] Here, the wearable device 100 may control the focus adjustment element 152 to have the same blur level.
[0155] refer to Fig.13(b), the wearable device 100 can display AR information corresponding to the object according to the distance information of each object. Here, the wearable device 100 can display the clarity differently for each AR information. Assume that the user is looking at the Sign. In addition, according to the above reference Fig.12 Assume that the distance to the sign may be 20m, the distance to the bear may be 30m, the distance to the vehicle may be 10m, and the distance to the building may be 100m.
[0156] In this case, the wearable device 100 can identify the distance information of each object and identify the focal difference. The focal difference may refer to the difference between the distance from the wearable device 100 to the position where the user is looking and the distance from the wearable device 100 to the object. The wearable device 100 can control the focus adjustment element 152 using the focal difference.
[0157] Since the user is looking at the Sign, and the Sign is 20m away from the wearable device 100, the standard of the focal length difference may be 20m. The difference in distance between the Sign and the focus is 0m, so the focal length difference of the Sign may be 0. The difference in distance between the bear and the focus is 10m, so the focal length difference of the bear may be 10. In addition, the difference in distance between the vehicle and the focus is -10m, so the focal length difference of the vehicle may be -10. In addition, the difference in distance between the building and the focus is 100m, so the focal length difference of the building may be 100.
[0158] The wearable device 100 can control the focus adjustment element 152 to display AR information with different clarity using the focal length difference. Fig.13 , when the focal length difference is 0, the wearable device 100 can clearly display the AR information corresponding to the corresponding object. When the focal length difference is not 0, the wearable device 100 can Fig.13 The blur level 1 in (b) blurrily displays AR information corresponding to the corresponding object.
[0159] At the same time, the wearable device 100 can display different clarity for each object using the focal length difference, and can display AR information of the object, such as Fig.13 As shown in (a). Specifically, when the refractive index of the focus adjustment element 152 included in the wearable device 100 changes, the focal length of the output light can change. When the focal length changes according to the changed refractive index of the focus adjustment element 152, the user can distinguishably recognize the clarity of the AR information corresponding to the object.
[0160] refer to Fig.13In (a), the wearable device 100 can clearly display the AR information 40 corresponding to the Sign object whose focal length difference is 0. The wearable device 100 can display the AR information 41, 42, and 43 corresponding to the object whose focal length difference is not 0 at a blur level 1.
[0161] at the same time, Fig.14 is a diagram illustrating an embodiment of displaying AR information at various definition levels using distance information from the wearable device 100 to the object. Here, the distance information may refer to the distance from the wearable device 100 to the object.
[0162] As referenced above Fig.13 As described above, the wearable device 100 can obtain the focal length difference of each object. Fig.13 In the example, the sharpness is divided into two types by dividing the objects with a focal length difference of 0 and the objects with a focal length difference of not 0, but the reference Fig.14 , the clarity can be specifically divided by dividing the focal length difference according to specific units.
[0163] For example, Fig.14 As shown in (b), if the focal length difference (absolute value) is 0 to 5, the clarity can be identified as clear, if the focal length difference is 6 to 30, the clarity can be identified as blur level 1, if the focal length difference is 31 to 50, the clarity can be identified as blur level 2, and if the focal length difference is 51 to 100, the clarity can be identified as blur level 3. Here, the higher the level, the blurrier the corresponding clarity. When the focal length difference (absolute value) is greater than 100, the clarity can be identified as "none" and can be set to not display AR information at all. Here, the wearable device 100 can control the focus adjustment element 152 or control the light-emitting element 151 to not display AR information at all.
[0164] Here, the reason for using the focal length difference in absolute value form is that the difference in distance between the focus (gaze) and the object is actually meaningful, and the difference between the negative value of the distance difference and the positive value of the distance difference is usually not distinguished by the user. However, in actual implementation, the clarity can be set differently to distinguish the negative value of the distance difference from the positive value of the distance difference.
[0165] Meanwhile, like a control method according to another embodiment of the present disclosure, if the focal length difference is 100 or more, the wearable device 100 may control not to generate AR information.
[0166] The wearable device 100 can be based on Fig.14 The AR information is displayed at the resolution specified in (b), and this Fig.14 is shown in (a).
[0167] refer to Fig.14In (a), the wearable device 100 can clearly display the AR information 40 corresponding to the sign. The wearable device 100 can display the AR information 41 and 42 corresponding to the bear and the vehicle at blur level 1 with a clarity of blur level 1. In addition, the wearable device 100 can display the AR information 43 corresponding to the building at blur level 3 with a clarity of blur level 3.
[0168] In reference Fig.13 and Fig.14 In the description of , the clarity is specified for the focal length difference based on a specific range, but in actual implementation, the wearable device 100 can divide each clarity according to the value of the focal length difference. For example, when the focal length difference is 1 to 100, the wearable device 100 can identify the clarity as 1 to 100.
[0169] Figures 15 to 18 is a diagram showing the amount of change of the focus adjustment element according to each embodiment.
[0170] Figures 15 to 18 is a diagram showing a change amount of the focus adjustment element according to a moving path of the scanning mirror 153. The moving path of the scanning mirror 153 may refer to a path along which light output by the light emitting element 151 and reflected by the scanning mirror 153 is projected.
[0171] refer to Figures 15 to 18 ,For the convenience of calculation, it is assumed that it takes 1 second to move horizontally or vertically from one square to another. In the actual implementation, the movement from one square to another may not take the same time, and a delay may occur in a specific part.
[0172] refer to Fig.15 (a), the wearable device 100 can control the scanning mirror 153 so that the flow path of the projected light starts from the lower left end, moves to the right, and passes through all parts. In this case, the time taken by the scanning mirror to move the projected light to all parts can be about 168 seconds.
[0173] Although the scanning mirror 153 moves the projection light to all parts, the wearable device 100 can output light through the light emitting element 151 only when AR information is displayed. Therefore, the wearable device 100 can control the laser projector 150 not to actually project light into a part where AR information is not displayed.
[0174] In this case, the focal length difference of the part where the AR information is not displayed can be set to maintain the original focus. This is to minimize the change of the focus adjustment element 152, because the part where the artificial intelligence is not displayed is irrelevant to the change of the focus adjustment element 152.
[0175] Along the basis Fig.15 The amount of change of the focus adjustment element of the flow path of the scanning mirror 153 (a) can be as follows Fig.15 The change in the focus adjustment element can be identified by the change in the focal length difference.
[0176] refer to Fig.15 (b), the focal length difference can change repeatedly between 40 seconds and 100 seconds. For ease of calculation, the time unit has been described as a unit of one second above, but the time unit can actually be implemented as 10 -3 s or 10 -6 s. Therefore, when the focal length difference changes repeatedly during a short period of time, the focus adjustment element 152 must also change during a short period of time. In such a case, the wearable device 100 may have to use a high-speed focus adjustment element 152.
[0177] refer to Fig.16 (a), the starting position of the light projection is the lower left end, and the moving path can reciprocate in the vertical direction to project the light to all parts. Fig.15 As described in (a), it may take 168 seconds to wait for the scanning mirror 153 to complete all movements.
[0178] When based on Fig.16 (a) is used to identify the change amount of the focus adjustment element 152, which can be compared with Fig.16 The same as shown in (b).
[0179] refer to Fig.16 (b), it can be confirmed that the focal length difference changes during a short period of time around 120 seconds, and the wearable device 100 may have to use a high-speed focus adjustment element 152.
[0180] Fig.17 (a) is a diagram showing an embodiment in which the wearable device 100 controls the moving path of the scanning mirror 153 to minimize the amount of change of the focus adjustment element 152. Fig.15 and Fig.16 Unlike in the embodiment of the present invention, the wearable device 100 can control the scanning mirror 153 to project light not to the entire path but only to the required portion.
[0181] In order to minimize the amount of change of the focus adjustment element 152, the wearable device 100 may move the scanning mirror 153 for each area having the same focal difference. In addition, the wearable device 100 may move the scanning mirror 153 from a portion having a small focal difference to a portion having a large focal difference, or from a portion having a large focal difference to a portion having a small focal difference. In this case, the total movement time may be 59 seconds.
[0182] When along Fig.17 When the moving path of (a) controls the scanning mirror 153, the change amount of the focus adjustment element 152 can be Fig.17 The same as in (b). Fig.17 (b), the focal difference may be changed at a specific time. However, since the focal difference is maintained once changed, the focus adjustment element 152 may not be changed repeatedly. For example, Fig.17 In an embodiment of the present invention, the wearable device 100 can use a normal focus adjustment element without using a high-speed focus adjustment element. In this case, the cost of the components of the wearable device 100 can be reduced.
[0183] Fig.18 (a) is a diagram showing an embodiment in which the wearable device 100 controls the scanning mirror 153 to minimize the moving path of the scanning mirror 153. In order to minimize the moving path of the scanning mirror 153, the wearable device 100 may simply consider only the area where the AR information is displayed, without considering the change of the focus adjustment element 152. In this case, the total time required may be 57 seconds.
[0184] When following the Fig.18 When the moving path of (a) controls the scanning mirror 153, the change amount of the focus adjustment element 152 can be Fig.18 The same as in (b). Fig.18 (b), the focal difference can be changed at a specific time. However, since the focal difference is maintained once changed, the focus adjustment element 152 may not be changed repeatedly. In this case, since the focus adjustment element 152 does not change repeatedly during a short period of time, the wearable device 100 can use a normal focus adjustment element 152, which can lead to cost reduction.
[0185] exist Fig.18 In the case of (b), the moving path of the scanning mirror 153 is not controlled in the order from the small focal length difference to the large focal length difference, but the scanning mirror 153 is only controlled to minimize the moving path of the scanning mirror 153. Therefore, the degree of change of the focus adjustment element 152 may be greater than Fig.17 The embodiment of is larger.
[0186] The wearable device 100 may control the moving path of the scanning mirror 153 to minimize the amount of change of the focus adjustment element 152 , and may control the moving path of the scanning mirror 153 to minimize the moving path of the scanning mirror 153 .
[0187] Meanwhile, in actual implementation, the wearable device 100 may control the scanning mirror 153 to select the optimal method by identifying the two situations described above. The wearable device 100 may apply weights to the main items to automatically select the optimal method. In addition, in actual implementation, the wearable device 100 may provide a UI for guiding the user to select one of the various methods.
[0188] Meanwhile, the wearable device 100 may have to select any one of the two methods. In this case, since the driving speed of the focus adjustment element 152 may be lower than the driving speed of the scanning mirror 153, the method of minimizing the amount of change of the focus adjustment element 152 may be more effective than the method of minimizing the moving path of the scanning mirror 153.
[0189] Fig.19 is a flowchart illustrating a method for controlling a wearable device according to an embodiment.
[0190] At the same time, according to an embodiment of the present disclosure, through a method for controlling a wearable device, a user's gaze can be identified based on sensing data obtained by a sensor (S1905), wherein the wearable device includes: a laser projector, including: a light-emitting element configured to emit laser; a focus adjustment element configured to adjust the focus of the emitted laser; and a scanning mirror configured to control the scanning direction of light with adjusted focus.
[0191] Furthermore, position information of at least one object included in an image obtained by a camera may be recognized ( S1910 ).
[0192] Furthermore, the driving state of the focus adjustment element may be controlled based on the recognized user's gaze and the position information of the object (S1915).
[0193] Furthermore, augmented reality (AR) information related to the object may be provided on the display based on the recognized user's gaze and the position information of the object ( S1920 ).
[0194] Here, step S1915 may include: identifying whether the position of the object corresponds to the gaze of the user by comparing the gaze of the user with the position information of the object; and controlling the driving state of the focus adjustment element based on the identified result.
[0195] In addition, step S1920 may include: based on the position of the object corresponding to the position of the user's gaze, controlling the focal length of the focus adjustment element to become a first focal length, and providing AR information related to the object on the display; and based on the position of the object not corresponding to the position of the user's gaze, controlling the focal length of the focus adjustment element to become a second focal length, and providing AR information related to the object on the display.
[0196] Here, the first focal length may be a unique focal length of the focus adjustment element, and the second focal length may be different from the unique focal length.
[0197] In addition, the method for controlling the wearable device may further include: identifying position information of each object among multiple objects included in the obtained image, wherein the driving state of controlling the focus adjustment element may include: based on providing first AR information related to a first object corresponding to the user's gaze position among the multiple objects, controlling the focal length of the focus adjustment element to become a first focal length; and based on providing second AR information related to a second object not corresponding to the user's gaze position among the multiple objects, controlling the focal length of the focus adjustment element to become a second focal length.
[0198] Step S1915 may include: based on providing the second AR information, controlling the focal length of the focus adjustment element to become a second focal length according to the depth information of the second object.
[0199] Step S1915 may include: obtaining a difference between a depth value included in the depth information of the second object and a depth value corresponding to the recognized gaze of the user; and controlling the focal length of the focus adjustment element to become a second focal length based on the obtained difference in depth values.
[0200] The control method may further include: based on the fact that the acquired image includes a plurality of objects, controlling the scanning mirror by determining at least one of a scanning start position, a scanning direction, and a scanning path based on position information of each of the plurality of objects.
[0201] Controlling the scanning mirror may include determining at least one of a scanning start position, a scanning direction, and a scanning path based on position information of each object of the plurality of objects, thereby changing the focal length of the focus adjustment element in stages.
[0202] Controlling the scanning mirror may include determining at least one of a scanning start position, a scanning direction, and a scanning path based on position information of each of the plurality of objects to minimize a rotation operation of the scanning mirror.
[0203] Fig.19 The method for controlling a wearable device may be provided in a Figure 5 The method may be executed on a wearable device having a configuration of , and may also be executed on a wearable device having other configurations.
[0204] Meanwhile, the method according to the above-described embodiments of the present disclosure may be implemented in the form of an application installable in a wearable device of the related art.
[0205] In addition, the method according to the above-mentioned embodiments of the present disclosure may be simply implemented through software upgrade or hardware upgrade in a wearable device of the related art.
[0206] Furthermore, the above disclosed embodiments may be executed by an embedded server provided in the electronic device or an external server of the electronic device.
[0207] Meanwhile, the method for controlling a wearable device according to an embodiment may be implemented as a program and provided in the wearable device. Specifically, the program including the method for controlling a wearable device may be stored and provided in a non-transitory computer-readable medium.
[0208] In addition, the above-described embodiments can be implemented in a computer or similar device readable recording medium using software, hardware or a combination thereof. According to the implementation of hardware, at least one of the following can be used to implement the embodiments of the present disclosure: an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, and an electronic unit for performing other functions. In some cases, the embodiments described in this specification can be implemented as the processor 140 itself. According to the implementation by software, the embodiments of the processing and functions described in this specification can be implemented as independent software modules. Each software module can perform one or more functions and operations described in this specification.
[0209] The computer instructions for performing processing operations on the wearable device according to the embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. When the computer instructions stored in such a non-transitory computer-readable medium are executed by a processor, the computer instructions may cause a specific machine to perform processing operations on the wearable device according to the embodiments described above.
[0210] A non-transitory computer-readable medium is not a medium that stores data for a short period of time (such as a register, cache, or memory), but may refer to a medium that stores data semi-permanently and can be read by a machine. Specific examples of non-transitory computer-readable media may include CDs, DVDs, hard disk drives, Blu-ray discs, USBs, memory cards, and ROMs.
[0211] Although the present disclosure has been shown and described with reference to the preferred embodiments of the present disclosure, the present disclosure is not limited to the above-mentioned specific embodiments, and it is obvious that various modifications can be made by a person skilled in the art of the present disclosure without departing from the gist of the present disclosure as claimed in the appended claims. In addition, it is intended that such modifications should not be interpreted as being independent of the technical ideas or prospects of the present disclosure.
[0212] Although the present disclosure has been shown and described with reference to the preferred embodiments of the present disclosure, the present disclosure is not limited to the above-mentioned specific embodiments, and it is obvious that various modifications can be made by a person skilled in the art of the present disclosure without departing from the gist of the present disclosure as claimed in the appended claims. In addition, it is intended that such modifications should not be interpreted as being independent of the technical ideas or prospects of the present disclosure.
Claims
1. A wearable device, include: camera; sensor; monitor; Laser projector; as well as The processor is configured as: identifying a user's gaze based on the sensing data obtained by the sensor, identifying position information of at least one object included in an image obtained by the camera, distinguishing between objects at the location of the user's gaze and objects not at the location of the user's gaze, Based on the recognized user's gaze and the position information of the object, the laser projector is controlled to provide augmented reality (AR) information related to the object on the display, wherein the laser projector comprises: a light emitting element configured to emit laser light; a focus adjustment element configured to adjust the focus of the emitted laser light; and a scanning mirror configured to control a scanning direction of the light whose focus is adjusted, and wherein the processor is configured to: controlling a driving state of the focus adjustment element based on the recognized user's gaze and the position information of the object, determining a focal length difference of the object as a difference between a distance from the wearable device to a location of gaze of the user and a distance from the wearable device to the object, controlling the display to display the AR information at a blur level corresponding to the focal length difference, and The focus adjustment element is controlled to vaguely display AR information corresponding to an object that is not at a gaze position of the user.
2. The wearable device according to claim 1, in, The processor is configured to identify whether the position of the object corresponds to the gaze of the user by comparing the gaze of the user with the position information of the object, and control a driving state of the focus adjustment element based on the identified result.
3. The wearable device according to claim 2, in, The processor is configured to: Based on the position of the object corresponding to the gaze of the user, providing AR information related to the object on the display by controlling the focal length of the focus adjustment element to become a first focal length; as well as Based on the fact that the position of the object does not correspond to the gaze of the user, AR information related to the object is provided on the display by controlling the focal length of the focus adjustment element to become a second focal length.
4. The wearable device according to claim 3, in, The first focal length is a unique focal length of the focus adjustment element, and Wherein, the second focal length is different from the unique focal length.
5. The wearable device according to claim 1, in, The processor is configured to: identifying position information of each of a plurality of objects included in the obtained image; controlling the focal length of the focus adjustment element to become a first focal length based on providing first AR information related to a first object corresponding to the gaze of the user among the plurality of objects; as well as Based on the provision of second AR information on a second object among the plurality of objects that does not correspond to the gaze of the user, the focal length of the focus adjustment element is controlled to become a second focal length.
6. The wearable device according to claim 5, in, The processor is configured to: based on providing the second AR information, control the focal length of the focus adjustment element to become the second focal length according to the depth information of the second object.
7. The wearable device according to claim 6, in, The processor is configured to: obtaining a difference between a depth value included in the depth information of the second object and a depth value corresponding to the identified gaze of the user; as well as Based on the difference between the obtained depth values, the focal length of the focus adjustment element is controlled to become the second focal length.
8. The wearable device according to claim 1, in, The processor is configured to control the scanning mirror by determining at least one of a scanning start position, a scanning direction, and a scanning path based on position information of each of the multiple objects based on the fact that the acquired image includes multiple objects.
9. The wearable device according to claim 8, in, The processor is configured to determine at least one of the scanning start position, the scanning direction, and the scanning path based on the position information of each object among the plurality of objects, so that the focal length of the focus adjustment element changes in stages.
10. The wearable device according to claim 8, in, The processor is configured to determine at least one of the scanning start position, the scanning direction, and the scanning path based on position information of each of the plurality of objects to minimize a rotation operation of the scanning mirror.
11. A method for controlling a wearable device comprising a laser projector, the laser projector include: a light emitting element configured to emit laser light; a focus adjustment element configured to adjust the focus of the emitted laser light; and a scanning mirror configured to control a scanning direction of the light whose focus is adjusted, the method comprising: identifying a gaze of a user based on sensing data obtained by a sensor; identifying position information of at least one object included in an image obtained by a camera; distinguishing between an object at a location of the user's gaze and an object not at a location of the user's gaze; controlling a driving state of the focus adjustment element based on the recognized user's gaze and the position information of the object; and Based on the recognized user's gaze and the position information of the object, providing augmented reality (AR) information related to the object on a display, Wherein providing the AR information on the display comprises: determining a focal length difference of the object as a difference between a distance from the wearable device to a location of gaze of the user and a distance from the wearable device to the object, controlling the display to display the AR information at a blur level corresponding to the focal length difference, and The focus adjustment element is controlled to vaguely display AR information corresponding to an object that is not at a gaze position of the user.
12. The method according to claim 11, in, Controlling the driving state of the focus adjustment element includes: identifying whether the position of the object corresponds to the user's gaze by comparing the user's gaze with the position information of the object, and controlling the driving state of the focus adjustment element based on the identified result.
13. The method according to claim 12, in, Providing AR information on the display includes: Based on the position of the object corresponding to the gaze of the user, providing AR information related to the object on the display by controlling the focal length of the focus adjustment element to become a first focal length; and Based on the fact that the position of the object does not correspond to the gaze of the user, AR information related to the object is provided on the display by controlling the focal length of the focus adjustment element to become a second focal length.
14. The method according to claim 13, in, The first focal length is a unique focal length of the focus adjustment element, and Wherein, the second focal length is different from the unique focal length.
15. The method according to claim 11, further comprising: include: identifying position information of each of a plurality of objects included in the obtained image, Wherein, controlling the driving state of the focus adjustment element includes: controlling the focal length of the focus adjustment element to become a first focal length based on providing first AR information about a first object corresponding to the gaze of the user among the plurality of objects; and Based on the provision of second AR information on a second object among the plurality of objects that does not correspond to the gaze of the user, the focal length of the focus adjustment element is controlled to become a second focal length.
