Electronic device and method for providing notification information

Abstract

The wearable device may include a camera, a display, and a processor. The processor may be configured to identify, based on the physical capability information of the user, one reference level corresponding to the agility of the user from a plurality of reference levels indicating the agility. The processor may be configured to identify at least one first visual object within the environment from an image acquired via the camera and representing the environment surrounding the wearable device. The processor may be configured to determine a risk level for the at least one first visual object. The processor may be configured to identify, from the at least one first visual object, at least one second visual object whose risk level has been determined to be higher than a reference level. The processor may be configured to display the at least one second visual object with a visual object for the at least one second visual object.

Description

Technical Field

[0001] The following description relates to an electronic device and method for providing notification information. Background Art

[0002] In an augmented reality (AR) environment, an electronic device can provide a user with a more diverse experience by processing information generated based on information obtained through an external device, etc. in a real environment. At this time, the user can provide a response to an image displayed through a display of the electronic device, and the electronic device can obtain user information. Summary of the invention

[0003] A wearable device is provided. The wearable device may include a camera. The wearable device may include a display. The wearable device may include a processor. The processor may be configured to identify a reference level corresponding to the agility of a user from among a plurality of reference levels indicating agility based on physical ability information of the user. The processor may be configured to identify at least one first visual object in the environment from an image obtained by the camera representing the environment around the wearable device. The processor may be configured to determine a risk level of the at least one first visual object. The processor may be configured to identify at least one second visual object whose risk level is determined to be higher than the reference level in the at least one first visual object. The processor may be configured to display the at least one second visual object and a visual object for the at least one second visual object.

[0004] A method performed in a wearable device is provided. The method may include identifying a reference level corresponding to the agility of a user from among a plurality of reference levels indicating agility based on physical ability information of the user. The method may include identifying at least one first visual object in an environment obtained through a camera of the wearable device and representing the environment around the wearable device. The method may include determining a risk level of the at least one first visual object. The method may include identifying at least one second visual object whose risk level is determined to be higher than a reference level among the at least one first visual object. The method may include displaying the at least one second visual object and a visual object for the at least one second visual object.

[0005] A wearable device is provided. The wearable device may include a camera. The wearable device may include a display. The wearable device may include a processor. The processor may be configured to identify a reference level corresponding to the agility of the user among multiple reference levels indicating agility based on the physical ability information of the user. The processor may be configured to identify a visual object corresponding to an external object in the environment from an image obtained by the camera representing the environment around the wearable device. The processor may be configured to identify at least one first visual object of the visual objects based on identifying a first reference level among the multiple reference levels, and display the at least one first visual object and a visual object for the at least one first visual object. The processor may be configured to identify a second visual object including the at least one first visual object among the visual objects based on identifying a second reference level lower than the first reference level among the multiple reference levels, and display the second visual object and a visual object for the second visual object. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A block diagram of an electronic device in a network environment according to various embodiments is shown.

[0007] Figure 2A An example of providing an augmented reality (AR) environment through an electronic device is shown.

[0008] Figure 2B An example of an electronic device providing an augmented reality (AR) environment is shown.

[0009] Figure 2C An example of providing different augmented reality (AR) environments according to user information is shown.

[0010] Figure 3 A flow chart illustrating an example of a method for providing notification information based on user information is shown.

[0011] Figure 4 A flow chart illustrating an example of a method for obtaining user information is shown.

[0012] Figure 5A and Figure 5B An example of obtaining user information through an electronic device is shown.

[0013] FIG. 6A to FIG. 6E An example of obtaining information about surrounding environment through an electronic device is shown.

[0014] Figure 7 An example of providing notification information for each object according to user information is shown.

[0015] Figure 8A flow chart illustrating an example of identifying a risk level of an object based on a state of a surrounding environment of an electronic device is shown. Specific embodiments

[0017] The terms used in the present disclosure are only used to describe specific embodiments and may not be intended to limit the scope of another embodiment. Singular expressions may include plural expressions unless clearly meant differently in the context. The terms used herein (including technical or scientific terms) may have the same meanings as those generally understood by those of ordinary skill in the art described in the present disclosure. The terms defined in the general dictionary of the terms used in the present disclosure may be interpreted with the same or similar meanings as the contextual meanings of the related art, and unless clearly defined in the present disclosure, are not interpreted with ideal or overly formal meanings. In some cases, even the terms defined in the present disclosure cannot be interpreted as excluding embodiments of the present disclosure.

[0018] In various embodiments of the present disclosure described below, a hardware method is described as an example. However, since various embodiments of the present disclosure include techniques using both hardware and software, various embodiments of the present disclosure do not exclude a software-based method.

[0019] For the convenience of description, terms related to the configuration of the device (e.g., processor, camera, display, module, etc.), terms for operating states (e.g., steps, operations, processes), terms related to signals (e.g., signals, information, etc.), and terms related to data (e.g., parameters, values, etc.) used in the following description are shown. Therefore, the present disclosure is not limited to the terms described below, and other terms with equivalent technical meanings may be used.

[0020] In addition, in the present disclosure, in order to determine whether a specific condition is satisfied or achieved, an expression of greater than or less than may be used, but this is merely a description for expressing an example, and does not exclude a description of greater than or equal to, or less than or equal to. A condition described as "greater than or equal to" may be replaced by "greater than", a condition described as "less than or equal to" may be replaced by "less than", and a condition described as "greater than or equal to and less than" may be replaced by "greater than and less than or equal to". In addition, hereinafter, "A" to "B" refers to at least one of the elements from A (including A) to B (including B).

[0021] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments.

[0022] Reference Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, and / or an antenna module 197. In some embodiments, at least one of the above components (e.g., the connection terminal 178) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some components (eg, sensor module 176 , camera module 180 , or antenna module 197 ) may be implemented as a single component (eg, display module 160 ).

[0023] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to an embodiment, as at least a part of the data processing or calculation, the processor 120 may store a command or data received from another component (e.g., sensor module 176 or communication module 190) in the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. For example, when the electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or as part of the main processor 121.

[0024] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (rather than the main processor 121) may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., a camera module 180 or a communication module 190) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated through machine learning. For example, such learning may be performed by the electronic device 101 where the artificial intelligence is executed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the artificial intelligence model may include a software structure in addition to a hardware structure.

[0025] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0026] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0027] The input module 150 may receive commands or data to be used by other components (e.g., the processor 120) of the electronic device 101 from outside (e.g., a user) of the electronic device 101. The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0028] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. Depending on the embodiment, the receiver can be implemented as a separate part from the speaker, or as part of the speaker.

[0029] The display module 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0030] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input module 150, or output sound via the sound output module 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0031] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) outside the electronic device 101, and then generate an electrical signal or a data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0032] The interface 177 may support one or more designated protocols for directly (e.g., wired) or wirelessly connecting the electronic device 101 with an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0033] The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0034] The haptic module 179 may convert the electric signal into mechanical stimulation (eg, vibration or motion) or electric stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electric stimulator.

[0035] The camera module 180 may capture still images or moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0036] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0037] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0038] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multiple components (e.g., multiple chips) separated from each other. The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0039] The wireless communication module 192 can support 5G networks after 4G networks and next-generation communication technologies (e.g., new radio (NR) access technology). NR access technology can support enhanced mobile broadband (eMBB), massive machine type communication (mMTC), or ultra-reliable low-latency communication (URLLC). The wireless communication module 192 can support high-frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 192 can support various technologies for ensuring performance on high-frequency bands, such as, for example, beamforming, massive multiple-input multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 can support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., a second network 199). According to an embodiment, the wireless communication module 192 may support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane delay for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less for a round trip).

[0040] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiation element formed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 197 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). A signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiation element may be additionally formed as a part of the antenna module 197.

[0041] According to various embodiments, the antenna module 197 may form a millimeter wave antenna module. According to an embodiment, the millimeter wave antenna module may include a printed circuit board, a radio frequency integrated circuit (RFIC), and a plurality of antennas (e.g., array antennas), wherein the RFIC is disposed on a first surface (e.g., bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a designated high frequency band (e.g., millimeter wave band), and the plurality of antennas are disposed on a second surface (e.g., top surface or side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the designated high frequency band.

[0042] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transfer signals (e.g., commands or data) therebetween.

[0043] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 or the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be executed in the electronic device 101 may be executed in one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices receiving the request may execute at least part of the requested function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology, for example, may be used. The electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology or IoT-related technologies.

[0044] Augmented reality (AR) is a technology that synthesizes any object or information for a real object so that the synthesized arbitrary object or information looks like an object existing in the original environment. An electronic device used for AR can enhance and provide information based on a real object. For example, an electronic device may include AR glasses to provide information to a user based on a real object. In this case, the electronic device may be referred to as a wearable electronic device.

[0045] In an AR environment, users wearing AR glasses can perform various experiences. In addition, AR glasses can obtain the user's physical ability information. For example, the user's physical ability information may include body information, such as iris information, facial information, hand information, etc. The user's physical ability information can be obtained through sensors such as 2-dimensional (2D) image cameras, 3-dimensional (3D) depth cameras, IR cameras, etc. An electronic device such as AR glasses can perform interaction with the user to obtain the user's physical ability information based on the user's response. The electronic device can provide the user with information generated based on human-friendly interactions such as hand movements, eye movements, facial movements, etc.

[0046] In the case of performing hobbies, driving, or concentrated and repetitive tasks, it is necessary to notify the user of unexpected or dangerous situations. The electronic device can detect dangers that may be imposed on the user based on information related to the user (e.g., the user's physical ability information), and can provide notifications. For example, a vehicle driven by a user can provide guidance by tracking the user's gaze or detecting drowsiness. In addition, medical electronic devices can identify cognitive abilities through interaction with the user, and can provide information for health management. However, the vehicle or medical electronic device does not provide distinguishing information for each user based on the user's physical ability information (or the user's cognitive ability). Hereinafter, in various embodiments of the present disclosure, in order to provide an effective AR experience for each user, an electronic device and method for providing notifications based on the user's physical ability information (hereinafter, a device and method for providing notifications to each user) are proposed.

[0047] Hereinafter, for ease of description, the method is described by a video see-through (VST) method that generates and provides new information about the actual environment displayed by the camera. However, the present disclosure is not limited thereto. Various embodiments of the present disclosure may also be applied to an optical see-through (OST) method that adds and provides information generated in the environment actually seen.

[0048] Figure 2A An example of providing an augmented reality (AR) environment through an electronic device is shown. Figure 2A In the embodiment, an electronic device 101 in the shape of glasses for providing an AR environment (e.g., AR glasses) is exemplarily shown, but the present disclosure is not limited thereto. Various embodiments of the present disclosure may be applied to all electronic devices capable of providing an AR environment. For example, the electronic device may include a mask capable of providing an AR environment, a lens capable of providing an AR environment, and the like. In addition, the electronic device 101 may be worn on the user's body and may be referred to as a wearable device.

[0049] refer to Figure 2A, the electronic device 101 can be used to receive information when the user performs various experiences. For example, the electronic device 101 can provide additional information or notifications based on the user's reaction speed for unexpected or dangerous situations that occur when the user is performing activities such as driving, exercising, etc. In this case, the user's reaction speed can mean the time it takes from the moment a specific situation (e.g., an unexpected situation or a dangerous situation) occurs to the occurrence of the user's response. Taking driving as an example, the object that is dangerous to the user may vary depending on the user's physical ability information (or cognitive ability). The electronic device 101 can provide information that may be a risk factor to users with poor physical ability information faster than to users with good physical ability information. In addition, taking skiing as an example, the electronic device 101 can provide tutorials based on the user's physical ability information.

[0050] According to the above, the information provided by the electronic device 101 and the time of providing the information can vary for each user. The electronic device 101 can provide a user-customized AR environment to the user. For example, the electronic device 101 can adjust the playback frequency of the user interface (UI) provided by the display. The electronic device 101 can adjust the size of the information provided by the display. Therefore, the electronic device 101 can improve the power consumption of the electronic device 101 by providing a user-customized UI / user experience (UX) for adjusting the input range for user response.

[0051] Figure 2B An example of an electronic device providing an augmented reality (AR) environment is shown. Figure 2B In the embodiment, an electronic device 101 in the shape of glasses for providing an AR environment (e.g., AR glasses) is exemplarily shown, but the present disclosure is not limited thereto. Various embodiments of the present disclosure may be applied to all electronic devices capable of providing an AR environment. For example, the electronic device may include a mask capable of providing an AR environment, a lens capable of providing an AR environment, and the like. In addition, the electronic device 101 may be worn on the user's body and may be referred to as a wearable device.

[0052] Reference Figure 2B , electronic devices (e.g. Figure 1 The electronic device 101 may include a memory configured to store instructions (eg, Figure 1 130), at least one display 200, a frame 240 supporting at least one display 200, and a processor (eg, Figure 1 processor 120).

[0053] According to an embodiment, the electronic device 101 may be referred to as a wearable device worn on a part of a user's body. The electronic device 101 may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) combining augmented reality and virtual reality to the user wearing the electronic device 101. For example, the electronic device 101 may display a virtual reality image provided by at least one optical device on at least one display 200 in response to a user-specified gesture obtained by a motion recognition camera.

[0054] According to an embodiment, the memory may store instructions executed by the processor. The instructions stored in the memory may be related to the specified functions of the electronic device 101. For example, the memory may store instructions related to the operation of a photoplethysmography (PPG) sensor, at least one microphone. When the instructions are loaded by the processor, the processor may control the electronic device 101 to perform the operations specified in the instructions.

[0055] According to an embodiment, at least one display 200 may provide visual information to a user. For example, at least one display 200 may include a transparent or translucent lens. At least one display 200 may include a first display 210 and / or a second display 220 spaced apart from the first display 210. For example, the first display 210 and the second display 220 may be respectively disposed at positions corresponding to the left eye and the right eye of the user.

[0056] Reference Figure 2B , at least one display 200 may provide visual information transmitted from external light and other visual information different from the visual information to the user through a lens included in the at least one display 200. For example, in the case of being worn by the user, the at least one display 200 may include a first surface close to the user and a second surface opposite to the first surface, and may have a display area on the first surface. When the user wears the electronic device 101, the external light may be transmitted to the user by being incident on the second surface and transmitted through the first surface. For another example, the at least one display 200 may display an augmented reality image combined with a virtual reality image transmitted by external light provided from at least one optical device on a real screen on the display area on the first surface. The at least one display 200 may include at least one waveguide that diffracts the light transmitted from the at least one optical device and transmits it to the user. The electronic device 101 may analyze an object included in a real image collected by a photographic camera (not shown), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display it on the at least one display 200. The virtual object may include at least one of a text and an image for various information related to the object included in the real image. A user wearing the electronic device 101 may view images displayed on the at least one display 200 .

[0057] According to an embodiment, the frame 240 may have a physical structure that the electronic device 101 can be used to be worn on the user's body. According to an embodiment, the frame 240 may be configured so that when the user wears the electronic device 101, the first display 210 and the second display 220 may be positioned to correspond to the user's left eye and right eye.

[0058] The frame 240 may support at least one display 200. For example, the frame 240 may support the first display 210 and the second display 220 to be positioned at positions corresponding to the left and right eyes of the user.

[0059] Reference Figure 2B , the frame 240 may include an area 230, and when the user wears the electronic device 101, at least a portion of the frame 240 contacts a portion of the user's body in the area 230. For example, the area 230 of the frame 240 that contacts a portion of the user's body may include an area that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that contacts the electronic device 101. According to an embodiment, the frame 240 may include a nose pad 270 that contacts a portion of the user's body. When the electronic device 101 is worn by the user, the nose pad 270 may contact a portion of the user's nose. The frame 240 may include a first temple 250 and a second temple 245, which contact another portion of the user's body that is different from the above-mentioned portions of the user's body.

[0060] For example, the frame 240 may include a first frame 260 surrounding at least a portion of the first display 210, a second frame 265 surrounding at least a portion of the second display 220, a bridge 272 disposed between the first frame 260 and the second frame 265, a first pad 273 disposed along a portion of an edge of the first frame 260 from one end of the bridge 272, a second pad 271 disposed along a portion of an edge of the second frame 265 from the other end of the bridge 272, a first temple 250 extending from the first frame 260 and fixed to a portion of an ear of the wearer, and a second temple 245 extending from the second frame 265 and fixed to a portion of an ear opposite to the above-mentioned ear. The first pad 273 and the second pad 271 may contact a portion of a user's nose, and the first temple 250 and the second temple 245 may contact a portion of the user's face and a portion of the user's ear. The temples 250 and 245 may be rotatably connected to the frame by a hinge unit. The first temple 250 may be rotatably connected to the first frame 260 by a first hinge unit 206 disposed between the first frame 260 and the first temple 250. The second temple 245 may be rotatably connected to the second frame 265 by a second hinge unit 207 disposed between the second frame 265 and the second temple 245.

[0061] According to an embodiment, the processor may refer to the above-mentioned processor. In an embodiment, the processor may be mounted on the frame 240 included in the electronic device 101. The processor may control the operation of the electronic device 101 by running instructions stored in the memory. For example, the processor may control at least one display 200.

[0062] According to an embodiment, the electronic device 101 may include electronic components that perform various functions. For example, the electronic components may include a battery module, an antenna module, at least one optical device, a sound output module, a light emitting module, and / or a printed circuit board. Various electronic components may be arranged in the frame 240 to perform designated functions.

[0063] According to an embodiment, at least one optical device may project a virtual object onto at least one display 200 so as to provide various image information to the user. For example, the at least one optical device may be a projector. The at least one optical device may be disposed adjacent to the at least one display 200, or may be provided as a part of the at least one display 200. According to an embodiment, the electronic device 101 may include a first optical device corresponding to the first display 210 and a second optical device corresponding to the second display 220. For example, the at least one optical device may include a first optical device disposed at an edge of the first display 210 and a second optical device disposed at an edge of the second display 220. The first optical device may transmit light to a first waveguide disposed on the first display 210, and the second optical device may transmit light to a second waveguide disposed on the second display 220.

[0064] In various embodiments, the electronic device 101 may include a camera. The camera may include a photographic camera, an eye tracking camera (ET CAM), and / or a motion recognition camera. The photographic camera, the eye tracking camera (ET CAM), and the motion recognition camera may be provided at different locations on the frame 240 and may perform different functions.

[0065] The photographic camera may capture an actual image or background to be matched with a virtual image in order to realize augmented reality or mixed reality content. The photographic camera may capture an image of a specific object present at a location where the user is viewing, and may provide the image to at least one display 200. At least one display 200 may display an image in which information about an actual image or background including an image of a specific object obtained using a photographic camera and a virtual image provided by at least one optical device overlap. In an embodiment, the photographic camera may be disposed on a bridge disposed between the first frame 260 and the second frame 265.

[0066] By tracking the gaze of the user wearing the electronic device 101, the eye tracking camera (ET CAM) can match the user's gaze with the visual information provided on at least one display 200 to achieve a more realistic augmented reality. For example, when the user looks forward, the electronic device 101 can naturally display environmental information related to the front of the user on at least one display 200 at the user's location. The eye tracking camera (ET CAM) can be configured to capture an image of the user's pupil in order to determine the user's gaze. For example, the eye tracking camera (ET CAM) can receive the gaze detection light reflected from the user's pupil, and can track the user's gaze based on the position and movement of the received gaze detection light. In an embodiment, the eye tracking camera (ET CAM) can be set at a position corresponding to the left and right eyes of the user. For example, the eye tracking camera (ET CAM) can be set in the first frame 260 and / or the second frame 265 to face the direction where the user wearing the electronic device 101 is located.

[0067] The motion recognition camera can provide a specific event to a screen provided on at least one display 200 by recognizing the movement of the whole or part of the user's body (e.g., the user's torso, hand, or face, etc.). The motion recognition camera can obtain a signal corresponding to the gesture by recognizing the user's gesture, and can provide a display corresponding to the signal to at least one display 200. The processor can recognize the signal corresponding to the gesture, and can perform a specified function based on the recognition. In an embodiment, the motion recognition camera can be set on the first frame 260 and / or the second frame 265.

[0068] According to an embodiment, the battery module may supply power to the electronic components of the electronic device 101. The battery module may be referred to as a battery as described above (eg, Figure 1 In an embodiment, the battery module may be provided in the first temple 250 and / or the second temple 245. For example, the battery module may be a plurality of battery modules. The plurality of battery modules may be provided on the first temple 250 and the second temple 245, respectively. In an embodiment, the battery module may be provided at the end of the first temple 250 and / or the second temple 245.

[0069] The antenna module may transmit a signal or power to the outside of the electronic device 101, or may receive a signal or power from the outside. The antenna module may be referred to as the antenna module as described above (eg, Figure 1 In an embodiment, the antenna module may be disposed in the first temple 250 and / or the second temple 245. For example, the antenna module may be disposed on a surface close to the first temple 250 and / or the second temple 245.

[0070] Figure 2C An example of providing different augmented reality (AR) environments according to user information is shown. Here, user information can be defined as the user's physical ability information or the user's recognition ability. Figure 2C In the embodiment, user information refers to the age of the user, but the present disclosure is not limited thereto. User information may include not only the age of the user, but also gender, reaction speed, and physical information (eg, height, arm length, etc.).

[0071] Reference Figure 2C , Figure 2C A first example 280 in which the age of the user of the electronic device 101 is relatively old and a second example 290 in which the age of the user of the electronic device 101 is relatively young are shown. Here, the relatively old age of the user may mean that the level of the user's physical ability information is low. In addition, the relatively young age of the user may mean that the level of the user's physical ability information is high.

[0072] Referring to the first example 280 and the second example 290, the electronic device 101 may provide information that may be a risk factor to a user having low-level physical ability information faster than to a user having high-level physical ability information. In the case where the user's recognition ability (e.g., recognition speed, reaction speed) is low as in the first example 280, the electronic device 101 may provide information faster than in the case where the user's recognition ability is high as in the second example 290. In addition, the electronic device 101 may provide more specific information to the user of the first example 280 than to the user of the second example 290. The electronic device 101 may change the information provided for each user and the time for providing the information.

[0073] According to an embodiment, the electronic device 101 may provide a user-customized AR environment to the user. For example, the electronic device 101 may adjust the playback frequency of a user interface (UI) provided through a display. The electronic device 101 may adjust the size of information provided through a display. In addition, the electronic device 101 may improve the power consumption of the electronic device 101 by providing a user-customized UI / user experience (UX) for adjusting an input range in response to a user.

[0074] Figure 3 A flowchart illustrating an example of a method for providing notification information based on user information is shown. Here, the method for providing notification information may be Figure 1 The electronic device 101 is executed. In addition, the user may refer to a person who wears and uses the electronic device 101.

[0075] Reference Figure 3In operation 300, the electronic device may obtain physical ability information of a user. Here, the physical ability information of the user may be referred to as the recognition ability of the user. The physical ability information of the user and the recognition ability of the user may include information for recognizing the agility of the user.

[0076] According to an embodiment, the user's physical ability information may include the user's reaction speed or the movable range of the user's body. For example, the user's reaction speed may include the reaction speed of the user's eyeballs (e.g., irises, etc.) or the reaction speed of the user's body (e.g., fingers, palms, arms, etc.). The movable range of the user's body may mean the moving range of the user's eyeballs (e.g., irises, etc.) or the moving range of the user's body (e.g., fingers, palms, arms, etc.). Here, the moving range may mean the range in the environment displayed to the user by the electronic device worn by the user. According to an embodiment, the electronic device may obtain the user's physical ability information through interaction with the user. In the following, Figures 4 to 5B The specific details related to this are described in.

[0077] In operation 305, the electronic device may identify one of the multiple reference levels based on the user's physical ability information. Here, one reference level may be a reference level for the user, and multiple reference levels may mean dividing the user's physical ability information into multiple levels in order to distinguish the user. For example, multiple reference levels may include level 1 to level N (where N is an integer greater than 1). The lower the reference level, the lower the level of the user's physical ability information may be, and the higher the reference level, the higher the level of the user's physical ability information may be. This is just an example, and the lower the reference level, the higher the level of the user's physical ability information may be. Here, the N value can be determined according to the AR environment provided by the electronic device. For example, in the case of an AR environment that is sensitive to the user, N can be set to a large value, and the electronic device can distinguish the user more specifically. On the contrary, in the case of an AR environment that is not sensitive to the user, N can be set to a low value, and the electronic device can distinguish the user more approximately.

[0078] According to an embodiment, the electronic device may identify a reference level of the physical ability information of the current user from among a plurality of predetermined reference levels based on the obtained physical ability information. The electronic device may identify a reference level for each physical ability information. For example, a reference level for the reaction speed of the user's eyeballs may be identified as 2, and a reference level for the reaction speed of the user's body may be identified as 3. In addition, a reference level for the movable range of the user's body may be identified as 4.

[0079] According to an embodiment, the electronic device may identify a representative reference level of the user's physical ability information based on a representative value of the reference level for each piece of physical ability information. Here, the representative reference level may be an identified reference level among a plurality of reference levels. The representative value may include a maximum value, a minimum value, an average value, and an intermediate value. For example, the electronic device may identify the minimum reference level (i.e., the lowest physical ability information) as the user's representative reference level. In addition, in the above example, the electronic device may identify 3 as the average value of the representative reference level (when the reference level for eyeball reaction speed is 2, the reference level for body reaction speed is 3, and the reference level for the movable range of the body is 4).

[0080] According to an embodiment, the electronic device may store a reference level or a representative reference level identified for each physical ability information of the user. For example, the electronic device may store a reference level or a representative reference level obtained by measurement and calculation of a processor in a memory.

[0081] In operation 310, the electronic device may recognize a visual object of a real-time image. The electronic device may display the surrounding environment (eg, space) of the electronic device to the user as an image through a display of the electronic device. The image of the surrounding environment may mean a real-time image. Figure 3 In the present invention, for the convenience of description, it is assumed that the electronic device is a video see-through (VST), and the real-time image may represent a virtual image of the surrounding environment. However, the embodiments of the present disclosure are not limited thereto. For example, in the case where the electronic device is an optical see-through (OST), the real-time image may represent a virtual image of the surrounding environment through a display of the electronic device (e.g., Figure 2B The visual object may represent an object shown in the real-time image. In other words, the visual object may include a virtual object contained in the virtual image or an actual object contained in the actual image.

[0082] According to an embodiment, the electronic device may recognize an object in the surrounding environment based on an input signal. For example, the input signal may include input from a red, green, and blue (RGB) camera, an IR camera, a depth camera, and a stereo camera. The electronic device may analyze and recognize how to configure the surrounding environment in three-dimensional (3D) coordinates based on the input signal. FIG. 6A to FIG. 6E The specific details of the surrounding environment information based on the input signal analysis are described in Figure 3 , operation 310 is shown as being sequentially performed relative to operation 305, but the present disclosure is not limited thereto. Operation 310 may also be performed before, after, or between operation 300 of obtaining the user's physical ability information and operation 305 of identifying the reference level.

[0083] In operation 315, the electronic device may determine a risk level for each of the visual objects. The visual object may be the visual object identified in operation 310. Here, the risk level may be one of a plurality of risk levels. For example, the risk level may include levels from 1 to M (M is an integer greater than 1). The lower the risk level, the lower the risk of the object may be, and the higher the risk level, the higher the risk of the object may be. This is merely an example, and the lower the risk level, the higher the risk of the object may be. Here, the M value may be determined based on the AR environment provided by the electronic device.

[0084] According to an embodiment, the risk level of a visual object can be determined based on information obtained from an image of the surrounding environment. For example, the risk level is determined based on the configuration of the object in the image of the surrounding environment, the movement of the object, or the state of the space, etc. For example, the risk level of a visual object can be determined based on whether the object is close to a user. In the case where the object is close to the user, the risk level of the object may increase. In addition, the risk level of a visual object can be determined based on the configuration of the object. In the case where the configuration of the object includes a bicycle and a person, the electronic device identifies the object as a cyclist, rather than a person or a bicycle, and can identify the risk level of the object accordingly.

[0085] According to an embodiment, the risk level of a visual object may be determined based on the risk level of the visual object in an image at a previous time for a current image (i.e., a real-time image). For example, in a case where the situation has not changed considering the difference between the image at a previous time and the current image, the risk level of the visual object in the current image may be determined as the risk level of the visual object in the image at a previous time. Even in a case where the situation has changed considering the difference between the image at a previous time and the current image, the risk level of the visual object in the current image may be determined as a high risk level considering the risk level of the visual object in the image at a previous time. Therefore, the electronic device may accurately identify the risk level of an object in an image of the surrounding environment. The accuracy of identifying the risk level of a visual object may be improved.

[0086] In operation 320, the electronic device may identify at least one of the visual objects based on the risk level and the reference level. Here, the risk level may mean the risk level identified in operation 315, and the reference level may mean the reference level identified in operation 305. According to an embodiment, the electronic device may identify at least one visual object by comparing the risk level with the reference level. For example, in the case where the reference level is X, the electronic device may identify at least one visual object having a risk level greater than or equal to X.

[0087] Optionally, in the case where the reference level is X, the electronic device may identify a At least one visual object with a risk level of w. Here, w may refer to a weight value. According to an embodiment, w may be determined according to an AR environment. For example, in the case of an AR environment in which a user participates in extreme sports, a space used for extreme sports may have a lower object risk than a general space, so w may be a value greater than 1. On the other hand, in the case of an AR environment related to an environment unfamiliar to the user, the risk of objects in the unfamiliar environment may be higher than the risk of objects in an environment familiar to the user, so w may be a value lower than 1. Therefore, the electronic device may provide specific information to the user in an unfamiliar space. In addition, in the case of an AR environment related to sports such as basketball, football, etc., objects such as basketball or football may be objects for the AR environment to be provided. Therefore, the electronic device may not identify the risk level of objects such as basketball or football, and may not provide a separate notification for this. According to an embodiment, in the case of an AR environment preset by a user, the electronic device 101 may identify at least one visual object based on a reference level to which a weight value is applied. Optionally, according to the state of the surrounding environment, the electronic device 101 may identify at least one visual object based on a reference level to which a weight value is applied. In Figure 8 The specific details related to this are described in .

[0088] In operation 325, the electronic device may visually highlight and display at least one object that is identified. Here, visually highlighting may mean visually adding information or notifications for at least one object. For example, the information or notifications for at least one object may include the speed of the object, the moving direction of the object, the state of the object, and a notification message for notifying the risk and the degree of risk. According to an embodiment, when at least one object is identified, the electronic device may obtain information for at least one object. In addition, the electronic device may display information for at least one object so that at least one object is visually prominent. Therefore, compared with objects that are not visually prominent, the user can easily identify at least one visually prominent object.

[0089] Despite Figure 3 Although not shown in the figure, the electronic device may provide an image including at least one visually prominent object to the user through a display of the electronic device. Here, providing to the user through the display means displaying an image in which at least one object is highlighted. For example, the electronic device may display a real-time image in which information on at least one object is added to the real-time image of operation 310 through the display.

[0090] According to an embodiment, it may be determined that information for at least one object is displayed through a display based on the user's physical ability information, the risk level of the object, or the state of the space. For example, based on the user's physical ability information, information for at least one object may be enlarged or reduced and displayed. In addition, based on the risk level of the object, information for at least one object may be enlarged or reduced and displayed. In addition, based on the state of the space, the time for displaying information for at least one object may be increased, or the period for displaying information for at least one object may be shortened.

[0091] According to the above, the electronic device can identify a reference level of agility for a user from multiple reference levels based on the user's physical ability information (or cognitive ability). In addition, the electronic device can obtain information about objects from an image of the user's surrounding environment, and can identify the risk level of the object. Based on the identified reference level and risk level, at least one of the objects in the environment can be identified and visually highlighted and displayed. At this time, the electronic device can visually highlight and display at least one object by adding information for at least one object. The electronic device can provide the user with an image in which at least one object is visually highlighted through a display. In other words, the electronic device can provide notification information to each user by identifying the risk level of each object for each user and providing information about some objects. Therefore, the user can experience a more effective AR environment.

[0092] Figure 4 A flow chart illustrating an example of a method for obtaining user information is shown. Figure 4 The flowchart of the method for obtaining user information may refer to Figure 3 Therefore, the method for obtaining user information can be performed by Figure 1 The electronic device 101 is executed. Here, the user information may include the user's physical ability information or the user's recognition ability. The user information may refer to information for identifying the user's agility.

[0093] In operation 400, the electronic device may display arbitrary information through a display. According to an embodiment, the arbitrary information may include information used for interaction between the electronic device and the user. For example, the arbitrary information may refer to a virtual object different from an object included in the image of the surrounding environment or some objects included in the image of the surrounding environment. In addition, the arbitrary information may include an interface displayed when the user first uses the electronic device or when the electronic device is reset and needs to be calibrated according to the user. The arbitrary information may include content about the AR environment that the user experiences through the electronic device. For example, when the user plays a game through the AR environment, the arbitrary information may be a character of the game. Optionally, in the case where the user runs content through the AR environment, the arbitrary information may include system alarms for the run, etc.

[0094] In operation 405, the electronic device may obtain the user's response to the displayed arbitrary information. According to an embodiment, the electronic device may identify and obtain the user's response to the arbitrary information through a sensor or the like. For example, the electronic device may use a sensor to obtain the user's eye or body movement information (e.g., reaction speed or movable range), thereby identifying that this is the user's response to the arbitrary information. In the case where the user's response to the displayed information is identified, the electronic device may store information related to the user's response.

[0095] In operation 410, the electronic device may obtain the user's physical ability information based on the user's response. According to an embodiment, the electronic device may obtain the difference between the time of displaying any information and the time of receiving the user's response through the display. The time from the time of displaying any information to the time of receiving the user's response may be referred to as the reaction speed. For example, the reaction speed of the user's eyeball may be the time consumed from generating any information from the display until the focus of the user's eyeball moves to indicate the any information. In addition, the reaction speed of the user's body may be the time consumed from generating any information from the display until the user's body moves to perform feedback on the any information. Here, feedback may include any information input by the user recognized by the sensor of the electronic device.

[0096] According to an embodiment, the electronic device may obtain information about whether a user's response to any information is displayed through a display. The user's response to any information may indicate the movable range of the user's eyeball or body. For example, the electronic device may identify information that is not within a certain range from the focus of the user's eyeball among any multiple pieces of information displayed through the display. Therefore, the electronic device may identify the range that the user's eyeball can recognize. In addition, the electronic device may identify the movable range of the user's body based on the user's body information and the user's position among any multiple pieces of information displayed through the display.

[0097] Despite Figure 4Although not shown in the figure, the electronic device may identify a reference signal among a plurality of reference levels for the agility of the user based on the obtained physical ability information of the user. In this case, the electronic device may identify a reference level for each physical ability information of the user, or may identify a representative reference level for the reference level of the physical ability information. The representative reference level may be determined based on a representative value of the reference level for the physical ability information.

[0098] According to an embodiment, the electronic device may obtain the user's physical ability information based on a response to any information displayed at a specific time and a response to any information displayed at other times. For example, the electronic device may obtain the user's physical ability information based on a response to an initial interface registered by the user at a first time and a response of the user to AR content at a second time. Therefore, the electronic device can obtain more accurate user physical ability information.

[0099] Figure 5A and Figure 5B An example of obtaining user information through an electronic device is shown. Figure 5A and Figure 5B The electronic device 101 can be understood as Figure 1 The same as the electronic device 101. Here, the user information may include the user's physical ability information or the user's recognition ability. The user information may refer to information for identifying the user's agility.

[0100] The method of obtaining user information by the electronic device 101 may include a manual method and an automatic method. Here, the standard for distinguishing between manual and automatic may vary depending on whether an event is required to obtain user information. For example, manual may mean a situation where an event is required to obtain user information, while automatic may mean a situation where no event is required. Figure 5A 500 shows a situation where the electronic device 101 automatically obtains user information. Figure 5B The situations 530 and 535 in which the electronic device 101 manually obtains the user information are shown.

[0101] exist Figure 5A In the case 500 where user information is automatically obtained in the AR environment, the user can wear the electronic device 101 and can execute content in the AR environment. For example, the user can play a game through the electronic device 101, and the game screen 515 can be displayed through the display of the electronic device 101. The user can perform feedback through the user's body 505 in response to the game screen 515. Alternatively, the user can input a response to the game screen 515 through another device 510 connected to the electronic device 101. As described above, the electronic device 101 can obtain the user's physical ability information by tracking the movement of the user's body or the movement of the user's eyeballs and analyzing the AR content and the actual environment such as the game screen 515.

[0102] Furthermore, in the case where the user executes AR content such as reading or watching a movie through the electronic device 101 , the user's physical ability information may be obtained by tracking responses to alarms required for runtime. Figure 5A An example of may be an example of automatically obtaining user information by obtaining the user's physical ability information based on information displayed while the user is enjoying AR content and a response thereto.

[0103] On the other hand, Figure 5B In the case of manually obtaining user information 530 and 535, the user can wear the electronic device 101 and can obtain user information in the AR environment in response to arbitrary information. For example, the electronic device 101 can display an image to which the first information 531, the second information 532, the third information 533, and the fourth information 534 are added to the user. In order to obtain the user's response, the electronic device 101 can change the third information 533 to a new third information 533-1 displayed by visually highlighting the third information 533. In other words, the electronic device 101 can visually highlight and display the third information 533 as the new third information 533-1. Thereafter, in the case where the user indicates the new third information 533-1, the electronic device 101 can obtain the user's physical ability information by tracking the user's response to the new third information 533-1. In the case of manually obtaining user information 530 and 535, it can be a user calibration process, such as using the electronic device 101 for the first time or resetting it. The case of manually obtaining user information can also include a case where arbitrary information is displayed by the electronic device 101 and the response is measured using the user's eyes or body. At this time, the electronic device 101 can obtain the user's physical ability information based on various input signals. For example, the electronic device 101 can obtain the user's physical ability information based on the input of the RGB camera, the IR camera, the depth camera, and the stereo camera.

[0104] FIG. 6A to FIG. 6E An example of obtaining information about surrounding environment through an electronic device is shown. FIG. 6A to FIG. 6E The electronic device can be understood as Figure 1 The same as the electronic device 101. Here, the user information may include the user's physical ability information or the user's recognition ability. The user information may refer to information for identifying the user's agility.

[0105] refer to FIG. 6A to FIG. 6E, the electronic device can obtain an image of the surrounding environment of the electronic device, and can obtain information about the surrounding environment based on the image. Here, the surrounding environment may mean an area adjacent to the user wearing the electronic device. The image of the surrounding environment can be obtained based on an input signal of a camera included in the electronic device. For example, the input signal may include inputs of an RGB camera, an IR camera, a depth camera, and a stereo camera. The information about the surrounding environment may be information about an object included in a 3D coordinate. In other words, the information about the surrounding environment may include 3D information about the real space, and the electronic device may obtain 3D information about the real space from the image about the surrounding environment.

[0106] refer to Fig. 6A In the first example 600, the electronic device may obtain 2D image information. In the first example 600, the electronic device may obtain a 2D image of a scene viewed by a user through a camera. The 2D image obtained by the electronic device may include a plurality of cars 605 and 606, a bicycle 601, a person 602, a cyclist 603, a truck 607, and a traffic signal 608. The electronic device may identify a plurality of cars 605 and 606, a bicycle 601, a person 602, a cyclist 603, a truck 607, and a traffic signal 608 based on the obtained 2D image and pre-stored information. In particular, the cyclist 603 may be identified based on the bicycle 601, the person 602, and the relative position of the person 602 relative to the bicycle 601.

[0107] refer to Figure 6B In the second example 610, the electronic device may obtain depth information of an object in the image through an image obtained from a camera. In the second example 610, the electronic device may obtain an image of a scene viewed by a user through a camera, and may obtain depth information of each object in the image through a sensor of the electronic device. The electronic device may obtain information about a depth 611 from a user position to a first object, a depth 612 to a second object, and a depth 613 to a third object. The electronic device may calculate a distance from the user to the object based on the information about the depth to each object.

[0108] refer to Figure 6CIn the third example 620, the electronic device can obtain information about the surface of the object in the image by obtaining an image from the camera. In the third example 620, the electronic device can obtain an image of a scene viewed by a user through a camera, and can obtain information about the surface of each of the objects by configuring the objects in the image into multiple patterns. Here, the pattern may include a mesh pattern, a dot pattern, etc. The third example 620 shows an image configured with multiple mesh patterns. The electronic device can identify the surface of each of the objects based on the configuration of the mesh pattern. For example, the electronic device can identify that the top surface of the first object 621 is configured with a circular surface 622. In addition, the electronic device can identify that the top surface of the second object 623 is configured with a rectangular surface 624.

[0109] refer to Fig.6D In the fourth example 630, 633 and 636, the electronic device can obtain the coordinate information about the 3D position of the object in the image through the image obtained from the camera. In the fourth examples 630, 633 and 636, the electronic device can obtain the image of the scene viewed by the user through the camera, and can identify the object in the image and the 3D position coordinates of the object. Here, the 3D position coordinates of each object can be obtained by adjusting the object to be positioned in the virtual 3D frame (or hexahedron). In the fourth example 630, the electronic device can obtain the 3D position coordinates of the chair 631 through the virtual 3D frame 632. In addition, in the fourth example 633, the electronic device can obtain the 3D position coordinates of the bicycle 634 through the virtual 3D frame 635. In the fourth example 636, the electronic device can obtain the 3D position coordinates of the long table 638 through the virtual 3D frame 637, and obtain the 3D position coordinates of the round table 640 through the virtual 3D frame 639. The fourth examples 630, 633 and 636 can be referred to as 3D object detection technology.

[0110] refer to Fig. 6E In the fifth example 650, the electronic device can obtain information about the object in the image by obtaining the image from the camera. In the fifth example 650, the electronic device can obtain an image of the scene viewed by the user through the camera, and can obtain information about the object by configuring the object in the image into a 3D model. Specifically, the electronic device can identify the object by classifying, detecting (or locating), and segmenting the object in the image. The electronic device can identify the object by dividing the image into 3D pixels and classifying the categories according to the number of 3D pixels. In the fifth example 650, the electronic device can identify multiple chairs 651, tables 653, floors 654, and doors 655. The fifth example 650 can be called a 3D semantic segmentation technology.

[0111] refer to FIG. 6A to FIG. 6E, the electronic device can obtain information about objects in the surrounding environment based on the image displayed by the camera. Here, the information about objects in the surrounding environment can be referred to as information about the image. The electronic device can obtain the location of each user and information about the object based on the information about the image. Here, the speed of the object, the distance from the object to the user, and the configuration information of the object can be obtained for each object. Here, the configuration information of the object can include an object configured by multiple objects (e.g. Fig. 6A The first example 600 of the cyclist 603) and by Fig. 6E The fifth example 650 classifies the categories to identify objects 651 to 655 .

[0112] In addition, regarding FIG. 6A to FIG. 6E The information of each image shown in is described as a separate example, but can also be obtained in association with each other. For example, the electronic device can obtain information about the images of the second example 610, the third example 620, the fourth example 630, 633, 636, and the fifth example 650 based on the image of the first example 600.

[0113] Figure 7 An example of providing notification information for each object according to user information is shown. Figure 7 The electronic device can be understood as Figure 1 The same as the electronic device 101 of the electronic device. Here, the object may refer to an object in an image of the surrounding environment viewed by a camera of the electronic device. In addition, the notification information may mean additional information generated based on the risk level of the user. The risk level may indicate the degree of risk of the object to the user of the electronic device.

[0114] Reference Figure 7 , the electronic device can obtain an image 700 of the surrounding environment viewed by the user. Referring to image 700, the electronic device can provide various information to the user according to the AR environment. Assume that a user wearing AR glasses and driving is an example. In image 700, the current speed 702, the moving distance and direction 704 and 706, the information about the building 708, and the moving path 740 can be provided to the user through the display of the electronic device.

[0115] Referring to image 700, a first object 710, a second object 720, and a third object 730 may exist in the user's surrounding environment. According to an embodiment, the electronic device may identify the objects in image 700 based on the analysis of image 700. For example, the electronic device may identify the first object 710 as a person riding a bicycle. The electronic device may identify the second object 720 as a pedestrian. The electronic device may identify the third object 730 as a car.

[0116] Referring to the image 700, the electronic device may identify the risk level for each object. For example, the electronic device may identify the risk level 711 of the first object 710 as 1 and the risk level 721 of the second object 720 as 3. According to an embodiment, the electronic device may identify at least one object by comparing the reference level and the risk level of the user. For example, as shown in the image 700, in the case where the reference level of the user is 1, the first object 710 and the second object 720 may be identified. Therefore, the electronic device may provide additional information about the first object 710 and the second object 720. For example, the electronic device may provide the user with information that it is riding a bicycle as additional information 712 for the first object 710. In addition, the electronic device may provide the user with information that a person is moving on foot as additional information 722 of the second object 720. In addition, the electronic device may provide additional information 723 that the second object 720 moves to the right of the user. If different from the image 700, in the case where the reference level of the user is 2, only the second object 720 may be identified. Therefore, the electronic device may provide additional information about the second object 720. For example, the electronic device may provide the user with information that a person is moving on foot as the additional information 722 for the second object 720. In addition, the electronic device may provide additional information 723 that the second object 720 is moving to the right side of the user.

[0117] Despite Figure 7 Although not shown in the figure, the electronic device may provide the risk notification information to the user through a phrase. For example, the electronic device may display and provide a phrase such as “Notification: Possible contact with external objects” through a partial area of ​​the image 700.

[0118] Figure 8 A flowchart illustrating an example of identifying the risk level of an object based on the state of the surrounding environment of an electronic device is shown. Figure 1 The electronic device 101 of the embodiment of the present invention performs a method for identifying a risk level of an object based on the state of the surrounding environment. Here, the object may refer to an object in an image of the surrounding environment viewed by a camera of the electronic device. The risk level may indicate the degree of risk of the object to the user of the electronic device.

[0119] Reference Figure 8 In operation 800, the electronic device may calculate a representative value of the risk level of the visual object. The representative value may include a maximum value, a minimum value, an average value, and a median value. Assume that the risk levels of three objects in the image are 1, 2, and 3. For example, the electronic device may calculate an average value of 2 of the risk levels of the objects as a representative value. Alternatively, the electronic device may calculate a minimum value of 1 among the risk levels of the objects as a representative value.

[0120] In operation 805, the electronic device may identify whether the state of the surrounding environment is a first state. According to an embodiment, the electronic device may identify whether the state of the surrounding environment is a first state by comparing a threshold value with a representative value of a risk level for a visual object. The first state may refer to an unusual state, such as an environment for extreme sports. The threshold value may be determined based on the user's physical ability information. For example, in the case where the user's physical ability information is low, the threshold value may be determined to be low. In addition, the threshold value may be preset by the user. For example, the user may preset the electronic device before wearing the electronic device. In the case where the state of the surrounding environment is identified as a first state, the electronic device may perform step 810. In the case where the state of the surrounding environment is identified as a state different from the first state (e.g., a second state), the electronic device may perform step 815.

[0121] In step 810, when the state of the surrounding environment is the first state, the electronic device may identify a new reference level obtained by applying the weight value to the reference level. Assume that the electronic device identifies the reference level for the user's physical ability information as N as an example. When the state of the surrounding environment is the first state, the electronic device may identify the new reference level for the user's physical ability information as N. w. w may refer to a weight value. According to an embodiment, w may be determined according to an AR environment. For example, in the case of an AR environment in which a user participates in extreme sports, a space used for extreme sports may have a lower object risk than a general space, and thus w may be a value greater than 1. On the other hand, in the case of an AR environment related to an environment with which the user is unfamiliar, the risk of objects in the unfamiliar environment may be higher than the risk of objects in an environment with which the user is familiar, and thus w may be a value lower than 1. Therefore, the electronic device may provide specific information to a user in an unfamiliar space. In this case, in the case of an AR environment related to sports such as basketball, football, and the like, objects such as basketballs or footballs may be objects for the AR environment to be provided. Therefore, the electronic device may identify the state of the space by excluding the risk level for objects such as basketballs or footballs. In addition, the electronic device may not provide separate notifications for objects such as basketballs or footballs.

[0122] In step 815, the electronic device may identify at least one of the visual objects based on the identified reference level, and may display the visual object through a display. For example, when the electronic device identifies that the surrounding environment is in the second state, at least one visual object may be identified and displayed based on the reference level to which the weight value is not applied and the risk level for each object.

[0123] exist Figure 8, an embodiment of determining the state of the surrounding environment based on the risk level of the object in the image of the surrounding environment of the electronic device and identifying at least one object to which additional information is to be provided among the objects by applying a reference level of a weight value is described. However, as described above, in an AR environment preset by a user, at least one object may be identified based on the reference level to which the weight value is applied, regardless of the risk level of the object in the image.

[0124] refer to Figures 1 to 8 According to an embodiment of the present disclosure, the apparatus and method for providing notifications to each user can provide additional information about some objects based on the user's physical ability information indicating the user's agility and the risk of objects in the surrounding environment. In other words, according to an embodiment of the present disclosure, the apparatus and method for providing notifications to each user can provide a customized AR environment for each user based on user information.

[0125] According to various embodiments, a wearable device (101) includes a camera (180). The wearable device (101) includes a display (160). The wearable device (101) includes a processor (120). The processor (120) is configured to identify (305) a reference level corresponding to the agility of a user among a plurality of reference levels indicating agility based on physical ability information of the user. The processor (120) is configured to identify (310) at least one first visual object in the environment from an image obtained by the camera (180) representing the environment around the wearable device (101). The processor (120) is configured to determine (315) a risk level of the at least one first visual object. The processor (120) is configured to identify (320) at least one second visual object whose risk level is determined to be higher than the reference level among the at least one first visual object. The processor (120) is configured to display (325) at least one second visual object and a visual object for the at least one second visual object.

[0126] In an embodiment, the processor (120) is configured to display a virtual object or interface for initial setup of the wearable device (101) through the display (160). The processor (120) is configured to receive a user's response to the virtual object or interface. The user's physical ability information is determined based on the user's response.

[0127] In an embodiment, the user's physical ability information includes at least one of the user's eye reaction speed, the user's body reaction speed, a range recognizable by the user's eyes in an image, or the user's body movement range in an environment.

[0128] In an embodiment, the processor (120) is further configured to obtain information for the image. The information for the image includes at least one of two-dimensional (2D) information of the image, information about the depth of the environment, information for configuring a unit of the environment, or 3D position information of at least one first visual object.

[0129] In an embodiment, the processor (120) is further configured to obtain information for each of the at least one first visual object and position information of the user based on the information for the image. The information for each visual object includes at least one of configuration information of the visual object, a distance from the visual object to the user, or a speed of the visual object.

[0130] In an embodiment, a risk level is determined for each of the at least one first visual object based on the location information of the user and the information for each visual object.

[0131] In an embodiment, the processor (120) is further configured to identify the state of the environment as a first state in response to an average value of the risk level being greater than or equal to a threshold value. The processor (120) is further configured to identify the state of the environment as a second state in response to an average value of the risk level being less than a threshold value. In response to the state of the environment being the first state, at least one second visual object has a risk level higher than a reference level to which a weight value is applied.

[0132] According to various embodiments, a method performed in a wearable device (101) includes: based on physical ability information of the user, identifying (305) a reference level corresponding to the agility of the user from a plurality of reference levels indicating agility. The method includes identifying (310) at least one first visual object in the environment from an image obtained by a camera (180) of the wearable device (101) representing the environment around the wearable device (101). The method includes determining (315) a risk level of the at least one first visual object. The method includes identifying (320) at least one second visual object whose risk level is determined to be higher than the reference level from among the at least one first visual object. The method includes displaying (325) at least one second visual object and a visual object for the at least one second visual object.

[0133] In an embodiment, the method includes displaying a virtual object or interface for initial setup of the wearable device (101) through a display (160) of the wearable device (101). The method includes receiving a user's response to the virtual object or interface. Based on the response, physical ability information of the user is determined.

[0134] In an embodiment, the user's physical ability information includes at least one of the user's eye reaction speed, the user's body reaction speed, a range recognizable by the user's eyes in an image, or the user's body movement range in an environment.

[0135] In an embodiment, the method comprises obtaining information for an image. The information for the image comprises at least one of two-dimensional (2D) information of the image, information for depth of the environment, information for configuring a unit of the environment, or 3D position information of at least one first visual object.

[0136] In an embodiment, the method includes obtaining information for each of at least one first visual object and position information of the user based on the information for the image. The information for each visual object includes at least one of configuration information of the visual object, a distance from the visual object to the user, or a speed of the visual object.

[0137] In an embodiment, a risk level is determined for each of the at least one first visual object based on the location information of the user and the information for each visual object.

[0138] In an embodiment, the method includes identifying the state of the environment as a first state in response to an average value of the risk level being greater than or equal to a threshold. The method includes identifying the state of the environment as a second state in response to the average value of the risk level being less than a threshold. In response to the state of the environment being the first state, at least one second visual object has a risk level that is higher than a reference level to which a weight value is applied.

[0139] According to various embodiments, a wearable device (101) includes a camera (180). The wearable device (101) includes a display (160). The wearable device (101) includes a processor (120). The processor (120) is configured to identify (305) a reference level corresponding to agility of a user from among a plurality of reference levels indicating agility based on physical ability information of the user. The processor (120) is configured to identify (310) a visual object corresponding to an external object in the environment from an image obtained by the camera (180) representing an environment around the wearable device (101). The processor (120) is configured to identify (320, 325) at least one first visual object of the visual objects based on identifying a first reference level from among the plurality of reference levels, and to display (320, 325) at least one visual object and a visual object for at least one visual object. The processor (120) is configured to identify (320, 325) a second visual object including at least one of the first visual objects based on identifying a second reference level lower than the first reference level among the plurality of reference levels, and to display (320, 325) the second visual object and a visual object for the second visual object.

[0140] In an embodiment, the processor (120) is configured to display a virtual object or interface for initial setup of the wearable device (101) through the display (160). The processor (120) is configured to receive a user's response to the virtual object or interface. The user's physical ability information is determined based on the user's response.

[0141] In an embodiment, the user's physical ability information includes at least one of the user's eye reaction speed, the user's body reaction speed, a range recognizable by the user's eyes in an image, or the user's body movement range in an environment.

[0142] In an embodiment, the processor (120) is further configured to obtain information for an image. The processor (120) is further configured to obtain information for each of the visual objects and position information of the user based on the information for the image. The processor (120) is further configured to: wherein the information for the image includes at least one of two-dimensional (2D) information of the image, information for the depth of the environment, information for configuring a unit of the environment, or 3D position information of the visual object. The information for each visual object includes at least one of configuration information of the visual object, a distance from the visual object to the user, or a speed of the visual object.

[0143] In an embodiment, each of the at least one first visual object and the second visual object is identified based on the position information of the user and the information for each visual object.

[0144] In an embodiment, at least one first visual object is identified based on a third reference level obtained by applying a weight value to the first reference level. A second visual object is identified based on a fourth reference level obtained by applying a weight value to the second reference level.

[0145] The electronic device according to various embodiments disclosed in this document may be various types of devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0146] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or alternative forms for the corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “coupled with another element (e.g., the second element)”, “coupled to another element (e.g., the second element)”, “connected with another element (e.g., the second element)”, or “connected to another element (e.g., the second element)” with or without the terms “operably” or “communicatively” being used, it means that the element may be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0147] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0148] The various embodiments described herein may be implemented as software (e.g., program #40) including one or more instructions stored in a storage medium (e.g., internal memory #36 or external memory #38) that can be read by a machine (e.g., electronic device #01). For example, under the control of a processor, a processor (e.g., processor #20) of the machine (e.g., electronic device #01) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0149] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be released in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be downloaded via an application store (e.g., Play Store TM ) the computer program product may be published (e.g., downloaded or uploaded) online, or the computer program product may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, a server of an application store, or a forwarding server).

[0150] According to various embodiments, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities, and some of the multiple entities may be separately arranged in different components. According to various embodiments, one or more of the above-mentioned components or operations may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (e.g., a module or a program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as the corresponding one of the multiple components performing one or more functions before integration. According to various embodiments, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more of the operations may be run or omitted in a different order, or one or more other operations may be added.

Claims

1. A wearable device (101), comprising: camera(180); a display (160); and Processor (120), Wherein, the processor (120) is configured to: identifying (305) a reference level of agility corresponding to the user from among a plurality of reference levels indicating agility based on the user's physical ability information, identifying (310) at least one first visual object in an environment surrounding the wearable device (101) from an image representing the environment surrounding the wearable device (101), the image being acquired by the camera (180), determining (315) a risk level of the at least one first visual object, identifying (320) at least one second visual object whose risk level is determined to be higher than the reference level among the at least one first visual object, and The at least one second visual object and a visual object for the at least one second visual object are displayed (325).

2. The wearable device (101) according to claim 1, in, The processor (120) is configured to: displaying a virtual object or interface for initial setting of the wearable device (101) through the display (160), and receiving a response from the user to the virtual object or the interface, Wherein, the physical ability information of the user is determined based on the response of the user.

3. The wearable device (101) according to any one of claims 1 and 2, in, The physical ability information of the user includes at least one of: the user's eye reaction speed, the user's body reaction speed, a range recognizable by the user's eyes in the image, or the user's body movement range in the environment.

4. The wearable device (101) according to any one of claims 1 to 3, in, The processor (120) is further configured to: obtain information for the image, The information used for the image includes at least one of: two-dimensional 2D information of the image, information about the depth of the environment, information for configuring a unit of the environment, or 3D position information of the at least one first visual object.

5. The wearable device (101) according to claim 4, in, The processor (120) is further configured to: obtain information for each of the at least one first visual object and position information of the user based on the information for the image, The information for each visual object includes: at least one of configuration information of the visual object, a distance from the visual object to the user, or a speed of the visual object.

6. The wearable device (101) according to claim 5, in, The risk level is determined for each of the at least one first visual object based on the location information of the user and information for each visual object.

7. The wearable device (101) according to any one of claims 1 to 6, in, The processor (120) is also configured to: In response to the average value of the risk level being greater than or equal to a threshold, identifying the state of the environment as a first state, and In response to the average of the risk levels being less than the threshold, identifying the state of the environment as a second state, Wherein, in response to the state of the environment being the first state, the at least one second visual object has a risk level higher than the reference level to which the weight value is applied.

8. A method performed in a wearable device (101), comprising: identifying (305) a reference level of agility corresponding to the user from among a plurality of reference levels indicating agility based on the user's physical ability information, identifying (310) at least one first visual object in an environment surrounding the wearable device (101) from an image representing the environment surrounding the wearable device (101), the image being obtained by a camera (180) of the wearable device (101), determining (315) a risk level of the at least one first visual object, identifying (320) at least one second visual object whose risk level is determined to be higher than the reference level among the at least one first visual object, and The at least one second visual object and a visual object for the at least one second visual object are displayed (325).

9. The method according to claim 8, comprising: displaying a virtual object or interface for initial setting of the wearable device (101) through the display (160) of the wearable device (101), and receiving a response from the user to the virtual object or the interface, Wherein, the physical ability information of the user is determined based on the response.

10. The method according to any one of claims 8 and 9, in, The physical ability information of the user includes at least one of: the user's eye reaction speed, the user's body reaction speed, a range recognizable by the user's eyes in the image, or the user's body movement range in the environment.

11. The method according to any one of claims 8 to 10, comprising: obtaining information for the image, The information used for the image includes at least one of: two-dimensional 2D information of the image, information about the depth of the environment, information for configuring a unit of the environment, or 3D position information of the at least one first visual object.

12. The method according to claim 11, comprising: obtaining information for each of the at least one first visual object and position information of the user based on the information for the image; The information for each visual object includes: at least one of configuration information of the visual object, a distance from the visual object to the user, or a speed of the visual object.

13. The method according to claim 12, in, The risk level is determined for each of the at least one first visual object based on the location information of the user and information for each visual object.

14. A method according to any one of claims 8 to 13, comprising: In response to the average value of the risk level being greater than or equal to a threshold, identifying the state of the environment as a first state, and In response to the average of the risk levels being less than the threshold, identifying the state of the environment as a second state, Wherein, in response to the state of the environment being the first state, the at least one second visual object has a risk level higher than the reference level to which the weight value is applied.

15. A wearable device (101), comprising: camera(180); a display (160); and Processor (120), Wherein, the processor (120) is configured to: identifying (305) a reference level of agility corresponding to the user from among a plurality of reference levels indicating agility based on the user's physical ability information, identifying (310) a visual object corresponding to an external object in an image representing an environment surrounding the wearable device (101), the image being acquired by the camera (180), Based on identifying a first reference level among the plurality of reference levels, identifying (320, 325) at least one first visual object among the visual objects, and displaying (320, 325) the at least one first visual object and visual objects for the at least one first visual object, Based on identifying a second reference level among the plurality of reference levels that is lower than the first reference level, identifying (320, 325) a second visual object among the visual objects including the at least one first visual object, and displaying (320, 325) the second visual object and a visual object for the second visual object.

Images