Augmented reality apparatus and method for recognizing object in image

Abstract

An augmented reality apparatus and a method for detecting an object within an image are provided. The method for detecting the object in the image of the augmented reality device comprises the following operations: acquiring a shot image; identifying a gaze of the user; identifying performance information of the augmented reality device and performance information of an external electronic device connected to the augmented reality device; and selecting a device for identifying the object among the augmented reality device and the external electronic device based on the performance information of the augmented reality device and the performance information of the external electronic device; and selecting an artificial intelligence model for identifying the object.

Description

Technical Field

[0001] The present disclosure relates to augmented reality devices and methods for identifying objects in images. Background Art

[0002] An augmented reality device is a device capable of expressing augmented reality, and generally includes not only an augmented reality device in the shape of glasses worn on the user's face, but also a head-mounted display device (HMD) worn on the head, an augmented reality helmet, etc. The augmented reality device has limitations in terms of installation space, and therefore there are limitations in increasing the battery capacity of the augmented reality device. In addition, since the augmented reality device lacks space for installing chips, the augmented reality device can be manufactured with limited computing power and memory size. Therefore, the augmented reality device may need to utilize a device using a high-end neural processing unit (NPU) (such as a smart phone and a server) in order to provide various services based on augmented reality. Summary of the invention

[0003] Solution to the problem

[0004] A first aspect of the present disclosure may provide a method comprising: acquiring a captured image; identifying a user's gaze; identifying performance information of an augmented reality device and performance information of an external electronic device connected to the augmented reality device; selecting a device for identifying an object between the augmented reality device and the external electronic device based on the performance information of the augmented reality device and the performance information of the external electronic device; selecting an artificial intelligence model for identifying an object based on the performance information of the augmented reality device and the performance information of the external electronic device; acquiring a partial image including an object related to the user's gaze from the captured image; and acquiring a recognition result of the object from the partial image by using the selected device and the selected artificial intelligence model.

[0005] In addition, a second aspect of the present disclosure can provide an augmented reality device, which includes: a communication interface, which is configured to communicate with an external electronic device; a camera; a gaze tracking sensor, which is configured to detect a user's gaze; a memory, which stores instructions; and a processor, which is operably connected to the communication interface, the camera, the gaze tracking sensor and the memory, and is configured to execute instructions, wherein the processor executes instructions to: acquire a captured image; control the gaze tracking sensor to identify the user's gaze; identify performance information of the augmented reality device and performance information of an external electronic device connected to the augmented reality device; based on the performance information of the augmented reality device and the performance information of the external electronic device, select a device for identifying an object from the augmented reality device and the external electronic device; based on the performance information of the augmented reality device and the performance information of the external electronic device, select an artificial intelligence model to identify an object; acquire a partial image including an object related to the user's gaze from the captured image; and obtain a recognition result of the object from the partial image by using the selected device and the selected artificial intelligence model.

[0006] In addition, a third aspect of the present disclosure may provide a computer-readable recording medium having recorded therein a program for executing the method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a schematic diagram of a system for enabling an augmented reality device to recognize an object in an image according to an embodiment.

[0008] Figure 2 A case where an augmented reality device recognizes an object in an image according to an embodiment is shown.

[0009] Figure 3 is a flowchart illustrating a method for acquiring a result of object recognition including a partial image of an object corresponding to a gaze direction of a user through an augmented reality device according to an embodiment.

[0010] Figure 4 is a flowchart illustrating a method for identifying a condition for providing a service related to object recognition by an augmented reality device according to an embodiment.

[0011] Figure 5 An example of determining the size of a partial image, a device for object recognition, and an artificial intelligence model for object recognition according to an embodiment by considering a condition for providing a service to be provided to a user is shown.

[0012] Figure 6is a flowchart illustrating a method for recognizing, by an augmented reality device, an object in a partial image by using at least one of the augmented reality device or an external electronic device according to an embodiment.

[0013] Figure 7 is a flowchart illustrating a method for recognizing, by an augmented reality device, an object in a partial image by using at least one of the augmented reality device, an external electronic device, or a server according to an embodiment.

[0014] Figure 8 An example is shown in which the augmented reality device recognizes an object using the augmented reality device or an external electronic device according to a service providing condition according to an embodiment.

[0015] Fig. 9 An example is shown in which an augmented reality device recognizes an object using an augmented reality device or a server according to a service providing condition according to an embodiment.

[0016] Fig.10 An example of recognizing an object by interworking an augmented reality device with an external electronic device and a server according to an embodiment is shown.

[0017] Fig.11 An example of recognizing text in an image through an augmented reality device according to an embodiment is shown.

[0018] Fig.12 An example is shown in which an augmented reality device selects an artificial intelligence model and device for object recognition based on the location of the augmented reality device according to an embodiment.

[0019] Fig.13 An example is shown in which an augmented reality device selects an artificial intelligence model and device for object recognition based on the location of the augmented reality device according to an embodiment.

[0020] Fig.14 An example is shown in which an augmented reality device acquires a partial image based on a sound obtained by the augmented reality device according to an embodiment.

[0021] Fig.15 is a schematic diagram of a system for enabling an augmented reality device to recognize an object in an image when the augmented reality device does not support processing of an artificial intelligence model according to an embodiment.

[0022] Fig.16 is a flowchart illustrating a method in which an augmented reality device obtains a result of object recognition for a partial image including an object corresponding to a gaze direction of a user when the augmented reality device does not support processing of an artificial intelligence model according to an embodiment.

[0023] Fig.17is a flowchart illustrating a method in which an augmented reality device recognizes an object in a partial image by using at least one of an external electronic device and a server when the augmented reality device does not support processing of an artificial intelligence model according to an embodiment.

[0024] Fig.18 is a block diagram of an augmented reality device according to an embodiment.

[0025] Fig.19 is a block diagram of an electronic device in a network environment according to various embodiments.

[0026] Fig. 20 is a flowchart illustrating a method for acquiring a result related to an object corresponding to a user's gaze through an augmented reality device for providing a service according to an embodiment. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily practice them with reference to the accompanying drawings. However, the present disclosure can be implemented in many different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present disclosure in the accompanying drawings, components that are not related to the description are omitted, and similar reference numerals are attached to similar components throughout the specification.

[0028] The terms used in this disclosure have been described as general terms currently used by considering the functions mentioned in this disclosure. However, various other terms may be implied depending on the intentions of those skilled in the art or precedents, the emergence of new technologies, etc. Therefore, the terms used in this disclosure should not be interpreted as just the names of the terms, but should be interpreted based on the meanings of the terms and the content throughout this disclosure.

[0029] In addition, terms such as "first", "second", etc. may be used to describe various elements, but the elements should not be limited by these terms. These terms are used to distinguish one element from another.

[0030] Throughout the specification, when an element is referred to as being “connected to” another element, this includes not only the case of being “directly connected to” another element, but also the case of being “electrically connected to” another element with an intervening element therebetween. In addition, when an element is referred to as “comprising” another element, unless otherwise specified, the element may further include another element rather than excluding the another element.

[0031] The appearances of phrases such as "in an embodiment" in various places in this disclosure are not necessarily all referring to the same embodiment.

[0032] Embodiments of the present disclosure can be expressed as function block configurations and various processing stages. Some or all of these function blocks can be implemented by various numbers of hardware and / or software components that perform specific functions. For example, the function blocks of the present disclosure can be implemented by one or more microprocessors or circuit configurations for predetermined functions. In addition, for example, the function blocks of the present disclosure can be implemented in various programming or scripting languages. Function blocks can be implemented as algorithms running on one or more processors. In addition, the present disclosure can adopt prior art to carry out electronic environment configuration, signal processing and / or data processing. Terms such as "mechanism", "element", "means" and "configuration" can be used extensively and are not limited to mechanical and physical configurations.

[0033] In addition, the connecting lines or connecting members between the elements shown in the drawings are only examples of functional connections and / or physical or circuit connections. In actual devices, the connections between the elements can be represented by various functional connections, physical connections or circuit connections that can be replaced or added.

[0034] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

[0035] In the present disclosure, “augmented reality (AR)” refers to showing a virtual image in a physical environment space of the real world or showing a real object and a virtual image together.

[0036] In addition, "augmented reality device" refers to a device that can express "augmented reality", and augmented reality devices generally include not only glasses-shaped augmented reality devices worn on the user's face, but also head-mounted display devices (HMDs) worn on the head, augmented reality helmets, etc.

[0037] Meanwhile, the "real scene" is a scene of the real world that the user sees through the augmented reality device, and may include real-world objects. In addition, the "virtual image" is an image generated by an optical engine, and may include both static images and dynamic images. Such a virtual image is observed together with the real scene, and may be an image representing information about real objects in the real scene, information about the operation of the augmented reality device, or a control menu.

[0038] Therefore, a typical augmented reality device includes an optical engine for generating a virtual image configured by light generated from a light source, and a waveguide made of a transparent material that guides the virtual image generated by the optical engine to the user's eyes and allows the user to also see the scene of the real world. As described above, the augmented reality device should be able to observe the scene of the real world, so in order to guide the light generated by the optical engine to the user's eyes through the waveguide, an optical element for changing the optical path with substantially straightness is required. In this case, the optical path can be changed by reflection through a reflector or the like, or the optical path can be changed by diffraction through a diffraction element (such as a diffraction optical element (DOE) or a hologram optical element (HOE)), but is not limited thereto.

[0039] According to the functions related to artificial intelligence of the present disclosure, the processor and the memory are operated. The processor may include one or more processors. In this case, the one or more processors may be a general-purpose processor such as a CPU, an AP or a digital signal processor (DSP), a graphics processor such as a GPU or a visual processing unit (VPU), or an artificial intelligence processor such as an NPU. One or more processors perform control to process input data according to predefined operating rules stored in the memory or the artificial intelligence model. Alternatively, when one or more processors are processors dedicated to artificial intelligence, the processor dedicated to artificial intelligence can be designed with a hardware structure dedicated to processing a specific artificial intelligence model.

[0040] The predefined action rules or artificial intelligence models are characterized by being generated by learning. Here, generation by learning means learning a basic artificial intelligence model using multiple learning data through a learning algorithm, thereby generating a predefined action rule or artificial intelligence model configured to perform the desired characteristics (or purposes). This learning can be performed in a device that executes artificial intelligence according to the present disclosure, or can be performed by a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the above examples.

[0041] The artificial intelligence model may be configured by multiple neural network layers. The artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and the neural network operation is performed by the operation between the multiple weight values ​​and the operation results of the previous layer. The multiple weights owned by the multiple neural network layers can be optimized by the learning results of the artificial intelligence model. For example, multiple weights can be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), such as a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN) or a deep Q network, but is not limited to the above examples.

[0042] According to an embodiment of the present disclosure, a method for identifying an object in an image through an augmented reality device may include: acquiring a captured image; identifying a user's gaze; identifying performance information of the augmented reality device and performance information of an external electronic device connected to the augmented reality device; selecting a device for identifying the object from among the augmented reality device and the external electronic device based on the performance information of the augmented reality device and the performance information of the external electronic device; selecting an artificial intelligence model for identifying the object based on the performance information of the augmented reality device and the performance information of the external electronic device; acquiring a partial image including an object related to the user's gaze from the captured image; and acquiring a recognition result of the object from the partial image by using the selected device and the selected artificial intelligence model.

[0043] In addition, the method may also include identifying conditions for providing services to be provided to users, wherein selecting an artificial intelligence model includes selecting an artificial intelligence model for recognizing objects from multiple artificial intelligence models trained to recognize objects based on the conditions for providing services, performance information of the augmented reality device, and performance information of the external electronic device.

[0044] In addition, the partial image may have a size corresponding to the selected artificial intelligence model.

[0045] Additionally, multiple AI models may be configured to process a local image by using different amounts of computation.

[0046] In addition, for object recognition regarding multiple captured images including captured images, at least one of a device for recognizing an object or an artificial intelligence model for recognizing an object is changeable based on a condition for providing a service, and a size of a partial image can change as at least one of the device for recognizing an object and an artificial intelligence model for recognizing an object changes.

[0047] Additionally, multiple AI models may be configured to process a local image by using different amounts of computation.

[0048] In addition, a plurality of artificial intelligence models may be configured such that the number of bits of an output value of an activation function and the number of bits of a weight configured between layers are different from each other.

[0049] Additionally, identifying the conditions for providing the service may include identifying the service providing conditions based on at least one of an attribute of the service to be provided to the user, a type of application executed in connection with recognition of the object, or a voice input of a user requesting recognition of the object.

[0050] In addition, the conditions for providing the service may be determined based on the accuracy and real-time nature of the recognition result of the object.

[0051] Additionally, the method may further include identifying a resolution of an input image configured for the selected artificial intelligence model.

[0052] Additionally, acquiring the partial image may include cropping the partial image from the captured image so that the partial image has the identified resolution.

[0053] In addition, the performance information of the augmented reality device and the performance information of the external electronic device may include specifications of at least one of a neural processing unit (NPU), a central processing unit (CPU), or a graphics processing unit (GPU) capable of executing at least one of a plurality of artificial intelligence models.

[0054] In addition, when the selected device is an augmented reality device, the selected artificial intelligence model includes a first artificial intelligence model stored in the augmented reality device, and obtaining the recognition result of the object may include recognizing the object in the partial image by applying the partial image to the first artificial intelligence model.

[0055] In addition, when the selected device is an external electronic device, the selected artificial intelligence model includes a second artificial intelligence model stored in the external electronic device, and obtaining the recognition result of the object may include sending a partial image to the external electronic device, and receiving from the external electronic device the recognition result of the object obtained by the external electronic device due to applying the partial image to the second artificial intelligence model.

[0056] Additionally, the method further includes identifying a communication state of the augmented reality device, and selecting the device for identifying the object may include selecting the device for identifying the object from among the augmented reality device and the external electronic device based on the communication state.

[0057] Additionally, selecting the artificial intelligence model for recognizing the partial image may include selecting the artificial intelligence model for recognizing the partial image based on a communication state.

[0058] According to an embodiment of the present disclosure, an augmented reality device for identifying an object in an image may include: a communication interface configured to communicate with an external electronic device; a camera; a gaze tracking sensor configured to detect a user's gaze; a memory configured to store instructions; and a processor operably connected to the communication interface, the camera, the gaze tracking sensor and the memory, the processor being configured to execute instructions, wherein the processor executes the instructions to: acquire a captured image; control the gaze tracking sensor to identify the user's gaze; identify performance information of the augmented reality device and performance information of an external electronic device connected to the augmented reality device; select a device for identifying an object from the augmented reality device and the external electronic device based on the performance information of the augmented reality device and the performance information of the external electronic device; select an artificial intelligence model for identifying an object based on the performance information of the augmented reality device and the performance information of the external electronic device; acquire a partial image including an object related to the user's gaze from the captured image; and acquire a recognition result of the object from the partial image by using the selected device and the selected artificial intelligence model.

[0059] Additionally, the processor may be configured to execute instructions to identify conditions for providing a service to be provided to the user.

[0060] In addition, the processor can be configured to execute instructions to select an artificial intelligence model for recognizing an object from among a plurality of artificial intelligence models trained to recognize objects based on conditions for providing services, performance information of the augmented reality device, and performance information of the external electronic device.

[0061] In addition, the partial image has a size corresponding to the selected artificial intelligence model.

[0062] In addition, for object recognition regarding multiple captured images including captured images, at least one of a device for recognizing an object or an artificial intelligence model for recognizing an object is changeable based on a condition for providing a service, and a size of a partial image can change as at least one of the device for recognizing an object and an artificial intelligence model for recognizing an object changes.

[0063] According to an embodiment of the present disclosure, a computer-readable recording medium is provided, in which a program for executing a method for identifying an object in an image is recorded, the program including instructions that, when executed, cause the medium to perform operations, the operations including: acquiring a captured image; identifying a user's gaze; identifying performance information of an augmented reality device and performance information of an external electronic device connected to the augmented reality device; selecting a device for identifying an object from among the augmented reality device and the external electronic device based on the performance information of the augmented reality device and the performance information of the external electronic device; selecting an artificial intelligence model for identifying an object based on the performance information of the augmented reality device and the performance information of the external electronic device; acquiring a partial image including an object related to the user's gaze from the captured image; and acquiring a recognition result of the object from the partial image by using the selected device and the selected artificial intelligence model.

[0064] According to an embodiment of the present disclosure, a method performed by an augmented reality device may include: acquiring a captured image of a physical environment around the AR device via a camera, the captured image including objects in the physical environment; receiving voice input from a user of the AR device; recognizing the user's gaze; identifying a service related to the object included in the captured image and to be performed based on the voice input and the user's gaze; identifying conditions for providing the service, wherein the conditions for providing the service are related to at least one of a target accuracy of the service, a target delay of the service, computing requirements for the AR device, or a communication state of the AR device; based on the conditions for providing the service, selecting one of an artificial intelligence model in the AR device, an artificial intelligence model in an external electronic device connected to the AR device, or an artificial intelligence model in a server; obtaining a result related to the object from the selected artificial intelligence model; and providing the service to the user.

[0065] According to an embodiment of the present disclosure, an augmented reality device may include: a communication interface configured to communicate with an external electronic device; a camera; a gaze tracking sensor configured to detect a user's gaze; a processor; and a memory storing instructions, which, when executed by the processor, cause the AR device to: obtain a captured image of a physical environment around the AR device via the camera, the captured image including objects in the physical environment; recognize the user's gaze; receive voice input; identify a service related to the object included in the captured image and to be performed based on the voice input and the user's gaze; identify conditions for providing the service, wherein the conditions for providing the service are related to at least one of a target accuracy of the service, a target delay of the service, a computing requirement for the AR device, or a communication state of the AR device; based on the conditions for providing the service, select one of an artificial intelligence model in the AR device, an artificial intelligence model in an external electronic device connected to the AR device, or an artificial intelligence model in a server; obtain a result related to the object from the selected artificial intelligence model; and provide the service to the user.

[0066] According to an embodiment of the present disclosure, a computer-readable recording medium may be provided, in which a program for executing a method is recorded, the program including instructions that, when executed, cause the medium to perform operations, the operations including: acquiring a captured image of a physical environment surrounding an augmented reality (AR) device via a camera, the captured image including objects in the physical environment; receiving voice input from a user of the AR device; recognizing the user's gaze; identifying a service to be performed based on the voice input and the user's gaze and related to the object included in the captured image; identifying conditions for providing the service, wherein the conditions for providing the service are related to at least one of a target accuracy of the service, a target delay of the service, a computing requirement for the AR device, or a communication status of the AR device; based on the conditions for providing the service, selecting one of an artificial intelligence model in the AR device, an artificial intelligence model in an external electronic device connected to the AR device, or an artificial intelligence model in a server; obtaining a result related to the object from the selected artificial intelligence model; and providing the service to the user.

[0067] Therefore, the augmented reality device can select an appropriate artificial intelligence model from artificial intelligence models with different computational loads by considering whether services related to object recognition should be provided accurately or quickly. In addition, since the augmented reality device can efficiently select an augmented reality device and an external electronic device for recognizing an object, the augmented reality device can quickly recognize an object in a captured image and provide services to the user.

[0068] In addition, since the augmented reality device is configured to crop a partial image to be input into the artificial intelligence model from the captured image based on the user's gaze and use the partial image, images containing unwanted objects are not input into the artificial intelligence model, so that accurate recognition of the object can be performed.

[0069] In addition, since the augmented reality device is configured to crop a partial image to be input to the artificial intelligence model from the captured image and use the partial image, the amount of calculation for object recognition processing can be reduced.

[0070] In addition, the augmented reality device can appropriately adjust the amount of calculation for recognizing an object according to the situation, and thus, can provide high-quality real-time services to users.

[0071] Figure 1 is a schematic diagram of a system for enabling an augmented reality device to recognize an object in an image according to an embodiment.

[0072] refer to Figure 1 , a system 100 for recognizing an object in an image may include an augmented reality device 1000 , an external electronic device 2000 , and a server 3000 .

[0073] The augmented reality device 1000 may be configured to capture (e.g., capture a still image or a moving image) an object around the augmented reality device 1000 (e.g., in the environment around the augmented reality device 1000) and communicate with at least one of the external electronic device 2000 or the server 3000 to identify the object in the captured image. The augmented reality device 1000 may be configured to recognize the object in the captured image by using at least one of the augmented reality device 1000, the external electronic device 2000, and the server 3000 according to the service provision condition of the service provided to the user based on the recognition result of the object in the captured image.

[0074] The augmented reality device 1000 may include at least one first artificial intelligence model 10 trained for object recognition, the external electronic device 2000 may include at least one second artificial intelligence model 20 trained for object recognition, and the server 3000 may include at least one third artificial intelligence model 30 trained for object recognition. Object recognition may include, for example, at least one of recognizing the type of an object, detecting an object, or recognizing the position of an object. Object recognition may include, for example, at least one of detecting a gesture of a user's hand and recognizing the type of a gesture (e.g., a gesture of a user's hand may be detected and the type of gesture corresponding to the detected gesture may be recognized).

[0075] The first artificial intelligence model 10, the second artificial intelligence model 20, and the third artificial intelligence model 30 can each recognize an object in a captured image by using a different amount of calculation. The first artificial intelligence model 10 can recognize an object in a captured image by using a first amount of calculation, the second artificial intelligence model 20 can recognize an object in a captured image by using a second amount of calculation, and the third artificial intelligence model 30 can recognize an object in a captured image by using a third amount of calculation.

[0076] The first computation amount of the first artificial intelligence model 10 may be smaller than the second computation amount of the second artificial intelligence model 20, and the second computation amount of the second artificial intelligence model 20 may be smaller than the third computation amount of the third artificial intelligence model 30. The first number of bits of the parameters of the first artificial intelligence model 10 may be smaller than the second number of bits of the parameters of the second artificial intelligence model 20, and the second number of bits of the parameters of the second artificial intelligence model 20 may be smaller than the third number of bits of the parameters of the third artificial intelligence model 30. The parameters of the artificial intelligence model may be parameters of a neural network included in the artificial intelligence model, and may include, for example, activation parameters and weight parameters, but are not limited thereto.

[0077] The augmented reality device 1000 may be configured to recognize an object in a captured image by using the first artificial intelligence model 10 of the augmented reality device 1000 when the real-time capability of the service to be provided to the user is important. The augmented reality device 1000 may be configured to recognize an object in a captured image by using at least one of the second artificial intelligence model 20 and the third artificial intelligence model 30 when the accuracy of the service to be provided to the user is important.

[0078] When an artificial intelligence model to be used to identify an object is determined among the first artificial intelligence model 10, the second artificial intelligence model 20, and the third artificial intelligence model 30, the augmented reality device 1000 can be configured to crop a partial image including the object from the captured image and input the partial image into the determined artificial intelligence model, thereby obtaining an object recognition result based on the determined artificial intelligence model.

[0079] The augmented reality device 1000 may be configured as a wearable device having a communication function and a data processing function, such as glasses and a hair band. However, the augmented reality device 1000 is not limited thereto, and may include all types of devices (including cameras) configured to provide an augmented reality service.

[0080] The external electronic device 2000 is a device capable of storing the second artificial intelligence model 20 and communicating with the augmented reality device 1000 and the server 3000, and may include, for example, a smart phone, a tablet PC, a PC, a smart TV, a cellular phone, a personal digital assistant (PDA), a laptop computer, a media player, a micro server, a global positioning system (GPS) device, a digital broadcast terminal, a navigation, a kiosk, a home appliance, and other mobile or non-mobile computing devices, but may not be limited thereto. According to an embodiment, the external electronic device 2000 is connected to the augmented reality device 1000 via short-range wireless communication.

[0081] The server 3000 is a device capable of storing the third artificial intelligence model 30 and communicating with at least one of the augmented reality device 1000 and the external electronic device 2000, and may be a network device capable of recognizing an object in a captured image by using the third artificial intelligence model 30. According to an embodiment, the server is connected to at least one of the external electronic device 2000 and the augmented reality device 1000 through long-range wireless communication.

[0082] The network can be implemented by a wired network such as a local area network (LAN), a wide area network (WAN) or a value-added network (VAN) or various wireless networks such as a mobile radio communication network or a satellite communication network. In addition, the network may include a combination of at least two of a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile communication network or a satellite communication network, and the network is a data communication network that enables network configuration entities to communicate smoothly with each other and includes wired Internet, wireless Internet and mobile wireless communication networks in a comprehensive sense. Wireless communication may include, for example, wireless LAN (Wi-Fi), Bluetooth, Bluetooth low energy, Zigbee, Wi-Fi Direct (WFD), ultra-wideband (UWB), infrared communication (Infrared Data Association (IrDA)), near field communication (NFC), etc., but may not be limited thereto.

[0083] Figure 2 A case where an augmented reality device recognizes an object in an image according to an embodiment is shown.

[0084] refer to Figure 2 , a user wearing the augmented reality device 1000 may be configured to input an input command to the augmented reality device 1000, the input command requesting a service related to an object 22 of the real world. For example, the user may input a voice input command "What is the name of that car?" to the augmented reality device 1000. The augmented reality device 1000 may be configured to capture (an image of) an object 22 of the real world in response to the user's input command, and may select a device and / or an artificial intelligence model for identifying the object 22 from the captured image. In addition, the augmented reality device 1000 may be configured to crop a partial image 24 including the object 22 corresponding to the user's gaze 20 from the captured image, and recognize the object 22 from the partial image 24 by using the selected artificial intelligence model. For example, the augmented reality device 1000 may be configured to crop a partial image 24 of a predetermined area from the captured image based on the user's gaze 20.

[0085] Figure 3 is a flowchart illustrating a method for acquiring a result of object recognition including a partial image of an object corresponding to a gaze direction of a user through an augmented reality device according to an embodiment.

[0086] In operation 300, the augmented reality device 1000 may be configured to acquire a captured image by capturing an image of an object using a camera. The augmented reality device 1000 may be configured to capture an image of an object in the real world using a camera installed toward the front of the augmented reality device 1000. For example, the augmented reality device 1000 may be configured to acquire a still image or a moving image by controlling the camera to capture an image of an object when an application requesting a camera function is run. For example, the augmented reality device 1000 may be configured to acquire a still image or a moving image by controlling the camera to capture an image of an object when a user input command requesting a service requiring a camera function is received.

[0087] In operation 310, the augmented reality device 1000 may be configured to recognize the gaze of the user. For example, the augmented reality device 1000 may be configured to detect the gaze of the user using a gaze tracking sensor installed in a direction toward the user's eyes. Detecting the user's gaze may include acquiring gaze information related to the user's gaze. The gaze tracking sensor may include, for example, at least one of an IR scanner or an image sensor, and when the augmented reality device 1000 is a glasses-type device, a plurality of gaze tracking sensors may be arranged around a left display and a right display of the augmented reality device 1000 toward the user's eyes, respectively (for example, the left display and the right display may correspond to a left lens and a right lens of the glasses-type device, respectively).

[0088] According to an embodiment, the augmented reality device 1000 may be configured to recognize the user's gaze detected by using a gaze tracking sensor, and may recognize an object corresponding to the user's gaze from a captured image (for example, the augmented reality device 1000 may be used to capture an image corresponding to an image visible to the user, wherein the image may include the object; and then the object may be recognized from the captured image).

[0089] For example, the augmented reality device 1000 may be configured to sense the user's eyes at predetermined time intervals and recognize the user's gaze when an application requesting a camera function is executed. For example, the augmented reality device 1000 may be configured to sense the user's eyes and recognize the user's gaze when a user input command requesting a service requiring a camera function is received. For example, the augmented reality device 1000 may be configured to sense the user's eyes and recognize the user's gaze, so that when the camera function needs to be executed, an image corresponding to the user's gaze can be captured in real time.

[0090] In operation 320, the augmented reality device 1000 may be configured to identify conditions for providing services to be provided to the user. The conditions for providing services (e.g., service provision conditions) may be preset conditions required to provide high-quality services to the user. For example, the service provision conditions may be conditions set for arithmetic processing of images for object recognition and / or object identification. According to an embodiment, the service provision conditions may be pre-configured according to predetermined criteria in order to provide high-quality services to the user, and may be configured differently, for example, depending on whether the accuracy of the service is important or the real-time capability of the service is important. The conditions for providing services may include, for example, conditions related to target accuracy, target latency, computing requirements, and / or communication status. For example, the target accuracy may include the degree to which an object is accurately recognized (e.g., the target accuracy may include a required level of detail for recognizing an object, and if the target accuracy includes a high level of detail, the augmented reality device 1000 may be configured to use at least one of the second artificial intelligence model 20 and the third artificial intelligence model 30 to recognize the object), the target delay may include a waiting time required for object recognition, and the computational requirements may include the amount of computation of the artificial intelligence model to be used for object recognition (e.g., if the computational requirements include an amount of computation that is too high for the first artificial intelligence model 10, the augmented reality device 1000 may be configured to use at least one of the second artificial intelligence model 20 and the third artificial intelligence model 30 to recognize the object). For example, the target accuracy may be digitized and configured by an accuracy level to be described later, and the target delay may be digitized and configured by a real-time level to be described later, but are not limited thereto.

[0091] The augmented reality device 1000 may be configured to identify a service provision condition based on at least one of, for example, a type of service to be provided to the user, a type of application executed in connection with recognition of an object, or a user input command requesting recognition of an object.

[0092] Reference Figure 4 and Figure 5 The service provision conditions and examples in which the augmented reality device 1000 may be configured to identify the service provision conditions are described in more detail.

[0093] In operation 330, the augmented reality device 1000 may be configured to identify performance information of the augmented reality device 1000 and performance information of the external electronic device 2000. The performance information of the augmented reality device 1000 and / or the external electronic device 2000 may include hardware performance information of the augmented reality device 1000 and / or the external electronic device 2000. The performance information of the augmented reality device 1000 and / or the external electronic device 2000 may include information about the performance of processes such as the CPU, NPU, and GPU of the augmented reality device 1000 and / or the external electronic device 2000. In addition, the performance information of the augmented reality device 1000 and / or the external electronic device 2000 may include the type of memory (e.g., SRAM and / or DRAM) within the augmented reality device 1000 and / or the external electronic device 2000, and information about the capacity of the memory.

[0094] The performance information of the augmented reality device 1000 may include specification information of a processor for executing the first artificial intelligence model 10. For example, the performance information of the augmented reality device 1000 may include specification information of an NPU of the augmented reality device 1000 that executes the first artificial intelligence model 10. The specification information of the NPU of the augmented reality device 1000 may include information about the amount of computation that can be processed by the NPU of the augmented reality device 1000. For example, the specification information of the NPU of the augmented reality device 1000 may include an identification value of an artificial intelligence model that can be processed by the NPU of the augmented reality device 1000, the precision of the artificial intelligence model, the accuracy of the artificial intelligence model, and / or information about the amount of computation of the artificial intelligence model. When the augmented reality device 1000 does not include an NPU, the specification information of the CPU and / or GPU of the augmented reality device 1000 may be used to determine the performance of the augmented reality device 1000.

[0095] The performance information of the external electronic device 2000 may include specification information of a processor configured to execute the second artificial intelligence model 20. For example, the performance information of the external electronic device 2000 may include specification information of an NPU of the external electronic device 2000 configured to execute the first artificial intelligence model 10. The specification information of the NPU of the external electronic device 2000 may include information about the amount of calculation that can be processed by the NPU of the external electronic device 2000. For example, the specification information of the NPU of the external electronic device 2000 may include an identification value of an artificial intelligence model that can be processed by the NPU of the external electronic device 2000, the precision of the artificial intelligence model, the accuracy of the artificial intelligence model, and / or information about the amount of calculation of the artificial intelligence model. When the external electronic device 2000 does not include an NPU, the specification information of the CPU and / or GPU of the external electronic device 2000 may be used to determine the performance of the augmented reality device 1000.

[0096] When the augmented reality device 1000 is communicatively connected to the external electronic device 2000, the augmented reality device 1000 may be configured to receive performance information (e.g., NPU identification value, NPU specification information, memory type, and / or memory capacity) of the external electronic device 2000 from the external electronic device 2000. When the augmented reality device 1000 is communicatively connected to the external electronic device 2000, the augmented reality device 1000 may be configured to receive an identification value (e.g., SSID) of the external electronic device 2000 from the external electronic device 2000. In this case, the augmented reality device 1000 may be configured to acquire performance information of the external electronic device 2000 based on the identification value (e.g., SSID) of the external electronic device 2000. When the external electronic device 2000 does not include an NPU, the augmented reality device 1000 may be configured to acquire specification information of a CPU or GPU of the external electronic device 2000.

[0097] According to an embodiment, an operation of the augmented reality device 1000 to identify performance information of the augmented reality device 1000 and performance information of the external electronic device 2000 may be omitted.

[0098] According to an embodiment, the augmented reality device 1000 may be configured to recognize an identification value of the first artificial intelligence model 10 to be executed in the augmented reality device 1000 or information about the computational amount of the first artificial intelligence model 10. For example, the computational amount of the first artificial intelligence model 10 may include the number of bits of the parameters of the first artificial intelligence model 10. The parameters of the first artificial intelligence model 10 may be parameters of a neural network included in the first artificial intelligence model 10, and may include, for example, activation parameters and weight parameters, but are not limited thereto.

[0099] In addition, the augmented reality device 1000 may be configured to recognize an identification value of the second artificial intelligence model 20 to be executed in the external electronic device 2000 or information about the calculation amount of the second artificial intelligence model 20. For example, the calculation amount of the second artificial intelligence model 20 may include the number of bits of the parameters of the second artificial intelligence model 20. The parameters of the second artificial intelligence model 20 may be parameters of a neural network included in the second artificial intelligence model 20, and may include, for example, activation parameters and weight parameters, but are not limited thereto.

[0100] According to an embodiment, operation 330 of the augmented reality device 1000 may be omitted.

[0101] In operation 340 , the augmented reality device 1000 may be configured to select a device and an artificial intelligence model for object recognition.

[0102] The augmented reality device 1000 may be configured to select a device and an artificial intelligence model for object recognition by considering at least one of the performance information of the augmented reality device 1000, the performance information of the external electronic device 2000, or the accuracy and real-time capability of the service to be provided to the user. For example, if the service to be provided to the user requires high accuracy, the external electronic device 2000 may be selected. In another example, if the service to be provided to the user requires real-time capability, the augmented reality device 1000 may be selected.

[0103] According to an embodiment, the augmented reality device 1000 may select an artificial intelligence model for recognizing an object based on a service providing condition, and may recognize the external electronic device 2000 or the server 3000 including the selected artificial intelligence model.

[0104] In this case, information about the accuracy and real-time capabilities of the services to be provided to the user may be preconfigured. For example, the accuracy level and real-time level of the services to be provided to the user may be configured according to the type of application and the functions to be provided to the user by the application.

[0105] For example, regarding a function of detecting the presence of an object in an application for providing object recognition, the accuracy level may be configured to be low and the real-time level may be configured to be high. For example, in the case of detecting whether a specific object exists in a captured image, the accuracy level may be configured to be low and the real-time level may be configured to be high. For example, when a gesture of a user's hand is detected from a captured image and the type of the gesture is classified, the accuracy level may be configured to be low and the real-time level may be configured to be high.

[0106] In addition, for example, regarding a function of providing an identification value and / or detailed information of an object in an application for providing object recognition, the accuracy level may be configured to be high and the real-time level may be configured to be low.

[0107] For example, regarding a function of providing information about an object while driving a vehicle among the functions of the navigation application, the accuracy level may be configured as low and the real-time level may be configured as high. For example, regarding a function of providing information about an object while the vehicle is stopped among the functions of the navigation application, the accuracy level may be configured as high and the real-time level may be configured as high.

[0108] For example, regarding a function of providing detailed information about an object among functions of an application for providing search information, the accuracy level may be configured to be high and the real-time level may be configured to be low.

[0109] For example, when a video playback application provides search information about an object in a video, the real-time level may be configured as high. For example, regarding a function of providing text translation information in a translation application, the accuracy level may be configured as high.

[0110] According to an embodiment, the augmented reality device 1000 may be configured to select an artificial intelligence model for object recognition based on the accuracy level and real-time level of the service to be provided to the user. For example, as shown in Table 1 below, the calculation amount of the artificial intelligence model according to the accuracy level and real-time level may be pre-configured.

[0111]

[0112] Level of accuracy Real-time level Artificial Intelligence Model 1 1 A4W4... 1 2 A4W1,A1W4… 1 3 A1W1… 2 1 A16W16… 2 2 A16W8,A8W16… 2 3 A8W8,A4W8,A8W4… 3 1 A32W32… 3 2 A16W32, A32W16… 3 3 A16W16…

[0113] For example, when the accuracy level is "1" and the real-time level is "3", an AI model with A1W1 accuracy and a smaller amount of calculation may be selected. For example, when the accuracy level is "3" and the real-time level is "1", an AI model with A32W32 accuracy and a larger amount of calculation may be selected. However, the AI ​​model corresponding to the accuracy level and the real-time level is not limited thereto.

[0114] For example, according to the communication status among the augmented reality device 1000, the external electronic device 2000, and the server 3000, the artificial intelligence model corresponding to the accuracy level and the real-time level may be configured differently from Table 1. In this case, the augmented reality device 1000 may be configured to select an artificial intelligence model for object recognition from among artificial intelligence models supported by devices available to the augmented reality device 1000 by considering the communication status.

[0115] According to an embodiment, for example, when a user's gesture is detected and the type of the gesture is recognized, an artificial intelligence model that has a lower computational complexity and uses a large-sized partial image as an input may be selected.

[0116] According to an embodiment, the accuracy level and the real-time level of the service to be provided to the user can be changed according to whether the user's gaze is kept on the object to be recognized. For example, when the user's gaze is directed to the object (or around the object), the real-time level can be set to high, and when the user's gaze is not directed to the object (or around the object), the accuracy level can be set to high.

[0117] According to an embodiment, the augmented reality device 1000 may be configured to select a device for performing object recognition by driving an artificial intelligence model. The augmented reality device 1000 may be configured to select a device for performing object recognition from among the augmented reality device 1000, the external electronic device 2000, and the server 3000. For example, as shown in Table 2, the artificial intelligence models supported by the augmented reality device 1000, the external electronic device 2000, and the server 3000 may be pre-configured.

[0118]

[0119]

[0120] For example, the augmented reality device 1000 may be configured to support the first artificial intelligence model 10 of A1W1 precision, A4W1 precision, and A4W4 precision. For example, the external electronic device 2000 may support the second artificial intelligence model 20 of A8W8 precision, A16W8 precision, and A8W16 precision. For example, the server 3000 may support the third artificial intelligence model 30 of A32W32 precision. However, examples of artificial intelligence models supported by the augmented reality device 1000, the external electronic device 2000, and the server 3000 are not limited thereto.

[0121] According to an embodiment, the device for identifying an object and / or the artificial intelligence model for identifying an object may be changed according to the service provision conditions for identifying or capturing images. For example, in the case of performing object recognition on continuously acquired captured images, the service provision conditions may be changed in the middle of performing object recognition on the captured images. In this case, the augmented reality device 1000 may be configured to change the device and AI model for object recognition by considering the service provision conditions, the performance information of the augmented reality device 1000, the performance information of the external electronic device 2000, and the accuracy and real-time capabilities of the service to be provided to the user.

[0122] When the device for recognizing an object and / or the artificial intelligence model for recognizing an object is changed, the augmented reality device 1000 may be configured to change the size of the partial image, which will be described later.

[0123] In operation 350, the augmented reality device 1000 may be configured to acquire a partial image including an object from the captured image. The augmented reality device 1000 may be configured to acquire a partial image including an object corresponding to the user's gaze from the captured image. The augmented reality device 1000 may be configured to identify a position corresponding to the user's gaze in the captured image, and to crop a partial image having a predetermined size around the identified position. In this case, the size of the partial image may be determined based on the size of the input image input to the selected artificial intelligence model. For example, the size of the input image of the artificial intelligence model may be pre-configured as shown in Table 2, and the augmented reality device 1000 may be configured to crop a partial image having the size of the input image of the selected artificial intelligence model from the captured image.

[0124] In Table 2, the input image of the artificial intelligence model is configured to have various sizes, but is not limited thereto. For example, the larger the computational complexity of the artificial intelligence model, the larger the configuration size of the input image corresponding to the artificial intelligence model.

[0125] The augmented reality device 1000 may be configured to additionally adjust the size of the partial image. The augmented reality device 1000 may be configured to additionally adjust the size of the partial image so that the partial image can be input to the artificial intelligence model.

[0126] In operation 360, the augmented reality device 1000 may be configured to obtain a recognition result of the object. The augmented reality device 1000 may be configured to obtain an object recognition result by using the acquired partial image and the selected artificial intelligence model.

[0127] When the first artificial intelligence model 10 of the augmented reality device 1000 is selected, the augmented reality device 1000 may be configured to input a partial image to the first artificial intelligence model 10 and obtain a result value output from the first artificial intelligence model 10 .

[0128] When the second artificial intelligence model 20 of the external electronic device 2000 is selected, the augmented reality device 1000 may be configured to provide a partial image to the external electronic device 2000, and the external electronic device 2000 may input the partial image to the second artificial intelligence model 20 of the external electronic device 2000. The external electronic device 2000 may acquire a result value output from the second artificial intelligence model 20, and provide the acquired result value of object recognition to the augmented reality device 1000. The external electronic device 2000 may be configured to acquire additional search information by using the object recognition result value, and may provide the acquired search information to the augmented reality device 1000.

[0129] When the third artificial intelligence model 30 of the server 3000 is selected, the augmented reality device 1000 may be configured to provide the server 3000 with a partial image through the external electronic device 2000. The augmented reality device 1000 may be configured to send a request to the external electronic device 2000 so that the server 3000 recognizes an object in the partial image, and the external electronic device 2000 may be configured to request the server for object recognition of the partial image in response to the request of the augmented reality device 1000. The server 3000 may be configured to receive the partial image from the external electronic device 2000 and input the partial image to the third artificial intelligence model so as to obtain an object recognition result. The server 3000 may be configured to provide the object recognition result to the external electronic device 2000, and the external electronic device 2000 may be configured to provide the object recognition result to the augmented reality device 1000. The external electronic device 2000 and / or the server 3000 may be configured to obtain additional search information by using the object recognition result value, and provide the obtained search information to the augmented reality device 1000.

[0130] The augmented reality device 1000 may provide a user with services related to an object. The augmented reality device 1000 may be configured to output an object recognition result and / or additional search information.

[0131] Figure 4 is a flowchart illustrating a method for identifying a condition for providing a service related to object recognition by an augmented reality device according to an embodiment. Figure 4 Operations 400 to 430 may correspond to Figure 3 Operation 320.

[0132] In operation 400, the augmented reality device 1000 may be configured to receive a user input command. The user may input a voice input requesting a service related to an object into the augmented reality device 1000. For example, the user may input a voice input such as “What is the name of that dog?”, “Please translate it”, and “What is the name of that building?” into the augmented reality device 1000.

[0133] In operation 410, the augmented reality device 1000 may be configured to identify an application being executed and a function of the application. The augmented reality device 1000 may be configured to identify an application providing a service according to a user's voice input and a function of the application. When an application providing a service according to a user's voice input is not being executed, the augmented reality device 1000 may be configured to execute an application providing a service according to the user's voice input. Alternatively, an application providing a service according to a user's voice input may be executed in the augmented reality device 1000 and / or the external electronic device 2000.

[0134] For example, when the user's voice input is "What is the name of that dog?", the augmented reality device 1000 may be configured to recognize the search function of the application providing the search service. For example, when the user's voice input is "Please translate it", the augmented reality device 1000 may be configured to recognize the translation function of the application providing the translation service. For example, when the user's voice input is "What is the name of that building?", the augmented reality device 1000 may be configured to recognize the search function of the navigation application.

[0135] In operation 420, the augmented reality device 1000 may be configured to identify the location of the augmented reality device 1000. The augmented reality device 1000 may be configured to identify the location of the augmented reality device 1000 by using a GPS sensor within the augmented reality device 1000. Alternatively, the augmented reality device 1000 may be configured to receive a location value of the external electronic device 2000 from the external electronic device 2000 connected to the augmented reality device 1000 (for example, the external electronic device 2000 may be wirelessly connected to the augmented reality device 1000), and use the location value of the external electronic device 2000 as the location value of the augmented reality device 1000.

[0136] According to an embodiment, operation 420 of the augmented reality device 1000 may be omitted.

[0137] In operation 430, the augmented reality device 1000 may be configured to identify a service provision condition for a service to be provided to the user. For example, the augmented reality device 1000 may be configured to determine the accuracy level and real-time level of the service to be provided to the user based on at least one of the type of application, the function of the application, and the location of the augmented reality device 1000. In this case, the accuracy level and real-time level corresponding to at least one of the type of application, the function of the application, and the location of the augmented reality device 1000 may be pre-configured (e.g., the type of application, the function of the application, and the location of the augmented reality device 1000 may each correspond to a predetermined accuracy level and real-time level). The augmented reality device 1000 may be configured to identify the accuracy level and real-time level corresponding to at least one of the type of application, the function of the application, and the location of the augmented reality device 1000 as a condition for providing the service to be provided to the user.

[0138] For example, in the case of executing an application for providing tourist destination information to provide a service, when the augmented reality device 1000 is located around a preconfigured landmark, the augmented reality device 1000 may be configured to recognize the accuracy level of the service to be provided to the user as high, and recognize its real-time level as low. For example, in the case of executing an application for providing tourist destination information to provide a service, when the augmented reality device 1000 is not located around a preconfigured landmark, the augmented reality device 1000 may be configured to recognize the accuracy level of the service to be provided to the user as low, and recognize its real-time level as high.

[0139] For example, in the case of executing a navigation application to provide a service, when the augmented reality device 1000 is located in an area prone to accidents, the augmented reality device 1000 may be configured to identify the accuracy level of the service to be provided to the user as high, and its real-time level as low. For example, in the case of executing a navigation application to provide a service, when the augmented reality device 1000 is not located in an area prone to accidents, the augmented reality device 1000 may be configured to identify the accuracy level of the service to be provided to the user as low, and its real-time level as high.

[0140] For example, the higher the real-time level of the service, the stricter the configuration conditions related to the communication status of the service (for example, it may be more important to provide the service to the user quickly, and therefore the configuration conditions may be stricter). In addition, for example, as the real-time level of the service decreases, the conditions related to the communication status of the service may be configured to be relaxed (for example, it may be less important to provide the service to the user quickly, and therefore the configuration conditions may be less strict). In addition, when the communication status of the augmented reality device 1000 and / or the external electronic device 2000 satisfies the communication status of the service provision condition, the service provision condition may be configured so that the augmented reality device 1000 selects the second artificial intelligence model 20 of the external electronic device 2000 or selects the third artificial intelligence model 30 of the server 3000.

[0141] Figure 5 An example of determining the size of a partial image, a device for object recognition, and an artificial intelligence model for object recognition according to an embodiment by considering a condition for providing a service to be provided to a user is shown.

[0142] refer to Figure 5, by considering information 50 about the processor specification of the external electronic device 2000, information 51 about the processor specification of the augmented reality device 1000, service provision conditions 52, communication status 53 of the augmented reality device 1000 and / or the external electronic device 2000, and user's gaze information 54, the augmented reality device 1000 can be configured to determine a cropped area of ​​a partial image from a captured image, select a device for object recognition, and select an artificial intelligence model for object recognition. The service provision conditions 52 may include target latency, computing requirements, and target accuracy. For example, the target latency may be a value quantified according to a real-time level, and the target accuracy may be a value quantified according to an accuracy level.

[0143] For example, the augmented reality device 1000 can be configured to crop a partial image 56-1 having a resolution of 300x300 pixels from a captured image, and configure an operation mode for object recognition so that the processor of the augmented reality device 1000 inputs the cropped partial image 56-1 into an artificial intelligence model with A4W1 precision to perform calculations.

[0144] For example, the augmented reality device 1000 can be configured to crop a partial image 56-2 having a resolution of 500x500 pixels from a captured image, and configure an operating mode for object recognition so that the processor of the augmented reality device 1000 inputs the cropped partial image 56-2 into an artificial intelligence model with A1W1 precision to perform calculations.

[0145] For example, the augmented reality device 1000 can be configured to crop a partial image 56-3 having a resolution of 400x400 pixels from a captured image, and configure an operation mode for object recognition so that the processor of the external electronic device 2000 inputs the cropped partial image 56-3 into an artificial intelligence model with A8W8 precision to perform calculations.

[0146] For example, the augmented reality device 1000 can be configured to crop a partial image 56-4 having a resolution of 300x300 pixels from a captured image, and configure an operation mode for object recognition so that the processor of the external electronic device 2000 inputs the cropped partial image 56-4 into an A16W8-precision artificial intelligence model to perform calculations.

[0147] For example, the augmented reality device 1000 may be configured to crop a partial image 56-5 having a resolution of 400x400 pixels from a captured image, and configure an operation mode for object recognition so that the processor of the server 3000 inputs the cropped partial image 56-5 into an artificial intelligence model with FP32 precision to perform calculations.

[0148] Figure 6is a flowchart illustrating a method for recognizing an object in a partial image by using at least one of an augmented reality device or an external electronic device according to an embodiment. Figure 6 The operation can correspond to Figure 3 Operations 350 and 360 .

[0149] In operation 605, the augmented reality device 1000 may be configured to determine whether the augmented reality device 1000 is selected. The augmented reality device 1000 may be configured to determine whether the augmented reality device 1000 among the augmented reality device 1000 and the external electronic device 2000 is selected as a device to execute an artificial intelligence model.

[0150] When it is determined in operation 605 that the augmented reality device 1000 is selected ("Yes"), the augmented reality device 1000 may be configured to acquire a first partial image having a size corresponding to the first artificial intelligence model 10 in operation 610. The augmented reality device 1000 may be configured to identify the resolution of the input image of the first artificial intelligence model 10, and to crop the first partial image having the identified resolution from the captured image. For example, the augmented reality device 1000 may be configured to crop a first partial image including a predetermined area from the captured image based on the user's gaze. For example, based on the user's gaze, a first partial image having the identified resolution may be cropped from the captured image, wherein the first partial image includes an object to be identified.

[0151] In operation 615, the augmented reality device 1000 may be configured to acquire the first artificial intelligence model 10 stored in the memory of the augmented reality device 1000, and in operation 620, the augmented reality device 1000 may be configured to input the first partial image to the first artificial intelligence model 10. The processor of the augmented reality device 1000 may be configured to extract the first artificial intelligence model 10 from the memory of the augmented reality device 1000 and input the first partial image to the first artificial intelligence model 10. For example, the NPU of the augmented reality device 1000 may be configured to extract the first artificial intelligence model 10 from the memory of the augmented reality device 1000 and input the first partial image to the first artificial intelligence model 10. Alternatively, for example, the CPU or GPU in the augmented reality device 1000 may be configured to extract the first artificial intelligence model 10 from the memory of the augmented reality device 1000 and input the first partial image to the first artificial intelligence model 10.

[0152] In operation 625, the augmented reality device 1000 may be configured to obtain an object recognition result output from the first artificial intelligence model 10. The augmented reality device 1000 may be configured to display a graphical user interface (GUI) indicating the object recognition result or output a sound indicating the object recognition result. Since the first artificial intelligence model 10 is a model that recognizes an object by using a small amount of calculation, when it is necessary to quickly provide services to the user, the augmented reality device 1000 may be configured to quickly recognize the object in the captured image and provide the user with information about the object recognition.

[0153] When it is determined in operation 605 that the augmented reality device 1000 is not selected (No), the augmented reality device 1000 may acquire a second partial image having a size corresponding to the second artificial intelligence model 20 in operation 630. The augmented reality device 1000 may be configured to identify the resolution of the input image of the second artificial intelligence model 20, and crop the second partial image having the identified resolution from the captured image. For example, the augmented reality device 1000 may be configured to crop the second partial image including a predetermined area from the captured image based on the user's gaze.

[0154] In operation 635, the augmented reality device 1000 may be configured to request object recognition for the second partial image from the external electronic device 2000. The augmented reality device 1000 may be configured to request object recognition while sending the second partial image to the external electronic device 2000. In this case, the augmented reality device 1000 may be configured to identify the communication state between the augmented reality device 1000 and the external electronic device 2000. When it is determined that the communication state between the augmented reality device 1000 and the external electronic device 2000 is good (for example, when the connection between the augmented reality device 1000 and the external electronic device 2000 is established and stable), the augmented reality device 1000 may be configured to request object recognition from the external electronic device 2000. When the communication connection between the augmented reality device 1000 and the external electronic device 2000 is disconnected or unstable, the augmented reality device 1000 may be configured to perform operation 610 instead of requesting object recognition from the external electronic device 2000.

[0155] In operation 640, the augmented reality device 1000 may be configured to receive, from the external electronic device 2000, an object recognition result obtained by the external electronic device 2000 due to applying the second partial image to the second artificial intelligence model 20. The external electronic device 2000, which has received the second partial image from the augmented reality device 1000, may be configured to input the second partial image to the second artificial intelligence model 20, and obtain the object recognition result output from the second artificial intelligence model 20. For example, the processor of the external electronic device 2000 may be configured to extract the second artificial intelligence model 20 from the memory of the external electronic device 2000, and input the second partial image to the second artificial intelligence model 20. For example, the NPU of the external electronic device 2000 may be configured to extract the second artificial intelligence model 20 from the memory of the external electronic device 2000 and input the second partial image to the second artificial intelligence model 20. Or, for example, the CPU or GPU in the external electronic device 2000 may be configured to extract the second artificial intelligence model 20 from the memory of the external electronic device 2000, and input the second partial image to the second artificial intelligence model 20. In addition, the external electronic device 2000 may be configured to transmit the object recognition result with respect to the second partial image to the augmented reality device 1000 .

[0156] The augmented reality device 1000 may be configured to display a graphical user interface (GUI) indicating the object recognition result received from the external electronic device 2000 or output a sound indicating the object recognition result. Since the second artificial intelligence model 20 is a model that recognizes objects more accurately than the first artificial intelligence model 10, when it is necessary to accurately provide services to users, the augmented reality device 1000 may be configured to accurately recognize objects in captured images and provide information on object recognition to users.

[0157] Figure 7 is a flowchart illustrating a method for recognizing an object in a partial image by using at least one of an augmented reality device, an external electronic device, and a server according to an embodiment. Figure 7 The operation can correspond to Figure 3 Operations 350 and 360 .

[0158] in addition, Figure 7 Operations 705 to 725 correspond to Figure 6 Operations 605 to 625, and Figure 7 Operations 730 to 740 may correspond to Figure 6 Therefore, descriptions of operations 705 to 725 and operations 730 to 740 will be omitted.

[0159] As a result of the determination in operation 705, when it is determined that the augmented reality device 1000 has not selected the augmented reality device 1000 ("No"), the augmented reality device 1000 may be configured to determine whether the external electronic device 2000 has been selected in operation 727. The augmented reality device 1000 may be configured to determine whether the external electronic device 2000 among the augmented reality device 1000, the external electronic device 2000, and the server 3000 is selected as a device for executing an artificial intelligence model.

[0160] As a result of the determination in operation 727 , when it is determined that the augmented reality device 1000 has selected the external electronic device 2000 (“Yes”), the augmented reality device 1000 may be configured to perform operation 730 .

[0161] As a result of the determination in operation 727, when it is determined that the augmented reality device 1000 has not selected the external electronic device 2000 ("No"), the augmented reality device 1000 may be configured to acquire a third partial image having a size corresponding to the third artificial intelligence model 30 in operation 745. The augmented reality device 1000 may be configured to identify the resolution of the input image of the third artificial intelligence model 30, and crop the third partial image having the identified resolution from the captured image. For example, the augmented reality device 1000 may be configured to crop the third partial image including a predetermined area from the captured image based on the user's gaze.

[0162] In operation 750, the augmented reality device 1000 may be configured to request object recognition for the third partial image from the server 3000. The augmented reality device 1000 may be configured to request object recognition for the third partial image from the server 3000 through the external electronic device 2000. For example, the augmented reality device 1000 may be configured to send a request to the external electronic device 2000 while sending the third partial image to the external electronic device 2000 so that the server 3000 requests object recognition. Alternatively, the augmented reality device 1000 may be configured to request object recognition directly from the server 3000. For example, the augmented reality device 1000 may be configured to request object recognition from the server 3000 while sending the third partial image to the server 3000.

[0163] In this case, the augmented reality device 1000 may be configured to identify at least one of a communication status between the augmented reality device 1000 and the external electronic device 2000 , a communication status between the external electronic device 2000 and the server 3000 , or a communication status between the augmented reality device 1000 and the server 3000 .

[0164] When it is determined that the communication status between the augmented reality device 1000 and the external electronic device 2000 and the communication status between the external electronic device 2000 and the server 3000 are good (for example, when the connection between the augmented reality device 1000 and the external electronic device 2000 is established and stable and the connection between the external electronic device 2000 and the server 3000 is established and stable), the augmented reality device 1000 can be configured to request object recognition from the server 3000 through the external electronic device 2000.

[0165] When the communication connection between the augmented reality device 1000 and the external electronic device 2000 is disconnected or unstable, and the communication status between the augmented reality device 1000 and the server 3000 is good (for example, when the connection between the augmented reality device 1000 and the server 3000 is established and stable), the augmented reality device 1000 can be configured to directly request object recognition from the server 3000.

[0166] When the communication status between the augmented reality device 1000 and the external electronic device 2000 is good (for example, when the connection between the augmented reality device 1000 and the external electronic device 2000 is established and stable), and the communication connection between the external electronic device 2000 and the server 3000 is disconnected or the communication status therebetween is unstable, the augmented reality device 1000 can be configured to perform operation 730 instead of requesting object recognition from the server 3000.

[0167] When the communication connection between the augmented reality device 1000 and the external electronic device 2000 is disconnected or the communication state therebetween is unstable, and when the communication connection between the augmented reality device 1000 and the server 3000 is disconnected or the communication state therebetween is unstable, the augmented reality device 1000 may be configured to perform operation 610 instead of requesting object recognition from the server 3000.

[0168] In operation 755, the augmented reality device 1000 may be configured to receive, from the server 3000, an object recognition result obtained by the server 3000 due to applying the third partial image to the third artificial intelligence model 30. The server 3000 may be configured to input the third partial image to the third artificial intelligence model 30, and obtain the object recognition result output from the third artificial intelligence model 30. For example, the processor of the server 3000 may be configured to extract the third artificial intelligence model 30 stored in the server 3000, and input the third partial image to the third artificial intelligence model 30. In addition, the server 3000 may be configured to directly transmit the object recognition result for the third partial image to the augmented reality device 1000 or to transmit it to the augmented reality device 1000 through the external electronic device 2000.

[0169] The augmented reality device 1000 may be configured to display a graphical user interface (GUI) indicating the object recognition result received from the server 3000 or output a sound indicating the object recognition result.

[0170] Since the accuracy and real-time performance of the first artificial intelligence model 10, the second artificial intelligence model 20, and the third artificial intelligence model 30 are different, the augmented reality device 1000 can be configured to effectively select and use at least one of the first artificial intelligence model 10, the second artificial intelligence model 20, and the third artificial intelligence model 30. Therefore, according to the attributes of the service to be provided to the user, the augmented reality device 1000 can be configured to effectively provide the user with the object recognition result in the captured image.

[0171] Figure 8 An example in which the augmented reality device recognizes an object by using the augmented reality device or an external electronic device according to a service providing condition according to an embodiment is shown.

[0172] refer to Figure 8 , with the user's gaze directed toward the middle dog among three dogs in the real world, the user may input a voice input 80 saying “What type of dog is that?” into the augmented reality device 1000 (eg, AR glasses).

[0173] The augmented reality device 1000 may be configured to recognize from the user's voice input 80 that the user needs to determine the type of dog in detail, and determine that the accuracy of the object recognition service to be provided to the user is important (e.g., the accuracy level is determined to be high). For example, since the accuracy of the object recognition service to be provided to the user is important, the external electronic device 2000 may be selected from the augmented reality device 1000 and the external electronic device 2000 as the device for executing the artificial intelligence model. Then, the augmented reality device 1000 may be configured to identify the communication state between the augmented reality device 1000 (e.g., a glasses-type device) and the external electronic device 2000 (e.g., a smart phone).

[0174] When the communication state between the augmented reality device 1000 and the external electronic device 2000 is good (for example, when the connection between the augmented reality device 1000 and the external electronic device 2000 is established and stable), the augmented reality device 1000 may be configured to crop the second partial image 83 from the captured image 82 and adjust the size of the second partial image 83 so that the second partial image is input to the second artificial intelligence model 20 of the external electronic device 2000. In this case, the augmented reality device 1000 may display a GUI for indicating the position of the second partial image 83 so that the user can check whether the second partial image 83 to be cropped includes an object (for example, a dog) corresponding to the voice input 80. In addition, the augmented reality device 1000 may be configured to request object recognition from the external electronic device 2000, and the external electronic device 2000 may be configured to input the second partial image 83 to the second artificial intelligence model 20 so as to recognize the type of the dog. For example, a smart phone may receive an object recognition request including the second partial image 83 from the augmented reality device 1000, and may recognize the type of the dog by using a high-precision artificial intelligence model in the smart phone. Thereafter, the augmented reality device 1000 may be configured to receive information indicating the type of dog from the external electronic device 2000, and output the received information about the type of dog to the display of the augmented reality device 1000 or output it as audio or voice. For example, the augmented reality device 1000 outputs the recognized type of dog to the display of the augmented reality device 1000, or outputs it as audio of voice.

[0175] When the communication state between the augmented reality device 1000 and the external electronic device 2000 is not good (for example, when the connection has failed / disconnected or the connection is unstable), the augmented reality device 1000 may be configured to crop the first partial image 84 from the captured image 82 and adjust the size of the first partial image 84 so that the first partial image is input to the first artificial intelligence model 10 of the augmented reality device 1000. In addition, the augmented reality device 1000 may be configured to identify the type of dog by inputting the first partial image 84 to the first artificial intelligence model 10. For example, the AR glasses can identify the type of dog in the first partial image 84 by using a low-precision artificial intelligence model within the AR glasses. Thereafter, the augmented reality device 1000 may be configured to output the identified type of dog to the display of the augmented reality device 1000 or output it as audio or voice. For example, when the user's gaze is maintained on the object corresponding to the voice input 80, the augmented reality device 1000 outputs the identified type of dog to the display of the augmented reality device 1000, or outputs it as audio of voice.

[0176] Fig. 9An example is shown in which the augmented reality device can recognize an object by using the augmented reality device or the server according to the service providing condition according to the embodiment.

[0177] refer to Fig. 9 , when the user's gaze is directed toward the middle dog among three dogs in the real world, the user may input a voice input 85 of “Where is the dog?” into the augmented reality device 1000 (eg, AR glasses).

[0178] The augmented reality device 1000 may be configured to recognize from the user's voice input 85 that information indicating the location of the dog should be provided for each frame of the captured image so as to notify the user of the location of the dog. Since the augmented reality device 1000 only needs to detect the dog and identify the location of the dog rather than accurately identify the type of dog, it can be determined that the real-time capability of the object recognition service to be provided to the user is more important than the accuracy of the object recognition service (for example, it can be determined that the real-time level is higher than the accuracy level). Thereafter, the augmented reality device 1000 may be configured to crop the first partial image 86 from the captured image 82 and adjust the size of the first partial image 86 so that the first partial image is input to the first artificial intelligence model 10 of the augmented reality device 1000. In this case, the captured image 82 may correspond to each frame of the captured video, and the first partial image 86 may be an image cropped and resized from each frame of the captured video. In addition, the augmented reality device 1000 may be configured to detect the dog and identify the location of the dog in the first partial image 86 by inputting the first partial image 86 into the first artificial intelligence model 10. For example, the AR glasses may detect the dog in the first partial image 86 and identify the location of the dog by using a low-precision artificial intelligence model within the AR glasses. Thereafter, the augmented reality device 1000 may be configured to output an indicator (e.g., a bounding box) indicating the position of the recognized dog on the display of the augmented reality device 1000. For example, when the user's gaze remains on the object corresponding to the voice input 85, the augmented reality device 1000 outputs an indicator indicating the position of the recognized dog on the display of the augmented reality device 1000.

[0179] Fig.10 An example of recognizing an object by interworking an augmented reality device with an external electronic device and a server according to an embodiment is shown.

[0180] refer to Fig.10 , when the user's gaze is directed toward a bottle of wine in the real world, the user may input a voice input 101 saying “What kind of wine is that bottle of wine?” into the augmented reality device 1000 (eg, a glasses-type device).

[0181] The augmented reality device 1000 can be configured to recognize from the user's voice input 101 that detailed information related to wine needs to be notified to the user, and recognize that the accuracy of the object recognition service to be provided to the user is important. For example, the augmented reality device 1000 can be configured to recognize that the accuracy of the object recognition service is high and the target delay is low.

[0182] The augmented reality device 1000 may be configured to crop the third partial image 102 from the captured image and adjust the size of the third partial image 102 so that the third partial image is input to the third artificial intelligence model 30 of the server 3000 (for example, the third partial image may correspond to the label of the wine bottle). In this case, the augmented reality device 1000 may display a GUI for indicating the position of the third partial image 102 so that the user can check whether the third partial image 102 to be cropped includes an object corresponding to the voice input 101 (for example: a wine bottle). In addition, the augmented reality device 1000 may be configured to send the third partial image 102 to the external electronic device 2000 (for example, a smart phone) (indicated by reference numeral 103) so that the server 3000 performs object recognition. The augmented reality device 1000 may be configured to request object recognition and additional information about the object while sending the third partial image 102 to the external electronic device 2000.

[0183] Thereafter, the external electronic device 2000 may transmit the received third partial image 102 to the server 3000 (indicated by reference numeral 104). The external electronic device 2000 may request object recognition and additional information about the object while transmitting the third partial image 102 to the server.

[0184] The server 3000 may be configured to input the third partial image 102 to the third artificial intelligence model 30 in order to identify the type of wine (indicated by reference numeral 105). For example, the server 3000 may be configured to receive an object recognition request including the third partial image 102 from a smartphone, and identify the type of wine within the third partial image 102 by using a high-precision artificial intelligence model in the server 3000. In addition, the server 3000 may be configured to obtain additional information about the identified wine (indicated by reference numeral 106). For example, the server 3000 may be configured to perform an Internet search based on the identified type of wine, and obtain additional information related to the name, price, taste, etc. of the wine as an Internet search result.

[0185] Thereafter, the server 3000 may be configured to send the type of wine and additional information to the external electronic device 2000 (e.g., a smart phone) (indicated by reference numeral 107), and the external electronic device 2000 may be configured to send the type of wine and additional information to the augmented reality device 1000 (e.g., AR glasses) (indicated by reference numeral 108).

[0186] Thereafter, the augmented reality device 1000 may be configured to output the received information about the wine to the display of the augmented reality device 1000 or output it as audio or voice. For example, when the user's gaze remains on the object corresponding to the voice input 101, the augmented reality device 1000 outputs the received information about the wine to the display of the augmented reality device 1000 or outputs it as audio of voice.

[0187] Fig.11 An example of recognizing text in an image through an augmented reality device according to an embodiment is shown.

[0188] refer to Fig.11 , with the user's gaze directed to a specific page of a book in the real world, the user may input a voice input 111 saying “translate it” to the augmented reality device 1000 (e.g., a glasses-type device).

[0189] The augmented reality device 1000 may be configured to recognize from the user's voice input 111 that the text of a portion of a book toward which the user's gaze is directed needs to be accurately translated, and recognize that the accuracy of the translation service to be provided to the user is important.

[0190] The augmented reality device 1000 may be configured to crop the second partial image 112 from the captured image 110 and adjust the size of the second partial image 112 so that the second partial image is input to the second artificial intelligence model 20 of the external electronic device 2000 (e.g., a smart phone). The augmented reality device 1000 may be configured to track the user's gaze within a predetermined period of time after receiving the user's voice input 111, and crop the second partial image 112 from the captured image 110 based on the gaze tracked within the predetermined period of time.

[0191] In addition, the augmented reality device 1000 may be configured to request translation from the external electronic device 2000 while providing the second partial image 112 to the external electronic device 2000. The external electronic device 2000 may input the second partial image 112 to the second artificial intelligence model 20 so as to perform text recognition and translation (indicated by reference numeral 113). In this case, the second artificial intelligence model 20 may be an artificial intelligence model trained to recognize and translate text. For example, a smart phone may receive a translation request including the second partial image 112 from the augmented reality device 1000, and may recognize the text in the second partial image 112 and translate the recognized text by using a high-precision artificial intelligence model in the smart phone. In the above, it has been described that a second artificial intelligence model 20 performs text recognition and translation, but is not limited thereto. The external electronic device 2000 may use multiple artificial intelligence models for text recognition and translation. For example, the external electronic device 2000 may be configured to recognize and translate the text in the second partial image 112 by using an artificial intelligence model trained for text recognition and an artificial intelligence model trained for translation.

[0192] Thereafter, the augmented reality device 1000 may be configured to receive a translation result from the external electronic device 2000 and output the received translation result on a display of the augmented reality device 1000 or output the received translation result as audio or voice.

[0193] On the other hand, according to an embodiment, in a case where the user's gaze is directed to a specific page of the book, the user may input a voice input 115 saying “Do you have text 'you'?” to the augmented reality device 1000 (eg, AR glasses).

[0194] The augmented reality device 1000 may be configured to recognize from the user's voice input 115 that information indicating the location of the text "you" (e.g., each instance of the word "you") should be provided for each frame of the captured image so as to find "you" in the text on the current page and provide a notification to the user. Since the augmented reality device 1000 should quickly recognize the text "you", the target latency of the object recognition service to be provided to the user may be determined to be low.

[0195] Thereafter, the augmented reality device 1000 may be configured to crop the first partial image 116 from the captured image 110 and adjust the size of the first partial image 116 so that the first partial image is input to the first artificial intelligence model 10 of the augmented reality device 1000. In this case, the captured image 110 may correspond to each frame of the captured video, and the first partial image 116 may correspond to an image cropped and resized from each frame of the captured video. In addition, the augmented reality device 1000 may be configured to search for the text "you" on the current page by using the first partial image 116 (indicated by reference numeral 117). The augmented reality device 1000 may be configured to detect the text "you" on the current page by inputting the first partial image 116 into the first artificial intelligence model 10, and identify the position of the text "you" in the first partial image 116. For example, the AR glasses may detect the text "you" in the first partial image 86, and identify the position of the text "you" by using a low-precision artificial intelligence model in the AR glasses. Thereafter, the augmented reality device 1000 may be configured to output an indicator indicating the location of the recognized text “you” to a display (indicated by reference numeral 118 ) of the augmented reality device 1000 .

[0196] Fig.12 An example is shown in which an augmented reality device selects an artificial intelligence model and device for object recognition based on the location of the augmented reality device according to an embodiment.

[0197] refer to Fig.12 , in a state where the navigation application of the augmented reality device 1000 is executed, the augmented reality device 1000 may be configured to move toward the destination. When the augmented reality device 1000 is located within a predetermined area 120 near the destination, the augmented reality device 1000 may be configured to determine that high accuracy of object recognition is required. The augmented reality device 1000 may be configured to crop a large-sized partial image from the captured image and provide the cropped partial image to the external electronic device 2000 to identify an object in the captured image (e.g., a destination location within a street map). The augmented reality device 1000 may be configured to send a request to the external electronic device 2000 so that a processor (e.g., NPU) of the external electronic device 2000 (e.g., a smart phone) recognizes the object by using a second artificial intelligence model 20 with high accuracy.

[0198] Fig.13 An example is shown in which an augmented reality device selects an artificial intelligence model and device for object recognition based on the location of the augmented reality device according to an embodiment.

[0199] refer to Fig.13, the augmented reality device 1000 may be configured to move in a state where a map application of the augmented reality device 1000 is executed (e.g., when the user walks). When the augmented reality device 1000 is located in an area 130 where traffic accidents frequently occur, the augmented reality device 1000 may be configured to determine that high accuracy of object recognition is required. The augmented reality device 1000 may be configured to crop a large-sized partial image from a captured image and provide the cropped partial image to the external electronic device 2000 in order to identify an object in the captured image (e.g., a portion of a street map near the user's current location). The augmented reality device 1000 may be configured to send a request to the external electronic device 2000 so that a processor (e.g., NPU) of the external electronic device 2000 (e.g., a smart phone) recognizes the object by using a second artificial intelligence model 20 with high accuracy.

[0200] Fig.14 An example is shown in which an augmented reality device acquires a partial image based on a sound obtained by the augmented reality device according to an embodiment.

[0201] refer to Fig.14 , the augmented reality device 1000 may be configured to crop a partial image from a captured image based on the ambient sound of the augmented reality device 1000 rather than the user's gaze. The augmented reality device 1000 may be configured to receive a sound signal (e.g., the sound of a dog barking) from a location around the augmented reality device 1000 by using at least one microphone included in the augmented reality device 1000, and identify the location of the source 140 in which the sound is generated from the received sound signal. The augmented reality device 1000 may be configured to identify the location corresponding to the sound source 140 in the captured image.

[0202] In this case, instead of Figure 3 In operation 310 of recognizing the user's gaze, an operation of recognizing a sound source around the augmented reality device 1000 may be performed. Therefore, the augmented reality device 1000 may be configured to crop a partial image 142 from the captured image based on the sound source 140 and adjust the size of the partial image 142. In addition, the augmented reality device 1000 may be configured to perform object recognition based on the cropped partial image 142.

[0203] Fig.15 is a schematic diagram of a system for enabling an augmented reality device to recognize an object in an image when the augmented reality device does not support processing of an artificial intelligence model according to an embodiment.

[0204] refer to Fig.15, the augmented reality device 1000 for recognizing an object in an image in the system 100 may be configured not to support processing of an artificial intelligence model. For example, when the augmented reality device 1000 does not include a separate NPU configured to execute an artificial intelligence model, or when the performance of the processor in the augmented reality device 1000 is low, the augmented reality device 1000 may be configured not to support processing of an artificial intelligence model. For example, when resources for object recognition of the augmented reality device 1000 are insufficient, the augmented reality device 1000 may be configured not to support processing of an artificial intelligence model.

[0205] The augmented reality device 1000 may be configured to capture images of objects around the augmented reality device 1000 and communicate with at least one of the external electronic device 2000 or the server 3000 to obtain a recognition result of the object in the captured image. The augmented reality device 1000 may be configured to recognize the object in the captured image by using at least one of the external electronic device 2000 or the server 3000 according to the service provision conditions of the service provided to the user based on the object recognition in the captured image.

[0206] The augmented reality device 1000 does not include the first artificial intelligence model 10, the external electronic device 2000 may include at least one second artificial intelligence model 20 trained for object recognition, and the server 3000 may include at least one third artificial intelligence model 30 trained for object recognition. Object recognition may include, for example, at least one of recognizing the type of an object, detecting an object, or recognizing the location of an object (as shown in FIG. Figures 8 to 14 described above).

[0207] The second artificial intelligence model 20 and the third artificial intelligence model 30 can respectively recognize objects in the captured image by using different computing amounts. The second artificial intelligence model 20 can recognize objects in the captured image by using the second computing amount, and the third artificial intelligence model 30 can recognize objects in the captured image by using the third computing amount.

[0208] The second calculation amount of the second artificial intelligence model 20 may be smaller than the third calculation amount of the third artificial intelligence model 30. The second number of bits of the parameter of the second artificial intelligence model 20 may be smaller than the third number of bits of the parameter of the third artificial intelligence model 30. The parameters of the artificial intelligence model may be parameters of a neural network included in the artificial intelligence model, and may include, for example, activation parameters and weight parameters, but are not limited thereto.

[0209] When the real-time capability of the service to be provided to the user is more important than the accuracy of the service, the augmented reality device 1000 may be configured to recognize the object in the captured image by using the second artificial intelligence model 20 of the external electronic device 2000. In addition, the augmented reality device 1000 may be configured to recognize the object in the captured image by using the third artificial intelligence model 30 of the server 3000 when the accuracy of the service to be provided to the user is more important than the real-time capability of the service.

[0210] When an artificial intelligence model to be used for object recognition is determined among the second artificial intelligence model 20 and the third artificial intelligence model 30, the augmented reality device 1000 can be configured to crop a partial image including the object from the captured image and input the partial image into the determined artificial intelligence model so as to obtain the result of object recognition based on the determined artificial intelligence model.

[0211] Fig.16 is a flowchart illustrating a method in which an augmented reality device obtains a result of object recognition for a partial image including an object corresponding to a gaze direction of a user when the augmented reality device does not support processing of an artificial intelligence model according to an embodiment.

[0212] Fig.16 Operations 1605 to 1620 correspond to Figure 3 Operations 300 to 320 are described below, and for ease of explanation, descriptions of operations 1605 to 1620 will be omitted.

[0213] In operation 1630, the augmented reality device 1000 may be configured to identify performance information of the external electronic device 2000. The performance information of the external electronic device 2000 may include specification information of a processor (e.g., NPU) of the external electronic device 2000 that executes the second artificial intelligence model 20. The specification information of the processor of the external electronic device 2000 may include information about the amount of calculation that the processor of the external electronic device 2000 can process. For example, the specification information of the processor of the external electronic device 2000 may include information about an identification value, accuracy, or amount of calculation of an artificial intelligence model that can be processed by the NPU of the external electronic device 2000.

[0214] When the augmented reality device 1000 is communicatively connected to the external electronic device 2000, the augmented reality device 1000 may be configured to receive performance information of the external electronic device 2000 from the external electronic device 2000. When the augmented reality device 1000 is communicatively connected to the external electronic device 2000, the augmented reality device 1000 may be configured to receive an identification value (e.g., SSID) of the external electronic device 2000 from the external electronic device 2000. In this case, the augmented reality device 1000 may be configured to acquire specification information of the NPU included in the external electronic device 2000 based on the identification value (e.g., SSID) of the external electronic device 2000.

[0215] According to an embodiment, an operation of the augmented reality device 1000 recognizing performance information of the external electronic device 2000 may be omitted.

[0216] According to an embodiment, the augmented reality device 1000 may be configured to recognize information about an identification value of the second artificial intelligence model 20 or a computational amount of the second artificial intelligence model 20 to be executed in the external electronic device 2000. For example, the computational amount of the second artificial intelligence model 20 may include the number of bits of a parameter of the second artificial intelligence model 20. The parameter of the second artificial intelligence model 20 may be a parameter of a neural network included in the second artificial intelligence model 20, and may include, for example, an activation parameter and a weight parameter, but may not be limited thereto.

[0217] According to an embodiment, operation 1630 of the augmented reality device 1000 may be omitted.

[0218] In operation 1640, the augmented reality device 1000 may be configured to select a device and an artificial intelligence model for recognizing an object. The augmented reality device 1000 may be configured to select a device and an artificial intelligence model for recognizing an object by considering specification information of a processor of the external electronic device 2000 and accuracy and real-time capability of a service to be provided to a user.

[0219] According to an embodiment, the augmented reality device 1000 may select an artificial intelligence model for recognizing an object based on a service providing condition, and may recognize the external electronic device 2000 or the server 3000 including the selected artificial intelligence model.

[0220] In this case, information about the accuracy and real-time capabilities of the services to be provided to the user may be preconfigured. For example, the accuracy level and real-time level of the services to be provided to the user may be configured according to the type of application and the functions to be provided to the user through the application.

[0221] For example, regarding a function for detecting the presence of an object in an application for providing object recognition, the accuracy level may be configured as low and the real-time level may be configured as high. For example, regarding a function for providing an identification value and / or detailed information of an object in an application for providing object recognition, the accuracy level may be configured as high and the real-time level may be configured as low.

[0222] For example, regarding a function of providing information about an object while driving a vehicle among the functions of the navigation application, the accuracy level may be configured as low and the real-time level may be configured as high. For example, regarding a function of providing information about an object while the vehicle is stopped among the functions of the navigation application, the accuracy level may be configured as high and the real-time level may be configured as high.

[0223] For example, regarding a function of providing detailed information about an object among functions of an application for providing search information, the accuracy level may be configured to be high and the real-time level may be configured to be low.

[0224] For example, when a video playback application provides search information about an object in a video, the real-time level may be configured as high. For example, regarding a function of providing text translation information in a translation application, the accuracy level may be configured as high.

[0225] According to an embodiment, the augmented reality device 1000 may be configured to select an artificial intelligence model for object recognition based on the accuracy level and real-time level of the service to be provided to the user. For example, the calculation amount of the artificial intelligence model according to the accuracy level and real-time level may be pre-configured.

[0226] According to an embodiment, the augmented reality device 1000 may be configured to select a device for performing object recognition by driving an artificial intelligence model. The augmented reality device 1000 may be configured to select a device for performing object recognition from among the external electronic device 2000 and the server 3000. For example, the artificial intelligence model supported by the external electronic device 2000 and the server 3000 may be configured to be pre-configured.

[0227] In operation 1650, the augmented reality device 1000 may be configured to acquire a partial image including an object from the captured image. The augmented reality device 1000 may be configured to acquire a partial image including an object corresponding to the user's gaze from the captured image. The augmented reality device 1000 may be configured to identify a position in the captured image corresponding to the user's gaze, and to crop a partial image having a predetermined size around the identified position. In this case, the size of the partial image may be determined according to the size of the input image input to the selected artificial intelligence model. For example, the augmented reality device 1000 may be configured to crop a partial image having the size of the input image of the selected artificial intelligence model from the captured image. The augmented reality device 1000 may be configured to additionally adjust the size of the partial image. The augmented reality device 1000 may be configured to additionally adjust the size of the partial image so that the partial image can be input to the artificial intelligence model.

[0228] In operation 1660, the augmented reality device 1000 may be configured to acquire a recognition result of the object. The augmented reality device 1000 may be configured to acquire the object recognition result by using the acquired partial image and the selected artificial intelligence model.

[0229] When the second artificial intelligence model 20 of the external electronic device 2000 is selected, the augmented reality device 1000 may be configured to provide a partial image to the external electronic device 2000, and the external electronic device 2000 may input the partial image to the second artificial intelligence model 20 of the external electronic device 2000. The external electronic device 2000 may acquire a result value output from the second artificial intelligence model 20, and provide the acquired result value of object recognition to the augmented reality device 1000. The external electronic device 2000 may acquire additional search information by using the object recognition result value, and may provide the acquired search information to the augmented reality device 1000.

[0230] When the third artificial intelligence model 30 of the server 3000 is selected, the augmented reality device 1000 may be configured to provide a partial image to the server 3000 through the external electronic device 2000 (for example, the external electronic device 2000 may receive the partial image from the augmented reality device 1000 and then send the partial image to the server 3000). The augmented reality device 1000 may be configured to send a request to the external electronic device 2000 so that the server 3000 recognizes an object in the partial image, and the external electronic device 2000 may be configured to request the server 3000 for object recognition of the partial image in response to the request of the augmented reality device 1000. The server 3000 may be configured to receive the partial image from the external electronic device 2000 and input the partial image to the third artificial intelligence model in order to obtain an object recognition result. The server 3000 may be configured to provide the object recognition result to the external electronic device 2000, and the external electronic device 2000 may be configured to provide the object recognition result to the augmented reality device 1000. The external electronic device 2000 and / or the server 3000 may be configured to acquire additional search information by using the object recognition result value, and provide the acquired search information to the augmented reality device 1000 .

[0231] The augmented reality device 1000 may be configured to output object recognition results and / or additional search information.

[0232] Fig.17 is a flowchart illustrating a method in which an augmented reality device recognizes an object in a partial image by using at least one of an external electronic device and a server when the augmented reality device does not support processing of an artificial intelligence model according to an embodiment.

[0233] In operation 1727, the augmented reality device 1000 may be configured to determine whether the external electronic device 2000 is selected. The augmented reality device 1000 may be configured to determine whether the external electronic device 2000 among the external electronic device 2000 and the server 3000 is selected as a device to execute an artificial intelligence model.

[0234] As a result of the determination in operation 1727, when it is determined that the external electronic device 2000 is selected by the augmented reality device 1000, the augmented reality device 1000 may be configured to perform operation 1730. Fig.17 Operations 1730 to 1740 correspond to Figure 7 Operations 1730 to 1740 are described below, and therefore, for convenience, descriptions of operations 1730 to 1740 will be omitted.

[0235] As a result of the determination in operation 1727, when it is determined that the external electronic device 2000 is not selected by the augmented reality device 1000, the augmented reality device 1000 may be configured to perform operation 1745. Fig.17 Operations 1745 to 1755 correspond to Figure 7 Operations 1745 to 1755 are described below, and therefore, for convenience, descriptions of operations 1745 to 1755 will be omitted.

[0236] Fig.18 is a block diagram of an augmented reality device according to an embodiment.

[0237] refer to Fig.18 , the augmented reality device 1000 according to the embodiment may include a user input unit 1100, a microphone 1200, a display 1300, a speaker 1350, a camera module 1400, a gaze tracking sensor 1500, a communication interface 1600, a memory 1700, and a processor 1800. In addition, the processor 1800 may include a CPU 1810 and an NPU 1820. The first artificial intelligence model 10 may be stored in the memory 1700 or in an internal memory of the NPU 1820.

[0238] The user input unit 1100 refers to a means by which a user inputs data for controlling the augmented reality device 1000. For example, the user input unit 1100 may include a keyboard, a dome switch, a touch pad (touch capacitance type, piezoresistive type, infrared beam sensing type, surface acoustic wave type, integral strain gauge type, piezoelectric effect type, etc.), a scroll wheel, and a jog switch, but is not limited thereto. The user input unit 1100 may receive input from a user to receive services related to object recognition.

[0239] The microphone 1200 receives an external sound signal and processes the sound signal into electrical voice data. For example, the microphone 1200 may receive a sound signal from an external device or a speaker. The microphone 1200 may use various noise elimination algorithms to remove noise generated in the process of receiving the external sound signal. The microphone 1200 may receive sounds generated around the augmented reality device 1000. The microphone 1200 may receive a voice input of a user for controlling the augmented reality device 1000.

[0240] The display 1300 displays and outputs information processed by the augmented reality device 1000. For example, the display 1300 may display a user interface for photographing the surroundings of the augmented reality device 1000 and information related to a service provided based on the photographed image of the surroundings of the augmented reality device 1000.

[0241] According to an embodiment, the display 1300 may provide an augmented reality (AR) image. The display 1300 according to an embodiment may include a light guide plate (not shown) and a display module (not shown). When the user wears the device 1000, the light guide plate (not shown) may be made of a transparent material, in which a portion of the rear surface is visible. The light guide plate (not shown) may be configured by a single-layer or multi-layer flat plate made of a transparent material, through which light is reflected and propagated. The light guide plate (not shown) may be positioned to face the light-emitting surface of the display module and receive light of a virtual image projected therefrom. Here, a transparent material means a material through which light can pass, and the transparency may not be 100% and may have a predetermined color. In an embodiment, since the light guide plate (not shown) is made of a transparent material, the user can not only see the virtual objects of the virtual image through the display 1300, but also the external real scene, so the light guide plate (not shown) may be referred to as a perspective display. The display 1300 may provide an augmented reality image by outputting the virtual objects of the virtual image through the light guide plate. When the augmented reality device 1000 is a glasses type device, the display 1300 may include a left display and a right display.

[0242] The speaker 1350 may output sounds generated by the augmented reality device 1000 .

[0243] The camera module 1400 can capture the surrounding environment of the augmented reality device 1000. When an application requesting a photo function is executed, the camera module 1400 can obtain an image frame such as a still image or a moving image through an image sensor. The image captured by the image sensor can be processed by the processor 1800 or a separate image processor (not shown). The camera module 1400 may include, for example, at least one of a rotatable RGB camera module or a plurality of depth camera modules, but is not limited thereto.

[0244] The gaze tracking sensor 1500 may track the gaze of the user wearing the augmented reality device 1000. The gaze tracking sensor 1500 may be installed in a direction toward the eyes of the user, and may detect the gaze direction of the user's left eye and the gaze direction of the user's right eye. Detecting the direction of the user's gaze may include acquiring gaze information related to the user's gaze. The gaze information is information related to the user's gaze, and may include, for example, information about the position of the pupil of the user's eye, the coordinates of the center point of the pupil, and the user's gaze direction. The user's gaze direction may be, for example, the direction of the gaze from the center point of the user's pupil to the position where the user is gazed.

[0245] The gaze tracking sensor 1500 may include, for example, at least one of an IR scanner or an image sensor, and when the augmented reality device 1000 is a glasses-type device, multiple gaze tracking sensors may be arranged around the left display and the right display of the augmented reality device 1000 respectively toward the user's eyes.

[0246] The gaze tracking sensor 1500 may detect data related to the gaze of the user's eyes. The user's gaze information may be generated based on the data related to the gaze of the user's eyes. The gaze information is information related to the user's gaze, and may include, for example, information about the position of the pupil of the user's eye, the coordinates of the center point of the pupil, and the direction of the user's gaze. The user's gaze direction may be, for example, the direction of the gaze from the center point of the user's pupil to the position where the user is gazing.

[0247] The communication interface 1600 may transmit or receive data for a service based on an image acquired by photographing the surrounding environment of the augmented reality device 1000 to or from the external electronic device 2000 and the server 3000 .

[0248] The memory 1700 may store a program to be executed by the processor 1800 described later, and may store data input to or output from the augmented reality device 1000 .

[0249] The memory 1700 may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card-type memory (for example, an SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, and an optical disk.

[0250] The programs stored in the memory 1700 may be classified into a plurality of modules according to their functions.

[0251] The processor 1800 is configured to control the overall operation of the augmented reality device 1000. The processor 1800 may be configured to execute a program stored in the memory 1700. Figures 1 to 17 For example, the processor 1800 may be configured to execute a program stored in the memory 1700 so as to perform overall control of the user input unit 1100, the microphone 1200, the display 1300, the speaker 1350, the camera module 1400, the gaze tracking sensor 1500, the communication interface 1600, the memory 1700, and the like.

[0252] The processor 1800 may be configured to execute a program stored in the memory 1700 so as to shoot an object by using a camera and acquire a shot image. The processor 1800 may shoot an object in the real world by controlling a camera installed toward the front of the augmented reality device 1000. For example, when an application for requesting a camera function is executed, the processor 1800 may shoot a still image or a moving image by controlling the camera to shoot an object. For example, the processor 1800 may be configured to acquire a still image or a moving image by controlling the camera to shoot an object when a user input command requesting a service requiring a camera function is received.

[0253] The processor 1800 may be configured to recognize the gaze of the user by executing the program stored in the memory 1700. The processor 1800 may be configured to detect the gaze of the user by using the gaze tracking sensor 1500 installed in a direction toward the eyes of the user.

[0254] The processor 1800 may be configured to identify conditions for providing services to be provided to the user by executing a program stored in the memory 1700. The service provision conditions may be pre-configured to provide high-quality services to the user, and may be configured differently, for example, depending on whether the accuracy of the service is important or the real-time capability of the service is important. The service provision conditions may include, for example, conditions related to target accuracy, target latency, computing requirements, and / or communication status. For example, the target accuracy may include the degree to which an object is accurately identified, the target latency may include the waiting time required for object identification, and the computing requirements may include the amount of computing of an artificial intelligence model for object identification. For example, the target accuracy may be quantified and configured by the accuracy level to be described later, and the target latency may be quantified and configured by the real-time level to be described later, but is not limited thereto.

[0255] The processor 1800 may be configured to identify the service provision condition based on, for example, at least one of a type of service to be provided to the user, a type of application executed regarding object recognition, or a user input command requesting object recognition.

[0256] The processor 1800 may be configured to identify the performance information of the augmented reality device 1000 and the performance information of the external electronic device 2000 by executing the program stored in the memory 1700. The performance information of the augmented reality device 1000 and / or the external electronic device 2000 may include information related to the hardware performance of the augmented reality device 1000 and / or the external electronic device 2000. The performance information of the augmented reality device 1000 and / or the external electronic device 2000 may include information about the performance of processes such as the CPU, NPU, and GPU of the augmented reality device 1000 and / or the external electronic device 2000. In addition, the performance information of the augmented reality device 1000 and / or the external electronic device 2000 may include a type of memory (e.g., SRAM and / or DRAM) in the augmented reality device 1000 and / or the external electronic device 2000, and information about the capacity of the memory. The performance information of the augmented reality device 1000 may include specification information of a processor for executing the first artificial intelligence model 10. For example, the performance information of the augmented reality device 1000 may include specification information of the NPU 1820 of the augmented reality device 1000 that executes the first artificial intelligence model 10. The specification information of the NPU 1820 of the augmented reality device 1000 may include information about the amount of calculation that the NPU 1820 of the augmented reality device 1000 can process. For example, the specification information of the NPU 1820 of the augmented reality device 1000 may include information about the identification value, accuracy, or amount of calculation of the artificial intelligence model that can be processed by the NPU 1820 of the augmented reality device 1000. When the augmented reality device 1000 does not include the NPU, the specification information of the CPU and / or GPU of the augmented reality device 1000 may be used to determine the performance of the augmented reality device 1000.

[0257] The performance information of the external electronic device 2000 may include specification information of a processor for executing the second artificial intelligence model 20. For example, the performance information of the augmented reality device 1000 may include specification information of an NPU of the external electronic device 2000 that executes the first artificial intelligence model 10. The specification information of the NPU of the external electronic device 2000 may include information about the amount of calculation that the NPU of the external electronic device 2000 can process. For example, the specification information of the NPU of the external electronic device 2000 may include information about an identification value, precision, or amount of calculation of an artificial intelligence model that can be processed by the NPU of the external electronic device 2000. When the external electronic device 2000 does not include an NPU, specification information of the CPU and / or GPU of the external electronic device 2000 may be used to determine the performance of the augmented reality device 1000.

[0258] When the augmented reality device 1000 is communicatively connected to the external electronic device 2000, the processor 1800 may be configured to receive an identification value or specification information of the NPU of the external electronic device 2000 from the external electronic device 2000. When the augmented reality device 1000 is communicatively connected to the external electronic device 2000, the processor 1800 may be configured to receive an identification value (e.g., SSID) of the external electronic device 2000 from the external electronic device 2000. In this case, the processor 1800 may be configured to acquire specification information of the NPU included in the external electronic device 2000 based on the identification value (e.g., SSID) of the external electronic device 2000.

[0259] According to an embodiment, the processor 1800 may be configured to recognize an identification value of the first artificial intelligence model 10 to be executed in the augmented reality device 1000 or information about the amount of calculation of the first artificial intelligence model 10. For example, the amount of calculation of the first artificial intelligence model 10 may include the number of bits of a parameter of the first artificial intelligence model 10. The parameter of the first artificial intelligence model 10 may be a parameter of a neural network included in the first artificial intelligence model 10, and may include, for example, an activation parameter and a weight parameter, but may not be limited thereto.

[0260] In addition, the processor 1800 may be configured to recognize an identification value of the second artificial intelligence model 20 to be executed in the external electronic device 2000 or information about the computation amount of the second artificial intelligence model 20. For example, the computation amount of the second artificial intelligence model 20 may include the number of bits of a parameter of the second artificial intelligence model 20. The parameter of the second artificial intelligence model 20 may be a parameter of a neural network included in the second artificial intelligence model 20, and may include, for example, an activation parameter and a weight parameter, but is not limited thereto.

[0261] The processor 1800 may be configured to select a device and an artificial intelligence model for recognizing an object by executing a program stored in the memory 1700 .

[0262] The processor 1800 may be configured to select a device and a smart model for object recognition by considering performance information of the NPU 1820 of the augmented reality device 1000 , performance information of the NPU of the external electronic device 2000 , and accuracy and real-time capabilities of a service to be provided to the user.

[0263] In this case, the accuracy and real-time capability information of the service to be provided to the user can be pre-configured. For example, the accuracy level and real-time level of the service to be provided to the user can be configured according to the type of application and the function to be provided to the user through the application.

[0264] For example, regarding a function of detecting the presence of an object in an application for providing object recognition, the accuracy level may be configured to be low and the real-time level may be configured to be high. For example, regarding a function of providing an identification value and / or detailed information of an object in an application for providing object recognition, the accuracy level may be configured to be high and the real-time level may be configured to be low.

[0265] For example, regarding a function of providing information about an object while driving a vehicle among the functions of the navigation application, the accuracy level may be configured as low and the real-time level may be configured as high. For example, regarding a function of providing information about an object while the vehicle is stopped among the functions of the navigation application, the accuracy level may be configured as high and the real-time level may be configured as high.

[0266] For example, regarding a function of providing detailed information about an object among functions of an application for providing search information, the accuracy level may be configured to be high and the real-time level may be configured to be low.

[0267] For example, when a video playback application provides search information about an object in a video, the real-time level may be configured as high. For example, regarding a function in a translation application that provides text translation information, the accuracy level may be configured as high.

[0268] According to an embodiment, the augmented reality device 1000 may be configured to select an artificial intelligence model for object recognition based on the accuracy level and real-time level of the service to be provided to the user. For example, the calculation amount of the artificial intelligence model according to the accuracy level and real-time level may be pre-configured.

[0269] According to an embodiment, the processor 1800 may be configured to select a device for performing object recognition by driving an artificial intelligence model. The augmented reality device 1000 may be configured to select a device for performing object recognition among the augmented reality device 1000, the external electronic device 2000, and the server 3000. For example, the artificial intelligence models supported by the augmented reality device 1000, the external electronic device 2000, and the server 3000 may be preconfigured.

[0270] According to an embodiment, according to the service provision conditions, the device for object recognition and / or the artificial intelligence model for object recognition may be changed for recognizing the captured image. In the case where object recognition is performed on continuously acquired captured images, the service provision conditions may be changed during the process of performing object recognition on the captured images. In this case, the processor 1800 may be configured to change the device for object recognition and the artificial intelligence model by considering the service provision conditions, the performance information of the augmented reality device 1000, the performance information of the external electronic device 2000, and the accuracy and real-time capability of the service to be provided to the user.

[0271] When the device for recognizing an object and / or the artificial intelligence model for recognizing an object is changed, the processor 1800 may change the size of a partial image which will be described later.

[0272] The processor 1800 may be configured to acquire a partial image including an object from a captured image by executing a program stored in the memory 1700. The processor 1800 may be configured to acquire a partial image including an object corresponding to the user's gaze from the captured image. The processor 1800 may be configured to identify a position corresponding to the user's gaze in the captured image, and to crop a partial image having a predetermined size around the identified position. In this case, the size of the partial image may be determined based on the size of the input image input to the selected artificial intelligence model. For example, the size of the input image of the artificial intelligence model may be preconfigured, and the processor 1800 may be configured to crop a partial image having the size of the input image of the selected artificial intelligence model from the captured image.

[0273] The processor 1800 may be configured to further adjust the size of the partial image. The augmented reality device 1000 may be configured to additionally adjust the size of the partial image so that the partial image can be input to the artificial intelligence model.

[0274] The processor 1800 may be configured to obtain a recognition result of the object. The processor 1800 may be configured to obtain an object recognition result by using the acquired partial image and the selected artificial intelligence model.

[0275] When the first artificial intelligence model 10 of the augmented reality device 1000 is selected, the NPU 1820 of the augmented reality device 1000 may be configured to input a partial image to the first artificial intelligence model 10 and obtain a result value output from the first artificial intelligence model 10 .

[0276] When the second artificial intelligence model 20 of the external electronic device 2000 is selected, the processor 1800 may be configured to provide a partial image to the external electronic device 2000, and the external electronic device 2000 may input the partial image to the second artificial intelligence model 20 of the external electronic device 2000. The external electronic device 2000 may acquire a result value output from the second artificial intelligence model 20, and provide the acquired object recognition result value to the augmented reality device 1000. The external electronic device 2000 may acquire additional search information by using the object recognition result value, and may provide the acquired search information to the augmented reality device 1000.

[0277] When the third artificial intelligence model 30 of the server 3000 is selected, the processor 1800 may be configured to provide the server 3000 with a partial image through the external electronic device 2000. The processor 1800 may be configured to send a request to the external electronic device 2000 so that the server 3000 recognizes an object in the partial image, and the external electronic device 2000 may be configured to request the server for object recognition of the partial image in response to the request of the augmented reality device 1000. The server 3000 may be configured to receive the partial image from the external electronic device 2000 and input the partial image to the third artificial intelligence model 30 to obtain an object recognition result. The server 3000 may be configured to provide the object recognition result to the external electronic device 2000, and the external electronic device 2000 may be configured to provide the object recognition result to the augmented reality device 1000. The external electronic device 2000 and / or the server 3000 may be configured to obtain additional search information by using the object recognition result value, and provide the obtained search information to the augmented reality device 1000.

[0278] Processor 1800 may be configured to output object recognition results and / or additional search information.

[0279] Fig.19 is a block diagram showing an electronic device in a network environment according to various embodiments. Fig.19 , the electronic device 20001 in the network environment 20000 may communicate with the electronic device 20002 via the first network 20098 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 20004 or the server 20008 via the second network 20099 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 20001 may communicate with the electronic device 20004 via the server 20008. According to an embodiment, the electronic device 20001 may include a processor 20020, a memory 20030, an input module 20050, a sound output module 20055, a display module 20060, an audio module 20070, a sensor module 20076, an interface 20077, a connection terminal 20078, a haptic module 20079, a camera module 20080, a power management module 20088, a battery 20089, a communication module 20090, a user identification module (SIM) 20096, or an antenna module 20097. In some embodiments, some components (e.g., the sensor module 20076, the camera module 20080, or the antenna module 20097) may be implemented as a single integrated component (e.g., the display module 20060).

[0280] The processor 20020 may run, for example, software (e.g., program 20040) to control at least one other component (e.g., hardware component or software component) of the electronic device 20001 coupled to the processor 20020, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 20020 may store a command or data received from another component (e.g., sensor module 20076 or communication module 20090) in the volatile memory 20032, process the command or data stored in the volatile memory 20032, and store the result data in the non-volatile memory 20034. According to an embodiment, the processor 20020 may include a main processor 20021 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 20023 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 20021 in operation. For example, when the electronic device 20001 includes a main processor 20021 and an auxiliary processor 20023, the auxiliary processor 20023 may be adapted to consume less power than the main processor 20021, or may be adapted to be dedicated to a specific function. The auxiliary processor 20023 may be implemented to be separate from the main processor 20021, or to be implemented as part of the main processor 20021.

[0281] When the main processor 20021 is in an inactive (e.g., sleep) state, the auxiliary processor 20023 (rather than the main processor 20021) may control at least some of the functions or states related to at least one component (e.g., display module 20060, sensor module 20076, or communication module 20090) among the components of the electronic device 20001, or when the main processor 20021 is in an active state (e.g., running an application), the auxiliary processor 20023 may control at least some of the functions or states related to at least one component (e.g., display module 20060, sensor module 20076, or communication module 20090) among the components of the electronic device 20001 together with the main processor 20021. According to an embodiment, the auxiliary processor 20023 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 20080 or communication module 20090) that is functionally related to the auxiliary processor 20023. According to an embodiment, the auxiliary processor 20023 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated by machine learning. For example, such learning may be performed by the electronic device 20001 where artificial intelligence is executed or via a separate server (e.g., server 20008). The learning algorithm may include, but is not limited to, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the artificial intelligence model may include a software structure in addition to the hardware structure.

[0282] The memory 20030 may store various data used by at least one component of the electronic device 20001 (e.g., the processor 20020 or the sensor module 20076). The various data may include, for example, software (e.g., the program 20040) and input data or output data for commands related thereto. The memory 20030 may include a volatile memory 20032 or a non-volatile memory 20034.

[0283] The program 20040 may be stored as software in the memory 20030 , and the program 20040 may include, for example, an operating system (OS) 20042 , middleware 20044 , or an application 20046 .

[0284] The input module 20050 may receive commands or data to be used by other components (e.g., processor 20020) of the electronic device 20001 from outside (e.g., user) of the electronic device 20001. The input module 20050 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0285] The sound output module 20055 can output sound signals to the outside of the electronic device 20001. The sound output module 20055 can include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. According to an embodiment, the receiver can be implemented as being separated from the speaker, or as part of the speaker.

[0286] The display module 20060 may visually provide information to the outside (e.g., user) of the electronic device 20001. The display device 20060 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display module 20060 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0287] The audio module 20070 can convert sound into an electrical signal, and vice versa. According to an embodiment, the audio module 20070 can obtain sound via the input module 20050, or output sound via the sound output module 20055 or an earphone of an external electronic device (e.g., electronic device 20002) directly (e.g., wired) connected to the electronic device 20001 or wirelessly connected.

[0288] The sensor module 20076 may detect an operating state (e.g., power or temperature) of the electronic device 20001 or an environmental state (e.g., a state of a user) outside the electronic device 20001, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 20076 may include, for example, a gesture sensor, a gyroscope sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0289] The interface 20077 may support one or more specific protocols to be used to connect the electronic device 20001 directly (e.g., wired) or wirelessly to an external electronic device (e.g., electronic device 20002). According to an embodiment, the interface 20077 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0290] The connection end 20078 may include a connector, wherein the electronic device 20001 can be physically connected to an external electronic device (e.g., the electronic device 20002) via the connector. According to an embodiment, the connection end 20078 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0291] The haptic module 20079 may convert the electrical signal into a mechanical stimulus (eg, vibration or motion) or an electrical stimulus that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 20079 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0292] The camera module 20080 may capture a still image or a moving image. According to an embodiment, the camera module 20080 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0293] The power management module 20088 may manage power supply to the electronic device 20001. According to an embodiment, the power management module 20088 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0294] The battery 20089 may power at least one component of the electronic device 20001. According to an embodiment, the battery 20089 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0295] The communication module 20090 can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 20001 and an external electronic device (e.g., electronic device 20002, electronic device 20004, or server 20008), and perform communication via the established communication channel. The communication module 20090 may include one or more communication processors capable of operating independently with the processor 20020 (e.g., an application processor (AP)) and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 20090 may include a wireless communication module 20092 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 20094 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate via a first network 20098 (e.g., a short-range communication network such as Bluetooth TM, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 20099 (for example, a long-distance communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (for example, a LAN or a wide area network (WAN))) to communicate with an external electronic device. These various types of communication modules can be implemented as a single component (for example, a single chip), or these various types of communication modules can be implemented as multiple components separated from each other (for example, multiple chips). The wireless communication module 20092 can identify and authenticate the electronic device 20001 in a communication network (such as the first network 20098 or the second network 20099) using user information (for example, an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 20096.

[0296] The wireless communication module 20092 can support 5G networks after 4G networks and next-generation communication technologies (e.g., new radio (NR) access technology). NR access technology can support enhanced mobile broadband (eMBB), massive machine type communication (mMTC), or ultra-reliable low-latency communication (URLLC). The wireless communication module 20092 can support high-frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 20092 can support various technologies for ensuring performance on high-frequency bands, such as, for example, beamforming, massive multiple-input multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 20092 can support various requirements specified in the electronic device 20001, an external electronic device (e.g., electronic device 20004), or a network system (e.g., a second network 20099). According to an embodiment, the wireless communication module 20092 can support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane delay for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less round trip).

[0297] The antenna module 20097 may transmit a signal or power to the outside of the electronic device 20001 (e.g., an external electronic device) or receive a signal or power from the outside of the electronic device 20001 (e.g., an external electronic device). According to an embodiment, the antenna module 20097 may include an antenna including a radiating element, the radiating element being formed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 20097 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 20098 or the second network 20099) may be selected from the plurality of antennas by, for example, the communication module 20090 (e.g., the wireless communication module 20092). Subsequently, a signal or power may be transmitted or received between the communication module 20090 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as part of the antenna module 20097.

[0298] According to various embodiments, the antenna module 20097 may form a millimeter wave antenna module. According to an embodiment, the millimeter wave antenna module may include a printed circuit board, an RFIC, and a plurality of antennas (e.g., array antennas), wherein the RFIC is disposed on a first surface (e.g., bottom surface) of the printed circuit board, or is adjacent to the first surface and is capable of supporting a specified high frequency band (e.g., millimeter wave band), and the plurality of antennas are disposed on a second surface (e.g., top surface or side surface) of the printed circuit board, or is adjacent to the second surface and is capable of transmitting or receiving signals of the specified high frequency band.

[0299] At least some of the above components can be connected to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0300] According to an embodiment, a command or data may be sent or received between the electronic device 20001 and the external electronic device 20004 via the server 20008 connected to the second network 20099. Each of the electronic device 20002 or the electronic device 20004 may be a device of the same type as the electronic device 20001, or a device of a different type from the electronic device 20001. According to an embodiment, all or some operations to be executed in the electronic device 20001 may be executed in one or more of the external electronic device 20002, the external electronic device 20004, or the server 20008. For example, if the electronic device 20001 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 20001 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 20001 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices receiving the request may execute at least part of the requested function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 20001. The electronic device 20001 may provide the result as at least a partial reply to the request in the case of further processing the result or in the case of not further processing the result. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology or client-server computing technology, for example, may be used. The electronic device 20001 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 20004 may include an Internet of Things (IoT) device. The server 20008 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 20004 or the server 20008 may be included in the second network 20099. The electronic device 20001 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars or health care) based on 5G communication technology or IoT-related technologies.

[0301] According to an embodiment of the present disclosure, the external electronic device 2000 may correspond to Fig.19 In this case, the NPU of the external electronic device 2000 may correspond to Fig.19 The auxiliary processor 20023 of the external electronic device, and the second artificial intelligence model 20 used by the external electronic device 2000 can be stored in the memory 20030 or in the internal memory of the NPU of the external electronic device.

[0302] Fig. 20 is a flowchart illustrating a method for acquiring a result related to an object corresponding to a user's gaze through an augmented reality device for providing a service according to an embodiment.

[0303] In operation 300-1, the augmented reality device 1000 may be configured to acquire a captured image including an object by using a camera. The augmented reality device 1000 may be configured to capture an image of an object in the real world using a camera installed toward the front of the augmented reality device 1000. For example, the augmented reality device 1000 may be configured to acquire a still image or a moving image by controlling the camera to capture an image of an object when an application requesting a camera function is run. For example, the augmented reality device 1000 may be configured to acquire a still image or a moving image by controlling the camera to capture an image of an object when a user input command requesting a service requiring a camera function is received.

[0304] In operation 300-2, the augmented reality device 1000 may be configured to receive a voice input. The user may input a voice input requesting a service related to an object into the augmented reality device 1000. For example, the user may input voice inputs such as “What is the name of that dog?”, “Please translate it”, and “What is the name of that building?” into the augmented reality device 1000.

[0305] In operation 310-1, the augmented reality device 1000 may be configured to recognize the gaze of the user. For example, the augmented reality device 1000 may be configured to detect the gaze of the user using a gaze tracking sensor installed in a direction toward the user's eyes. Detecting the user's gaze may include acquiring gaze information related to the user's gaze. The gaze tracking sensor may include, for example, at least one of an IR scanner or an image sensor, and when the augmented reality device 1000 is a glasses-type device, a plurality of gaze tracking sensors may be arranged around the left display and the right display of the augmented reality device 1000 toward the user's eyes, respectively (for example, the left display and the right display may correspond to the left lens and the right lens of the glasses-type device, respectively).

[0306] According to an embodiment, the augmented reality device 1000 can be configured to recognize the gaze of the user detected by using a gaze tracking sensor, and an object corresponding to the user's gaze can be identified from a captured image (for example, the augmented reality device 1000 can be used to capture an image corresponding to an image visible to the user, where the image may include an object; and then the object can be identified from the captured image).

[0307] For example, the augmented reality device 1000 may be configured to sense the user's eyes at predetermined time intervals and recognize the user's gaze in the case of executing an application requesting a photographing function. For example, the augmented reality device 1000 may be configured to sense the user's eyes and recognize the user's gaze in the case of receiving a user input command requesting a service requiring a photographing function. For example, the augmented reality device 1000 may be configured to sense the user's eyes and recognize the user's gaze, so that when the photographing function needs to be executed, an image corresponding to the user's gaze can be captured in real time.

[0308] In operation 320 - 1 , the augmented reality device 1000 may be configured to recognize a service to be executed based on a voice input and a user's gaze.

[0309] The augmented reality device 1000 may identify a service related to the recognition and / or identification of the object corresponding to the user's gaze based on the interpretation result of the voice input. For example, the augmented reality device 1000 may identify a service based on at least one of the type of service to be provided to the user, the type of application executed on the object, or the function of the application executed on the object.

[0310] According to an embodiment, the augmented reality device 1000 may be configured to identify an application being executed and a function of the application. The augmented reality device 1000 may be configured to identify an application providing a service based on a user's voice input and the function of the application. When an application providing a service based on a user's voice input is not being executed, the augmented reality device 1000 may be configured to execute an application providing a service based on a user's voice input. Alternatively, an application for providing a service based on a user's voice input may be executed in the augmented reality device 1000 and / or the external electronic device 2000.

[0311] For example, when the user's voice input is "What is the name of that dog?", the augmented reality device 1000 may be configured to recognize the search function of an application that provides a search service. For example, when the user's voice input is "Please translate it", the augmented reality device 1000 may be configured to recognize the translation function of an application that provides a translation service. For example, when the user's voice input is "What is the name of that building?", the augmented reality device 1000 may be configured to recognize the search function of a navigation application.

[0312] In operation 320-2, the augmented reality device 1000 may be configured to identify conditions for providing services to be provided to the user. The conditions for providing services (e.g., service provision conditions) may be preset conditions required to provide high-quality services to the user. For example, the service provision conditions may be conditions set regarding arithmetic processing of images for object recognition and / or object identification. According to an embodiment, the service provision conditions may be pre-configured according to predetermined criteria in order to provide high-quality services to the user, and may be configured differently, for example, depending on whether the accuracy of the service is important or the real-time capability of the service is important. The conditions for providing services may include, for example, conditions related to target accuracy, target latency, computing requirements, and / or communication status. For example, the target accuracy may include the degree to which an object is accurately recognized (e.g., the target accuracy may include a level of detail required for recognizing an object, and if the target accuracy includes a high level of detail, the augmented reality device 1000 may be configured to use at least one of the second artificial intelligence model 20 and the third artificial intelligence model 30 to recognize the object), the target delay may include a waiting time required for object recognition, and the computational requirements may include the amount of computation of the artificial intelligence model to be used for object recognition (e.g., if the computational requirements include an amount of computation that is too high for the first artificial intelligence model 10, the augmented reality device 1000 may be configured to use at least one of the second artificial intelligence model 20 and the third artificial intelligence model 30 to recognize the object). For example, the target accuracy may be digitized and configured by the accuracy level, and the target delay may be digitized and configured by the real-time level, but are not limited thereto.

[0313] In operation 340 - 1 , the augmented reality device 1000 may be configured to select an artificial intelligence model among the first artificial intelligence model 10 in the augmented reality device 1000 , the second artificial intelligence model 20 in the external electronic device 2000 , or the third artificial intelligence model 30 in the server 3000 .

[0314] The augmented reality device 1000 may be configured to select an artificial intelligence model for object recognition by considering the accuracy and / or real-time capability of the service to be provided to the user. For example, if the service to be provided to the user requires high accuracy, the second artificial intelligence model 20 in the external electronic device 2000 may be selected. In another example, if the service to be provided to the user requires real-time capability, the first artificial intelligence model 10 in the augmented reality device 1000 may be selected.

[0315] According to an embodiment, the augmented reality device 1000 may select an artificial intelligence model for recognizing an object based on a service providing condition, and may recognize the external electronic device 2000 or the server 3000 including the selected artificial intelligence model.

[0316] In this case, information about the accuracy and real-time capabilities of the services to be provided to the user may be preconfigured. For example, the accuracy level and real-time level of the services to be provided to the user may be configured according to the type of application and the functions to be provided to the user through the application.

[0317] For example, regarding a function of detecting the presence of an object in an application for providing object recognition, the accuracy level may be configured to be low and the real-time level may be configured to be high. For example, in the case of detecting whether a specific object exists in a captured image, the accuracy level may be configured to be low and the real-time level may be configured to be high. For example, when a gesture of a user's hand is detected from a captured image and the type of the gesture is classified, the accuracy level may be configured to be low and the real-time level may be configured to be high.

[0318] In addition, for example, regarding a function of providing an identification value and / or detailed information of an object in an application for providing object recognition, the accuracy level may be configured to be high and the real-time level may be configured to be low.

[0319] For example, regarding a function of providing information about an object while driving a vehicle among the functions of the navigation application, the accuracy level may be configured as low and the real-time level may be configured as high. For example, regarding a function of providing information about an object while the vehicle is stopped among the functions of the navigation application, the accuracy level may be configured as high and the real-time level may be configured as high.

[0320] For example, regarding a function of providing detailed information about an object among functions of an application for providing search information, the accuracy level may be configured to be high and the real-time level may be configured to be low.

[0321] For example, when a video playback application provides search information about an object in a video, the real-time level may be configured as high. For example, regarding a function of providing text translation information in a translation application, the accuracy level may be configured as high.

[0322] According to an embodiment, the augmented reality device 1000 may be configured to select an artificial intelligence model for object recognition based on the accuracy level and real-time level of the service to be provided to the user. For example, as shown in Table 1, the calculation amount of the artificial intelligence model according to the accuracy level and real-time level may be pre-configured.

[0323] According to an embodiment, for example, when a user's gesture is detected and the type of the gesture is recognized, an artificial intelligence model that has a low computational amount and uses a large-sized partial image as an input may be selected.

[0324] According to an embodiment, the accuracy level and the real-time level of the service to be provided to the user can be changed according to whether the user's gaze is kept on the object to be recognized. For example, when the user's gaze is directed to the object (or around the object), the real-time level can be set to high, and when the user's gaze is not directed to the object (or around the object), the accuracy level can be set to high.

[0325] According to an embodiment, the augmented reality device 1000 may be configured to select a device for performing object recognition by driving an artificial intelligence model. The augmented reality device 1000 may be configured to select a device for performing object recognition from among the augmented reality device 1000, the external electronic device 2000, and the server 3000. For example, as shown in Table 2, the artificial intelligence models supported by the augmented reality device 1000, the external electronic device 2000, and the server 3000 may be pre-configured.

[0326] According to an embodiment, according to the service provision conditions, the device for identifying an object and / or the artificial intelligence model for identifying an object may be changed for identifying a captured image. For example, in the case of performing object recognition on continuously acquired captured images, the service provision conditions may be changed in the middle of performing object recognition on the captured images. In this case, the augmented reality device 1000 may be configured to change the device and AI model for object recognition by considering the service provision conditions, the performance information of the augmented reality device 1000, the performance information of the external electronic device 2000, and the accuracy and real-time capabilities of the service to be provided to the user.

[0327] When the device for recognizing an object and / or the artificial intelligence model for recognizing an object is changed, the augmented reality device 1000 may be configured to change the size of the partial image.

[0328] In operation 350-1, the augmented reality device 1000 may be configured to acquire a partial image including an object from the captured image. The augmented reality device 1000 may be configured to acquire a partial image including an object corresponding to the user's gaze from the captured image. The augmented reality device 1000 may be configured to identify a position in the captured image corresponding to the user's gaze, and to crop a partial image having a predetermined size around the identified position. In this case, the size of the partial image may be determined based on the size of the input image input to the selected artificial intelligence model. For example, the size of the input image of the artificial intelligence model may be pre-configured as shown in Table 2, and the augmented reality device 1000 may be configured to crop a partial image having the size of the input image of the selected artificial intelligence model from the captured image.

[0329] In Table 2, the input image of the artificial intelligence model is configured to have various sizes, but is not limited thereto. For example, the larger the computational complexity of the artificial intelligence model, the larger the configuration size of the input image corresponding to the artificial intelligence model.

[0330] The augmented reality device 1000 may be configured to additionally adjust the size of the partial image. The augmented reality device 1000 may be configured to additionally adjust the size of the partial image so that the partial image can be input to the artificial intelligence model.

[0331] In operation 360-1, the augmented reality device 1000 may be configured to obtain a result related to the object by using the selected artificial intelligence model to provide a service. The augmented reality device 1000 may be configured to obtain an object recognition result by using the acquired partial image and the selected artificial intelligence model.

[0332] When the first artificial intelligence model 10 of the augmented reality device 1000 is selected, the augmented reality device 1000 may be configured to input the partial image to the first artificial intelligence model 10 and obtain a result value output from the first artificial intelligence model 10 .

[0333] When the second artificial intelligence model 20 of the external electronic device 2000 is selected, the augmented reality device 1000 may be configured to provide the partial image to the external electronic device 2000, and the external electronic device 2000 may input the partial image to the second artificial intelligence model 20 of the external electronic device 2000. The external electronic device 2000 may acquire a result value output from the second artificial intelligence model 20, and provide the acquired result value for object recognition to the augmented reality device 1000. The external electronic device 2000 may be configured to acquire additional search information by using the object recognition result value, and may provide the acquired search information to the augmented reality device 1000.

[0334] When the third artificial intelligence model 30 of the server 3000 is selected, the augmented reality device 1000 may be configured to provide the server 3000 with a partial image through the external electronic device 2000. The augmented reality device 1000 may be configured to send a request to the external electronic device 2000 so that the server 3000 recognizes an object in the partial image, and the external electronic device 2000 may be configured to request the server for object recognition of the partial image in response to the request of the augmented reality device 1000. The server 3000 may be configured to receive the partial image from the external electronic device 2000 and input the partial image to the third artificial intelligence model so as to obtain an object recognition result. The server 3000 may be configured to provide the object recognition result to the external electronic device 2000, and the external electronic device 2000 may be configured to provide the object recognition result to the augmented reality device 1000. The external electronic device 2000 and / or the server 3000 may be configured to obtain additional search information by using the object recognition result value, and provide the obtained search information to the augmented reality device 1000.

[0335] The augmented reality device 1000 may provide a user with services related to an object. The augmented reality device 1000 may be configured to output an object recognition result and / or additional search information.

[0336] In a first example of the present disclosure, a method for identifying an object in an image through an augmented reality device is provided, the method comprising: acquiring a captured image including an object; identifying a user's gaze; identifying hardware performance information of the augmented reality device, hardware performance information of an external electronic device connected to the augmented reality device, and hardware performance information of a server; selecting a device for identifying the object from among the augmented reality device, the external electronic device, and the server based on the hardware performance information of the augmented reality device, the hardware performance information of the external electronic device, and the hardware performance information of the server; selecting an artificial intelligence model for identifying the object based on the hardware performance information of the augmented reality device, the hardware performance information of the external electronic device, and the hardware performance information of the server; acquiring a partial image including the object related to the user's gaze from the captured image; and acquiring a recognition result of the object from the partial image by using the selected device and the selected artificial intelligence model.

[0337] In a second example, the method of the first example is provided and further includes: identifying conditions for providing services to be provided to users, wherein the conditions for providing services include target accuracy, target latency, computing requirements, and communication status; wherein selecting an artificial intelligence model includes selecting an artificial intelligence model for identifying objects from a plurality of artificial intelligence models trained to identify objects based on the conditions for providing services, hardware performance information of the augmented reality device, hardware performance information of the external electronic device, and hardware performance information of the server.

[0338] In a third example, a second example is provided in which a condition for providing a service is identified based on at least one of: at least one type of service to be provided to a user, a type of application executed regarding recognition of an object, and a user input command requesting recognition of an object.

[0339] In a fourth example, the first example is provided, wherein the partial image has a size corresponding to the computational amount of the selected artificial intelligence model.

[0340] In a fifth example, a second example is provided in which a plurality of artificial intelligence models are configured to process a partial image by using different amounts of computation.

[0341] In a sixth example, a second example is provided, wherein, for object recognition regarding a plurality of captured images including captured images, at least one of a device for recognizing an object or an artificial intelligence model for recognizing an object is changeable based on a condition for providing a service, and a size of a partial image changes as at least one of the device for recognizing an object and an artificial intelligence model for recognizing an object is changed.

[0342] In a seventh example, a fourth example is provided, in which a plurality of artificial intelligence models are configured so that the number of bits of an output value of an activation function and the number of bits of a weight configured between layers are different from each other.

[0343] In an eighth example, a second example is provided, wherein identifying conditions for providing a service includes identifying the service provision conditions based on at least one of an attribute of the service to be provided to the user, a type of application executed in connection with identification of an object, or a voice input of a user requesting identification of an object.

[0344] In a ninth example, the eighth example is provided, wherein a condition for providing a service is determined based on the accuracy and real-time capability of a recognition result of an object.

[0345] In a tenth example, the first example is provided, further comprising identifying a resolution of an input image configured for the selected artificial intelligence model, wherein acquiring the partial image comprises cropping the partial image from the captured image so that the partial image has the identified resolution.

[0346] In an eleventh example, an augmented reality device for identifying objects in an image, the augmented reality device comprising: a communication interface configured to communicate with an external electronic device; a camera; a gaze tracking sensor configured to detect a user's gaze; a memory configured to store instructions; and a processor operably connected to the communication interface, the camera, the gaze tracking sensor, and the memory, and configured to execute instructions, wherein the processor executes the instructions to: acquire a captured image; control the gaze tracking sensor to identify the user's gaze; identify hardware performance information of the augmented reality device, hardware performance information of an external electronic device connected to the augmented reality device, and hardware performance of a server; based on the hardware performance information of the augmented reality device, the hardware performance information of the external electronic device, and the hardware performance of the server, select a device for identifying the object from the augmented reality device and the external electronic device; based on the hardware performance information of the augmented reality device, the hardware performance information of the external electronic device, and the hardware performance information of the server, select an artificial intelligence model for identifying the object; acquire a partial image including an object related to the user's gaze from the captured image; and acquire a recognition result of the object from the partial image by using the selected device and the selected artificial intelligence model.

[0347] In a twelfth example, the device of the eleventh example is provided, wherein a processor executes instructions to identify conditions for providing services to be provided to a user, and wherein the processor executes instructions to select an artificial intelligence model for recognizing an object from among a plurality of artificial intelligence models trained to recognize objects based on the conditions for providing the services, performance information of the augmented reality device, and performance information of the external electronic device.

[0348] In a thirteenth example, the apparatus of the eleventh example is provided, wherein the partial image has a size corresponding to the selected artificial intelligence model.

[0349] In a fourteenth example, the device of the twelfth example is provided, wherein, for object recognition of a plurality of captured images including captured images, at least one of a device for recognizing an object or an artificial intelligence model for recognizing an object is changeable based on a condition for providing a service, and a size of a local image changes as at least one of the device for recognizing an object and an artificial intelligence model for recognizing an object is changed.

[0350] In a fifteenth example, a computer-readable recording medium is provided, in which a program for executing a method for identifying an object in an image is recorded, the program including instructions that, when executed, cause the medium to perform operations including: acquiring a captured image including an object; identifying a user's gaze; identifying hardware performance information of an augmented reality device, hardware performance information of an external electronic device connected to the augmented reality device, and hardware performance information of a server; selecting a device for identifying an object from among the augmented reality device and the external electronic device based on the hardware performance information of the augmented reality device, the hardware performance information of the external electronic device, and the hardware performance information of the server; selecting an artificial intelligence model for identifying an object based on the hardware performance information of the augmented reality device, the hardware performance information of the external electronic device, and the hardware performance information of the server; acquiring a partial image including an object related to the user's gaze from the captured image; and acquiring a recognition result of the object from the partial image by using the selected device and the selected artificial intelligence model.

[0351] In a sixteenth example of the present disclosure, a method performed by an augmented reality device is provided, the method including: acquiring a captured image of a physical environment around the augmented reality device via a camera, the captured image including objects in the physical environment; receiving voice input from a user of the augmented reality device; recognizing the user's gaze; identifying a service related to the object included in the captured image and to be performed based on the voice input and the user's gaze; identifying conditions for providing the service, wherein the conditions for providing the service are related to at least one of a target accuracy of the service, a target delay of the service, computing requirements for the augmented reality device, or a communication state of the augmented reality device; based on the conditions for providing the service, selecting one of an artificial intelligence model in the augmented reality device, an artificial intelligence model in an external electronic device connected to the augmented reality device, or an artificial intelligence model in a server; obtaining a result related to the object from the selected artificial intelligence model; and providing the service to the user.

[0352] In the augmented reality device of the seventeenth example, the augmented reality device includes: a communication interface configured to communicate with an external electronic device; a camera; a gaze tracking sensor configured to detect the gaze of a user; a processor; and a memory storing instructions that, when executed by the processor, cause the augmented reality device to: acquire a captured image of the physical environment surrounding the augmented reality device via the camera, the captured image including objects in the physical environment; recognize the gaze of the user; receive voice input; identify a service to be performed based on the voice input and the user's gaze and related to the object included in the captured image; identify conditions for providing the service, wherein the conditions for providing the service are related to at least one of the target accuracy of the service, the target delay of the service, the computing requirements for the augmented reality device, or the communication status of the augmented reality device; based on the conditions for providing the service, select one of an artificial intelligence model in the augmented reality device, an artificial intelligence model in an external electronic device connected to the augmented reality device, or an artificial intelligence model in a server; obtain results related to the object from the selected artificial intelligence model; and provide the service to the user.

[0353] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0354] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or alternative forms for corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “combined with another element (e.g., the second element)”, “combined to another element (e.g., the second element)”, “connected with another element (e.g., the second element)”, or “connected to another element (e.g., the second element)” when the terms “operably” or “communicatively” are used or when the terms “operably” or “communicatively” are not used, it means that the element can be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0355] As used in connection with various embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, depending on the embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0356] The various embodiments described herein may be implemented as software (e.g., program 20040) including one or more instructions stored in a storage medium (e.g., internal memory 20036 or external memory 20038) that can be read by a machine (e.g., electronic device 20001). For example, under the control of a processor, a processor (e.g., processor 20020) of the machine (e.g., electronic device 20001) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be run by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), but the term does not distinguish between data being semi-permanently stored in a storage medium and data being temporarily stored in a storage medium.

[0357] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be released in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be downloaded via an application store (e.g., PlayStore). TM ) The computer program product may be published (e.g., downloaded or uploaded) online, or the computer program product may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, a server of an application store, or a forwarding server).

[0358] According to various embodiments, each component (for example, module or program) in the above-mentioned components can include a single entity or multiple entities, and some entities in multiple entities can be separately arranged in different components. According to various embodiments, one or more components in the above-mentioned components can be omitted, or one or more other components can be added. Selectively or additionally, multiple components (for example, module or program) can be integrated into a single component. In this case, according to various embodiments, the integrated component can still perform the one or more functions of each component in the multiple components in the same or similar manner as a corresponding component in the multiple components before integration. According to various embodiments, the operation performed by a module, program or another component can be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more operations in the operation can be run or omitted in different orders, or one or more other operations can be added.

Claims

1. A method performed by an augmented reality device including a camera, the method comprising: Acquire, via the camera, a captured image of a physical environment surrounding the augmented reality device, wherein the captured image includes objects in the physical environment; receiving a voice input from a user of the augmented reality device; identifying a gaze of the user; identifying a service to be performed based on the voice input and the user's gaze, related to the object included in the captured image; identifying a condition for providing the service, wherein the condition for providing the service is related to at least one of a target accuracy of the service, a target latency of the service, a computational demand on the augmented reality device, or a communication state of the augmented reality device; selecting, based on a condition for providing the service, one of an artificial intelligence model in the augmented reality device, an artificial intelligence model in an external electronic device connected to the augmented reality device, or an artificial intelligence model in a server; Obtaining results related to the object from the selected artificial intelligence model; and The service is provided to the user.

2. The method according to claim 1, further comprising acquiring a partial image including the object from the captured image, the partial image having a size corresponding to the selected artificial intelligence model, and in, Acquiring a result related to the object includes acquiring a result related to the object based on the partial image.

3. The method according to claim 2, wherein: Obtaining results related to the object includes: According to the selected artificial intelligence model in the augmented reality device, a result related to the object is obtained by using the artificial intelligence model in the augmented reality device; requesting a result related to the object while providing the partial image to the external electronic device according to the selected artificial intelligence model in the external electronic device; and According to the selected artificial intelligence model in the server, a result related to the object is requested while providing the partial image to the server through the external electronic device.

4. The method according to claim 1, wherein: The augmented reality device is connected to the external electronic device via short-range wireless communication, and The server is connected to at least one of the external electronic device or the augmented reality device via long-distance wireless communication.

5. The method according to claim 1, wherein: The artificial intelligence model in the augmented reality device, the artificial intelligence model in the external electronic device, and the artificial intelligence model in the server are configured so that the number of bits of the output value of the activation function and the number of bits of the weights configured between the layers are different from each other.

6. The method according to claim 2, wherein: For object recognition with respect to a plurality of photographed images including the photographed image, the artificial intelligence model selected for recognizing the object is changeable based on a condition for providing the service, and According to the selected artificial intelligence model being changed, the size of the partial image to be used by the selected artificial intelligence model is changed.

7. The method of claim 1, further comprising identifying a resolution of an input image configured for the selected artificial intelligence model, and in, Acquiring the partial image includes cropping the partial image from the captured image so that the partial image has the identified resolution.

8. The method according to claim 1, wherein: Select AI models include: When the communication state of the augmented reality device is stable, one of the artificial intelligence model in the external electronic device or the artificial intelligence model in the server is selected, and When the communication state of the augmented reality device is unstable, the artificial intelligence model in the augmented reality device is selected.

9. The method according to claim 1, wherein: Select AI models include: When the object is text and the service is a translation service, one of an artificial intelligence model in the external electronic device or an artificial intelligence model in the server is selected, and When the object is text and the service is a word search service, an artificial intelligence model in the augmented reality device is selected.

10. The method according to claim 1, wherein: The target accuracy and the target delay vary depending on whether the user's gaze is maintained relative to the object.

11. An augmented reality device, comprising: a communication interface configured to communicate with an external electronic device; camera; a gaze tracking sensor configured to detect a user's gaze; processor; as well as A memory storing instructions, which, when executed by the processor, cause the augmented reality device to: Acquire, via the camera, a captured image of a physical environment surrounding the augmented reality device, wherein the captured image includes objects in the physical environment; identifying a gaze of the user; Receive voice input; identifying a service to be performed based on the voice input and the user's gaze, related to an object included in the captured image; identifying a condition for providing the service, wherein the condition for providing the service is related to at least one of a target accuracy of the service, a target latency of the service, a computational demand on the augmented reality device, or a communication state of the augmented reality device; selecting, based on a condition for providing the service, one of an artificial intelligence model in the augmented reality device, an artificial intelligence model in an external electronic device connected to the augmented reality device, or an artificial intelligence model in a server; Obtaining results related to the object from the selected artificial intelligence model; and The service is provided to the user.

12. The augmented reality device according to claim 11, wherein: The memory stores instructions, which, when executed by the processor, cause the augmented reality device to: acquiring a partial image including the object from the captured image, the partial image having a size corresponding to the selected artificial intelligence model; and A result related to the object is acquired based on the partial image.

13. The augmented reality device according to claim 12, wherein: The memory stores instructions, which, when executed by the processor, cause the augmented reality device to: According to the selected artificial intelligence model in the augmented reality device, a result related to the object is obtained by using the artificial intelligence model in the augmented reality device; requesting a result related to the object while providing the partial image to the external electronic device according to the selected artificial intelligence model in the external electronic device; as well as According to the selected artificial intelligence model in the server, a result related to the object is requested while providing the partial image to the server through the external electronic device.

14. The augmented reality device according to claim 11, wherein: The artificial intelligence model in the augmented reality device, the artificial intelligence model in the external electronic device, and the artificial intelligence model in the server are configured so that the number of bits of the output value of the activation function and the number of bits of the weights configured between the layers are different from each other.

15. A computer-readable recording medium having a program for executing a method therein, the program comprising instructions which, when executed, cause the medium to perform operations comprising: Acquire, via a camera, a captured image of a physical environment surrounding the augmented reality device, wherein the captured image includes objects in the physical environment; receiving a voice input from a user of the augmented reality device; Identify the user’s gaze; identifying a service to be performed based on the voice input and the user's gaze, related to the object included in the captured image; identifying a condition for providing the service, wherein the condition for providing the service is related to at least one of a target accuracy of the service, a target latency of the service, a computational demand on the augmented reality device, or a communication state of the augmented reality device; selecting, based on a condition for providing the service, one of an artificial intelligence model in the augmented reality device, an artificial intelligence model in an external electronic device connected to the augmented reality device, or an artificial intelligence model in a server; Obtaining results related to the object from the selected artificial intelligence model; and The service is provided to the user.

Images