Apparatus and method for displaying content on an augmented reality device

Abstract

A system for displaying content on an augmented reality (AR) device includes: a capture module configured to capture a user's field of view; a recording module configured to record the captured field of view; a user input controller configured to track the user's vision toward one or more objects; and a server. The server includes a determination module, an identifier, and an analyzer. The determination module is configured to determine at least one object of interest. The identifier is configured to identify a frame containing the disappearance of the determined object of interest. The analyzer is configured to analyze the identified frame based on at least one disappearance of the object of interest and generate analyzed data. The system also includes: a display module configured to display the content of the object of interest on the AR device.

Description

Technical Field

[0001] The present disclosure relates generally to augmented reality content, and particularly, but not exclusively, to systems and methods for displaying content on an augmented reality (AR) device. Background Art

[0002] Augmented reality (AR) is an interactive experience of the real-world environment, whereby objects residing in the real world are enhanced by computer-generated perceptual information across multiple sensory modalities (including vision, hearing, etc.). Augmented reality (AR) supports real-time enhancement of content by understanding the user's field of view and by using scene and object detection technology. Users can experience augmented reality (AR) using wearable augmented reality (AR) devices.

[0003] Existing augmented reality systems use one or more object tracking technologies, such as digital cameras and / or other optical sensors, accelerometers, global positioning systems (GPS), solid-state compasses, and radio frequency identification (RFID) systems. Object tracking technologies provide varying levels of accuracy and precision. One of the technologies is to track eye gaze and the position and orientation of the user's head. Eye tracking in an AR device is the process of measuring the gaze point that the user is looking at or the movement of the eyes relative to the head.

[0004] Generally, each person has his own speed of reading / viewing / observing information / text / object. If the object or the person is moving, his ability to read / view / observe the information / text / object is further reduced. During motion, the user needs more time to comprehend the information. The user can view any object only during a specific time interval, during which the object is in the user's field of view and there is no obstacle between the user and the object.

[0005] Obstacles have one or more characteristics, such as, an obstacle is an object that is large enough to partially or completely hide the "user's object of interest" so that the user cannot properly see the object of interest (e.g., a train or car comes between the user and the object of interest). Obstacles can be environmental conditions such as reduced visibility due to rain, fog, etc. Obstacles can be a reduction in the size of the object of interest due to relative motion. In addition, an obstacle can be a change in the state / direction / orientation of the object containing the object of interest, which causes the object of interest to disappear (e.g., the user is interested in reading a slogan on a T-shirt, and someone else wearing the T-shirt turns to another direction). Obstacles may be caused by limitations in the user's visual area.

[0006] The user cannot prevent any obstacles from coming between him and the object of interest. The user has no control over the speed and direction of the object of interest's movement, its appearance (e.g., color / size, etc.), and / or environmental factors (e.g., visibility, wind, etc.).

[0007] US Patent No. 9633479 discloses time-constrained augmented reality. In this patent, a predefined list of items is used to identify documents from a scene. The identified document may not be of interest to the user. US 9633479 discloses displaying content related to the document until the moment when the document is available in the AR view. The virtual content overlaps the document in AR. In addition, the viewing time of the scene is determined based on the relative motion between the AR device and the scene. The viewing time is an estimated time period during which the scene will be visible to the AR device. In addition, based on the determined viewing time, virtual content is selected from a predetermined range of virtual content. For example, if the viewing time is 10 seconds, a video with a playback time of approximately 10 seconds is selected. The type of virtual content (image / video) is selected based on a comparison of the determined viewing time and based on the relative motion of the object and the user. US 9633479 relates to displaying advertisements associated with posters.

[0008] U.S. Patent Application Publication No. 20150113454 discloses the use of eye tracking technology to deliver contextual data to a computing device. US20150113454 utilizes eye gaze instead of a mouse / touchpad. US20150113454 does not disclose any features for calculating the content complexity of an object of interest. Here, objects that are not in view are not considered. In addition, once the user starts viewing the object from a minimum dwell time (i.e., a threshold time), an action is taken. Here, only objects in view are considered. In addition, the dilation of the pupil of the eye is not considered when identifying an object of interest.

[0009] European patent application publication No. EP1422922 relates to generating and saving data related to a user's region of interest. It considers eye gaze to identify the user's region of interest, and saves the user's region of interest with the captured image. In this publication, objects that have moved out of view are not considered. The content complexity of the user's region of interest is also not calculated. EP1422922 discloses determining eye information including the eye gaze direction of the user's eye of a camera system during an image capture sequence, and then associating the eye information with the captured image.

[0010] None of the prior art documents determine that the user is interested in seeing the disappearing object. In addition, none of the prior art documents determine the unintentional disappearance or intentional disappearance of the user's object of interest, and perform an analysis of obstacles between the user and the object of interest. None of the prior art documents considers the following items to automatically identify the object of interest in real time and display the object of interest on the AR device: by the unintentional disappearance of the object of interest or intentional disappearance (despite the user's interest), any threshold eye gaze time on the object to be considered as an "object of interest", the content complexity of the object of interest used to determine the duration required for the user to properly see the object, and environmental changes (such as lighting conditions, fog, rain) that cause the visibility of the object of interest to be reduced as a disappearance condition. The prior art documents also do not display the object of interest for an extended duration without the user's explicit request, and do not disclose a predefined location at which the user in the AR device can gaze to request the display of the AR content of the object of interest even when the object of interest is clearly visible to the user.

[0011] Therefore, there is a need for a technique for displaying content on an augmented reality (AR) device that limits the above-mentioned disadvantages. Summary of the invention

[0012] Solution to the problem

[0013] This summary is provided to introduce concepts related to displaying content on augmented reality (AR) devices. This summary is neither intended to identify essential features of the disclosure nor to be used to determine or limit the scope of the disclosure.

[0014] For example, various embodiments herein may include one or more systems and methods for displaying content on an augmented reality (AR) device.

[0015] According to various embodiments of the present disclosure, a computer-implemented method for displaying content on an augmented reality (AR) device may include: capturing a user's field of view, recording the captured field of view, tracking the user's vision toward one or more objects in the recorded field of view, determining at least one object of interest in the recorded field of view, identifying a frame containing a determined object of interest that disappears in the recorded field of view, analyzing the identified frame based on at least one disappearance of the object of interest, generating analyzed data based on the analyzed frame, and displaying content of the object of interest on the augmented reality (AR) device based on the analyzed data.

[0016] According to various embodiments of the present disclosure, a computer-implemented system for displaying content on an augmented reality (AR) device includes: a memory configured to store predefined rules; a processor configured to cooperate with the memory, wherein the processor is configured to generate a system processing command based on the predefined rules; a capture module configured to capture a user's field of view; a recording module configured to record the captured field of view; a user input controller configured to track the user's vision toward one or more objects in the recorded field of view; and a server configured to cooperate with the AR device, the recording module, and the user input controller. The server may include: a determination module configured to determine at least one object of interest in the recorded field of view; an identifier configured to cooperate with the determination module to receive the determined object of interest, wherein the identifier is configured to identify a frame containing the determined object of interest disappearing in the recorded field of view; and an analyzer configured to cooperate with the identifier to receive the identified frame, wherein the analyzer is configured to analyze the identified frame based on at least one disappearance of the object of interest. The system further includes a display module configured to display the content of the object of interest based on the analyzed data on an augmented reality (AR) device.

[0017] According to various embodiments of the present disclosure, a computer-implemented method for displaying content on an augmented reality (AR) device includes: capturing a user's field of view, tracking the user's eye gaze toward an object of interest in the captured field of view, identifying frames containing intentional disappearance of the object of interest in the captured field of view, analyzing the identified frames based on the intentional disappearance, generating analyzed data based on the analyzed frames, and displaying content of the object of interest on the augmented reality (AR) device based on the analyzed data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The detailed description is described with reference to the accompanying drawings. In the accompanying drawings, the leftmost digit of a reference number identifies the drawing in which the reference number first appears. The same reference numbers are used throughout the drawings to refer to the same features and modules.

[0019] Figure 1 A schematic diagram is shown depicting a computer-implemented system for displaying content on an augmented reality (AR) device according to an exemplary embodiment of the present disclosure.

[0020] Figure 2 FIG. 1 shows an exemplary embodiment according to the present disclosure. Figure 1 A block diagram of a computer-implemented system for displaying content on an augmented reality (AR) device.

[0021] Figure 3A schematic diagram showing an unintentional disappearance of a depiction according to an exemplary embodiment of the present disclosure is shown.

[0022] Figure 4 A schematic diagram depicting intentional disappearance of a first object (object 1) according to an exemplary embodiment of the present disclosure is shown.

[0023] Figure 5 A flow chart depicting a process for displaying content on an augmented reality (AR) device according to an exemplary embodiment of the present disclosure is shown.

[0024] Figure 6 A flow chart depicting a process for displaying content on an augmented reality (AR) device while a user is looking at an object of interest, according to an exemplary embodiment of the present disclosure, is shown.

[0025] Figure 7 A flow chart depicting a process for removing multimedia content from an AR device when a user moves his gaze away from the multimedia content according to an exemplary embodiment of the present disclosure is shown.

[0026] Figure 8 A flow chart depicting a process for filtering / enhancing and augmenting recorded content and displaying the content to a user and removing the multimedia content from an AR device when the user moves his gaze away from the multimedia content according to an exemplary embodiment of the present disclosure is shown.

[0027] Fig. 9 A flow chart depicting a process for displaying multimedia content of an object of interest to a user in an AR device within a calculated duration according to an exemplary embodiment of the present disclosure is shown.

[0028] Fig.10 A flow chart depicting a process for displaying content on an augmented reality (AR) device according to an exemplary embodiment of the present disclosure is shown.

[0029] Fig.11 A flow chart depicting a process for displaying content on an augmented reality (AR) device with intentional disappearance according to an exemplary embodiment of the present disclosure is shown.

[0030] Fig.12 A flow chart depicting a process for displaying content on an augmented reality (AR) device in the event of an unintentional disappearance of an object of interest or despite an intentional disappearance of an object of interest according to an exemplary embodiment of the present disclosure is shown.

[0031] Fig.13 A use case scenario of depicting an intentional disappearance of a user's interest according to an exemplary embodiment of the present disclosure is shown.

[0032] Fig.14A use case scenario of depicting an intentional disappearance of a user's interest according to an exemplary embodiment of the present disclosure is shown.

[0033] Fig.15 A use case scenario of depicting unintentional disappearance according to an exemplary embodiment of the present disclosure is shown.

[0034] FIG. 16A to FIG. 16C A use case scenario in which depiction is unintentional and obstructive disappearance is shown according to an exemplary embodiment of the present disclosure.

[0035] FIG. 17A to FIG. 17B A use case scenario depicting unintentional disappearance of an object due to loss of visibility of an obstacle between a user and the object is shown according to an exemplary embodiment of the present disclosure.

[0036] 18A to 18D A use case scenario of depicting an intentional disappearance of a user's interest according to an exemplary embodiment of the present disclosure is shown.

[0037] FIG. 19A to FIG. 19F A use case scenario of depicting an intentional disappearance of a user's interest according to an exemplary embodiment of the present disclosure is shown.

[0038] FIG. 20A to FIG. 20C A use case scenario depicting intentional disappearance of an object of interest due to relative motion between a user and an object is shown according to an exemplary embodiment of the present disclosure.

[0039] FIG. 21A to FIG. 21F A use case scenario of disappearance of augmented reality (AR) content depicting a previous object of interest is illustrated according to an exemplary embodiment of the present disclosure.

[0040] FIG. 22A to FIG. 22C A use case scenario of depicting intentional and non-obstructive disappearance according to an exemplary embodiment of the present disclosure is shown.

[0041] FIG. 23A to FIG. 23F A use case scenario depicting an unintentional obstructive disappearance of a high-speed race car rapidly disappearing from a user's view is shown in accordance with an exemplary embodiment of the present disclosure.

[0042] Fig.24 A use case scenario depicting a process of performing mirroring on content of an object of interest displayed on one or more computing devices is shown according to an exemplary embodiment of the present disclosure.

[0043] It should be understood by those skilled in the art that any block diagram herein represents a conceptual view of an illustrative system that embodies the principles of the present disclosure. Similarly, it should be understood that any flow chart, flow diagram, etc. represent various processes that can be substantially represented in a computer-readable medium and thus executed by a computer or processor, regardless of whether such a computer or processor is explicitly shown. DETAILED DESCRIPTION

[0044] In the following description, for the purpose of explanation, specific details are set forth in order to provide an understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these details. Those skilled in the art will recognize that embodiments of the present disclosure (some of which are described below) may be incorporated into multiple systems.

[0045] Various embodiments of the present disclosure provide a computer-implemented system and method for displaying content on an augmented reality (AR) device.

[0046] Furthermore, the connections between components and / or modules in the drawings are not intended to be limited to direct connections. Instead, these components and modules may be modified, reformatted or otherwise altered by intermediate components and modules.

[0047] References in this disclosure to "one embodiment" or "an embodiment" mean that a particular feature, structure, characteristic, or function described in conjunction with the embodiment is included in at least one embodiment of the present invention. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment.

[0048] A computer-implemented system for displaying content on an AR device (hereinafter referred to as the "system") provides analysis of a user's viewing behavior at a certain moment through the user's eye movement, head movement, and gaze time, which can be easily captured using an eye-tracking camera present on the AR device. In addition, the system also identifies whether an object in the user's field of view is of interest to him by using multiple parameters (e.g., eye gaze, eye size, pupil size enlargement, movement of eyebrows, pupils, and irises toward an object, blinking, head movement, sudden high-speed movement of the head in the direction of an object, head movement for tracking an object until the object disappears, repeated viewing of the same object with multiple short gazes, and a multi-model based on adaptive learning artificial intelligence (AI) for identifying the user's interest in the object).

[0049] The system is configured to identify a user's interest in an object. Using object recognition and artificial intelligence (AI) techniques, the system is configured to identify any type of obstacle between the user and the object of interest. The system also determines the intentional or unintentional disappearance of the object of interest.

[0050] In an embodiment, the AR device's sensors collect the user's real-world interactions and communicate them for processing and interpretation. Cameras are also located outside the AR device and visually scan to collect data about the surrounding area. The device acquires information that typically determines where the surrounding physical objects are located and then develops a digital model to determine the appropriate output. For example, in the case of the Microsoft HoloLens, specific cameras perform specific duties, such as depth sensing. The depth sensing camera works in conjunction with two "environmental understanding cameras" on each side of the AR device. Another common type of camera is a standard multi-megapixel camera (similar to the cameras used in smartphones) that is used to record pictures, videos, and sometimes information to help enhance.

[0051] In another embodiment, eye tracking is a technique that measures eye activity and gaze movement. Eye tracking technology solves human-computer interaction problems with AR and improves the overall visual quality of the AR experience. Different types of eye movements are being studied and used in eye gaze research and applications to collect information about user intent, cognitive processing, behavior, and attention analysis. There are different types of eye movements, such as fixation, saccade, smooth pursuit, scan path, fixation duration, pupil size and blink, vergence, and vestibulo-ocular reflex.

[0052] Gaze: During gaze, the eyes are stationary between movements and visual input. Gazes are those times when the eyes essentially stop scanning around the scene, keeping the foveal vision in a position that allows the visual system to obtain detailed information about what is being looked at. Gaze-related measurement variables include total gaze duration, average gaze duration, gaze spatial density, number of gaze regions, gaze sequence, and gaze rate.

[0053] Saccades: Saccades are rapid and involuntary eye movements that occur between fixations. Measurable saccade-related parameters include the number of saccades, amplitude, and fixation-saccade ratio. Saccades can be triggered voluntarily or involuntarily. In this, both eyes move in the same direction.

[0054] Smooth pursuit is used to keep the fovea aligned with the moving object. In the absence of a moving target, smooth pursuit cannot be triggered autonomously.

[0055] Scanpath: A scanpath consists of a series of alternating short fixations and saccades before the eyes reach a target location on the screen. Movement measures derived from the scanpath include scanpath direction, duration, length, and area covered.

[0056] Fixation duration: the sum of all fixations made in the visual field of interest before the eyes leave that field and the proportion of time spent in each field.

[0057] Pupil Size and Blinking: Measures pupil size and blink rate.

[0058] Vergence movement is used to help us focus on objects placed at different distances with the left and right eyes moving in opposite directions.

[0059] The vestibulo-ocular reflex is used to keep the fovea pointed toward a point of interest as the head and body move.

[0060] Figure 1 A schematic diagram is shown depicting a computer-implemented system 100 for displaying content on an AR device according to an exemplary embodiment of the present disclosure.

[0061] System 100 includes an AR device 102 , a network 104 , and a server 106 .

[0062] The AR device 102 is associated with a user (A). The AR device 102 is communicatively connected to a server 106 via a network 104. In an embodiment, the AR device 102 includes a display screen, a global positioning system (GPS), a speaker, and a headset or earphones that can be worn on the user's ears. The AR device 102 also includes one or more sensors for sensing the user's motion and action. In one embodiment, the sensor includes at least one of an acceleration sensor, a tilt sensor, a gyroscope sensor, a three-axis magnetic sensor, and a proximity sensor.

[0063] In one embodiment, the network 104 includes a wired network and a wireless network. Examples of wired networks include a wide area network (WAN) or a local area network (LAN), a client-server network, a peer-to-peer network, etc. Examples of wireless networks include Wi-Fi, a global system for mobile communications (GSM) network and a general packet radio service (GPRS) network, an enhanced data GSM environment (EDGE) network, an 802.5 communication network, a code division multiple access (CDMA) network, a Bluetooth network, or a long term evolution (LTE) network, an advanced LTE (LTE-A) network, or a fifth generation (5G) network.

[0064] Figure 2 FIG. 1 shows a diagram depicting an exemplary embodiment of the present disclosure. Figure 1 Block diagram 200 of a computer-implemented system for displaying content on an AR device 102 .

[0065] The system 100 includes an AR device 102 , a server 106 , and a database 220 .

[0066] The system 100 also includes a memory 202, a processor 204, a capture module 206, a recording module 208, a user input controller 210, and a display module 218. In an embodiment, the memory 202, the processor 204, the capture module 206, the recording module 208, the user input controller 210, and the display module 218 are near the AR device 102 or external to the AR device 102 and independently perform various functions.

[0067] The memory 202 is configured to store predetermined rules related to the identification of locations, data extraction, determination of information, identification of text and images, and content complexity. In an embodiment, the memory 202 may include any computer-readable medium known in the art, including, for example, volatile memory (such as static random access memory (SRAM) and dynamic random access memory (DRAM)) and / or non-volatile memory (such as read-only memory (ROM), erasable programmable ROM, flash memory, hard disk, optical disk, and tape). The memory 202 also includes a cache memory to work more efficiently with the system 100.

[0068] The processor 204 is configured to cooperate with the memory 202 to read or load the predetermined rules. The processor 204 is also configured to generate system processing commands. In an embodiment, the processor 204 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits and / or any device that manipulates signals based on operating instructions. In other capabilities, the processor 204 is configured to extract the predetermined rules from the memory 202 and run different modules of the system 100.

[0069] The capture module 206 is configured to capture the user's field of view. In one embodiment, the capture module 206 includes a camera or a scanner. The capture module 206 is configured to capture a real-time view of a user associated with the AR device 102.

[0070] The recording module 208 is configured to cooperate with the capture module 206 to receive the captured user's field of view. The recording module 208 is also configured to record the captured user's field of view.

[0071] The user input controller 210 is configured to cooperate with the recording module 208 to receive the recorded field of view. The user input controller 210 is also configured to track the user's vision towards one or more objects in the recorded field of view. In an embodiment, the user input controller 210 is configured to track the user's vision based on at least one of the following parameters: eye gaze, eye movement direction, head movement direction, pupil dilation, number of blinks, head rotation angle, head rotation speed, facial expression, eye size dilation, pupil and iris movement, voice recognition, and / or other similar parameters.

[0072] The user input controller 210 includes an eye gaze tracking module 212 , a head movement tracking module 214 , and a speech recognition module 216 .

[0073] The eye gaze tracking module 212 is configured to track the user's vision by identifying eye gaze, eye movement direction, pupil dilation, number of blinks, facial expressions, eye size enlargement, pupil and iris movement, and / or other vision-related parameters. The eye gaze tracking module 212 is configured to obtain information about the user's focus area to determine which real object (or enhancement of a real object) the user is currently focusing on. The eye gaze tracking module 212 is configured to track the eye movement of multiple real objects and enhanced objects relative to the head. In one embodiment, the eye gaze tracking module 212 is configured to collect eye movement data and eye gaze time of real objects that will be sent as input to the estimation module 232 and eye gaze time of enhanced content. When there is no relative movement between the user and the object of interest, there will be no or minimal eye movement. The eyes will remain at a comfortable angle, usually in a "looking forward" position (3D focus area). The user will continue to look at the object, and an obstacle will appear between the user and the object of interest. When there is relative movement between the user and the object of interest, the user continues to turn his eyes to follow the object until the object disappears behind the obstacle. The pupil of the eye will remain dilated at the object of interest vanishing point, showing the user's interest in the object of interest at the vanishing point. The number of blinks will remain less than normal. At the vanishing point, the eyes will remain larger than usual.

[0074] The head movement tracking module 214 is configured to track the direction of head movement, the angle of head rotation, the speed of head rotation and other head related parameters. The head movement tracking module 214 is configured to collect information of head movement expressed in degrees per second, which will be sent as input to the estimation module 232. The head movement tracking module 214 is configured to track whether the user turns his head to the maximum comfortable degree to follow any object. When there is no relative movement between the user and the object of interest, there will be no or minimal head movement. The head will remain at a comfortable angle, usually in a "look forward" position (3D focus area). The user continues to look at the object, and an obstacle appears between the user and the object. When there is relative movement between the user and the object of interest, the user continues to turn his head to follow the object until the object disappears behind the obstacle.

[0075] The speech recognition module 216 is configured to track speech recognition of a user.

[0076] In an embodiment, the system 100 further comprises a database 220. The database 220 is configured to store recorded fields of view, predefined time intervals, predefined object categories, and data related to one or more objects, predefined rotation angles, and a rotation speed of the user about at least one angle toward the one or more objects.

[0077] The server 106 is configured to cooperate with the AR device 102, the recording module 208, and the user input controller 210. The server 106 includes a determination module 224, an identifier 226, and an analyzer 230.

[0078] The determination module 224 is configured to determine at least one object of interest in the recorded field of view.

[0079] The identifier 226 is configured to cooperate with the determination module 224 to receive the determined object of interest. The identifier 226 is also configured to identify frames containing the determined object of interest disappearing from the recorded field of view. In an embodiment, the identifier 226 is configured to identify frames based on the intentional disappearance or unintentional disappearance of the object of interest in the recorded field of view. In another embodiment, the identifier 226 is configured to identify frames based on visual time and threshold time.

[0080] In an embodiment, the recognizer 226 includes an object recognition module 228. The object recognition module 228 is configured to detect and recognize the object of interest from the recognized frames using deep learning techniques. The deep learning techniques include object recognition techniques, deep structured learning techniques, hierarchical learning techniques, or neural networks.

[0081] The analyzer 230 is configured to cooperate with the recognizer to receive the recognized frame. The analyzer 230 is also configured to analyze the recognized frame based on at least one intended disappearance and generate analyzed data. In an embodiment, the analyzer 230 is configured to analyze the recognized frame based on the intentional disappearance or unintentional disappearance of the object of interest in the recorded field of view. In another embodiment, the analyzer 230 is configured to analyze the recognized frame based on the intentional disappearance by identifying the user's lack of interest in the one or more objects.

[0082] In another embodiment, the analyzer 230 further includes an estimation module 232. The estimation module 232 is configured to determine a desired probability of the user being in the field of view based on the user's interest in the object. The analyzer 230 is also configured to identify a user's lack of interest in one or more objects.

[0083] In another embodiment, the analyzer 230 is configured to analyze the identified frames based on the unintentional disappearance by determining an obstruction between the user and the object of interest. In an embodiment, the obstruction between the user and the object of interest is determined based on visual analysis, object recognition, eye movement, head movement, or fixation gaze time for another object after the object of interest disappears. In an embodiment, the obstruction between the user and the object of interest is determined by using multiple sensors.

[0084] In an embodiment, the analyzer 230 is configured to analyze the identified frames based on intentional disappearance. The analyzer 230 is configured to analyze the user's vision for the object of interest within a predefined time interval. The analyzer 230 is also configured to use an artificial intelligence-based multi-parameter model to identify intentional non-obstructive disappearance of the object of interest. The model includes parameters related to the following items: the user's eye movement, the user's level of engagement, the companions with the user, and the object category. The analyzer 230 is also configured to perform analysis of the data received from multiple sensors and the viewing data within a threshold time, and then identify the gaze at a predetermined position of the AR device 102. In one embodiment of multiple embodiments, the intentional disappearance of an object is a disappearance despite the user's interest in the object.

[0085] The display module 218 is configured to display the content of the object of interest based on the analyzed data on the AR device 102. The display module 218 is configured to display the content of the object of interest on the AR device 102, wherein the content includes a fragment of the object of interest or a modified version of the object of interest. The modified version of the object of interest is generated by filtering and enhancing the fragment of the object of interest. The steps of filtering and enhancing the fragment of the object of interest include filtering and enhancing images, videos, enhanced information related to the object of interest, and a combination of images, videos, and enhanced information related to the object of interest.

[0086] The display module 218 is further configured to highlight the object of interest based on gaze and threshold time at a predefined location in the AR device 102. In an embodiment, the threshold time is variable based on a plurality of parameters.

[0087] In an embodiment, the display module 218 is configured to perform mirroring on the AR device 102 of the content of the object of interest displayed on one or more computing devices in the event of an unintentional disappearance of the object of interest or despite the intentional disappearance of the object of interest in which the user is interested, and when the user views the object of interest for more than a threshold amount of time. In an exemplary embodiment, when the user views the display for more than a threshold amount of time, the mirroring of the display is completed using the AR device 102 when the display disappears. For example, there is a heartbeat monitor in the operating room. The doctor needs to frequently view the heartbeat monitor during surgery. The doctor wears the AR device 102 and views the heartbeat monitor for a threshold amount of time. The doctor turns his head away from the heartbeat monitor and starts the operation. A real-time display of the heartbeat monitor is shown in the doctor's AR device 102. Therefore, he does not need to turn his head to the heartbeat monitor every time to view the heartbeat data. Multiple computing devices can be mirrored to a single AR device.

[0088] In an embodiment, the server 106 includes a transmitter 234. The transmitter is configured to cooperate with the analyzer 230 to receive the analyzed data. The transmitter 234 is also configured to send the analyzed data to the AR device 102. In an embodiment, the transmitter 234 is also configured to send the analyzed data from the AR device 102 to another AR device by using the location of the other AR device, artificial intelligence (AI) technology, or by using a profile. In an embodiment, the display module 218 is configured to display the content of the object of interest based on the analyzed data on the other AR device.

[0089] In an embodiment, the server 106 further includes a complexity identifier 236. The complexity identifier 236 is configured to calculate the time required for the user to view the object of interest based on the total time required for the user to view the object of interest and the duration for which the user has viewed the object of interest before the object of interest disappears. In an embodiment, the time required for the user to view the object of interest is calculated by subtracting the duration for which the user has viewed the object of interest before the object of interest disappears from the total time required for the user to view the object of interest.

[0090] In another embodiment, the complexity identifier 236 determines the amount of information (e.g., object complexity) in the user's object of interest to a satisfactory level, thereby calculating the total time required for the user to properly view / understand the object of interest. In this case, the duration that the object of interest is available in the user's view is less than the total time required for the user to properly view / understand the object, and the user remains interested in the object of interest. The complexity identifier 236 provides the digital image or multimedia content (e.g., video) of the user's object of interest on the AR device 102 within the calculated duration for the additional time the user needs to properly view / understand the object of interest. The formula for calculating the duration is as follows:

[0091] Duration = (total time required to view the object as calculated by the system) - (duration the user has viewed the object)

[0092] In an exemplary embodiment, Table 1 shows a list of some parameters among the parameters used to calculate the content complexity, but is not limited thereto. The complexity identifier 236 may assign different weights to different parameters in different scenarios to accurately calculate the content complexity.

[0093]

[0094]

[0095] Table 1

[0096] In another embodiment, the system 100 includes a content rendering module 238. The content rendering module 238 is configured to render the location of content on the AR device 102 based on the user's gaze, the user's preferences, and the current view content.

[0097] In another embodiment, the capture module 206 is configured to capture the user's field of view. The user input controller 210 is configured to track the user's eye gaze toward the object of interest in the captured field of view. The identifier 226 is configured to identify frames containing intentional disappearance of the object of interest in the captured field of view. The analyzer 230 is configured to analyze the identified frames based on the intentional disappearance and generate analyzed data. The display module 218 is configured to display the content of the object of interest on the AR device 102 based on the analyzed data.

[0098] In another embodiment, the capture module 206 is configured to capture the user's field of view. The recording module 208 is configured to record the captured field of view. The user input controller 210 is configured to track the user's eye gaze toward the object of interest in the recorded field of view. The identifier 226 is configured to identify frames containing at least one of an unintentional disappearance of the object of interest in the recorded field of view or an intentional disappearance despite interest. The analyzer 230 is configured to analyze the identified frames based on at least one of the unintentional disappearance of the object of interest or the intentional disappearance despite interest, and generate analyzed data. The display module 218 is configured to display the content of the object of interest on the AR device 102 based on the analyzed data.

[0099] In an embodiment, the system 100 identifies the unintentional disappearance or intentional disappearance of the object of interest, and determines the user's interest in viewing the object of interest even after the object disappears. More specifically, the system 100 provides a digital image or multimedia content of the object of interest on an AR device 102 (e.g., AR glasses).

[0100] In an exemplary embodiment, when a user is interested in seeing an object in a surrounding environment in a different surrounding environment, it is mainly because of the disappearance of the object of interest. The system 100 determines whether the disappearance of the object of interest is unintentional or intentional. The system 100 considers the following disappearance scenarios:

[0101] Unintentional disappearance (based on obstacles): In this scenario, the disappearance of the object of interest is not under the control of the user. The object disappears due to factors that are not under the control of the user. For example, an obstacle appears between the user and the object of interest, or the user wants to see the object of interest but the object of interest moves behind a wall. In an embodiment, visual analysis, object recognition, and monitoring of eye and head movements using data from various sensors of the AR device 102 are used to determine the obstacle between the user and the object of interest. If the disappearance is unintentional (i.e., caused by an obstacle), multimedia content of the object of interest is created and displayed on the AR device 102.

[0102] Intentional disappearance despite user interest in the object (based on non-obstruction): Here are some example embodiments of intentional disappearance:

[0103] i. While the object of interest is still available in the user's view, the user moves the gaze to some other object. For example, the user looks at a person while avoiding direct eye contact with him / her.

[0104] ii. The user is forced to look away from the object of interest because the object is beyond the user's comfortable viewing range. For example, the object of interest walks past the user and goes behind the user.

[0105] iii. The user may gaze at a predefined location in the AR device 102 after moving his gaze away from the object of interest.

[0106] In this scenario, after the user moves his gaze away from the object of interest, the user's interest in viewing the object of interest is determined by using an AI model, profiling the user data or data from various sensors of the AR device 102. In an embodiment, multimedia content is created for the object of interest and displayed in the AR device 102.

[0107] Intentional disappearance due to user's lack of interest in the object (non-obstruction-based): The user intentionally moves his gaze away from the object because the user is no longer interested in viewing the object. In this scenario, after the user moves his gaze away from the object, the user's lack of interest in viewing the object of interest is determined by profiling the user data or data from various sensors of the AR device 102.

[0108] In an embodiment, the system 100 is configured to identify an object of interest to the user based on data from various sensors in the AR device 102. Various sensors available in the AR device 102 may be used to identify the identity of the object under gaze and determine the duration of gaze on the object. The following is a list of sensor data that may be used for this purpose:

[0109] i. Eye and head movements

[0110] a. Head movement

[0111] b. Eye movement

[0112] c. Pupil dilation

[0113] d. Number of blinks

[0114] e. Eye enlargement

[0115] f. Pupil and iris movement

[0116] ii. Gaze time

[0117] a. Stare

[0118] b. Scan

[0119] c. Smooth tracking

[0120] d. Cumulative duration of fixation on the object

[0121] When the user looks at an object for more than a threshold time, the object is considered to be an object of interest. The total time the user has looked at the object is tracked.

[0122] Figure 3 A schematic diagram 300 depicting unintentional disappearance is shown according to an exemplary embodiment of the present disclosure.

[0123] Figure 3 The unintentional disappearance due to an obstacle is shown. The obstacle is caused by the wall 5. Specifically, Figure 3, when the object of interest disappears from the user's field of view 10, when the object of interest has disappeared behind the wall 5, the system 100 displays the multimedia content of the object of interest. At point 1, the object appears in the user's field of view 10. The user begins to continuously look at the object at time t1 (i.e., at 16:10:40). At point 2, the user looks at the object for a threshold time, and the object becomes an "object of interest". In an embodiment, the user looks at the object for two seconds, for example, the user looks at the object of interest from t1 to t2 (i.e., t2=16:10:42). The system automatically calculates the total time (t2-t1) of looking at the object of interest. At point 3, the user is still watching the object of interest, but the object of interest goes behind the obstacle at time t3 (i.e., at 16:10:44). This point 3 is called an unintentional disappearance. The obstacle can be a wall 5. At point 4, the system 100 is configured to determine that the disappearance of the object of interest from the user's view is unintentional because an obstacle appears between the user and the object of interest. The system 100 then displays the content (i.e., digital images or multimedia content) of the object of interest on the AR device 102. In an embodiment, when following the object of interest, the system 100 tracks the direction of head movement as shown at point 6, the direction of eye gaze as shown at point 7, the change in pupil size when viewing the object of interest as shown at point 8, and other parameters such as blinking, facial expressions, etc. as shown at point 9.

[0124] Figure 4 A schematic diagram 400 depicting intentional disappearance of a first object (Object 1) according to an exemplary embodiment of the present disclosure is shown.

[0125] Figure 4 Indicates intentional disappearance and the user shows a lack of interest in the object. Figure 4 In the example, the user moves his gaze from one object to another. At point 1, the first object appears in the user's field of view. Figure 5a. The user starts to continuously look at the first object as shown in point 6 at time t1 (i.e., at 16:10:40). At point 2, the user looks at the object for a threshold time, and the object becomes an "object of interest". In an embodiment, the user looks at the object for two seconds, for example, the user looks at the object of interest from t1 to t2 (t2=16:10:42). The system automatically calculates the total time (t2-t1) of looking at the object of interest. At point 3, the user turns his head from the first object even before the first object leaves the sight, and starts to continuously look at the second object as shown in point 7 at time t3 (i.e., at 16:10:44). This point 3 is called the intentional disappearance of the first object from the user's field of view 5a. The obstacle can be a wall 4. Since the user continuously looks at the first object from the user's field of view 5a, and then turns his head to look at the second object, the disappearance of the first object from the user's field of view 5a is intentional, because the user has the opportunity to continuously look at the first object for a period of time while the obstacle reaches 4, but he moves his gaze to see another object from the field of view 5b. In this case, even if the user views the object for the threshold time, the first object is not displayed on the AR device 102 .

[0126] Figure 5 A flow chart 500 is shown depicting a process for displaying content on an AR device according to an exemplary embodiment of the present disclosure.

[0127] Flowchart 500 begins at step 502, where, based on continuous automatic tracking of the user's eyes and movements, video / images of objects in the user's field of view are monitored and stored. In an embodiment, capture module 206 is configured to capture video / images of multiple objects in the user's field of view, and recording module 208 is configured to record, monitor, and store video / images of objects in the user's field of view. User input controller 210 is configured to track the user's eyes and movements.

[0128] At step 504, processing includes determining an object of interest based on the threshold gaze time and other parameters. In an embodiment, the determination module 224 determines an object of interest based on the threshold gaze time and other parameters.

[0129] At step 506, processing includes detecting disappearance of an "object of interest." In one embodiment, the identifier 226 is configured to detect disappearance of an object of interest.

[0130] At step 508, processing includes checking whether the disappearance of the object is intentional or unintentional. In an embodiment, the identifier 226 is configured to identify whether the disappearance of the object is intentional or unintentional. If the disappearance of the object of interest as shown in step 510 is intentional, the intentional disappearance is also identified by two conditions, namely, the user's lack of interest as shown in step 514, or despite interest as shown in step 518. In an embodiment, despite interest is based on non-obstruction. If the intentional disappearance is caused by the user's lack of interest, the system 100 discards the object of interest from further processing 516, and the user will not be unnecessarily disturbed at point 2. If the disappearance of the object of interest as shown in step 518 is based on despite interest, as shown in step 520, the system displays the multimedia content of the "object of interest" to the user in the AR device 102. In an embodiment, the display module 218 is configured to display the multimedia content of the "object of interest" to the user in the AR device 102.

[0131] If the disappearance of the object of interest as shown in step 512 is unintentional, the system displays the multimedia content of the “object of interest” to the user in the AR device 102 as shown in step 520. In an embodiment, the display module 218 is configured to display the multimedia content of the “object of interest” to the user in the AR device 102. In one embodiment, the system 100 provides auxiliary display when the user needs auxiliary display for viewing the object of interest until the moment when the user is interested in viewing the object of interest at point 1.

[0132] Figure 6 A flowchart 600 is shown depicting a process for displaying content on an AR device when a user gazes at an object of interest, according to an exemplary embodiment of the present disclosure.

[0133] Flowchart 600 begins at step 602, where, based on continuous automatic tracking of the user's eyes and movements, video / images of objects in the user's field of view are monitored and stored. In an embodiment, capture module 206 is configured to capture video / images of multiple objects in the user's field of view, and recording module 208 is configured to record, monitor, and store video / images of objects in the user's field of view. User input controller 210 is configured to track the user's eyes and movements.

[0134] At step 604, the process includes determining an object of interest based on the threshold gaze time and other parameters. In an embodiment, the determination module 224 is configured to determine an object of interest based on the threshold gaze time and other parameters.

[0135] At step 606, the process includes detecting disappearance of the "object of interest." In an embodiment, the identifier 226 is configured to detect disappearance of the object of interest.

[0136] At step 608, the process includes checking whether the disappearance of the object is intentional or unintentional. In an embodiment, the identifier 226 is configured to identify whether the disappearance of the object is intentional or unintentional. If the disappearance of the object of interest as shown in step 610 is intentional, the intentional disappearance is also identified by two conditions, namely, the user's lack of interest as shown in step 614, or despite interest as shown in step 618. In an embodiment, despite interest is based on non-obstruction. If the intentional disappearance is caused by the user's lack of interest, the system 100 discards the object of interest from further processing (616), and the user will not be unnecessarily disturbed at point 2. If the disappearance of the object of interest as shown in step 618 is based on despite interest, as shown in step 620, the system displays the multimedia content of the "object of interest" to the user in the AR device 102 until the moment when the user is looking at the object of interest. In an embodiment, the display module 218 is configured to display the multimedia content of the "object of interest" to the user in the AR device 102 until the moment when the user is looking at the object of interest.

[0137] If the disappearance of the object of interest as shown in step 612 is unintentional, the system displays the multimedia content of the “object of interest” to the user in the AR device 102 until the moment when the user is looking at the object of interest, as shown in step 620. In an embodiment, the display module 218 is configured to display the multimedia content of the “object of interest” to the user in the AR device 102 until the moment when the user is looking at the object of interest. In an embodiment, the system 100 provides auxiliary display when the user needs auxiliary display to view the object of interest until the moment when the user is interested in viewing the object of interest at point 1.

[0138] Figure 7 A flowchart 700 is shown depicting a process for removing multimedia content from an AR device when a user moves his gaze away from the multimedia content, according to an exemplary embodiment of the present disclosure.

[0139] Flowchart 700 begins at step 702, where, based on continuous automatic tracking of the user's eyes and movements, video / images of objects in the user's field of view are monitored and stored. In an embodiment, capture module 206 is configured to capture video / images of multiple objects in the user's field of view, and recording module 208 is configured to record, monitor, and store video / images of objects in the user's field of view. User input controller 210 is configured to track the user's eyes and movements.

[0140] At step 704, the process includes determining an object of interest based on the threshold gaze time and other parameters. In an embodiment, the determination module 224 is configured to determine an object of interest based on the threshold gaze time and other parameters.

[0141] At step 706, the process includes detecting disappearance of the "object of interest." In an embodiment, the identifier 226 is configured to detect disappearance of the object of interest.

[0142] At step 708, the process includes checking whether the disappearance of the object is intentional or unintentional. In an embodiment, the identifier 226 is configured to identify whether the disappearance of the object is intentional or unintentional. If the disappearance of the object of interest as shown in step 710 is intentional, the intentional disappearance is also identified by two conditions, namely, the user's lack of interest as shown in step 714, or despite interest as shown in step 718. In an embodiment, despite interest is based on non-obstruction. If the intentional disappearance is caused by the user's lack of interest, the system 100 discards the object of interest from further processing 716, and the user will not be unnecessarily disturbed at point 2. If the disappearance of the object of interest as shown in step 718 is based on despite interest, as shown in step 720, the system displays the multimedia content of the "object of interest" to the user in the AR device 102 until the moment when the user is looking at the object of interest. In an embodiment, the display module 218 is configured to display the multimedia content of the "object of interest" to the user in the AR device 102 until the moment when the user is looking at the object of interest.

[0143] If the disappearance of the object of interest as shown in step 712 is unintentional, the multimedia content of the “object of interest” is displayed to the user in the AR device 102 until the moment when the user is looking at the object of interest as shown in step 720. In an embodiment, the display module 218 is configured to display the multimedia content of the “object of interest” to the user in the AR device 102 until the moment when the user is looking at the object of interest. In an embodiment, the system 100 provides an auxiliary display until the moment when the user is interested in viewing it, wherein the auxiliary display quickly disappears at point 1.

[0144] At step 722 , the user moves his gaze away from the multimedia content displayed on the AR device 102 .

[0145] At step 724, the process includes removing the multimedia content from the AR device 102. In an embodiment, if the user moves his gaze away from the multimedia content, the system 100 automatically removes the multimedia content.

[0146] Figure 8 A flow chart 800 is shown depicting a process for filtering / enhancing and augmenting recorded content and displaying the content to a user, and removing the multimedia content from an AR device when the user moves his gaze away from the multimedia content, according to an exemplary embodiment of the present disclosure.

[0147] Flowchart 800 begins at step 802, where, based on continuous automatic tracking of the user's eyes and movements, video / images of objects in the user's field of view are monitored and stored. In an embodiment, capture module 206 is configured to capture video / images of multiple objects in the user's field of view, and recording module 208 is configured to record, monitor and store video / images of objects in the user's field of view. User input controller 210 is configured to track the user's eyes and movements.

[0148] At step 804, the process includes determining an object of interest based on the threshold gaze time and other parameters. In one embodiment, the determination module 224 is configured to determine an object of interest based on the threshold gaze time and other parameters.

[0149] At step 806, the process includes detecting disappearance of an "object of interest." In an embodiment, the identifier 226 is configured to detect disappearance of an object of interest.

[0150] At step 808, the process includes checking whether the disappearance of the object is intentional or unintentional. In one embodiment, the identifier 226 is configured to identify whether the disappearance of the object is intentional or unintentional. If the disappearance of the object of interest as shown in step 810 is intentional, the intentional disappearance is also identified by two conditions, namely, the user's lack of interest as shown in step 814, or despite interest as shown in step 818. In an embodiment, despite interest is based on non-obstruction. If the intentional disappearance is caused by the user's lack of interest, the system 100 discards the object of interest from further processing 816, and the user will not be unnecessarily disturbed at point 2. If the disappearance of the object of interest as shown in step 818 is based on despite interest, in step 820, the system 100 generates auxiliary content by filtering / enhancing and amplifying the recorded content. In an embodiment, the display module 218 is configured to display the content of the object of interest on the AR device 102, wherein the content of the object of interest includes a fragment of the object of interest or a modified version of the object of interest. The modified version of the object of interest is generated by filtering and enhancing the fragment of the object of interest. The step of filtering and enhancing the segment of the object of interest includes filtering and enhancing images, videos, enhanced information related to the object of interest, and a combination of images, videos, and enhanced information related to the object of interest.

[0151] At step 822, the process includes displaying the multimedia content of the “object of interest” to the user in the AR device 102 until the moment when the user is looking at the object of interest. In an embodiment, the display module 218 is configured to display the multimedia content of the “object of interest” to the user in the AR device 102 until the moment when the user is looking at the object of interest.

[0152] If the disappearance of the object of interest as shown in step 812 is unintentional, then in step 820, the system 100 generates auxiliary content by filtering / enhancing and amplifying the recorded content. In an embodiment, the display module 218 is configured to display the content of the object of interest on the AR device 102, wherein the content of the object of interest includes a fragment of the object of interest or a modified version of the object of interest. The modified version of the object of interest is generated by filtering and enhancing the fragments of the object of interest. The steps of filtering and enhancing the fragments of the object of interest include filtering and enhancing images, videos, enhanced information related to the object of interest, and combinations of images, videos, and enhanced information related to the object of interest. In an embodiment, the system 100 provides an auxiliary display until the moment when the user is interested in viewing it, wherein the auxiliary display disappears quickly at point 1.

[0153] At step 824 , the user moves his gaze away from the multimedia content displayed on the AR device 102 .

[0154] At step 826, the process includes removing the multimedia content from the AR device 102. In an embodiment, if the user moves his gaze away from the multimedia content, the system 100 automatically removes the multimedia content.

[0155] Fig. 9 A flowchart 900 is shown depicting a process for displaying multimedia content of an object of interest to a user in an AR device for a calculated duration according to an exemplary embodiment of the present disclosure. The process begins at step 902, where video / images of objects in a user's field of view are monitored and stored based on continuous automatic tracking of the user's eyes and movements. In an embodiment, the capture module 206 is configured to capture video / images of multiple objects in the user's field of view, and the recording module 208 is configured to record, monitor, and store video / images of objects in the user's field of view. The user input controller 210 is configured to track the user's eyes and movements.

[0156] At step 904, the process includes determining the object of interest based on the threshold gaze time and other parameters and determining the start time (ToA) of the user's gaze on the object. In one embodiment, the determination module 224 is configured to determine the object of interest based on the threshold gaze time and other parameters and determine the start time (ToA) of the user's gaze on the object.

[0157] At step 906, the process includes detecting disappearance of the "object of interest" and finding the time of disappearance (ToD) of the "object of interest". In an embodiment, the identifier 226 is configured to detect disappearance of the object of interest and find the time of disappearance (ToD) of the object of interest.

[0158] At step 908, the processing includes checking whether the disappearance of the object is intentional or unintentional. In one embodiment, the identifier 226 is configured to identify whether the disappearance of the object is intentional or unintentional. If the disappearance of the object of interest as shown in step 910 is intentional, the intentional disappearance is also identified by two conditions, namely, the user's lack of interest as shown in step 914, or despite interest as shown in step 918. In an embodiment, despite interest is based on non-obstruction. If the intentional disappearance is caused by the user's lack of interest, the system 100 discards the object of interest from further processing 916, and the user will not be unnecessarily disturbed at point 2.

[0159] If the disappearance of the object of interest as shown in step 918 is based on the fact that the object of interest is not present, the system 100 determines the content complexity of the "object of interest" and calculates the time (TR) required for the user to read / view the object of interest. In an embodiment, the complexity identifier 236 is configured to determine the content complexity of the "object of interest" and calculate the time (TR) required for the user to read / view the object of interest.

[0160] At step 920, the process includes calculating the duration for showing the content to the user by using the following formula:

[0161] Duration = TR - (ToD - ToA)

[0162] In an embodiment, the complexity identifier 236 is configured to calculate the duration for which the content is shown to the user.

[0163] At step 922, the process as shown in step 922 includes displaying the multimedia content of the “object of interest” to the user in the AR device 102 for the calculated duration. In an embodiment, the display module 218 is configured to display the multimedia content of the “object of interest” to the user in the AR device 102 for the calculated duration.

[0164] If the disappearance of the object of interest as shown in step 912 is unintentional, the system 100 determines the content complexity of the "object of interest" and calculates the time (TR) required for the user to read / view the object of interest. In an embodiment, the complexity identifier 236 is configured to determine the content complexity of the "object of interest" and calculate the time (TR) required for the user to read / view the object of interest. In an embodiment, the system 100 provides auxiliary display when the user needs auxiliary display for viewing the object of interest until the user is interested in viewing it at point 1.

[0165] Fig.10 A flowchart 1000 is shown depicting a process for displaying content on an AR device according to an exemplary embodiment of the present disclosure.

[0166] Flowchart 1000 begins with capturing a user's field of view at step 1002. In an embodiment, capture module 206 is configured to capture a user's field of view.

[0167] At step 1004, the process includes recording the captured field of view. In an embodiment, the recording module 208 is configured to record the captured field of view.

[0168] At step 1006, the process includes tracking the user's vision towards one or more objects in the recorded field of view. In an embodiment, the user input controller 210 is configured to track the user's vision towards one or more objects in the recorded field of view.

[0169] At step 1008, the process includes determining at least one object of interest in the recorded field of view. In an embodiment, the determination module 224 is configured to determine at least one object of interest in the recorded field of view.

[0170] At step 1010, the process includes identifying frames containing a determined disappearance of an object of interest in a recorded field of view. In one embodiment, the identifier 226 is configured to identify frames containing a determined disappearance of an object of interest in a recorded field of view.

[0171] At step 1012, the process includes analyzing the identified frame based on at least one disappearance of intent. In an embodiment, the analyzer 230 is configured to analyze the identified frame based on at least one disappearance of intent.

[0172] At step 1014, the process includes generating analyzed data based on the analyzed frames. In one embodiment, the analyzer 230 is configured to generate analyzed data based on the analyzed frames.

[0173] At step 1016 , the process includes displaying the content of the object of interest based on the analyzed data on the AR device 102 . In an embodiment, the display module 218 is configured to display the content of the object of interest based on the analyzed data on the AR device 102 .

[0174] Fig.11 A flowchart 1100 is shown depicting a process for displaying content on an AR device with intentional disappearance according to an exemplary embodiment of the present disclosure.

[0175] Flowchart 1100 begins with capturing a user's field of view at step 1102. In an embodiment, capture module 206 is configured to capture a user's field of view.

[0176] At step 1104, the process includes tracking the user's eye gaze towards the object of interest in the captured field of view. In one embodiment, the user input controller 210 is configured to track the user's eye gaze towards the object of interest in the captured field of view.

[0177] At step 1106, the process includes identifying frames that contain an intentional disappearance of an object of interest in the captured field of view. In one embodiment, the identifier 226 is configured to identify frames that contain an intentional disappearance of an object of interest in the captured field of view.

[0178] At step 1108, the process includes analyzing the identified frames based on the intentional disappearance. In an embodiment, the analyzer 230 is configured to analyze the identified frames based on the intentional disappearance.

[0179] At step 1110, the process includes generating analyzed data based on the analyzed frames. In one embodiment, the analyzer 230 is configured to generate analyzed data based on the analyzed frames.

[0180] At step 1112, the process includes displaying content of the object of interest on the AR device 102 based on the analyzed data. In an embodiment, the display module 218 is configured to display content of the object of interest on the AR device 102 based on the analyzed data.

[0181] Fig.12 A flow chart 1200 is shown depicting a process of displaying content on an AR device in the event of an unintentional disappearance of an object of interest or despite an intentional disappearance of an object of interest, according to an exemplary embodiment of the present disclosure.

[0182] Flowchart 1200 begins with capturing a user's field of view at step 1202. In an embodiment, capture module 206 is configured to capture a user's field of view.

[0183] At step 1204, the process includes recording the captured field of view. In an embodiment, the recording module 208 is configured to record the captured field of view.

[0184] At step 1206, the process includes tracking the user's eye gaze towards the object of interest in the recorded field of view. In an embodiment, the user input controller 210 is configured to track the user's eye gaze towards the object of interest in the recorded field of view.

[0185] At step 1208, the process includes identifying frames containing at least one of an unintentional disappearance of an object of interest in the recorded field of view or an intentional disappearance despite interest. In one embodiment, the identifier 226 is configured to identify frames containing at least one of an unintentional disappearance of an object of interest in the recorded field of view or an intentional disappearance despite interest.

[0186] At step 1210, the process includes analyzing the identified frames based on at least one of unintentional disappearance of the object of interest or despite intentional disappearance of the object of interest. In one embodiment, the analyzer 230 is configured to analyze the identified frames based on at least one of unintentional disappearance of the object of interest or despite intentional disappearance of the object of interest.

[0187] At step 1212, the process includes generating analyzed data based on the analyzed frames. In one embodiment, the analyzer 230 is configured to generate analyzed data based on the analyzed frames.

[0188] At step 1214, the process includes displaying the content of the object of interest on the AR device 102 based on the analyzed data. In an embodiment, the display module 218 is configured to display the content of the object of interest on the AR device 102 based on the analyzed data.

[0189] In various embodiments as described above, the AR device 102 displays the content of the object of interest in response to the intentional or unintentional disappearance of the object. In another embodiment, the AR device may display an indication (e.g., a pop-up window for an alarm or reminder) before displaying the content of the object of interest. For example, the indication may include a message notifying that the content of the object is recorded, at least one item (e.g., a button) for inputting a command to display the content.

[0190] In an embodiment, the indication is displayed at the AR device 102 or another device. For example, the other device may be a wearable device such as a watch paired with the AR device 102. In this case, if the user touches the indication displayed on the other device, the AR device 102 displays the content of the object of interest on the display of the AR device 102.

[0191] In an embodiment, the indication is displayed after a predefined time has passed since the moment when the object of interest disappears. In this case, before the predefined time has passed, the AR device 102 may display the content of the object of interest during at least a portion of the time period between the object disappearing and displaying the indication.

[0192] Fig.13 A use case scenario 1300 depicting an intentional disappearance of a user's interest is shown according to an exemplary embodiment of the present disclosure.

[0193] exist Fig.13In the example, Mr. A is looking at his crush intermittently because he feels embarrassed in direct eye contact. In an exemplary embodiment, Mr. A is looking at his crush (Ms. B) by looking at him intermittently to avoid the embarrassing situation of making direct eye contact. The duration of each gaze of Mr. A is less than the threshold time. The system 100 finds that Mr. A has looked at Ms. B many times. The system 100 also analyzes various parameters, such as the total duration of all eye gazes, the gaze of the eye gaze at Ms. B, the direction and speed of head movement, the pupil width of the eye when looking at Ms. B, the avoidance of blinking when looking at Ms. B, facial expressions, etc. When the total gaze time for Ms. B exceeds the threshold time, the system 100 regards her as an object of interest to Mr. A. The system 100 determines Mr. A's strong intention to look at Ms. B's image through various parameters. The system 100 determines that the image of Ms. B should be shown in Mr. A's AR device. The system 100 creates a virtual image of Ms. B and shows the virtual image to Mr. A in the AR device 102 so that he can see the virtual image without looking directly at Ms. B. Table 2 shows the total time Mr. A looked at Ms. B.

[0194]

[0195] Table 2

[0196] Fig.14 A use case scenario 1400 depicting an intentional disappearance of a user's interest is shown according to an exemplary embodiment of the present disclosure.

[0197] exist Fig.14 , the user looks at a predefined area in the AR device 102 to see his object of interest in the AR device 102 before the object of interest disappears. Mr. A wants to see his object of affection (Ms. B). But he does not want to make eye contact with her. Mr. A looks at Ms. B for a threshold amount of time. He moves his gaze away from Ms. B and looks at a predefined location in the AR device 102. The system 100 determines this as a request to show the object of interest in the AR device 102. The system 100 prepares multimedia content based on Ms. B's face and displays the content in the AR device 102. Now, Mr. A can see Ms. B in the AR device 102 without looking directly at her.

[0198] Fig.15 A use case scenario 1500 depicting unintentional and obstructive disappearance is shown according to an exemplary embodiment of the present disclosure.

[0199] exist Fig.15, the best available view of the object of interest is provided to the user. Here, the state of the object of interest changes, that is, the object of interest turns his head and cannot be properly seen. At 1502, the point at which Mr. B appears is in the view of Mr. A's AR device 102. Mr. A is looking somewhere else. At 1504, Mr. A starts to look at Mr. B. At 1506, Mr. B disappears from Mr. A's view. This disappearance is called an unintentional disappearance. Even if the best available view of Mr. B has never been seen by Mr. A, the system 100 uses the best available view of Mr. B to show Mr. A in AR. At 1508, Mr. A starts to look at Mr. B a little later and misses the frontal view of his face. The system 100 determines that Mr. A is interested in seeing Mr. B, and finds that Mr. A cannot have the opportunity to see the frontal view of Mr. B. Since Mr. A's gaze and head movement follow Mr. B until the vanishing point, Mr. B's disappearance from Mr. A's view is unintentional. The system 100 obtains the frontal view of Mr. B from the recorded content and displays it to Mr. A. The system 100 considers parameters such as: the direction of head movement while following an object 1510 , the time period 1512 that Mr. A's AR device looks at Mr. B, the time period 1514 that Mr. A looks at Mr. B, and obstacles 1516 .

[0200] FIG. 16A to FIG. 16C A use case scenario 1600 depicting unintentional disappearance of an object due to an obstacle between a user and the object is shown according to an exemplary embodiment of the present disclosure.

[0201] exist Fig.16A In , Mr. A has been reading an advertisement board on the other side of the railway track for more than a threshold amount of time. A train is approaching. Fig. 16B In the video, while he is still reading the billboard, an approaching train comes between Mr. A and the billboard. Mr. A cannot read the billboard completely. Fig. 16C In the example, the system 100 determines that Mr. A is still interested in the billboard. The system 100 creates virtual content and displays the virtual content in Mr. A's AR device.

[0202] FIG. 17A to FIG. 17B A use case scenario 1700 depicting unintentional disappearance of an object due to loss of visibility of an obstacle between a user and the object is shown according to an exemplary embodiment of the present disclosure.

[0203] exist Fig.17A In , the user is diving in the ocean. He looks at fish for more than a threshold amount of time. Fig. 17B In the video, the fish suddenly starts to swim very fast and disappears in the dark water in front of the user. The system 100 displays the image of the fish in the user's AR device 102.

[0204] 18A to 18DA use case scenario 1800 depicting an intentional disappearance of a user's interest is shown according to an exemplary embodiment of the present disclosure.

[0205] exist Fig.18A In the example, the user is walking down a street, looking at various objects that appear on his way. Another person wearing a slogan T-shirt enters the user's field of view. Fig.18B In , the user sees a slogan on a woman's t-shirt from the opposite direction. The user wants to read the slogan but feels embarrassed to keep looking at the t-shirt. Therefore, the user looks at the t-shirt with intermittent fixations. These intermittent fixations exceed the threshold time. Fig.18C In , the user cannot fully read the slogan on the t-shirt due to the viewing angle, obstacles in between, and the complexity of the content. The user keeps moving his head and eyes towards the slogan with intermittent gaze. Fig.18D In the example above, the lady walks past the user. Now the slogan is out of the user's view. The system 100 recreates the best possible image of the slogan along with the additional information. This image is shown in the AR device 102. Now the user can read the slogan in his AR device 102 without any embarrassment.

[0206] FIG. 19A to FIG. 19F A use case scenario 1900 depicting intentional disappearance of an object of interest due to relative motion between a user and an object is shown in accordance with an exemplary embodiment of the present disclosure.

[0207] exist Fig.19A In , the user is wearing the AR device 102 and is traveling in a car. At 16:02:15, he is looking at various objects that appear on his way. Fig.19B In , at 16:02:16, the user is looking at a specific object (a movie poster) that appears in his path. Fig.19C In , the user keeps looking at the movie poster for more than a threshold time (~3 seconds). The system calculates the content complexity of the poster. The system determines that it takes the user 10 seconds to properly view the poster. Fig.19D , the user continues to turn his head and move his eyes to continue looking at the movie poster. Fig.19E In , as the car continues to move, the movie poster moves out of the user's sight. The user can only view the poster for 5 seconds (from 16:02:16 to 16:02:21). The user keeps turning his head and eyes to look at the movie poster as long as possible. The system determines that the user is still interested in viewing the poster. The user cannot get enough time (such as 10 seconds calculated by the system) to look at the poster. Fig.19FIn the example above, the system recreates the best possible image of the poster along with the additional information and shows the image with the additional information in the AR device 102. This AR content does not block the normal view. The user will be shown this content until "16:02:26" (5 more seconds) or until the user is interested in the content.

[0208] FIG. 20A to FIG. 20C A use case scenario 2000 depicting intentional disappearance of an object of interest due to light / visibility and other environmental conditions is shown in accordance with an exemplary embodiment of the present disclosure.

[0209] exist Fig. 20A In , a user is examining text written on a glass in a store. Fig. 20B In , the user shows his interest in text written on the glass by turning his head and looking towards the text. The text is difficult to read due to glare, font color, background and lighting. The user continues to move his head and eyes towards the text. Fig. 20C In FIG. 1 , a user walks through a store. Due to glare, font color, background, and lighting, he cannot fully read the text on the glass. System 100 recognizes that the text is not clearly visible to the user. It recreates the text and shows it to the user in AR device 102 along with augmented information.

[0210] FIG. 21A to FIG. 21F A use case scenario 2100 depicting disappearance of AR content of a previous object of interest is shown according to an exemplary embodiment of the present disclosure.

[0211] exist Fig.21A In , the user is traveling in a car. He has been reading the roadside Ad-1 board for more than a threshold amount of time. His car moves forward before he finishes reading the Ad-1 board. Fig. 21B In , Ad-1 is out of Mr. A's view. The system 100 determines that the user is still interested in Ad-1. The system 100 creates a virtual image of Ad-1 and displays the virtual image in the user's AR device 102. Fig. 21C In FIG, the user has not yet finished reading the content of Ad-1 in AR device 101. However, he moves his head and eye gaze toward another roadside advertisement-2 of interest in his view. System 100 removes the AR content of Ad-1. Fig.21D In , the user views Ad-2 for more than a threshold amount of time. Ad-2 is no longer in the user's view. The system 100 determines that the user is still interested in Ad-2. Fig.21E In the example, the system 100 also creates virtual content of Ad-2 and displays the virtual content in the user's AR device. Fig.21F In the example, the virtual image of Ad-2 times out and disappears after the user has finished viewing the virtual image. The user continues to travel in the vehicle and continues to look at various objects that appear on his way.

[0212] FIG. 22A to FIG. 22C A use case scenario 2200 depicting intentional and non-obstructive disappearance is shown according to an exemplary embodiment of the present disclosure.

[0213] exist Fig.22A In , the user sees a stranger on an escalator going in the opposite direction. He wonders if he has seen the stranger somewhere before. He keeps turning his head to see the stranger's face. Fig. 22B In , the user has less time to properly look at the stranger. He keeps looking at the stranger. This makes them both uncomfortable. Fig. 22C In the example, a stranger walks past the user. The user obtains the information of the stranger in his AR device 102.

[0214] FIG. 23A to FIG. 23F A use case scenario 2300 depicting an unintentional obstructive disappearance of a high-speed race car rapidly disappearing from a user's view is shown in accordance with an exemplary embodiment of the present disclosure.

[0215] exist Fig.23A In , the user is watching a car race. The car passes him at a very high speed. The car enters the user's field of view and leaves the user's field of view even before the threshold time. Fig. 23B In , the user only looks at any car for 2 seconds before it leaves his view. This time is less than the threshold time required to consider a car as an object of interest. When the car approaches him, he only looks at the car and his head turns quickly to follow the car. The system considers multiple factors to identify objects of interest. In this case, once an object is visible, the user only looks at the object and looks at it until it disappears. Since head movement and eye gaze are more prominent, the weight of the threshold time is reduced and the car is considered an object of interest. Fig.23C In , the user must quickly turn his head to see the car. Fig.23D In the example above, the car quickly moves away from the user's view. The user continues to move his head and eyes towards the car. In this case, once an object is visible, the user only looks at that object and only looks at that object until it disappears. The total time the object is visible is very short. Since the head movement and eye gaze are more prominent, the threshold time is reduced and the car is considered to be an object of interest. Multiple factors are considered to identify the object of interest. Fig.23E In the example above, the car leaves the user’s line of sight. The system calculates object complexity and recognizes that the car is moving very fast. Fig.23F As shown, the system 100 creates a video from the camera recording and shows the video in the AR device 102 at a slow playback speed.

[0216] Fig.24A use case scenario 2400 is shown depicting a process for performing mirroring of content of an object of interest displayed on one or more computing devices, according to an exemplary embodiment of the present disclosure.

[0217] Specifically, Fig.24 Shows the real-time reading of the heart rate monitor on the AR glasses of the doctor performing the surgery. During the surgery, the doctor wears the AR device and continues to watch the heart rate monitor. The AR device continues to track the heart rate monitor for a threshold time. When the doctor turns their head towards the operating table, the real-time data from the heart rate monitor is transmitted to the corresponding AR device.

[0218] In an embodiment, a computer-implemented method for displaying content on an augmented reality (AR) device may include capturing a user's field of view, recording the captured field of view, tracking the user's vision toward one or more objects in the recorded field of view, determining at least one object of interest in the recorded field of view, identifying a frame containing a determined object of interest disappearing from the recorded field of view, analyzing the identified frame based on at least one disappearance of the object of interest, generating analyzed data based on the analyzed frame, and displaying content of the object of interest on the augmented reality (AR) device based on the analyzed data.

[0219] Frames may be identified based on the intentional or unintentional disappearance of an object of interest from the recorded field of view.

[0220] The identified frames may be analyzed based on the intentional or unintentional disappearance of the object of interest from the recorded field of view.

[0221] The method may further include storing in a database the recorded fields of view, predefined time intervals, predefined object categories, and data related to one or more objects, predefined rotation angles, and a rotation speed of the user at at least one angle toward the one or more objects.

[0222] The identified frame may be analyzed based on the intentional disappearance, and analyzing the identified frame may include identifying a user's lack of interest in the one or more objects.

[0223] The step of determining at least one object of interest may be performed using a threshold time.

[0224] Frames may be identified based on visual time and threshold time.

[0225] The identified frame may be analyzed based on the unintentional disappearance, and analyzing the identified frame may include determining an obstruction between the user and the object of interest.

[0226] Obstructions between the user and the object of interest may be determined based on visual analysis, object recognition, eye movement, head movement, or fixation gaze time on another object after the object of interest disappears.

[0227] Obstacles between the user and the object of interest can be determined by using multiple sensors.

[0228] The user's vision may be tracked based on at least one of parameters including: eye gaze, eye movement direction, head movement direction, pupil dilation, number of blinks, head rotation angle, head rotation speed, facial expressions, eye size dilation, pupil and iris movement, and voice recognition.

[0229] The step of analyzing the identified frames based on intentional disappearance may include: analyzing the user's vision of the object of interest within a predefined time interval, using an artificial intelligence-based multi-parameter model to identify intentional non-obstructive disappearance of the object of interest, wherein the model includes parameters related to the user's eye movement, the user's engagement level, companions with the user, and the object category, analyzing data received from sensors and viewing data within a threshold time, and identifying gaze at a predetermined location of the AR device.

[0230] An intentional disappearance of an object may be a disappearance despite a user's interest in the object.

[0231] The method may also include determining a probability of the user's anticipation of the field of view based on the user's interest in the object.

[0232] In another embodiment, the method may further include identifying a user's lack of interest in the object.

[0233] Displaying the content of the object of interest may include displaying a segment of the object of interest or a modified version of the object of interest on an augmented reality (AR) device.

[0234] A modified version of the object of interest may be generated by filtering and enhancing the segments of the object of interest.

[0235] The method may further include calculating a time required for the user to view the object of interest based on the object information.

[0236] The time required for the user to view the object of interest may be calculated based on the total time required for the user to view the object of interest and the duration for which the user has viewed the object of interest before disappearing.

[0237] The step of filtering and enhancing the segments of the object of interest may include filtering and enhancing images, videos, enhancement information related to the object of interest, and a combination of images, videos, and enhancement information related to the object of interest.

[0238] The method may also include rendering the positioning of the content on the AR device based on the user's gaze, the user's preference, and the current view content.

[0239] The method may also include highlighting the object of interest based on gaze and a threshold time at a predefined location in the AR device.

[0240] The method may further include detecting and identifying the object of interest from the identified frames using deep learning techniques, wherein the deep learning techniques include object recognition techniques, deep structured learning techniques, hierarchical learning techniques, or neural networks.

[0241] The method may also include sending the analyzed data from the augmented reality (AR) device to another augmented reality (AR) device by using the location of the other augmented reality (AR) device, artificial intelligence technology, or a usage profile, and displaying content of the object of interest on the other augmented reality (AR) device based on the analyzed data.

[0242] The method may also include performing, on the AR device, mirroring content of the object of interest displayed on the one or more computing devices in the event of an unintentional disappearance of the object of interest or despite an intentional disappearance of the object of interest and when the user views the object of interest for more than a threshold amount of time.

[0243] In another embodiment, a computer-implemented system for displaying content on an augmented reality (AR) device includes: a memory configured to store predefined rules; a processor configured to cooperate with the memory and configured to generate system processing commands based on the predefined rules; a capture module configured to capture a user's field of view; a recording module configured to record the captured field of view; a user input controller configured to track the user's vision toward one or more objects in the recorded field of view; and a server configured to cooperate with the AR device, the recording module, and the user input controller. The server may include: a determination module configured to determine at least one object of interest in the recorded field of view; an identifier configured to cooperate with the determination module to receive the determined object of interest, wherein the identifier is configured to identify a frame containing the determined object of interest disappearing in the recorded field of view; an analyzer configured to cooperate with the identifier to receive the identified frame, wherein the analyzer is configured to analyze the identified frame based on at least one disappearance of the object of interest and generate analyzed data. The system also includes: a display module configured to display content of the object of interest on the augmented reality (AR) device based on the analyzed data.

[0244] The AR device may be associated with the user.

[0245] The identifier may be configured to identify frames based on the intentional or unintentional disappearance of an object of interest from the recorded field of view.

[0246] The analyzer may be configured to analyze the identified frames based on intentional or unintentional disappearance of the object of interest in the recorded field of view.

[0247] The system may further include a database, wherein the database is configured to store recorded fields of view, predefined time intervals, predefined object categories, and data related to one or more objects, predefined rotation angles, and a rotation speed of the user at at least one angle toward the one or more objects.

[0248] The analyzer may be configured to analyze the identified frames based on intentional disappearance by identifying a user's lack of interest in the one or more objects.

[0249] The identifier may be configured to identify frames based on visual time and threshold time.

[0250] The analyzer may be configured to analyze the identified frames based on unintentional disappearance by determining an obstruction between the user and the object of interest.

[0251] The user input controller can be configured to track the user's vision based on at least one of the following parameters: eye gaze, eye movement direction, head movement direction, pupil dilation, number of blinks, head rotation angle, head rotation speed, facial expression, eye size dilation, pupil and iris movement, and voice recognition.

[0252] The analyzer may be configured to analyze the identified frames based on intentional disappearance, and the analyzer may be configured to analyze the user's vision of the object of interest within a predefined time interval, identify intentional non-obstructive disappearance of the object of interest using an artificial intelligence-based multi-parameter model, analyze data received from the sensor and viewing data within a threshold time, and identify gazes at predetermined locations of the AR device, wherein the model includes parameters related to the user's eye movements, the user's engagement level, companions with the user, and object categories.

[0253] An intentional disappearance of an object may be a disappearance despite a user's interest in the object.

[0254] The analyzer may include an estimation module, wherein the estimation module is configured to determine a probability of the user's desire for the field of view based on the user's interest in the object.

[0255] The analyzer may be configured to identify a user's lack of interest in one or more objects.

[0256] The display module may be configured to display content of the object of interest on an augmented reality (AR) device, wherein the content of the object of interest includes a segment of the object of interest or a modified version of the object of interest.

[0257] The server may further include a complexity identifier, wherein the complexity identifier is configured to calculate a time required for a user to view the object of interest based on the object information.

[0258] The complexity identifier may be configured to calculate the time required for the user to view the object of interest based on the total time required for the user to view the object of interest and the duration for which the user has viewed the object of interest before the object of interest disappears.

[0259] The system may further include a content rendering module, wherein the content rendering module is configured to render the location of the content on the AR device based on the user's gaze, the user's preference, and the current view content.

[0260] The display module may be configured to highlight an object of interest based on gaze and a threshold time at a predefined location in the AR device.

[0261] The recognizer may also include an object recognition module, wherein the object recognition module is configured to detect and recognize objects of interest from the recognized frames using deep learning technology, wherein the deep learning technology includes object recognition technology, deep structured learning technology, hierarchical learning technology or neural network.

[0262] The user input controller may include an eye gaze tracking module, wherein the eye gaze tracking module is configured to track the user's vision by identifying eye gaze, eye movement direction, pupil dilation, number of blinks, facial expressions, eye size enlargement, or pupil and iris movement.

[0263] The user input controller may include a head movement tracking module, wherein the head movement tracking module is configured to track a head movement direction, a head rotation angle, or a head rotation speed.

[0264] The user input controller may include a speech recognition module configured to track speech recognition of a user.

[0265] The server may further include a transmitter, wherein the transmitter is configured to transmit the analyzed data from the augmented reality (AR) device to another augmented reality (AR) device by using a location of the other augmented reality (AR) device, artificial intelligence technology, or a usage profile.

[0266] The display module may be configured to display content of the object of interest on another augmented reality (AR) device based on the analyzed data.

[0267] The display module may be configured to perform mirroring on the AR device of content of the object of interest displayed on the one or more computing devices in the event of unintentional disappearance of the object of interest or despite intentional disappearance of the object of interest and when the user views the object of interest for more than a threshold amount of time.

[0268] In another embodiment, a computer-implemented method for displaying content on an augmented reality (AR) device includes capturing a user's field of view, tracking the user's eye gaze toward an object of interest in the captured field of view, identifying frames containing an intentional disappearance of the object of interest in the captured field of view, analyzing the identified frames based on the intentional disappearance, generating analyzed data based on the analyzed frames, and displaying content of the object of interest on the augmented reality (AR) device based on the analyzed data.

[0269] In another embodiment, a computer-implemented system for displaying content on an augmented reality (AR) device includes: a memory configured to store predefined rules; a processor configured to cooperate with the memory and configured to generate system processing commands based on the predefined rules; a capture module configured to capture a user's field of view; a user input controller configured to track the user's eye gaze toward an object of interest in the captured field of view; and a server configured to cooperate with the AR device, the capture module, and the user input controller, and the server may include: an identifier configured to identify a frame containing an intentional disappearance of an object of interest in the captured field of view; and an analyzer configured to cooperate with the identifier to receive the identified frame, wherein the analyzer is configured to analyze the identified frame based on the intentional disappearance and generate analyzed data. The system also includes: a display module configured to display content of the object of interest on the augmented reality (AR) device based on the analyzed data.

[0270] In another embodiment, a computer-implemented method for displaying content on an augmented reality (AR) device includes capturing a user's field of view, recording the captured field of view, tracking the user's eye gaze toward an object of interest in the recorded field of view, identifying frames containing at least one of an unintentional disappearance of the object of interest in the recorded field of view or an intentional disappearance of the object of interest despite the recorded field of view, analyzing the identified frames based on at least one of the unintentional disappearance of the object of interest or the intentional disappearance of the object of interest despite the recorded field of view, generating analyzed data based on the analyzed frames, and displaying content of the object of interest on the augmented reality (AR) device based on the analyzed data.

[0271] In another embodiment, a computer-implemented system for displaying content on an augmented reality (AR) device includes: a memory configured to store predefined rules; a processor configured to cooperate with the memory and configured to generate system processing commands based on the predefined rules; a capture module configured to capture a user's field of view; a recording module configured to record the captured field of view; a user input controller configured to track the user's eye gaze toward an object of interest in the recorded field of view; and a server configured to cooperate with the AR device, the recording module, and the user input controller, wherein the server includes: an identifier configured to identify a frame containing at least one of an unintentional disappearance of an object of interest in the recorded field of view or an intentional disappearance of an object of interest despite interest in the recorded field of view; an analyzer configured to cooperate with the identifier to receive the identified frame, wherein the analyzer is configured to analyze the identified frame based on at least one of the unintentional disappearance of the object of interest or the intentional disappearance of the object of interest despite interest, and generate analyzed data. The system also includes: a display module configured to display content of the object of interest on the augmented reality (AR) device based on the analyzed data.

[0272] The above-described exemplary embodiments may be implemented as a software program including instructions stored on a machine (eg, computer) readable storage medium. A machine is a device that can call stored instructions from a storage medium and operate according to the called instructions.

[0273] When the command is executed by the processor, the processor may directly execute the function corresponding to the command, or use other components under the control of the processor to execute the function corresponding to the command. The command may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" only means that the storage medium does not include a signal but is tangible, and does not distinguish between the case where data is semi-permanently stored in the storage medium and the case where data is temporarily stored in the storage medium.

[0274] According to example embodiments, the methods according to the various example embodiments described above may be provided as included in a computer program product. The computer program product may be traded between a seller and a consumer as a product. The computer program product may be distributed in the form of (e.g., a compact disc read-only memory (CD-ROM)) or through an application store. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated in a manufacturer's server, an application store's server, or a storage medium such as a memory.

[0275] Each component (e.g., module or program) according to various exemplary embodiments may include a single entity or multiple entities, and some of the above-mentioned corresponding subcomponents may be omitted, or another subcomponent may be added to various exemplary embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into one entity, and the functions performed by each corresponding component may be performed in the same or similar manner as before the integration. Alternatively or additionally, some components (e.g., modules or programs) may be combined to form a single entity, wherein the single entity performs the same or similar functions as the corresponding elements before being combined. According to various exemplary embodiments, the operations performed by modules, programs or other components may be sequential, parallel, or both sequential and parallel, and the operations performed by modules, programs or other components may be performed iteratively or heuristically, or at least some operations may be performed in a different order, at least some operations may be omitted, or other operations may be added.

[0276] It should be noted that this description only illustrates the principles of the present disclosure. Therefore, it should be understood that those skilled in the art will be able to design various arrangements that embody the principles of the present disclosure, although not explicitly described herein. In addition, all examples described herein are primarily and explicitly intended to be for illustrative purposes only, to help the reader understand the principles of the present invention and the concepts contributed by the inventors to promote this area, and should be interpreted as not being limited to these specifically described examples and conditions. In addition, all statements describing the principles, aspects and embodiments of the present invention and their specific examples herein are intended to cover their equivalents.

Claims

1. A computer-implemented method for displaying content on an augmented reality (AR) device, the method comprising: Capturing the user’s field of view; recording the captured field of view; tracking the user's visual gaze toward one or more objects in the recorded field of view; an object of interest in the field of view recorded based on a tracking-based visual determination; identifying, from the recorded field of view, a frame containing a determination that the object of interest disappears from the recorded field of view; analyzing the identified frames based on at least one disappearance of the object of interest; generating analyzed data based on the analyzed frames; and displaying the content of the object of interest on the AR device based on the analyzed data, The step of displaying the content of the object of interest on the AR device based on the analyzed data includes: determining the content complexity of the object of interest, calculating a duration for showing the content to the user based on the content complexity, The content of the object of interest is displayed on the AR device for the calculated duration.

2. The method according to claim 1, wherein: The method further comprises: After the user turns away from the object of interest, determining that the user is interested in viewing the object of interest by profiling the user data or data from the AR device, and The frame is identified based on intentional non-obstruction-based disappearance or unintentional obstacle-based disappearance of the object of interest in the recorded field of view.

3. The method according to claim 1, wherein: The method further comprises: identifying a lack of interest in the object by the user, wherein the identified frame is analyzed based on intentional non-obstruction-based disappearance or unintentional obstacle-based disappearance of the object of interest in the recorded field of view, and Wherein the step of analyzing the identified frame based on the unintentional disappearance includes determining an obstacle between the user and the object of interest.

4. The method according to claim 1, wherein: The method further comprises: storing in a database the recorded fields of view, predefined time intervals, predefined object categories, and data related to one or more objects, predefined head rotation angles, and rotation speed of the user at at least one angle toward the one or more objects; Rendering the content on the AR device based on the user's gaze, the user's preference, and current view content, highlighting the object of interest based on gaze and a threshold time at a predefined location in the AR device; Detecting and identifying the object of interest from the identified frame using a deep learning technique, wherein the deep learning technique includes an object recognition technique, a deep structured learning technique, a hierarchical learning technique, or a neural network; sending the analyzed data from the AR device to the other AR device by using a location, artificial intelligence technology, or a usage profile of the other AR device; displaying, on the other AR device, the content of the object of interest based on the analyzed data; and In the event of an unintentional obstacle-based disappearance of the object of interest or an intentional non-obstruction-based disappearance of the object of interest despite being of interest, and when the user views the object of interest for more than a threshold amount of time, mirroring is performed on the AR device on content of the object of interest displayed on one or more computing devices.

5. The method according to claim 3, wherein: determining an obstruction between the user and the object of interest based on visual analysis, object recognition, eye movement, head movement, or a fixation gaze time on another object after the object of interest disappears, and Wherein, obstacles between the user and the object of interest are determined by using a plurality of sensors.

6. The method according to claim 3, wherein: The step of analyzing the identified frame based on the intentional disappearance comprises: analyzing the user's vision of the object of interest within a predefined time interval; identifying the intentional non-obstructive disappearance of the object of interest using an artificial intelligence-based multi-parameter model, wherein the multi-parameter model includes at least one of parameters related to: eye movement of the user, engagement level of the user, a companion with the user, and object category; parsing data received from sensors within a threshold time and viewing the data; identifying a gaze at a predetermined location of the AR device; and The time required for the user to view the object of interest is calculated based on the object information.

7. The method according to claim 1, wherein: The step of displaying the content of the object of interest comprises: displaying a fragment of the object of interest or a modified version of the object of interest on the AR device, wherein a modified version of the object of interest is generated by filtering and enhancing segments of the object of interest, and The step of filtering and enhancing the segment of the object of interest includes filtering and enhancing images, videos, enhanced information related to the object of interest, and a combination of images, videos and enhanced information related to the object of interest.

8. A computer-implemented system for displaying content on an augmented reality (AR) device, the system comprising: a memory configured to store predefined rules; a processor configured to cooperate with the memory, wherein the processor is configured to generate a system processing command based on the predefined rule; a capture module configured to capture a user's field of view; a recording module configured to record the captured field of view; a user input controller configured to track the user's visual gaze toward one or more objects in the recorded field of view; and A server is configured to cooperate with the AR device, the recording module and the user input controller, Wherein, the server includes: a determination module configured to determine an object of interest in the recorded field of view based on the tracked vision; an identifier configured to cooperate with the determination module to receive the determined object of interest, wherein the identifier is configured to identify a frame from the recorded field of view containing a determined disappearance of the object of interest in the recorded field of view; and an analyzer configured to cooperate with the recognizer to receive the recognized frame, wherein the analyzer is configured to analyze the recognized frame based on at least one disappearance of the object of interest and generate analyzed data; and The system further includes: a display module configured to display the content of the object of interest on the AR device based on the analyzed data, The server further comprises a complexity identifier, wherein the complexity identifier is configured to determine the content complexity of the object of interest and calculate a duration for showing the content to the user based on the content complexity, and The display module is configured to display the content of the object of interest on the AR device within the calculated duration.

9. The system according to claim 8, wherein: The identifier is configured to identify the frame based on a non-obstruction-based intentional disappearance or an obstacle-based unintentional disappearance of the object of interest in the recorded field of view, and The analyzer is further configured to determine that the user is interested in viewing the object of interest by analyzing user data or data from the AR device after the user moves his gaze away from the object of interest.

10. The system according to claim 8, wherein: The analyzer is configured to: analyzing the identified frames based on a non-obstruction-based intentional disappearance or an obstacle-based unintentional disappearance of the object of interest in the recorded field of view; analyzing the identified frame based on the unintentional disappearance by determining an obstruction between the user and the object of interest; A lack of interest by the user in the one or more objects is identified.

11. The system according to claim 8, wherein: The system further comprises: a database configured to store the recorded fields of view, predefined time intervals, redefined object categories, and data related to the one or more objects, predefined head rotation angles, and rotation speed of the user at at least one angle toward the one or more objects, a content rendering module configured to render the content on the AR device based on the user's gaze, the user's preference, and the current view content, The recognizer further comprises an object recognition module, wherein the object recognition module is configured to detect and recognize the object of interest from the recognized frame using a deep learning technique, wherein the deep learning technique comprises an object recognition technique, a deep structured learning technique, a hierarchical learning technique or a neural network, The server further includes a transmitter, wherein the transmitter is configured to transmit the analyzed data from the AR device to the other AR device by using a location of the other AR device, artificial intelligence technology, or a usage profile, wherein the display module is configured to highlight the object of interest based on a gaze and a threshold time at a predefined position in the AR device, and the display module is configured to display content of the object of interest on the other AR device based on the analyzed data, and Wherein, the display module is configured to perform mirroring on the AR device of the content of the object of interest displayed on one or more computing devices in the event of an unintentional obstacle-based disappearance of the object of interest or an intentional non-obstruction-based disappearance of the object of interest despite interest, and when the user views the object of interest for more than a threshold amount of time.

12. The system according to claim 8, wherein: The analyzer is configured to analyze the identified frame based on unintentional disappearance by determining an obstacle between the user and the object of interest, wherein the user input controller is configured to track the user's vision based on at least one of the following parameters: eye gaze, eye movement direction, head movement direction, pupil dilation, number of blinks, head rotation angle, head rotation speed, facial expression, eye size dilation, pupil and iris movement, and voice recognition, and Wherein the identifier is configured to identify the frame based on visual time and threshold time.

13. The system according to claim 10, wherein: The analyzer is configured to analyze the identified frame based on the intentional disappearance, wherein the analyzer is configured to: analyzing the user's vision of the object of interest within a predefined time interval; identifying the intentional non-obstructive disappearance of the object of interest using a multi-parameter model based on artificial intelligence, wherein the model includes at least one of parameters related to: eye movement of the user, engagement level of the user, a companion with the user, or object category; parsing data received from the sensor within a threshold time and viewing the data; and identifying a gaze at a predetermined location of the AR device, The complexity identifier is further configured to calculate the time required for the user to view the object of interest based on the object information.

14. The system according to claim 8, wherein: The user input controller includes an eye gaze tracking module, wherein the eye gaze tracking module is configured to track the user's vision by identifying eye gaze, eye movement direction, pupil dilation, number of blinks, facial expressions, eye size enlargement, or pupil and iris movement, The user input controller includes a head movement tracking module, wherein the head movement tracking module is configured to track the head movement direction, the head rotation angle or the head rotation speed, The user input controller includes a speech recognition module, wherein the speech recognition module is configured to track speech recognition of the user.

15. A computer-implemented method for displaying content on an augmented reality (AR) device, the method comprising: Capturing the user’s field of view; recording the captured field of view; tracking eye gaze of the user toward a recorded object of interest in the field of view; identifying, from the recorded field of view based on the tracked eye gaze, a frame containing at least one of an unintentional disappearance of the object of interest from the recorded field of view based on an obstacle or an intentional disappearance of the object of interest from the recorded field of view despite being of interest based on a non-obstruction; analyzing the identified frame based on said at least one of an unintentional disappearance of said object of interest or an intentional disappearance of said object of interest despite being interested; generating analyzed data based on the analyzed frames; and displaying the content of the object of interest on the AR device based on the analyzed data, The step of displaying the content of the object of interest on the AR device based on the analyzed data includes: determining the content complexity of the object of interest, calculating a duration for showing the content to the user based on the content complexity, The content of the object of interest is displayed on the AR device for the calculated duration.

Images