Method and system to provide dynamic overlay on xr device

Abstract

A method and system for providing a dynamic overlay of an XR device are disclosed. The method for providing a dynamic overlay of an XR device may include the step of recognizing a user situation based on AI (artificial intelligence); and the step of providing a dynamic overlay according to the user situation through an XR (Extended Reality) display.

Description

Technology Field

[0001] The following description concerns technology that provides an XR (Extended Reality) interface.

[0002] This patent was created as part of the New Industry Intellectual Property Convergence Talent Development Project under the Ministry of Education's University Innovation Support Program. Background Technology

[0003] An XR device is a device that displays an environment to a user through, for example, a head-mounted display (HMD), glasses, a mobile handset, or other devices.

[0004] In this case, the environment differs at least partially from the real-world environment where the user is located. Generally, the user can interactively change their view of the environment by tilting or moving the HMD, for example by moving the user's head, or by tilting or moving other devices.

[0005] Virtual reality (VR), augmented reality (AR), and mixed reality (MR) are examples of XR.

[0006] With the recent launch of Apple Vision Pro, interest is focusing on the XR (AR, MR, etc.) device market.

[0007] As an example of XR interface technology, Korean Published Patent No. 10-2024-0033226 (published on March 12, 2024) discloses a technology that provides a virtual touch-based interface in XR. The problem to be solved

[0008] Through an XR device, it is possible to implement a dynamic overlay that recognizes objects in the user's line of sight and provides information or animations about those objects.

[0009] It is possible to provide real-time user-customized overlays based on information that considers the user's current situation, context, preferences, etc. means of solving the problem

[0010] A method for providing a dynamic overlay for an XR (Extended Reality) device comprising at least one processor, the method comprising: a step of recognizing a user situation based on AI (Artificial Intelligence) by the at least one processor; and a step of providing a dynamic overlay according to the user situation through an XR (Extended Reality) display by the at least one processor.

[0011] According to one aspect, the recognizing step may recognize an object in the surrounding environment associated with the user event through on-device-based AI or cloud-based AI for the user event detected by the XR device.

[0012] According to another aspect, the recognizing step may include a step of collecting environmental data through a sensor embedded in the XR device to supplement object recognition.

[0013] According to another aspect, the recognizing step may include a step of refusing the use of the cloud-based AI in the case of data collected for the dynamic overlay that raises concerns about privacy infringement.

[0014] According to another aspect, the recognizing step may include the step of recognizing an object corresponding to the operating condition when the operating condition for the dynamic overlay is selected.

[0015] According to another aspect, the recognizing step may include a step of recognizing whether there is an object set in the operation condition in the user's line of sight when the operation condition for the dynamic overlay is selected.

[0016] According to another aspect, the step provided above may include a step of determining whether to expose the object information based on user-customized data set by the user as association information of the object recognized as the user situation.

[0017] According to another aspect, the step provided above may include rendering an animation searched or generated based on association information of an object recognized as the user situation onto the XR display.

[0018] According to another aspect, the steps provided above may further include a step of adjusting the overlay position by recognizing the screen viewed by the user in real time.

[0019] According to another aspect, the step provided above may further include the step of caching information exposed through the dynamic overlay on the XR device.

[0020] According to another aspect, the step provided above may include the step of searching for an advertisement for an object recognized as the user situation and overlaying the searched advertisement.

[0021] According to another aspect, the overlaying step may include: a step of labeling preferred objects among the recognized objects based on user-related data for user-customized advertisements; and a step of searching for the advertisement using the labeled data.

[0022] According to another aspect, the overlaying step may include a step of selecting an advertisement through a real-time bidding method when multiple advertisements are searched for a single object.

[0023] According to another aspect, the steps provided above may further include: a step of collecting user feedback regarding the advertisement; and a step of analyzing the user's sentiment regarding the advertisement based on the user feedback.

[0024] The present invention provides an XR device comprising at least one processor implemented to execute readable commands on a computer-implemented XR (Extended Reality) device, wherein the at least one processor processes a process of recognizing a user situation based on AI (Artificial Intelligence); and a process of providing a dynamic overlay according to the user situation through an XR display. Effects of the invention

[0025] According to embodiments of the present invention, by providing a real-time user-customized overlay based on information that takes into account the user's current situation, context, preferences, etc., the user can obtain immersive and personalized information and maximize the usability of the XR device. Brief explanation of the drawing

[0026] FIG. 1 is a block diagram illustrating an example of the internal configuration of a computer device in an embodiment of the present invention. FIG. 2 illustrates the overall architecture of an XR overlay providing system in one embodiment of the present invention. FIG. 3 illustrates an example of a method for providing an XR overlay in an embodiment of the present invention. FIG. 4 illustrates a system configuration for providing a dynamic overlay according to user conditions in an embodiment of the present invention. Figures 5 and 6 illustrate an example of providing a dynamic overlay depending on the user's situation while wearing an XR device. FIG. 7 illustrates another example of a method for providing an XR overlay in one embodiment of the present invention. FIG. 8 illustrates a system configuration for providing a dynamic overlay according to a user situation in an embodiment of the present invention. FIGS. 9 and FIGS. 10 illustrate an example of providing a dynamic overlay based on the type of object recognized in the surroundings of a user wearing an XR device. FIG. 11 illustrates the overall architecture of an XR overlay-based advertising system in one embodiment of the present invention. FIG. 12 illustrates another example of a method for providing an XR overlay in one embodiment of the present invention. FIG. 13 illustrates a system configuration for applying a dynamic overlay to an advertisement in one embodiment of the present invention. Figure 14 illustrates an example of providing a dynamic overlay with an advertisement to a user wearing an XR device. Specific details for implementing the invention

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0029] Embodiments of the present invention relate to technology that provides an XR (Extended Reality) interface.

[0030] Embodiments including those specifically disclosed in this specification can provide a real-time user-customized overlay based on information that takes into account the user's current situation, context, preferences, etc.

[0031] An XR overlay providing system according to embodiments of the present invention may be implemented by at least one computer device, and an XR overlay providing method according to embodiments of the present invention may be performed through at least one computer device included in the XR overlay providing system. At this time, a computer program according to one embodiment of the present invention may be installed and run on the computer device, and the computer device may perform an XR overlay providing method according to embodiments of the present invention under the control of the run computer program. The above-described computer program may be stored on a computer-readable recording medium to be combined with the computer device to execute the XR overlay providing method on the computer.

[0032] FIG. 1 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. For example, an XR overlay providing system according to embodiments of the present invention may be implemented by a computer device (100) illustrated in FIG. 1.

[0033] As illustrated in FIG. 1, a computer device (100) may include a memory (110), a processor (120), a communication interface (130), and an input / output interface (140) as components for executing an XR overlay providing method according to embodiments of the present invention.

[0034] Memory (110) is a computer-readable recording medium and may include a non-perishable mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Here, a non-perishable mass storage device such as a ROM and a disk drive may be included in the computer device (100) as a separate permanent storage device distinct from memory (110). Additionally, an operating system and at least one program code may be stored in memory (110). These software components may be loaded into memory (110) from a computer-readable recording medium separate from memory (110). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. In another embodiment, software components may be loaded into memory (110) through a communication interface (130) rather than a computer-readable recording medium. For example, software components can be loaded into the memory (110) of the computer device (100) based on a computer program installed by files received through the network (160).

[0035] The processor (120) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (120) via memory (110) or a communication interface (130). For example, the processor (120) may be configured to execute instructions received according to program code stored in a recording device such as memory (110).

[0036] The communication interface (130) may provide a function for the computer device (100) to communicate with other devices through a network (160). For example, requests, commands, data, files, etc. generated by the processor (120) of the computer device (100) according to program code stored in a recording device such as memory (110) may be transmitted to other devices through the network (160) under the control of the communication interface (130). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (100) through the communication interface (130) of the computer device (100) via the network (160). Signals, commands, data, etc. received through the communication interface (130) may be transmitted to the processor (120) or memory (110), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (100) may further include.

[0037] The communication method is not limited and may include not only communication methods utilizing communication networks (e.g., mobile communication networks, wired internet, wireless internet, broadcasting networks) that the network (160) may include, but also short-range wired / wireless communication between devices. For example, the network (160) may include any one or more networks such as a PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the Internet. Additionally, the network (160) may include any one or more network topologies such as a bus network, star network, ring network, mesh network, star-bus network, tree or hierarchical network, but is not limited thereto.

[0038] The input / output interface (140) may be a means for interfacing with an input / output device (150). For example, the input device may include a device such as a microphone, keyboard, camera, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (140) may be a means for interfacing with a device in which the functions for input and output are integrated into one, such as a touchscreen. The input / output device (150) may be composed of a computer device (100) and a single device.

[0039] Additionally, in other embodiments, the computer device (100) may include fewer or more components than the components of FIG. 1. However, it is not necessary to clearly illustrate most of the prior art components. For example, the computer device (100) may be implemented to include at least some of the input / output devices (150) described above, or may include other components such as a transceiver, a camera, various sensors, a database, etc.

[0040] Below, specific embodiments of the dynamic overlay provision technology for XR devices will be described.

[0041] Augmented reality technology is a technology that integrates virtual objects with the real world to allow users to experience immersion.

[0042] XR technology is currently being utilized in various fields such as entertainment, education, and healthcare.

[0043] XR devices possess the ability to detect interactions with the user in real time and recognize objects in the surrounding environment to provide useful information to the user. This capability of XR can enhance the user experience and is applicable to various fields.

[0044] In this embodiment, a dynamic overlay can be implemented that recognizes an object in the user's line of sight through an XR device and provides information or animation about the object.

[0045] The present invention can provide real-time user-customized overlays based on information that considers the user's current situation, context, preferences, etc. Through this, users can obtain immersive and personalized information and maximize the usability of XR devices. This can be utilized in various fields such as advertising, education, manufacturing, and entertainment.

[0046] FIG. 2 illustrates the overall architecture of an XR overlay providing system in one embodiment of the present invention.

[0047] Referring to FIG. 2, the XR overlay providing system (200) according to the present invention can identify surrounding environments such as IoT devices (201), other home appliances (202), and objects (203) through a network (20), and can display such environments to a user.

[0048] At this time, the XR overlay providing system (200) can be implemented on any XR device that provides augmented reality functionality.

[0049] The network (20) may refer to a cloud-based AI for improving the accuracy of AI, i.e., on-device AI, which is installed within an XR device, targeting an XR device in which an XR overlay providing system (200) is implemented.

[0050] The XR overlay providing system (200) may be implemented as a computer device (100) described through FIG. 1. The processor (120) of the computer device (100) may be implemented as a component for performing the following XR overlay providing method. Depending on the embodiment, the components of the processor (120) may be optionally included in or excluded from the processor (120). Additionally, depending on the embodiment, the components of the processor (120) may be separated or merged to represent the function of the processor (120).

[0051] These processors (120) and components of the processor (120) can control the computer device (100) to perform steps included in the following XR overlay providing method. For example, the processor (120) and components of the processor (120) may be implemented to execute instructions according to the code of an operating system included in memory (110) and the code of at least one program.

[0052] Here, the components of the processor (120) may be representations of different functions performed by the processor (120) according to instructions provided by program code stored in the computer device (100).

[0053] The processor (120) can read necessary instructions from memory (110) in which instructions related to the control of the computer device (100) are loaded. In this case, the read instructions may include instructions for controlling the processor (120) to execute the steps to be described later.

[0054] The steps included in the XR overlay provision method described below may be performed in a different order than the illustrated order, and some of the steps may be omitted or additional processes may be included.

[0055] FIG. 3 illustrates an example of an XR overlay provision method in an embodiment of the present invention. FIG. 3 shows a dynamic overlay provision procedure according to user conditions as an example of an XR overlay provision method.

[0056] Referring to FIG. 3, in step (S301), the processor (120) can detect user events in real life. It is assumed that the user is wearing an XR device that includes an XR overlay providing system according to the present invention. The user can convey requirements to the XR device through gestures or commands. For example, the user can convey requirements such as, "I need to finish urgent work quickly, so please sound an alert if I look away from where I am currently looking. I'm going to work for 30 minutes!"

[0057] In step (S302), the processor (120) can recognize the surroundings of the user based on AI (artificial intelligence). In order to recognize "where the user is currently looking" among the user's requirements, the processor (120) can identify specific objects, locations, structures, etc. of the place the user is looking at in detail. When analyzing user requirements, the processor (120) can proceed to the next step depending on whether additional data collection is necessary as a result of the requirements analysis.

[0058] In step (S303), the processor (120) can proceed with additional data collection and utilization based on the results of the requirements analysis. If additional actions are required as a result of the analysis, it can perform actions such as connecting to the IoT device (201) and dynamic overlay setting. If there are no specific requirements from the user regarding dynamic overlay setting, the user interaction management module (430 in FIG. 4), which will be described below, can utilize AI-based user pattern data to recommend dynamic overlay settings for situations similar to the user's stored patterns and determine whether to apply them based on the user's response. Data transmission from the IoT device can be received through the existing connected Wi-Fi, network, or home automation hub, and if a device other than the existing connected device is required, the object connection module (830 in FIG. 8), which will be described below, can proceed with device discovery and connection.

[0059] In step (S304), the processor (120) can provide a dynamic overlay that is appropriate for the situation around the user. Since the user has requested 30 minutes of concentration time in the user's requirements, the timer can be activated, and if the user is not concentrating and looks at the door behind them, a notification can be overlaid with a timer image showing the remaining time on the door and the notification to turn back to the opposite side from where the user turned.

[0060] User event detection on XR devices

[0061] For example, an (event detection) XR device can detect various inputs in real time, such as user gestures, voice commands, and touch.

[0062] As another example, (gesture recognition) cameras and AI-based image processing can be used to recognize the user's hand movements, posture, etc.

[0063] As another example, (voice command recognition) a user's voice command can be analyzed using a microphone and voice recognition software, and in this process, technologies such as natural language processing are used to interpret and understand the user's command.

[0064] As another example, user input can be detected by utilizing various sensors within the XR device.

[0065] Recognition of surrounding environment objects associated with user events

[0066] The object detection module (420 in FIG. 4), which will be described below, can recognize important objects in the surrounding environment associated with user events.

[0067] (AI-based object recognition) Before using network-based AI, the policy controller (441 in Fig. 4) rejects the use of network-based AI if there is sensitive information or data that raises concerns about privacy infringement. XR: Image and object recognition can be performed using on-device AI, and if necessary, the accuracy of object recognition can be further improved through cloud-based AI (network-based AI).

[0068] (Context Recognition) Object recognition can be complemented by collecting environmental data through various sensors. For example, information about the user's surrounding environment can be collected based on sensor data such as location, lighting, and noise. As another example, the precision of object recognition can be improved by integrating with AI-based object recognition.

[0069] Request to generate association information for recognized objects

[0070] A request is sent to a user interaction management module (430 in FIG. 4) to obtain additional information about the recognized object. At this time, the request information may include an object ID (an identifier that enables identification of the recognized object), connected device data, location information (information indicating the spatial location of the object), user state information (state of the user's current activity, such as whether they are standing, sitting, or moving), object state information (the current state of the object), time information (the time when the request occurred, etc.), user-customized data, dynamic overlay setting information (using user-customized data), etc., and the request details may also be changed according to the situation.

[0071] The above-mentioned request information can be queried through a separate Domain Name System (DNS) or directory service for IoT devices. Alternatively, a specific database, etc., can be queried using device information previously registered by the user.

[0072] Response regarding association information of the recognized object

[0073] The user interaction management module (430 in FIG. 4) can generate response data for a request for association information of a recognized object, and the generated data can be transmitted to the policy controller (441 in FIG. 4).

[0074] Receiving object information on an XR device

[0075] The XR device can receive information about the requested recognition object.

[0076] Determining whether object information is exposed on the XR device

[0077] The policy controller (441 in FIG. 4) can determine the degree of dynamic overlay exposure based on user setting values ​​(Dynamic Overlay setting, privacy protection, etc.) before overlaying through the overlay rendering engine (442 in FIG. 4) based on data received from the user interaction management module (430 in FIG. 4). Object information with a determined exposure degree is transmitted to the overlay rendering engine (442 in FIG. 4).

[0078] Exposing object information on XR devices

[0079] The overlay rendering engine (442 in FIG. 4) can render on the XR display based on the received information. At this time, AI can be used to search for or generate and implement a suitable animation for the overlay. The overlay position can be optimized by recognizing the screen the user is looking at in real time. The exposed information can be cached on the user's XR device terminal in preparation for additional use by the user.

[0080] FIG. 4 illustrates a system configuration for providing a dynamic overlay according to user conditions in an embodiment of the present invention.

[0081] Referring to FIG. 4, an XR overlay providing system implemented on an XR device may include a user event detection module (410), an object detection module (420), a user interaction management module (430), and a dynamic overlay module (440).

[0082] The user event detection module (410) can detect events through user commands or gestures. For example, if a user speaks, such as “I need to finish urgent work quickly, so if I look away from where I am currently looking, sound an alarm. I will be working for 30 minutes!”, this can be recognized as a command.

[0083] The object detection module (420) may include AI-based object recognition and context awareness.

[0084] In the case of AI-based object recognition, it is possible to identify the object that the user is observing. Image and object recognition can be performed by utilizing on-device AI within the XR device, which enables the identification of IoT devices and the understanding of the context of user interactions. If accurate recognition is difficult, object recognition can be carried out by additionally utilizing network-based AI.

[0085] For context awareness, surrounding environment data (such as the space where the user is located, lighting, and noise levels) can be collected in real time through various sensors embedded in the XR device. By combining this with AI-based object recognition, objects around the user can be recognized and distinguished in greater detail (e.g., the brand of an object the user is looking at). By collecting and analyzing information such as the user's activity, time, and location in real time, the user's current situation can be understood. If the data requires additional settings (such as setting do not disturb mode levels, utilizing user patterns, etc., in addition to simple actions), the data can be transmitted to the user interaction management module (430), and if it is a simple action, the data can be transmitted to the dynamic overlay module (440).

[0086] The user interaction management module (430) may include dynamic overlay settings. The initial user may input settings for the Do Not Disturb mode stages and situations. An example of dynamic overlay settings is shown in Table 1.

[0087] Step by step Step 1: Apply Full-Dynamic Overlay to Connectable Devices Step 2: Applying a simplified Dynamic Overlay to connectable devices Step 3: Disable Dynamic Overlay By situation Cleaning Mode: Dynamic Overlay applied only to washing machines, dryers, vacuum cleaners, air purifiers, and robot vacuums. Game Mode: Apply Dynamic Overlay only to monitor and speakers

[0088] The user interaction management module (430) can determine whether to apply a dynamic overlay based on selected information when the user selects a Do Not Disturb mode level set by the user. The user interaction management module (430) can manage personalized data. After consent to the collection of information, it can collect user data (data for providing user-customized services, such as preferences and purchase history) in real time and store specific recurring routines as user patterns. The stored data and patterns can be appropriately utilized for recommending dynamic overlay settings, etc., according to the user's requirements.

[0089] The user interaction management module (430) can manage the integration environment with other devices. When the object detection module (420) needs to provide visual feedback to the user through the integration of additional devices, it can receive information or data from the device by linking with the other device.

[0090] The dynamic overlay module (440) may include a policy controller (441) and an overlay rendering engine (442).

[0091] The policy controller (441) may refuse the use of network-based AI when processing data collected for dynamic overlays if the data includes sensitive information or data that may infringe on privacy. Based on the data received from the user interaction management module (430), the data may be adjusted to match user settings before overlay rendering and then transmitted to the overlay rendering engine (442).

[0092] The overlay rendering engine (442) can implement a suitable animation through the received data, and can implement the animation by searching for or generating a suitable animation through AI, a network, etc. It can also adjust the position of the overlay by analyzing the screen viewed by the user in real time.

[0093] Figures 5 and 6 illustrate an example of providing a dynamic overlay depending on the user's situation while wearing an XR device.

[0094] Context-aware dynamic overlays can be utilized to induce user immersion; work efficiency can be increased by providing dynamic overlays that understand, interact with, and support the user's situation. Furthermore, they can be applied to user learning, as even first-time tasks can be easily performed through the guidance provided by dynamic overlays, making them suitable for use in the education sector.

[0095] FIG. 7 illustrates another example of an XR overlay providing method in one embodiment of the present invention. FIG. 7 shows a dynamic overlay providing procedure according to the type of object as another example of an XR overlay providing method.

[0096] Referring to FIG. 7, in step (S701), the processor (120) can define an initial setting. The user can set or modify the situational and step-by-step operation conditions of the dynamic overlay. The setting can be configured via gestures or voice commands within the XR device (On-device AI processes the command or gesture). For example, the operation conditions can be configured via voice commands such as, "I'm going to create a cleaning mode. (Pointing at the washing machine with a finger) Add that, and show the overlay in full."

[0097] In step (S702), the processor (120) can select an operation condition for the dynamic overlay. If the user selects an operation condition after wearing the XR device, or changes the operation condition in the middle, the dynamic overlay can operate under that condition. For example, a cleaning mode can be set (the dynamic overlay is operated only on washing machines, dryers, and robot vacuum cleaners, and the devices set to that mode also perform that operation).

[0098] In step (S703), the processor (120) can perform object recognition that meets the conditions. It recognizes objects that meet the conditions in real time, and in the case of devices in a specific space, the user can set the relevant information to reduce problems such as errors in device recognition. For example, it can recognize whether a device (washing machine, dryer) set to cleaning mode is in the user's line of sight.

[0099] In step (S704), the processor (120) can perform device connection based on object recognition. If the devices corresponding to the object recognition are already connected to the XR device, the device connection process is omitted. When selecting the operation condition, the device can be connected immediately, or devices that are not connected can be searched for and connected through the device connection module. Once a device is connected, it can be connected immediately for subsequent use.

[0100] In step (S705), the processor (120) can collect data from the connected device. It can collect data from the connected or already connected device and check if there is a UI for the XR overlay provided by the original device.

[0101] In step (S706), the processor (120) can provide a dynamic overlay that reflects the collected data. The processor (120) can utilize AI to provide a dynamic overlay suitable for the situation. If there is a UI for the XR overlay, that UI can be applied. For example, if you look at the washing machine from a distance without going in front of it, an overlay showing how much washing has been done, which stage is complete and how much is left, and an additional video of the washing machine's interior can be applied to give the impression that the washing is being done well. Additionally, if you look at an air conditioner emitting cold air, a wave animation showing cold air coming out of the inlet can be overlaid along with a settings window next to the air conditioner.

[0102] FIG. 8 illustrates a system configuration for providing a dynamic overlay according to a user situation in an embodiment of the present invention.

[0103] Referring to FIG. 8, an XR overlay providing system implemented on an XR device may include a dynamic overlay setting module (810), an object detection module (820), an object connection module (830), and a dynamic overlay module (840).

[0104] The dynamic overlay setting module (810) may include dynamic overlay settings. The initial user may input settings for the Do Not Disturb mode stages and situations. An example of dynamic overlay settings is shown in Table 1. When the user selects a Do Not Disturb mode stage they have set, they can determine whether to apply the dynamic overlay based on the selected information.

[0105] The object detection module (820) provides an AI-based object recognition function and can recognize what kind of object the user is looking at. It can perform image and object recognition by utilizing on-device AI within the XR device, thereby identifying IoT devices and processing contextual understanding of user interaction. If accurate recognition is difficult, object recognition can be performed by additionally utilizing AI through a network. At this time, the object detection module (820) can transmit the recognized data to the object connection module (830).

[0106] The object connection module (830) can perform a connection with a connectable device near the user. When a request for a specific device is received from the dynamic overlay setting module (810), it can provide data for that device. The object connection module (830) can provide various data for applying a dynamic overlay, such as sensor data from the device and an XR-specific UI provided by the device. The object connection module (830) can provide a connection with a connectable device through Wi-Fi, a network, an automation hub acting as a central node for IoT devices, etc.

[0107] The dynamic overlay module (840) may include a policy controller (841) and an overlay rendering engine (842).

[0108] The policy controller (841) may refuse the use of network-based AI when processing data collected for dynamic overlays if the data includes sensitive information or data that may infringe on privacy. Based on the data received from the dynamic overlay setting module (810), the data may be adjusted to match user setting values ​​before overlay rendering and then transmitted to the overlay rendering engine (842).

[0109] The overlay rendering engine (842) can implement a suitable animation through the received data, and can implement the animation by searching for or generating a suitable animation through AI, a network, etc. It can also adjust the position of the overlay by analyzing the screen viewed by the user in real time.

[0110] FIGS. 9 and FIGS. 10 illustrate an example of providing a dynamic overlay based on the type of object recognized in the surroundings of a user wearing an XR device.

[0111] Dynamic overlays based on the type of object can be utilized to provide information about home appliances. This can be used to enhance the user experience of electronic devices when a user is wearing an XR device; for instance, it can maximize the advantages of XR devices by providing a vivid interface that makes an air conditioner feel cooler.

[0112] FIG. 11 illustrates the overall architecture of an XR overlay-based advertising system in one embodiment of the present invention.

[0113] Referring to FIG. 11, the XR overlay providing system (200) according to the present invention can apply a dynamic overlay to an advertisement by linking with an advertisement system (1101) that provides external advertisement data. At this time, the XR overlay providing system (200) can be implemented on any XR device that provides augmented reality functions.

[0114] FIG. 12 illustrates another example of a method for providing an XR overlay in an embodiment of the present invention. FIG. 12 illustrates a procedure for providing a dynamic overlay in an advertisement as another example of a method for providing an XR overlay.

[0115] Assume a situation where users consume XR content and content providers must display advertisements to generate revenue.

[0116] Referring to FIG. 12, in step (S1201), the processor (120) can recognize objects within the user's line of sight. Object recognition involves recognizing surrounding objects in advance before the advertisement starts, and can perform AI-based object recognition. Before using network-based AI, if there is sensitive information or data that raises concerns about privacy infringement, the use of network-based AI can be refused. Image and object recognition can be performed using AI within the XR device (On-device AI), and if necessary, the accuracy of object recognition can be further improved through cloud-based AI (Network-based AI). The processor (120) performs context recognition, and can supplement object recognition by collecting environmental data through various sensors. At this time, information regarding the user's surrounding environment, such as location, lighting, and noise, can be collected as sensor data. The precision of object recognition can be improved by linking with AI-based object recognition (linked recognition). The recognized objects are transmitted to a customized advertisement module (1320 in FIG. 13).

[0117] In step (S1202), the processor (120) can label the recognized object data with customized data. Data for user-customized advertisements that the device has collected, such as cookies, logs, purchase history, and feedback received after watching an advertisement, can be utilized. After checking whether there is an object among the recognized objects that can be labeled as a key advertisement using user-customized data information, if such an object exists, it can be labeled as a key advertisement. At this time, if there is an object preferred by the user, it is labeled, and if not, the data is transmitted as is. Data predicting preferences, interests, purchasing tendencies, etc., using on-device AI can be additionally utilized. The labeled data can be transmitted to an advertisement query module (1330 in FIG. 13) for ad search.

[0118] In step (S1203), the processor (120) can search for advertisements corresponding to the recognized object data. The processor (120) can search for advertisements in an advertisement DB (1340 in FIG. 13), which is a storage for storing advertisements. The advertisement DB may store general advertisements and advertisements dedicated to XR content. General advertisements refer to general viewing advertisements, and advertisements dedicated to XR content may include experiential advertisements that apply a dynamic overlay to the recognized object (e.g., overlaying a new product onto an old product) to allow users to experience the features of the new product in augmented reality. The processor (120) can search for appropriate advertisements in the advertisement DB based on the data labeled for the recognized object. Depending on whether Real-time Bidding is applied, the search may yield various results, such as multiple advertisements for a single object or multiple advertisements for multiple objects. The searched advertisements are transmitted to an advertisement placement module (1350 in FIG. 13).

[0119] In step (S1204), the processor (120) can perform ad placement and real-time bidding on the searched ads. If multiple user-customized ads are found for a single object, the optimal ad can be selected through a real-time bidding method. If only one ad is found for a single object, the process proceeds as is, and the data with the applied content is transmitted to the policy controller (1361 in FIG. 13).

[0120] In step (S1205), the processor (120) can determine whether to display a policy-based advertisement. It can determine whether to display an advertisement based on received advertisement information, and at this time, the advertisement display settings set by the user can be applied. The determined whether to display an advertisement is transmitted to the overlay rendering engine (1362 in FIG. 13).

[0121] In step (S1206), the processor (120) can apply a dynamic overlay advertisement to the XR device. The processor (120) can overlay an advertisement received through a policy controller. When the user's gaze changes, it can recognize an object in the direction the gaze was turned and play one of the prepared advertisements. If an experiential advertisement cannot be shown, a standard video advertisement can be played on the display. The position of the dynamic overlay advertisement can be optimized by analyzing the user's gaze in real time.

[0122] In step (S1207), the processor (120) can collect and analyze feedback regarding the advertisement. Once the user’s experience (or viewing) of the advertisement is finished, feedback can be collected through the user’s gestures, voice recognition, or actions. For example, feedback can be collected through gestures such as raising a thumb or lowering a thumb 180 degrees while raising it. It is also possible to collect feedback based on voice, such as “This ad is good,” or actions such as the user’s purchase or installation of the advertised product. Based on the collected feedback, the processor (120) can analyze emotions corresponding to the user’s voice, tone, facial expression, and actions. The processor (120) can collect and analyze various feedback data and utilize them as user-customized data. User feedback analysis data can be provided to advertisers and advertising databases after obtaining the user’s consent to provide user information. By utilizing user feedback analysis data to improve the quality of the advertisement, the quality of the advertisement experienced or viewed by the user can be enhanced and the advertising effect can be maximized.

[0123] In step (S1208), the processor (120) can update user-customized data. User-customized data can be updated based on collected and analyzed feedback data, and can be utilized together with user-customized data to provide advertising optimization.

[0124] FIG. 13 illustrates a system configuration for applying a dynamic overlay to an advertisement in one embodiment of the present invention.

[0125] Referring to FIG. 13, an XR overlay providing system implemented on an XR device may include an object detection module (1310), a custom advertisement module (1320), an advertisement query module (1330), an advertisement DB (1340), an advertisement placement module (1350), a dynamic overlay module (1360), and a feedback analysis module (1370).

[0126] The object detection module (1310) may include AI-based object recognition and context awareness.

[0127] In the case of AI-based object recognition, it is possible to identify the object that the user is observing. Image and object recognition can be performed by utilizing on-device AI within the XR device, which enables the identification of IoT devices and the understanding of the context of user interactions. If accurate recognition is difficult, object recognition can be carried out by additionally utilizing network-based AI.

[0128] In terms of context awareness, ambient environmental data (such as the space where the user is located, lighting, and noise levels) can be collected in real time through various sensors embedded in XR devices. By combining this with AI-based object recognition, objects around the user can be recognized and distinguished in greater detail (e.g., the brand of the object the user is focusing on). By collecting and analyzing information such as the user's activity, time, and location in real time, the user's current situation can be understood.

[0129] The object detection module (1310) transmits data related to object detection to the customized advertising module (1320).

[0130] The customized advertising module (1320) can collect data from the context recognition module through the object detection module (1310), as well as information such as user cookies, logs, and purchase history, which are essential for user-customized advertising. The collected data can be preprocessed and analyzed, and the analyzed data can be used to perform AI-based user behavior pattern analysis (preferences, interests, purchasing propensity, etc.), such as machine learning.

[0131] The ad query module (1330) can query the ad DB (1340) for user-customized ads based on the received data.

[0132] The advertising DB (1340) serves as a storage facility for all advertising content that can be provided through dynamic overlays. XR content advertising data from advertisers can be stored in the advertising DB (1340), and the advertising DB (1340) can include all advertisements that users can experience. For example, if it is a washing machine advertisement, it may include an advertisement for XR devices that allows users to experience the washing machine's functions.

[0133] The ad placement module (1350) is a module that performs the roles of placing ads, selecting platforms, and real-time bidding. It transmits the ads queried from the ad DB (1340) through the ad query module (1330) to the dynamic overlay module (1360).

[0134] The dynamic overlay module (1360) may include a policy controller (1361) and an overlay rendering engine (1362).

[0135] The policy controller (1361) may refuse the use of network-based AI when processing data collected for dynamic overlays if the data includes sensitive information or data that may infringe on privacy. The data may be adjusted to match user settings before overlay rendering and then transmitted to the overlay rendering engine (1362).

[0136] The overlay rendering engine (1362) can implement a suitable animation through the received data, and can implement the animation by searching for or generating a suitable animation through AI, a network, etc. It can also adjust the position of the overlay by analyzing the screen the user is looking at in real time. When the user changes their gaze, it can apply an ad overlay after recognizing an object in the gaze in real time.

[0137] The feedback analysis module (1370) can collect and analyze feedback on advertisements.

[0138] The feedback analysis module (1370) can provide a form for inputting feedback directly on the XR after the advertisement ends, as a feedback collection function. For example, it can provide a quantitative feedback UI such as a simple star rating or a like / dislike button. Alternatively, it can collect advertising feedback (qualitative feedback) through voice commands. Alternatively, it is also possible to collect feedback through user gestures. For example, feedback on the advertisement can be collected through expressions of opinion via gestures such as raising or lowering the thumb.

[0139] The feedback analysis module (1370) is a sentiment analysis function for feedback, and can organize collected feedback data and perform preprocessing as needed. The feedback analysis module (1370) can classify sentiments and derive insights by utilizing a sentiment analysis algorithm. Through sentiment analysis, advertisements with a lot of positive feedback can be maintained, while advertisements with a lot of negative feedback can be delivered to advertisers or advertising agencies (after the user consents to provide information).

[0140] Figure 14 illustrates an example of providing a dynamic overlay with an advertisement to a user wearing an XR device.

[0141] By shifting from the traditional method of watching boring advertisements to one where users directly experience advertised products through XR devices, user ad viewing and purchase rates can be increased. Content creators for XR devices can generate revenue through dynamic overlay-based advertising methods.

[0142] Accordingly, according to the embodiments, an immersive user experience can be provided by reducing the boundary between reality and virtuality through a user-customized overlay, and an overlay that responds in real time according to the user's situation and needs can maximize interaction with the user.

[0143] Furthermore, according to these embodiments, in the field of education and training, educational materials can be visualized through dynamic overlays, allowing users to learn through direct experience, thereby enhancing educational effectiveness. In the field of entertainment, additional dynamic overlays can also be applied to real-world objects to provide users with a more vivid experience.

[0144] In addition, according to the embodiments, work efficiency can be enhanced by providing step-by-step overlay explanations through dynamic overlays tailored to the user's situation when learning a new task or process as a work guide (recognizing tools that can be used as substitutes nearby if the necessary tools are unavailable in the guide). Concentration can be improved by blocking out other distractions and providing necessary information in a timely manner when the user needs to focus.

[0145] In addition, according to the embodiments, advertising effects can be maximized by analyzing user preferences and behavioral patterns and providing customized advertisements through dynamic overlays, and purchase conversion rates can be increased by providing users with experiential advertisements utilizing the advantages of XR.

[0146] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0147] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0148] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or several hardware combined, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0149] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0150] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A method for providing a dynamic overlay for an XR (Extended Reality) device comprising at least one processor, comprising: a step of recognizing a user situation based on AI (Artificial Intelligence) by the at least one processor; and a step of providing a dynamic overlay according to the user situation through an XR (Extended Reality) display by the at least one processor, wherein the recognizing step involves recognizing an object in a surrounding environment associated with a user event through an on-device-based AI or a cloud-based AI for a user event detected by the XR device, and comprises a step of collecting environmental data through a sensor embedded in the XR device to supplement the object recognition, wherein the supplementing step includes: a step of collecting user surrounding environment information including location, lighting, and noise; and a step of performing object recognition in conjunction with the AI-based object recognition, and wherein the providing step comprises: a step of searching for an advertisement on an object within the user's gaze recognized as the user situation through the AI-based object recognition and overlaying the searched advertisement; and a step of collecting user feedback on the overlaid advertisement. A dynamic overlay providing method comprising the step of analyzing a user’s sentiment regarding the overlaid advertisement based on the user feedback, wherein the overlaying step includes: the step of labeling a preferred object among objects recognized as the user situation based on user-related data for a user-customized advertisement; the step of searching for an advertisement related to the preferred object using the labeled data; and the step of selecting one advertisement among the multiple advertisements through a real-time bidding method when multiple advertisements are searched, and wherein user-related data for the user-customized advertisement is updated based on the results of the analysis of the user’s sentiment regarding the overlaid advertisement. Claim 2 delete Claim 3 delete Claim 4 A method for providing a dynamic overlay according to claim 1, wherein the recognizing step includes the step of refusing the use of the cloud-based AI in the case of data among the data collected for the dynamic overlay that is a concern regarding privacy infringement. Claim 5 A dynamic overlay providing method according to claim 1, wherein the recognizing step includes the step of recognizing an object corresponding to the operating condition when the operating condition for the dynamic overlay is selected. Claim 6 A dynamic overlay providing method according to claim 1, wherein the recognizing step includes the step of recognizing whether there is an object set in the operating condition in the user's line of sight when the operating condition for the dynamic overlay is selected. Claim 7 A dynamic overlay providing method according to claim 1, wherein the step of providing includes determining whether to expose the object information based on user-customized data set by the user as association information of the object recognized as the user situation. Claim 8 A dynamic overlay providing method according to claim 1, wherein the step of providing the above includes the step of rendering an animation searched or generated based on association information of an object recognized as the user situation onto the XR display. Claim 9 In claim 8, the step provided further comprises a dynamic overlay providing method including a step of adjusting the overlay position by recognizing the screen viewed by the user in real time. Claim 10 In claim 8, the step of providing the dynamic overlay further comprises the step of caching information exposed through the dynamic overlay on the XR device. Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 It includes at least one processor implemented to execute readable commands on a computer-implemented XR (Extended Reality) device, and said at least one processor recognizes user context based on AI (Artificial Intelligence); The process of providing a dynamic overlay according to the user situation through an XR display is processed, and the at least one processor recognizes objects in the surrounding environment associated with the user event through on-device-based AI or cloud-based AI regarding the user event detected by the XR device, and collects environmental data through sensors embedded in the XR device to supplement object recognition, the at least one processor collects user surrounding environment information including location, lighting, and noise, and performs object recognition in conjunction with AI-based object recognition, the at least one processor searches for advertisements on objects within the user's gaze recognized as the user situation through the AI-based object recognition, overlays the searched advertisements, collects user feedback on the overlaid advertisements, and analyzes the user's sentiment regarding the overlaid advertisements based on the user feedback, the at least one processor labels preferred objects among the objects recognized as the user situation based on user-related data for user-customized advertisements, searches for advertisements related to the preferred objects using the labeled data, and performs real-time bidding when multiple advertisements are searched. An XR device characterized by selecting one of the plurality of advertisements through a method (Real-time Bidding) and updating user-related data for the user-customized advertisement based on the results of user sentiment analysis for the overlaid advertisement.

Images