Method and electronic device for generating XR environment

Abstract

Embodiments herein disclose a method of generating an augmented reality (XR) environment by an electronic device (100). The method comprises: for a user of a plurality of users, generating an infrastructure associated with an XR ground in an XR environment based on a distance between the user and other users of the plurality of users (404); configuring the size and position of the infrastructure based on the distance between the users; identifying a landmark point for the user from the plurality of landmark points after measuring the distance between the users with reference to the infrastructure; and generating an XR environment based on the size of the infrastructure, the location of the infrastructure, and the plurality of landmark points identified for the user.

Description

Technical Field

[0001] Embodiments disclosed herein relate to virtual world systems, such as metaverse systems, extended reality (referred to herein as "XR") systems, mixed reality systems, see-through optical devices, see-through head-mounted devices, optical see-through displays, etc., and more specifically, to methods and systems or electronic devices for providing a multi-user based adaptive XR environment. Background Art

[0002] Most current XR environments are static and adapt to the spatial data of only one user. However, in many use cases, such as a photo booth as a scenario, the XR environment should be adaptive / responsive enough to instinctively cater to multiple users that are completely within the camera's field of view "FOV". By making the XR environment dynamic, it will be more engaging and interactive for multiple users or participants. Existing methods and systems do nothing to generate adaptive XR environments. This degrades the user experience in such scenarios.

[0003] It would be desirable to address the above-mentioned or other disadvantages, or at least provide a useful alternative. Summary of the invention

[0004] The main purpose of the embodiments of this document is to disclose a method and system (or electronic device) for providing an adaptive XR environment based on multiple users (referred to as "multi-users" in this document), wherein multiple users can be identified in a frame and one or more parameters (such as virtual objects, landmark points, etc.) can be used to automatically generate an adaptive XR environment in real time.

[0005] Another object of embodiments herein is to provide a multi-user based adaptive XR environment, wherein the generated XR environment is responsive and interactive based on the proximity of the users and one or more landmark points of the users in the XR environment.

[0006] Another object of embodiments herein is to generate environment valid user data in real time based on a combination of user valid data, i.e., different groups of multiple users detected in the XR environment and the proximity (or distance) between users in the XR environment.

[0007] Another purpose of the embodiments herein is to create a responsive simulation to adjust the base (e.g., XR floor, etc.) and infrastructure based on environment valid user data, and to adjust decorative details within the infrastructure of the XR environment based on the movement of user body landmarks.

[0008] Therefore, embodiments of the present invention provide a method for generating an XR environment by an electronic device. The method may include: after determining the number of users in the physical world, generating an XR ground associated with the XR environment. The method may include: configuring the size of the XR ground based on the determined number of users. The method may include: for a user among a plurality of users, based on the distance between the user and other users among the plurality of users, generating an infrastructure associated with the XR ground in the XR environment. The method may include configuring at least one of the following based on the distance between the user and the other users: the size of the infrastructure and the location of the infrastructure. The method may include: after measuring the distance between the users with reference to the infrastructure, identifying a landmark point for the user from a plurality of landmark points. The landmark point is a user body landmark point and is used to determine the movement of the users relative to each other. In an example, the landmark point may be a hand landmark point, a head landmark point, a leg landmark point, etc. The method may include: generating an XR environment based on the size of the infrastructure, the location of the infrastructure, and at least one of the identified landmark points.

[0009] In an embodiment, the method may include: generating a decoration detail associated with a plurality of landmark points in the XR environment. In addition, the method may include: locating the decoration detail in the adaptive XR environment with reference to the plurality of landmark points.

[0010] In an embodiment, the method may include detecting an event. The event may include at least one of: a new user is associated with the number of users, a user is detached from the number of users, a physical location of a user changes, a user action changes, and a user behavior changes. The method may include: after determining the number of users in the physical world based on the detected event, generating a second XR ground associated with the XR environment. The method may include: configuring the size of the second XR ground based on the determined number of users. The method may include: for a user of the number of users, based on the distance between the user and other users of the number of users, generating a second infrastructure associated with the second XR ground in the XR environment. The method may include configuring at least one of the following based on the distance between the users: the size of the second infrastructure and the location of the second infrastructure. The method may include: after measuring the distance between the users with reference to the second infrastructure, identifying a second landmark point for the user from a plurality of landmark points. The method may include: generating an adaptive XR environment based on the size of the second infrastructure, the location of the second infrastructure, and the identified at least one landmark point.

[0011] In an embodiment, configuring the size of the XR ground based on the determined number of users may include: determining a physical location of each of a plurality of users; grouping the plurality of users into a first location and a second location based on the physical location of each of the plurality of users and a distance between the users; determining different user groups from the plurality of users based on a predefined threshold distance; and, configuring the size of the XR ground based on the determined different user groups.

[0012] In an embodiment, the XR ground associated with the adaptive XR environment may be dynamically generated based on the number and group of users in the physical world.

[0013] In embodiments, an XR floor may control one or more areas in an XR environment.

[0014] In an embodiment, a base structure and landmark points from a plurality of landmark points may be placed within the XR ground.

[0015] In embodiments, the infrastructure may control the visual composition of the XR environment.

[0016] In an embodiment, a landmark point from a plurality of landmark points may be associated with a base structure.

[0017] In embodiments, an adaptive XR environment may correspond to dynamic reaction and adaptation to updates within the XR environment.

[0018] Therefore, the embodiments herein provide a method for generating an adaptive XR environment. The method includes: determining, by an electronic device, the distance between a user among a plurality of users in an XR environment. In addition, the method includes: determining, by an electronic device, the distance between an object in the XR environment and a user among a plurality of users. In addition, the method includes: generating, by an electronic device, user effective data based on the determined distance between the users among the plurality of users and the determined distance between the object in the XR environment and the user among the plurality of users. In addition, the method includes: determining, by an electronic device, environmental effective data, wherein the environmental effective data includes user body landmark points of each user detected in the XR environment. In addition, the method includes: generating, by an electronic device, an adaptive XR environment based on the user effective data and the environmental effective data.

[0019] In an embodiment, the XR environment is generated by generating an XR ground associated with the XR environment after determining the number of users in the physical world, configuring the size of the XR ground based on the determined number of users, generating an infrastructure associated with the XR environment for a user among a plurality of users after determining the distance between the users, configuring at least one of the size of the infrastructure and the location of the infrastructure based on the distance between the users, and dynamically generating the XR environment based on the configuration.

[0020] Therefore, an embodiment of the present invention provides an electronic device comprising a memory and at least one processor coupled to the memory. At least one processor may be configured to generate an XR ground associated with an XR environment after determining the number of users in the physical world. At least one processor may be configured to configure the size of the XR ground based on the determined number of users or user groups. At least one processor may be configured to: for a user among a plurality of users, based on the distance between the user and other users among the plurality of users, generate an infrastructure associated with the XR ground in the XR environment. At least one processor may be configured to: configure at least one of the size of the infrastructure and the position of the infrastructure based on the distance between the user and the other users. At least one processor may be configured to identify a landmark point for the user from a plurality of landmark points after measuring the distance between the users relative to the infrastructure. At least one processor may be configured to generate an XR environment based on the size of the infrastructure, the position of the infrastructure, and at least one landmark point identified.

[0021] Therefore, embodiments of the present invention provide a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of an electronic device, may cause the electronic device to perform operations. The operation may include generating an XR ground associated with the XR environment after determining the number of users in the physical world. The method may include configuring the size of the XR ground based on the determined number of users. The operation may include generating an infrastructure associated with the XR ground in the XR environment for a user from a plurality of users based on the distance between the user and other users from a plurality of users. The operation may include configuring at least one of the size of the infrastructure and the location of the infrastructure based on the distance between the user and the other users. The operation may include identifying a landmark point for the user from a plurality of landmark points after measuring the distance between the users relative to the infrastructure. The operation may include generating an XR environment based on the size of the infrastructure, the location of the infrastructure, and at least one of the identified landmark points.

[0022] Therefore, an embodiment of the present invention provides an electronic device, which includes an adaptive XR environment controller coupled to a processor and a memory. The adaptive XR environment controller is configured to determine the distance between users among a plurality of users in the XR environment. In addition, the adaptive XR environment controller is configured to determine the distance between an object in the XR environment and a user among the plurality of users. In addition, the adaptive XR environment controller is configured to generate user valid data based on the determined distance between the users among the plurality of users and the determined distance between the object in the XR environment and the user among the plurality of users. In addition, the adaptive XR environment controller is configured to determine environment valid data, wherein the environment valid data includes user body landmark points of each user detected in the XR environment. In addition, the adaptive XR environment controller is configured to generate an adaptive XR environment based on the user valid data and the environment valid data.

[0023] These and other aspects of the embodiments herein will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings.It should be understood, however, that while the following description indicates at least one embodiment and numerous specific details thereof, this is given by way of illustration. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The embodiments disclosed herein are shown in the accompanying drawings, in which like reference numerals indicate corresponding parts in the various figures. The embodiments of the present invention will be better understood from the following description with reference to the accompanying drawings, in which:

[0025] Figure 1 Various hardware components of an electronic device according to embodiments disclosed herein are shown;

[0026] Figure 2 and Figure 3 is a flowchart illustrating a method for generating an adaptive XR environment according to an embodiment disclosed herein;

[0027] Figure 4A-4D An example scenario is shown in which a procedurally generated environment profile for generating an adaptive XR environment is illustrated according to an embodiment disclosed herein;

[0028] Figure 5 An example sequence diagram is shown in which an electronic device generates an adaptive XR environment for multiple users according to an embodiment disclosed herein;

[0029] Fig. 6A and Figure 6B An example scenario is shown in which an electronic device generates valid environment data when generating an adaptive XR environment according to an embodiment disclosed herein;

[0030] Figure 7shows a responsive environment flow in generating an adaptive XR environment according to an embodiment disclosed herein;

[0031] Figure 8 An example flow of responsiveness simulation in generating an adaptive XR environment according to an embodiment disclosed herein is shown;

[0032] 9A to 9D is an example scenario in which scale-based proximity simulation is depicted according to embodiments disclosed herein;

[0033] FIG. 10A to FIG. 10D is an example scenario in which multi-user based adaptive XR environment generation is depicted according to an embodiment disclosed herein; and

[0034] Fig.11A and Fig. 11B is another example scenario in which multi-user based adaptive XR environment generation is depicted according to embodiments disclosed herein. DETAILED DESCRIPTION

[0035] The embodiments herein and their various features and advantageous details are explained more fully with reference to the embodiments shown in the accompanying drawings and described in detail in the following description. Descriptions of well-known components and processing techniques are omitted so as not to unnecessarily obscure the embodiments herein. The examples herein are intended only to facilitate understanding of the manner in which the embodiments herein may be implemented and to further enable those skilled in the art to implement the embodiments herein.

[0036] For the purpose of interpreting this specification, the definitions (as defined herein) shall apply and, where appropriate, terms used in the singular shall also include the plural and vice versa. Unless otherwise stated, the terms "including", "having" and "comprising" shall be interpreted as open-ended terms.

[0037] The words / phrases "exemplary," "example," "illustrative," "in an instance," "such as," "etc," "etc," "and others," "for example," and "i.e." are used herein merely to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the subject matter described using the words / phrases "exemplary," "example," "illustrative," "in an instance," "such as," "etc," "etc," "and others," "for example," and "i.e." is not necessarily to be construed as preferred or advantageous over other embodiments.

[0038] The embodiments herein may be described and illustrated with blocks that perform the function or functions. These blocks (which may be referred to herein as managers, units, modules, hardware components, etc.) are physically implemented by analog circuits and / or digital circuits (e.g., logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.), and may optionally be driven by firmware. For example, the circuit may be specifically implemented in one or more semiconductor chips, or on a substrate support such as a printed circuit board. The circuits constituting the blocks may be implemented by dedicated hardware or by a processor (e.g., one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware for performing some functions of the block and a processor for performing other functions of the block. Without departing from the scope of the present disclosure, each block of the embodiment may be physically divided into two or more interactive and discrete blocks. Similarly, the blocks of the embodiment may be physically combined into more complex blocks.

[0039] It should be noted that the elements in the drawings are shown for the purpose of this description and ease of understanding, and may not necessarily be drawn to scale. For example, a flow chart / sequence diagram illustrates a method in the form of steps required to understand aspects of the embodiments disclosed herein. In addition, with respect to the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may only show those specific details relevant to understanding the present embodiment, so as not to obscure the drawings with details that would be obvious to a person of ordinary skill in the art having the benefit of the description herein. In addition, with respect to the system, one or more components / modules comprising the system may have been represented in the drawings by conventional symbols, and the drawings may only show those specific details relevant to understanding the present embodiment, so as not to obscure the drawings with details that would be obvious to a person of ordinary skill in the art having the benefit of the description herein.

[0040] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. The use of words such as first, second, third, etc. to describe components / elements / steps is for the purpose of this description and should not be interpreted as sequential ordering / placement / occurrence unless otherwise explicitly stated.

[0041] Embodiments of the present invention implement a method for generating an adaptive XR environment. The method includes: generating an XR ground associated with the XR environment by an electronic device after determining the number of users in the physical world. In addition, the method includes: configuring the size of the XR ground proportional to the determined number of users by the electronic device. In addition, the method includes: generating an infrastructure associated with the XR ground in the XR environment by the electronic device for a user among multiple users based on the distance between the user and other users among the multiple users. In addition, the method includes: configuring at least one of the size of the infrastructure and the position of the infrastructure by the electronic device based on the distance between the users. In addition, the method includes: identifying a landmark point for the user from a plurality of landmark points by the electronic device after measuring the distance between the users with reference to the infrastructure. In addition, the method includes: generating an adaptive XR environment by the electronic device based on the size of the infrastructure, the position of the infrastructure, and a plurality of landmark points identified for the user.

[0042] Unlike conventional methods and systems, the proposed method can be used to provide a multi-user based adaptive XR environment. In the proposed method, multiple users can be identified in a frame, and one or more parameters (e.g., virtual objects, landmark points, etc.) can be used to automatically generate an adaptive XR environment in real time. The method can be used to provide an adaptive XR environment based on a multi-user situation, wherein the generated environment can be responsive and interactive based on the proximity of the user and one or more landmark points in the XR environment.

[0043] Based on the proposed method, in a virtual meeting or event with multiple participants, an adaptive XR environment or a responsive virtual environment can self-adjust based on the number of users. Adding responsiveness to the background XR environment can create an engaging experience, resulting in a sense of liveliness. The adaptive XR environment or the responsive virtual environment enhances the user experience by creating a more attractive and personalized programmatic environment based on the settings of the electronic device. The settings are completed by the user or the electronic device.

[0044] By making the XR environment dynamic, it becomes more engaging and interactive for multiple users or users and participants. In addition, responsiveness to the actions of multiple users can significantly improve the overall user experience.

[0045] The proposed method can be implemented in various augmented reality applications or XR applications, such as AR lenses, gaming and virtual event applications, virtual tourism and marketing / advertising, gaming parties, and virtual seminar environments.

[0046] Referring now to the drawings, and more particularly to the Figures 1 to 11B , at least one embodiment is shown, wherein like reference numerals consistently represent corresponding features throughout the drawings.

[0047] Figure 1 Various hardware components of an electronic device 100 according to an embodiment disclosed herein are shown. The electronic device 100 may be, for example, but not limited to: a laptop, a desktop computer, a notebook, a device-to-device "D2D" device, a vehicle-to-everything "V2X" device, a smartphone, a foldable phone, a smart TV, a tablet, an immersive device, a camera, a virtual studio technology "VST" device, a head-mounted display "HMD", a server, augmented reality "AR" glasses, a see-through optical device, a see-through head-mounted device, an optical see-through display, and an Internet of Things "IoT" device. The electronic device 100 is also appropriately referred to as an augmented reality "AR" device, a mixed reality "MR" device, a virtual reality "VR" device, an XR device, or a metaverse device. This patent application is described in an XR environment, but is not limited to an XR environment, and is also applicable to an AR environment, an MR environment, a metaverse environment, a VR environment, and the like.

[0048] In an embodiment, the electronic device 100 includes a processor 110, a communicator 120, a memory 130, one or more applications 140a-140n, an adaptive XR environment controller 150, a sensor 160, and a data-driven controller 170. The processor 110 is communicatively coupled with the communicator 120, the memory 130, the adaptive XR environment controller 150, the sensor 160, and the data-driven controller 170. One or more applications 140a-140n are stored or run in the memory 130. One or more applications 140a-140n may be, for example, but not limited to: VR applications, XR applications, MR applications, AR applications, social network applications (such as Facebook®, etc.), game applications, etc. In the following, the label of the application is 140. The sensor 160 may be, for example, but not limited to: a proximity sensor, a distance determination sensor, a depth sensor, etc.

[0049] The adaptive XR environment controller 150 determines the number of users in the physical world. After determining the number of users in the physical world, the adaptive XR environment controller 150 generates an XR floor or base 402, such as Figure 4BAs shown. This is also referred to as the first XR ground. The XR ground 402 is associated with the XR environment. The XR ground controls one or more areas in the XR environment. For example, the electronic device 100 creates the XR ground by using visual plane detection technology. In the visual plane detection technology, the electronic device 100 uses one or more sensors 160 and cameras (not shown) to map the real world environment in the AR environment, or to create a virtual environment in the VR environment. The visual plane detection technology involves the device using its camera to scan horizontal surfaces in the environment. Once the device has detected a horizontal surface, the electronic device 100 can create a virtual ground aligned with the horizontal surface. In other words, the electronic device 100 creates the XR ground by using an environmental scan. In an example, an XR application can use the device's camera and sensor 160 to scan the real world environment to detect a flat surface, such as the ground. This is accomplished through a process called simultaneous localization and mapping ("SLAM") or similar technology. Based on the scan, the electronic device 100 creates the XR ground.

[0050] In addition, the adaptive XR environment controller 150 configures the size of the XR ground in the XR environment in proportion to the determined number of users. In an embodiment, the adaptive XR environment controller 150 determines the physical location of each of the multiple users. In addition, the adaptive XR environment controller 150 groups the multiple users into a first location and a second location based on the physical location of each of the multiple users and the distance between the users. In addition, the adaptive XR environment controller 150 determines different user groups from the multiple users based on a predefined threshold distance. The predefined threshold distance is set by a user of the electronic device 100 or the electronic device 100. Based on the determined different user groups, the adaptive XR environment controller 150 configures the size of the XR ground.

[0051] In addition, the adaptive XR environment controller 150 determines the distance between the user in the XR environment and other users in the plurality of users. Based on the distance between the user and the other users, the adaptive XR environment controller 150 generates an infrastructure 404, such as a virtual table, etc. Figure 4C As shown. For a user among the plurality of users, an infrastructure 404 is associated with an XR ground in the XR environment. The infrastructure 404 controls the visual composition of the XR environment. The infrastructure 404 is placed within the XR ground. Figure 4C Information related to infrastructure 404 is illustrated.

[0052] Based on the distance between the users, the adaptive XR environment controller 150 configures the size of the infrastructure 404 and the location of the infrastructure 404. In addition, the adaptive XR environment controller, after measuring the distance between the users with reference to the infrastructure 404, identifies a landmark point for the user from among the plurality of landmark points.

[0053] Based on the size of the infrastructure 404, the location of the infrastructure 404, and the plurality of landmark points identified for the user, the adaptive XR environment controller 150 generates an adaptive XR environment. The adaptive XR environment corresponds to dynamic reaction and adaptation to updates within the XR environment. In an embodiment, the XR floor associated with the adaptive XR environment is dynamically generated based on the number of active users in the physical world.

[0054] In addition, the adaptive XR environment controller 150 generates decoration details 406a-406c, such as Figure 4D As shown. Decorative details 406a-406c, such as virtual bubbles, flowers, virtual walls, virtual tables, etc., are associated with multiple landmark points in the adaptive XR environment. In the following, the label of the decorative details is 406. The decorative details 406 are generated based on the perception data of the scene, the user's position information, and the environmental factors in the XR environment. In addition, the adaptive XR environment controller 150 locates the decorative details 406 with reference to the multiple landmark points in the adaptive XR environment. The landmark points from the multiple landmark points are placed in the XR ground.

[0055] In addition, the adaptive XR environment controller 150 detects one or more events. The one or more events include at least one of the following: a new user is associated with the number of users, a user is separated from the number of users, a physical position of the user changes, a user action changes, and a user behavior changes. The user action may be, for example, a change in the position of the head, a change in the position of the hand, etc. The user behavior may be, for example, starting to talk to another user, walking towards other users, etc. Based on the one or more detected events, the adaptive XR environment controller 150 generates another XR ground associated with the XR environment after determining the number of users in the physical world, that is, the second XR ground. The second XR ground is different from the first XR ground. In addition, the adaptive XR environment controller 150 configures the size of the second XR ground in the XR environment that is proportional to the determined number of users. In addition, the adaptive XR environment controller 150 determines the distance between the user and other users of the plurality of users. For a user of the plurality of users, the adaptive XR environment controller 150 generates a second infrastructure associated with the second XR ground in the XR environment based on the distance. Based on the distance between the users, the adaptive XR environment controller 150 configures the size of the second infrastructure and the position of the second infrastructure. In addition, the adaptive XR environment controller 150 measures the distance between the users with reference to the second infrastructure. After measuring the distance between the users with reference to the second infrastructure, the adaptive XR environment controller 150 identifies a second landmark point for the user from the plurality of landmark points. Based on the size of the second infrastructure, the position of the second infrastructure, and the plurality of landmark points identified for the user, the adaptive XR environment controller 150 generates an adaptive XR environment.

[0056] In another embodiment, the adaptive XR environment controller 150 determines the distance between users among the multiple users in the XR environment. In addition, the adaptive XR environment controller 150 determines the distance between an object in the XR environment and a user among the multiple users. Based on the determined distances between the users among the multiple users and the determined distances between the objects in the XR environment and the users among the multiple users, the adaptive XR environment controller 150 generates user valid data. FIG. 6 illustrates the generation of user valid data. In addition, the adaptive XR environment controller 150 determines environment valid data. The environment valid data corresponds to user body landmark points of each user detected in the XR environment. The user body landmark points may be, for example, but not limited to: hands, heads, torsos, etc. Based on the user valid data and the environment valid data, the adaptive XR environment controller 150 generates an adaptive XR environment.

[0057] In the example, 9A to 9D A scale-based proximity simulation is shown, and FIG. 10A to FIG. 10D as well as Fig.11A and Fig. 11B Multi-user based adaptive XR environment generation is shown.

[0058] The adaptive XR environment controller 150 is physically implemented by analog or digital circuits (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.) and may optionally be driven by firmware.

[0059] In addition, the processor 110 is configured to execute instructions stored in the memory 130 and perform various processes. The communicator 120 is configured for internal communication between internal hardware components and for communication with external devices through one or more networks. The memory 130 also stores instructions executed by the processor 110. The memory 130 may include a non-volatile storage element. Examples of such non-volatile storage elements may include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or an electrically programmable memory EPROM, or an electrically erasable programmable EEPROM memory. In addition, in some examples, the memory 130 may be considered to be a non-transitory storage medium. The term "non-transitory" may mean that the storage medium is not implemented by a carrier or a propagation signal. However, the term "non-transitory" should not be interpreted as the memory 130 being non-removable. In some examples, a non-transitory storage medium may store data that changes over time (e.g., in a random access memory "RAM" or a cache).

[0060] In addition, at least one of the multiple modules / controllers may be implemented using an AI / machine learning "ML" model using a data-driven controller 170. The data-driven controller 170 may be an ML model-based controller and an AI model-based controller. Functions associated with the AI ​​model may be performed by non-volatile memory, volatile memory, and processor 110. Processor 110 may include one or more processors. At this time, one or more processors may be a general-purpose processor (e.g., a central processing unit "CPU", an application processor "AP", etc.), a graphics processing unit (e.g., a graphics processing unit "GPU", a visual processing unit "VPU"), and / or an AI-specific processor (e.g., a neural processing unit "NPU"). The processor 100 and the adaptive XR environment controller (150) may be collectively referred to as at least one processor.

[0061] One or more processors control the processing of input data according to predefined operating rules or AI models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0062] Here, providing by learning means: making a predefined operating rule or AI model of a desired characteristic by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device itself that executes the AI ​​according to the embodiment, and / or can be implemented by a separate server / system.

[0063] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and the layer operation is performed by the calculation of the previous layer and the operation of multiple weights. Examples of neural networks include, but are not limited to: convolutional neural network "CNN", deep neural network "DNN", recurrent neural network "RNN", restricted Boltzmann machine "RBM", deep belief network "DBN", bidirectional recurrent deep neural network "BRDNN", generative adversarial network "GAN" and deep Q network.

[0064] A learning algorithm is a method of training a predetermined target device (e.g., a robot) using multiple learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms may include, but are not limited to: supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0065] although Figure 1 Various hardware components of the electronic device (100) are shown, but it should be understood that other embodiments are not limited thereto. In other embodiments, the electronic device 100 may include fewer or greater numbers of components. In addition, the labels or names of the components are for illustrative purposes only and do not limit the scope of the present invention. One or more components may be integrated to perform the same or substantially similar functions in the electronic device 100.

[0066] Figure 2 and Figure 3 Flowcharts 200 and 300 , respectively, illustrate methods for generating an adaptive XR environment according to embodiments disclosed herein.

[0067] like Figure 2 As shown, operations S202-S212 are processed by the adaptive XR environment controller 150. In step 202, the method includes generating an XR ground associated with the XR environment after determining the number of users in the physical world. In step 204, the method includes configuring a size of the XR ground proportional to the determined number of users. In step 206, the method includes: for a user among the plurality of users, based on the distance between the user and other users among the plurality of users, generating a base structure 404 associated with the XR ground in the XR environment.

[0068] At step 208, the method includes configuring a size of the infrastructure 404 and a location of the infrastructure 404 based on a distance between the users. At step 210, the method includes identifying a landmark point for the user from a plurality of landmark points after measuring the distance between the users with reference to the infrastructure 404. At step 212, the method includes generating an adaptive XR environment based on the size of the infrastructure 404, the location of the infrastructure 404, and the plurality of landmark points identified for the users.

[0069] like Figure 3 As shown, operations S302-S310 are processed by the adaptive XR environment controller 150. In step 302, the method includes determining a distance between a user of a plurality of users in the XR environment. In step 304, the method includes determining a distance between an object in the XR environment and a user of the plurality of users. In step 306, the method includes generating user validity data based on the determined distances between the users of the plurality of users and the determined distances between the objects in the XR environment and the users of the plurality of users.

[0070] At step 308, the method includes determining environment validity data. The environment validity data includes user body landmarks of each user detected in the XR environment. At step 310, the method includes generating an adaptive XR environment based on the user validity data and the environment validity data.

[0071] The proposed method can be used to provide a multi-user based adaptive XR environment. The generated adaptive XR environment can respond and interact based on the proximity of the user and one or more landmark points in the XR environment. Based on the proposed method, in a virtual meeting or event with multiple participants, the adaptive XR environment or responsive virtual environment can self-adjust based on the number of users. Adding responsiveness to the background XR environment can create an engaging experience, creating a sense of liveliness. The adaptive XR environment or responsive virtual environment enhances the user experience by creating a more attractive and personalized programmatic environment based on the settings of the electronic device 100. By making the XR environment dynamic, it will be more attractive and interactive for multiple users or participants. In addition, the responsiveness to the actions of multiple users can significantly enhance the overall user experience.

[0072] Figure 4A-4D An example scene 400 is shown, in which a programmatically generated environment analysis for generating an adaptive XR environment is illustrated, according to an embodiment disclosed herein. Each XR environment can be mainly analyzed into three parts, such as a base 402, a basic structure 404, and decorative details 406.

[0073] like Figure 4B As shown, base 402 controls the scale and overall composition of the XR environment. The main canvas of base 402 provides an initial platform or backdrop for placing and organizing XR environment elements. Base 402 is a bounding box that controls the area covered by the entire environment (i.e., the scale of the environment). All infrastructure and decorative elements can be placed within base 402 and can be controlled programmatically.

[0074] like Figure 4C As shown, infrastructure 404 controls the composition of base 402 of the XR template. The XR template has been configured or defined by a user or electronic device 100. Infrastructure 404 can be a solid block, a grid, or a plane. In most cases, infrastructure 404 is a static object, but this is not required. Infrastructure 404 controls the visual composition of the environment. Infrastructure 404 controls the main composition at the top of the canvas, which helps with the alignment and spacing of elements, thereby promoting the organizational composition and visual balance of elements in the XR environment.

[0075] like Figure 4D As shown, the decorative details 406 control the visual language or presentation of the XR environment. The decorative details 406 include fill details, which mainly control the visual language and / or the mood of the environment. These elements can be static or dynamic, that is, respond based on triggers. The decorative details 406 provide visual decorations, decorative elements, and decorative embellishments that provide a decorative style and enhance the overall aesthetic of the XR template and establish a unique visual identity in the XR environment.

[0076] Figure 5 An example sequence diagram 500 is shown in which an electronic device 100 generates an adaptive XR environment for multiple users according to an embodiment disclosed herein. In step 502, from the camera field of view "FOV" of one or more cameras, the electronic device 100 can detect users and / or user groups present in the FOV in steps 504 and 506, respectively. In step 508, the electronic device 100 identifies different user groups. Different user groups are determined based on predefined threshold distances between users. The predefined threshold distance can be, for example, 30 centimeters "cm", 100 cm, etc. In step 510, the electronic device 100 identifies valid user data based on the distance between the identified user groups.

[0077] In step 512, the electronic device 100 determines the proximity (or distance) between the user and the user and the virtual environment. In step 514, the electronic device 100 determines the environment validity data after determining the proximity between the user and the user and the virtual environment. In step 516, the electronic device 100 identifies the width from the environment validity data and the valid user data. In step 518, the electronic device 100 obtains the valid data to be used for XR environment generation.

[0078] At step 520, the electronic device 100 provides a responsive environment. A responsive environment refers to an environment that dynamically adjusts and adapts based on parameters or inputs from user data. At step 522, the electronic device 100 provides a responsive simulation. Responsive simulation refers to a simulation that dynamically reacts and adapts to changes within the XR environment. Responsive simulation aims to create a realistic interactive experience by simulating various aspects of the XR environment and allowing them to respond to user interactions. In addition, the electronic device 100 generates a dynamic, interactive, programmatic XR environment that is consistent with user actions and behaviors.

[0079] Fig. 6A and Figure 6B An example scenario 600 is shown in which an electronic device 100 generates valid environment data when generating an adaptive XR environment according to an embodiment disclosed herein. In step 602, the electronic device 100 detects one or more users and creates a bounding box around the detected person. The creation of the bounding box is based on the prior art. In step 604, based on the proximity between the users, the electronic device 100 blocks / aggregates the users and creates a group bounding box. In step 606, based on the proximity between the groups, the electronic device 100 divides the group into multiple blocks. In step 608, based on factors such as but not limited to visual coverage, % of FOV, visual composition guidelines (such as golden ratio, rule of thirds, rule of thumb, etc.), the electronic device 100 derives user valid data. The user valid data is determined based on the group division. In step 610, the electronic device 100 uses the user valid data and the proximity between the valid group and the virtual environment coordinates to derive the environment valid data.

[0080] In the example, the electronic device 100 focuses on extracting effective data about the environment of each user. The electronic device 100 achieves this by detecting and tracking the body landmarks of each user. These body landmarks include key body parts such as hands, head, and torso. By analyzing landmarks, user gestures, and body language, the electronic device 100 can meaningfully create the width of the main canvas and the base 402.

[0081] Figure 7 A responsive environment flow 700 is shown when generating an adaptive XR environment according to an embodiment disclosed herein. Using the environment effective user data, the electronic device 100 can generate a programmatic environment. The generated environment can include one or more instances of decorative details 406 placed on the surface of the base 404. There can be one or more instances of the base 404 placed on the surface of the base 402.

[0082] like Figure 7 As shown, after analyzing the environment validity data and the user group data, a responsive environment is generated through the interaction of the base 402, the infrastructure 404 and the decorative details 406. The base 402 controls the width of the total composition, which determines the overall size, dimension, depth, surface or proportion of the generated XR environment. The infrastructure 404 controls the composition of the XR template. By manipulating the infrastructure 404, the arrangement and organization of various elements in the XR environment can be changed by changing the proportion or surface of the infrastructure 404. The decorative details 406 are responsible for adding decorative assets. The decorative details 406 allow the user to customize the aesthetic elements of the generated XR environment and the theme aspects of the environment by, for example, adjusting the position of the decorative details 406 and rotating the decorative details 406. By adjusting the base 402, the infrastructure 404 and the decorative details 406, the electronic device 100 can create a responsive environment and obtain the flexibility to generate a variety of environments for multiple users.

[0083] Figure 8 An example flow 800 of response simulation in generating an adaptive XR environment according to embodiments disclosed herein is shown. Figure 8 The parameters described are examples, and it will be apparent to one of ordinary skill in the art that more simulations can be created by using a combination of one or more parameters. These include, for example, map ranges, vector directions, triggers, and the like. The map range performs one or more operations. The one or more operations can be, for example, controlling the scale and effective range of elements in the XR environment, controlling the direction of movement of each element in the XR environment, the position offset and rotation angle in the respective vector directions, and controlling the start and end frames of animations in the XR environment and triggering death animations. The vector direction corresponds to the direction between the user and each element.

[0084] 9A to 9D FIG. 9 is an example scene 900 depicting a scale-based proximity simulation according to an embodiment disclosed herein. 9A to 9D In , the cylinder representing the user is simulated only up to a certain distance from the user. Beyond that, each cylinder is scaled differently. This approach uses two conditions, such as the first condition that sets the amount that the simulation should affect, and the second condition that scales a single element or multiple elements based on the user's proximity (e.g., the proximity of the cylinder). Both conditions can be triggered based on the user's position and their proximity to an element (e.g., a cylinder, a virtual object, as shown here). Fig.9A As shown in frame 902 in FIG. 4 , the first user 910a stands close to the infrastructure 404, and the distance between the first user 910a and the user 910b is such that the virtual object (eg, represented in white) is placed in the frame 902. Fig. 9B As shown in frame 904 in FIG. 4 , the first user 910a stands close to the infrastructure 404, and the distance between the user 910a and the user 910b is minimized, and the third user 910c joins the XR environment. Accordingly, the virtual object is placed in frame 904. Fig. 9C As shown in frame 906 in FIG. 1 , the second user 910b and the third user 910c move toward the first user 910a. Based on the movement, an adaptive XR environment is generated in real time. Fig.9D As shown in frame 908 in FIG. 1 , the second user 910 b and the third user 910 c move very close to the first user 910 . Based on this movement, an adaptive XR environment is again generated on the fly, where virtual objects are placed close to the users 910 a-910 c.

[0085] FIG. 10A to FIG. 10D is an example scenario 1000 depicting multi-user based adaptive XR environment generation according to an embodiment disclosed herein.

[0086] and 9A to 9D Similar, such as Fig. 10A As shown in frame 1002 in FIG. 1 , a first user 1010a stands near the infrastructure 404a and 404b. Based on the distance between the first user 1010a and the infrastructure 404a and 404b, an adaptive XR environment is generated on the fly, wherein the decorative detail 406a is placed near the user 1010a.

[0087] like Fig. 10BAs shown in frame 1004, a first user 1010a is standing near infrastructure 404b, and a second user 1010b joins the XR environment, and the second user 1010b is standing near infrastructure 404b. Based on the distance between the first user 1010a and the infrastructures 404a and 404b and the distance between the first user 1010a and the second user 1010b, an adaptive XR environment is again generated on the fly, where decorative details 406a and 406b are placed near users 1010a and 1010b.

[0088] like Fig. 10C As shown in frame 1006, the first user 1010a and the second user 1010b are standing near the infrastructure 404a and 404b, the distance between the first user 1010a and the second user 1010b is very small, and the third user 1010c joins the XR environment. Therefore, an adaptive XR environment is again generated on the fly, where the decorative details 406a and 406b are placed near the users 1010a-1010c based on the current context (i.e., the first user 1010a and the second user 1010b are standing near the infrastructure 404a and 404b, the distance between the first user 1010a and the second user 1010b is very small, and the third user 1010c joins the XR environment).

[0089] like Fig. 10D As shown in frame 1008 in FIG. 1 , all three users 1010a-1010c are standing close to each other next to infrastructure 404a and 404b. An adaptive XR environment is again generated on the fly, where decorative detail 406a and 406b objects are placed close to users 1010a-1010c based on the updated context.

[0090] Fig.11A and Fig. 11B 1 is another example scenario 1100 in which multi-user based adaptive XR environment generation is depicted according to embodiments disclosed herein.

[0091] and FIG. 10A to FIG. 10D Similarly, in games and virtual events, the user of the electronic device 100 can explore the virtual world with other users 1110a and 1110b. Users 1110a and 1110b can interact with the environment and space. The dynamic XR environment can respond to the user's actions, thereby providing a more engaging experience.

[0092] Similarly, by using the proposed method, in virtual tours, dynamic and interactive elements can enhance the virtual tourism experience, where users can interact with dynamic environments and interactive historical information.

[0093] Similarly, by using the proposed method, in marketing and advertising campaigns, the proposed method can help create interactive and engaging campaigns. The dynamic XR environment can respond to user behavior to provide targeted, personalized interactive experiences.

[0094] Various actions, operations, blocks, steps, etc. in flow charts 200-300 can be performed in the order shown, in a different order, or simultaneously. In addition, in some embodiments, some of the actions, operations, blocks, steps, etc. can be omitted, added, modified, or skipped.

[0095] The embodiments disclosed herein can be implemented by running on at least one hardware device and performing network management functions to control at least one software program of each element. These elements can be at least one of the hardware devices, or a combination of a hardware device and a software module.

[0096] The above description of the specific embodiments fully discloses the general nature of the embodiments herein, so that other technicians can easily modify these specific embodiments and / or adjust them for various applications without departing from the overall concept by applying existing knowledge.

Claims

1. A method for generating an extended reality (XR) environment by an electronic device (100), the method comprising: After determining the number of users in the physical world, generating an XR ground associated with the XR environment; Configuring a size of the XR floor based on the determined number of users; generating, for a user among the number of users, at least one infrastructure in the XR environment associated with the XR ground surface based on distances between the user and other users among the number of users (404); Based on the distance between the user and other users, configuring at least one of: the size of the at least one infrastructure (404) and the location of the at least one infrastructure (404); identifying at least one landmark point for the user from a plurality of landmark points after measuring the distance between the users with reference to the at least one base structure (404); as well as The XR environment is generated based on the size of the at least one base structure (404), the location of the at least one base structure (404), and the identified at least one landmark point.

2. The method according to claim 1, further comprising: generating, in the XR environment, at least one decorative detail associated with the plurality of landmark points ( 406 ); as well as The at least one decorative detail is located in the XR environment with reference to the plurality of landmark points ( 406 ).

3. The method according to claim 1, further comprising: detecting at least one event, wherein the at least one event comprises at least one of: a new user being associated with the number of users, a user being detached from the number of users, a change in a user's physical location, a change in a user's action, and a change in user behavior; generating a second XR ground surface associated with the XR environment after determining a number of users in the physical world based on at least one detected event; configuring a size of the second XR ground based on the determined number of users; generating, for a user among the number of users, at least one second infrastructure in the XR environment associated with the second XR ground surface based on distances between the user and other users among the number of users; configuring, based on the distance between the users, at least one of: a size of the at least one second infrastructure and a position of the at least one second infrastructure; identifying at least one second landmark point for the user from among the plurality of landmark points after measuring the distance between the users with reference to the at least one second infrastructure; and The XR environment is generated based on a size of the at least one second base structure, a location of the at least one second base structure, and a plurality of landmark points identified for the user.

4. The method according to claim 1, wherein: Configuring the size of the XR floor based on the determined number of users includes: determining a physical location of each of a plurality of users; grouping the plurality of users into a first location and a second location based on a physical location of each of the plurality of users and a distance between the users; determining different user groups from the plurality of users based on a predefined threshold distance; and Based on the determined different user groups, the size of the XR ground is configured.

5. The method according to claim 1, wherein: An XR ground plane associated with the XR environment is dynamically generated based on the number of active users in the physical world.

6. The method according to claim 1, wherein: The XR ground controls at least one area in the XR environment, wherein the at least one infrastructure (404) and the at least one landmark point from the plurality of landmark points are placed within the XR ground, and wherein the at least one infrastructure (404) controls a visual composition of the XR environment.

7. The method according to claim 1, wherein: The at least one landmark point from the plurality of landmark points is associated with the infrastructure (404), and wherein the XR environment corresponds to dynamically reacting and adapting to updates within the XR environment.

8. An electronic device (100), comprising: Memory (130); as well as At least one processor (110, 150) coupled to the memory, wherein the at least one processor is configured to: After determining the number of users in the physical world, generating an XR ground associated with the XR environment; Configuring a size of the XR floor based on the determined number of users; generating, for a user among the number of users, at least one infrastructure in the XR environment associated with the XR ground surface based on distances between the user and other users among the number of users (404); Based on the distance between the user and other users, configuring at least one of: the size of the at least one infrastructure (404) and the location of the at least one infrastructure (404); identifying at least one landmark point for the user from a plurality of landmark points after measuring the distance between the users with reference to the at least one base structure (404); as well as The XR environment is generated based on a size of the at least one base structure (404), a location of the at least one base structure (404), and the identified at least one landmark point.

9. The electronic device according to claim 8, wherein: The at least one processor (110, 150) is further configured to: generating, in the XR environment, at least one decorative detail associated with the plurality of landmark points (406); and The at least one decorative detail is located in the XR environment with reference to the plurality of landmark points ( 406 ).

10. The electronic device according to claim 8, wherein: The at least one processor (110, 150) is further configured to: detecting at least one event, wherein the at least one event comprises at least one of: a new user being associated with the number of users, a user being detached from the number of users, a change in a user's physical location, a change in a user's action, and a change in user behavior; After determining a number of users in the physical world based on at least one detected event, generating a second XR ground surface associated with the XR environment; configuring a size of the second XR ground based on the determined number of users; generating, for a user among the number of users, at least one second infrastructure in the XR environment associated with the second XR ground surface based on distances between the user and other users among the number of users; configuring, based on the distance between the users, at least one of: a size of the at least one second infrastructure and a position of the at least one second infrastructure; identifying at least one second landmark point for the user from among the plurality of landmark points after measuring the distance between the users with reference to the at least one second infrastructure; and The XR environment is generated based on a size of the at least one second base structure, a location of the at least one second base structure, and a plurality of landmark points identified for the user.

11. The electronic device according to claim 8, wherein: To configure the size of the XR floor based on the determined number of users, the at least one processor (110, 150) is configured to: determining a physical location of each of a plurality of users; grouping the plurality of users into a first location and a second location based on a physical location of each of the plurality of users and a distance between the users; determining different user groups from the plurality of users based on a predefined threshold distance; as well as Based on the determined different user groups, the size of the XR ground is configured.

12. The electronic device according to claim 8, wherein: An XR ground plane associated with the XR environment is dynamically generated based on the number of active users in the physical world.

13. The electronic device according to claim 8, wherein: The XR ground controls at least one area in the XR environment, wherein the at least one infrastructure (404) and the at least one landmark point from the plurality of landmark points are placed within the XR ground, and wherein the at least one infrastructure (404) controls a visual composition of the XR environment.

14. The electronic device according to claim 8, wherein: The at least one landmark point from the plurality of landmark points is associated with the infrastructure (404), and wherein the XR environment corresponds to dynamically reacting and adapting to updates within the XR environment.

15. A non-transitory computer-readable storage medium storing instructions which, when executed by at least one processor (110, 150) of an electronic device (100), cause the electronic device (100) to perform the operations of the method according to one of claims 1 to 7.

Images