Method for operating mixed reality telepresence system and mixed reality telepresence system performing same

Abstract

A method for operating a mixed reality telepresence system comprises the steps of: extracting, from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients, a first subspace in which a target object is set and a plurality of second subspaces in which the target object is set; on the basis of the first subspace, the plurality of second subspaces, and an optimization function, generating a plurality of alignment spaces in which the plurality of second subspaces overlap the first subspace; determining a plurality of first positions on the plurality of alignment spaces; and on the basis of the plurality of first positions, setting a plurality of first interactable areas in the first virtual space.

Description

A method of operating a mixed reality telepresence system and a mixed reality telepresence system performing the same

[0001] The present invention relates to a mixed reality system, and more particularly, to an operating method of a mixed reality telepresence system and a mixed reality telepresence system performing the operating method.

[0002] Recently, mixed reality (MR) telepresence (or remote collaboration) systems, which connect people through augmented reality (AR) and virtual reality (VR) technologies, have been attracting continued attention. In MR telepresence, users can achieve a high level of copresence with other remote users as avatars placed in a mutual workspace. This space is typically generated based on the host user's AR space, and remote users are virtually transported while wearing a VR head-mounted display (HMD). In fact, most research on MR telepresence over the past decade has focused on the technical asymmetrical system involving the AR host and VR client.

[0003] One of the key challenges of asymmetric MR telepresence is understanding each user's heterogeneous space and generating an optimized interactivity zone where the AR host and VR client can effectively coexist. Past methods have attempted to find the maximum intersection of relevant subspaces, but as the number of subspaces increases, their size decreases, placing further constraints on the user's movements.

[0004] One object of the present invention is to provide a method of operating a mixed reality telepresence system capable of providing excellent spatial consistency according to a collaborative context.

[0005] One object of the present invention is to provide a mixed reality telepresence system that performs the operating method of the mixed reality telepresence system.

[0006] In order to achieve the above object, the operating method of the mixed reality telepresence system according to embodiments of the present invention includes the steps of extracting a first subspace in which a target object is set and a plurality of second subspaces in which the target object is set from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients, generating a plurality of alignment spaces overlapping the plurality of second subspaces with the first subspace based on the first subspace, the plurality of second subspaces, and an optimization function, determining a plurality of first positions on the plurality of alignment spaces, and setting a plurality of first interactable areas in the first virtual space based on the plurality of first positions.

[0007] In order to achieve the above object, a mixed reality telepresence system according to embodiments of the present invention includes a subspace extraction unit that extracts a first subspace in which a target object is set and a plurality of second subspaces in which the target object is set from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients, a subspace alignment unit that receives the first subspace and the plurality of second subspaces from the subspace extraction unit and, based on the first subspace, the plurality of second subspaces and an optimization function, generates a plurality of alignment spaces that overlap the plurality of second subspaces with the first subspace, and a subspace setting unit that receives the plurality of alignment spaces from the subspace alignment unit, determines a plurality of first positions on the plurality of alignment spaces, and sets a plurality of first interactable areas in the first virtual space based on the plurality of first positions.

[0008] In order to achieve the above object, an operating method of a mixed reality telepresence system according to embodiments of the present invention comprises the steps of: obtaining initial subspaces from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients; calculating a target object from the initial subspaces; obtaining a first subspace corresponding to the first virtual space and a plurality of second subspaces corresponding to the second virtual space based on the initial subspaces and the target object; determining an optimization function based on a collaboration context; setting a second position and a second angle for each of the plurality of second subspaces based on the first subspace, the plurality of second subspaces, and the optimization function; obtaining a plurality of alignment spaces overlapping the plurality of second subspaces with the first subspace based on the second position and the second angle; determining a plurality of first positions on the plurality of alignment spaces; setting different first to fourth markers in the form of lines at the plurality of first positions; and moving the first to fourth markers until a first condition is satisfied. It includes a step of moving in the first to fourth directions, and a step of generating polygons formed by extending the first to fourth markers as the plurality of first interactable areas.

[0009] In the operating method of the mixed reality telepresence system according to the embodiments of the present invention described above, interactable areas can be derived using data that overlaps (or aligns) the subspace of each of the plurality of clients with the subspace of the host. When the interactable areas are assigned to each of the plurality of clients, excellent spatial consistency according to the collaborative context can be provided regardless of an increase in the number of clients or heterogeneity with the host space.

[0010] FIG. 1 is a flowchart illustrating an operation method of a mixed reality telepresence system according to embodiments of the present invention.

[0011] FIG. 2 is a block diagram illustrating a mixed reality telepresence system according to embodiments of the present invention.

[0012] FIG. 3 is a flowchart illustrating an example of extraction of a first subspace and a plurality of second subspaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0013] FIG. 4 is a diagram for explaining the acquisition of initial subspaces of a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0014] FIG. 5 is a flowchart illustrating an example of target object extraction from initial subspaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0015] FIGS. 6, 7, 8 and 9 are drawings for explaining the extraction of target objects from initial subspaces in the operating method of the mixed reality telepresence system according to embodiments of the present invention.

[0016] FIG. 10 is a flowchart illustrating an example of generating multiple alignment spaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0017] FIGS. 11 and 12 are drawings for explaining the creation of multiple alignment spaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0018] FIG. 13 is a flowchart illustrating an example of determining a plurality of first positions in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0019] FIGS. 14, 15, 16 and 17 are drawings for explaining the determination of a plurality of first positions of a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0020] FIG. 18 is a flowchart illustrating an example of setting a plurality of first interactive areas in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0021] FIGS. 19, 20, 21, 22, 23 and 24 are drawings for explaining the setting of a plurality of first interactable areas in the operating method of the mixed reality telepresence system according to embodiments of the present invention.

[0022] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. Identical components in the drawings are designated by the same reference numerals, and redundant descriptions of identical components are omitted.

[0023] FIG. 1 is a flowchart illustrating an operation method of a mixed reality telepresence system according to embodiments of the present invention.

[0024] Referring to FIG. 1, the operating method of the mixed reality telepresence system according to embodiments of the present invention can be applied to a telepresence system in which a host and a plurality of clients collaborate remotely using an augmented reality (AR) device and / or a virtual reality (VR) device. For example, the mixed reality telepresence system according to embodiments of the present invention can refer to a system in which a plurality of clients are virtually transported while wearing VR devices based on the AR space of the host.

[0025] A first subspace and a plurality of second subspaces are extracted from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients (step S100). For example, the first virtual space and the plurality of second virtual spaces may represent data collected through an AR / VR device. For example, the first subspace may represent a portion of the first virtual space data, and the plurality of second subspaces may represent a portion of the data of the plurality of second virtual spaces.

[0026] For example, the plurality of clients may be implemented as three-dimensional avatars in the first virtual space of the host. For example, the host may visually display the avatars of the plurality of clients in the first virtual space. For example, the plurality of clients may visually display a portion of the first virtual space implemented in the plurality of second virtual spaces.

[0027] For example, the host may be one or more. If there are multiple hosts, this may indicate that there are multiple hosts sharing the first virtual space. In other words, the hosts may be multiple, but the first virtual space may be unique.

[0028] For example, a target object may be set in the first subspace and the plurality of second subspaces. For example, the target object may represent an interactable object. For example, the interactable object may represent an object that the host or the plurality of clients can directly sense through their five senses. For example, the target object may represent a table, a wall, a floor, etc. Specific examples of the first subspace and the plurality of second subspaces will be described below with reference to FIGS. 3 to 9.

[0029] Based on a first subspace, a plurality of second subspaces, and an optimization function, a plurality of alignment spaces are generated (step S200). For example, the optimization function may be used when aligning or overlapping the plurality of second subspaces with the first subspace. For example, the optimization function may be determined based on a collaboration context. For example, when a remote conference is conducted at a table, the optimization function may be set so that the tables existing in the plurality of second subspaces and the tables existing in the first subspace overlap as much as possible, i.e., so that they have the largest overlapping area possible. For example, the plurality of alignment spaces may represent data that overlaps the data of the plurality of second subspaces with the data of the first subspace. A specific process of generating the plurality of alignment spaces will be described below with reference to FIGS. 10 to 12.

[0030] A plurality of first positions on a plurality of alignment spaces are determined (step S300). For example, the plurality of first positions may represent initial positions of the plurality of clients. For example, the initial positions of the plurality of clients may represent positions of the plurality of clients at the initial operation of the mixed reality telepresence system. For example, the plurality of first positions may include not only the positions of the plurality of clients but also the position of the host. For example, the plurality of first positions may be used to set first interactable areas. A specific process of determining the plurality of first positions will be described below with reference to FIGS. 13 to 17.

[0031] Based on a plurality of first locations, a plurality of first interactable areas are set in a first virtual space (step S400). For example, the plurality of first interactable areas may include different interactable areas of each of the plurality of clients. For example, each of the plurality of clients may be located on the first virtual space of the host within the range of the different interactable areas. The specific process of setting the plurality of first interactable areas will be described below with reference to FIGS. 18 to 24.

[0032] In the operating method of the mixed reality telepresence system according to the embodiments of the present invention as described above, a plurality of first interactable areas can be derived using data that overlaps (or aligns) a plurality of second subspaces of each of a plurality of clients with the first subspace of the host. When the plurality of first interactable areas are allocated to each of the plurality of clients, excellent spatial consistency according to the collaboration context can be provided regardless of an increase in the number of clients or heterogeneity with the host space.

[0033] FIG. 2 is a block diagram illustrating a mixed reality telepresence system according to embodiments of the present invention.

[0034] Referring to FIG. 2, the mixed reality telepresence system (1000) includes a partial space extraction unit (100), a partial space alignment unit (200), and a partial space setting unit (300). As described above, the mixed reality telepresence system (1000) may represent a system in which multiple clients are virtually transported while wearing VR devices based on the AR space of the host.

[0035] The subspace extraction unit (100) extracts a first subspace (SS1) in which a target object is set and a plurality of second subspaces (SS2) in which the target object is set from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients. For example, the subspace extraction unit (100) may receive data of the first virtual space and the plurality of second virtual spaces from an AR / VR device. For example, the subspace extraction unit (100) may obtain initial subspaces from the first virtual space and the plurality of second virtual spaces. For example, the subspace extraction unit (100) may calculate the target object from the initial subspaces. For example, the subspace extraction unit (100) may obtain the first subspace (SS1) and a plurality of second subspaces (SS2) by setting the target object in the initial subspaces. For example, the subspace extraction unit (100) may perform step S100 of FIG. 1. For example, the subspace extraction unit (100) can perform steps S110, S120, and S130 of FIG. 3, which will be described later. For example, the subspace extraction unit (100) can perform steps S121 and S122 of FIG. 5, which will be described later.

[0036] The subspace alignment unit (200) receives a first subspace (SS1) and a plurality of second subspaces (SS2) from the subspace extraction unit (100). In addition, the subspace alignment unit (200) generates a plurality of alignment spaces (AS) in which the plurality of second subspaces (SS2) overlap the first subspace (SS1) based on the first subspace (SS1), the plurality of second subspaces (SS2), and the optimization function. For example, the subspace alignment unit (200) may determine the optimization function based on a collaboration context. For example, the subspace alignment unit (200) may set a second position and a second angle for each of the plurality of second subspaces (SS2) based on the first subspace (SS1), the plurality of second subspaces (SS2), and the optimization function, and may obtain a plurality of alignment spaces based on the second position and the second angle. For example, the subspace alignment unit (200) can perform step S200 of FIG. 1. For example, the subspace alignment unit (200) can perform steps S210, S220, and S230 of FIG. 10, which will be described later.

[0037] The partial space setting unit (300) receives a plurality of alignment spaces (AS) from the partial space alignment unit (200). In addition, the partial space setting unit (300) determines a plurality of first positions on the plurality of alignment spaces (AS), and sets a plurality of first interactable areas in the first virtual space based on the plurality of first positions. For example, the partial space setting unit (300) can output the first virtual space in which the plurality of first interactable areas are set. For example, a mixed reality telepresence system can be operated based on the first virtual space in which the plurality of first interactable areas are set. For example, the partial space setting unit (300) can perform steps S300 and S400 of FIG. 1. For example, the partial space setting unit (300) can perform steps S310, S320, and S330 of FIG. 13, which will be described later. For example, the partial space setting unit (300) can perform steps S410, S420, and S430 of FIG. 18, which will be described later.

[0038] FIG. 3 is a flowchart illustrating an example of extraction of a first subspace and a plurality of second subspaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0039] Referring to FIG. 3, steps S110, S120, and S130 may represent an example of step S100 of FIG. 1.

[0040] Initial subspaces can be acquired from a first virtual space and a plurality of second virtual spaces (step S110). For example, the initial subspaces may include information about a floor, first objects that can be interacted with, second objects on which the host and a plurality of clients can sit, and third objects representing obstacles. For example, the initial subspaces may represent data expressing the floor, the first objects, the second objects, and the third objects in the form of polygons.

[0041] A target object can be derived from the initial subspaces (step S120). For example, the target object can be selected from among the first objects. For example, a scene graph can be used when selecting the target object from among the first objects. The specific process of generating the target object will be described below with reference to FIGS. 5 to 9.

[0042] A first subspace and a plurality of second subspaces can be acquired based on the initial subspaces and the target object (step S130). For example, the first subspace and the plurality of second subspaces can represent data that sets the target object in the initial subspaces.

[0043] FIG. 4 is a diagram for explaining the acquisition of initial subspaces of a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0044] Referring to FIG. 4, initial subspaces (PSS) can be obtained from a first virtual space (VS1) and a plurality of second virtual spaces (VS2). For example, the initial subspaces (PSS) can include a first initial subspace (PSS1) corresponding to the first virtual space (VS1) and a plurality of second initial subspaces (PSS2) corresponding to a plurality of second virtual spaces (VS2). For example, the initial subspaces (PSS) can include information about a floor (FL), first objects (O1), second objects (O2), and third objects (O3). For example, the initial subspaces (PSS) can represent data in which the floor (FL), first objects (O1), second objects (O2), and third objects (O3) are each expressed in the form of a rectangle.

[0045] FIG. 5 is a flowchart illustrating an example of target object extraction from initial subspaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0046] Referring to FIG. 5, steps S121 and S122 may represent an example of step S120 of FIG. 3.

[0047] A target object can be selected from among first objects on a first initial subspace (step S121). For example, in the first initial subspace (PSS1) of FIG. 4, there is only one first object, but there may be multiple first objects on the first initial subspace. For example, if there is only one first object on the first initial subspace, the first object can be automatically selected as the target object. For example, if there are multiple first objects on the first initial subspace, the target object in the first initial subspace can be selected by a user (e.g., a host).

[0048] Using a target object and a scene graph on a first initial subspace, a target object can be determined from among first objects on a plurality of second initial subspaces (step S122). For example, the scene graph can represent a data structure including a set of nodes in a graph or tree structure. For example, the scene graph can be used to infer not only objects but also relationships between objects for an input image. For example, the scene graph can classify objects by instance from the input image and express relationships using a connection graph. A specific process for selecting a target object using the scene graph will be described below with reference to FIG. 7.

[0049] FIGS. 4, 6, 7, 8 and 9 are drawings for explaining the extraction of target objects from initial subspaces in the operating method of the mixed reality telepresence system according to embodiments of the present invention.

[0050] Referring to FIGS. 6, 7, 8 and 9, a process of obtaining a first subspace (SS1) and a plurality of second subspaces (SS2) from a first initial subspace (PSS1) and a plurality of second initial subspaces (PSS2) can be illustrated.

[0051] As illustrated in FIG. 6, the target object (TO) may be selected as one of the first objects (O1) in the first initial subspace (PSS1). For example, since there is only one first object (O1) in the first initial subspace (PSS1), the target object (TO) may be selected as one first object (O1).

[0052] As illustrated in FIG. 7, using the scene graph, one target object (TO) among the first objects (O1) on a plurality of second initial subspaces (PSS2) can be determined. Hereinafter, a second-first initial subspace (PSS2_1) obtained from a virtual space of a first client among a plurality of clients will be described as an example. For example, the second-first initial subspace (PSS2_1) may include two first objects (O1_1, O1_2). For example, a first scene graph (SG1) may be generated for the target object (TO_H) of the first initial subspace (PSS1). For example, the first scene graph (SG1) may generate nodes (SG_TO, SG_O2) for the target object (TO_H) and surrounding objects, and connect the nodes (SG_TO, SG_O2). For example, the first scene graph (SG1) may include relationship information between the target object (TO_H) and the second objects (O2). Similarly, scene graphs may also be generated for the first objects (O1_1, O1_2) in the second-first initial subspace (PSS2_1). Thereafter, the scene graphs of the first scene graph (SG1) and the first objects (O1_1, O1_2) may be compared to select the first object that most closely matches the target object (TO_H) by considering the size of the first objects (O1_1, O1_2) as well as the relationship with surrounding objects.

[0053] As illustrated in Fig. 8, through comparison of the new graphs, the first object (O1_1) among the first objects (O1_1, O1_2) of Fig. 7 can be selected as the target object. For example, the remaining first objects that were not selected as the target objects can be treated as third objects representing obstacles.

[0054] As illustrated in FIG. 9, a first subspace (SS1) in which one target object (TO) is set in the first initial subspace (PSS1) of FIG. 4 can be obtained, and a plurality of second subspaces (SS2) in which one target object (TO) is set in the plurality of second initial subspaces (PSS2) of FIG. 4 can be obtained.

[0055] FIG. 10 is a flowchart illustrating an example of generating multiple alignment spaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0056] Referring to FIG. 10, steps S210, S220 and S230 may represent an example of step S200 of FIG. 1.

[0057] Based on the collaborative context, an optimization function can be determined (step S210). For example, if the target object is a table, the optimization function can be determined so that the tables existing in multiple second subspaces and the tables existing in the first subspace overlap as much as possible, i.e., so that they have the widest possible overlapping area.

[0058] Based on a first subspace, a plurality of second subspaces, and an optimization function, a second position and a second angle may be set for each of the plurality of second subspaces (step S220). For example, the second position and the second angle may be calculated by the optimization function so that a target object of the first subspace and a target object of the plurality of second subspaces overlap as much as possible. For example, the second position may represent a relative position of the plurality of second subspaces in relation to the first subspace. For example, the second angle may represent a relative rotation degree of the plurality of second subspaces in relation to the first subspace.

[0059] Based on the second position and the second angle, a plurality of alignment spaces can be acquired (step S230). For example, the plurality of alignment spaces can include first partial space data and a plurality of second partial space data that are moved by the second position and rotated by the second angle.

[0060] FIGS. 11 and 12 are drawings for explaining the creation of multiple alignment spaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0061] Referring to FIGS. 11 and 12, a process of obtaining a plurality of aligned spaces (AS1, AS2, AS3, AS4) overlapping a plurality of second subspaces (SS2_1, SS2_2, SS2_3, SS2_4) in a first subspace (SS1) based on a second position (L2_1, L2_2, L2_3, L2_4) and a second angle (A2_1, A2_2, A2_3, A2_4) can be illustrated.

[0062] As illustrated in Fig. 11, the second positions (L2_1, L2_2, L2_3, L2_4) and the second angles (A2_1, A2_2, A2_3, A2_4) can be calculated based on the optimization function. For example, the second angle (A2_1) of the 2-1 subspace (SS2_1) can be calculated as 0°, the second angle (A2_2) of the 2-2 subspace (SS2_2) can be calculated as 90°, the second angle (A2_3) of the 2-3 subspace (SS2_3) can be calculated as 0°, and the second angle (A2_4) of the 2-4 subspace (SS2_4) can be calculated as 90°.

[0063] As illustrated in FIG. 12, based on the second positions (L2_1, L2_2, L2_3, L2_4) and the second angles (A2_1, A2_2, A2_3, A2_4) of FIG. 11, a plurality of alignment spaces (AS1, AS2, AS3, AS4) can be obtained. For example, when the plurality of clients include first to fourth clients, the first alignment space (AS1) can represent data overlapping (or aligned) the subspaces of the host and the first client. Similarly, the second alignment space (AS2) can represent data overlapping (or aligned) the subspaces of the host and the second client, the third alignment space (AS3) can represent data overlapping (or aligned) the subspaces of the host and the third client, and the fourth alignment space (AS4) can represent data overlapping (or aligned) the subspaces of the host and the fourth client.

[0064] FIG. 13 is a flowchart illustrating an example of determining a plurality of first positions in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0065] Referring to FIG. 13, steps S310, S320 and S330 may represent an example of step S300 of FIG. 1.

[0066] A plurality of first sample positions of a plurality of clients may be set to be spaced apart from a first overlap area by a first distance (step S310). For example, the first overlap area may represent an area where a target object of a first subspace and target objects of a plurality of second subspaces overlap. For example, the first overlap area may be included in a plurality of alignment spaces. For example, the first distance may be set such that the host or the plurality of clients can move around the target object without being restricted by the target object. For example, when the target object is a table, the first distance may be set to an extent (e.g., 20 cm) that allows a person to move around the table without being restricted by the table. For example, the plurality of first sample positions may be expressed in the form of a plurality of dots around the first overlap area.

[0067] A plurality of second sample locations may be selected from a plurality of first sample locations based on obstacle information, body information of a plurality of clients, etc. (step S320). For example, the obstacle information may indicate information related to third objects on the plurality of aligned spaces. For example, the body information of the plurality of clients may include shoulder width, stride length, etc. of the plurality of clients. For example, the plurality of second sample locations may be obtained by removing some of the plurality of first sample locations. For example, locations overlapping with the location of an obstacle and locations spaced apart from the obstacle by a distance equal to or less than the shoulder width of the client may be excluded from the plurality of first sample locations.

[0068] Based on the second sample locations and the proxy mix between the plurality of clients, a plurality of first locations can be acquired (step S330). For example, the proxy mix can represent the distances between the plurality of clients. For example, if the plurality of clients include first, second, and third clients, the plurality of first locations can be acquired by considering the distances between the first client and the second client, the distances between the second client and the third client, and the distances between the third client and the first client.

[0069] FIGS. 14, 15, 16 and 17 are drawings for explaining the determination of a plurality of first positions of a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0070] Referring to FIGS. 14, 15, 16, and 17, a process for determining a plurality of first positions (PL1) on a plurality of alignment spaces (AS1, AS2, AS3, AS4) can be illustrated. Hereinafter, for convenience of explanation, it is assumed that the plurality of clients include first to fourth clients, and the plurality of alignment spaces (AS1, AS2, AS3, AS4) include a first alignment space (AS1), a second alignment space (AS2), a third alignment space (AS3), and a fourth alignment space (AS4).

[0071] As illustrated in FIG. 14, a plurality of first sample locations (PSL1) of a plurality of clients can be set to be spaced apart from a first overlap area (OA1) by a first distance (DS1). For example, the first overlap area (OA1) can represent an area where a target object of a first subspace and target objects of a plurality of second subspaces overlap.

[0072] As illustrated in FIG. 15, a plurality of second sample locations (PSL2) can be obtained by excluding locations that overlap with the location of an obstacle among a plurality of first sample locations (PSL1) and locations that are spaced apart from the obstacle by a distance less than the shoulder width of the client.

[0073] As illustrated in Fig. 16, a plurality of first positions (PL1) may be set considering the distances between the first to fourth clients. For example, the plurality of first positions (PL1) may include a first position (PL1_1) of the first client, a first position (PL1_2) of the second client, a first position (PL1_3) of the third client, and a first position (PL1_4) of the fourth client. In this case, for the convenience of distance comparison between the first to fourth clients, the plurality of alignment spaces (AS1, AS2, AS3, AS4) may be integrated into one integrated alignment space (TAS).

[0074] As illustrated in FIG. 17, in addition to the plurality of first positions (PL1), a plurality of host positions (PHL1) may also be set in the integrated alignment space (TAS) or the plurality of alignment spaces (AS1, AS2, AS3, AS4). As described above with reference to FIG. 1, the number of hosts may be plural, and the plurality of host positions (PHL1) may include positions (HL1, HL2, HL3) of the first to third hosts. However, the plurality of host positions (PHL1) may be set at any time after the plurality of first positions (PL1) are set.

[0075] FIG. 18 is a flowchart illustrating an example of setting a plurality of first interactive areas in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

[0076] Referring to FIG. 18, steps S410, S420, and S430 may represent an example of step S400 of FIG. 1.

[0077] Different first to fourth markers in the form of line segments may be set at a plurality of first locations (step S410). For example, the first to fourth markers may represent an example of a method for generating first interactable areas. For example, the first to fourth markers may be arranged to surround the plurality of first locations. For example, the first to fourth markers may be implemented in the form of line segments or rectangular bars.

[0078] The first to fourth markers may be moved in the first to fourth directions, respectively, until a first condition is satisfied (step S420). For example, the first condition may include that the first to fourth markers do not come into contact with third objects or a target object. For example, the first condition may include that the first to fourth markers do not come into contact with floor boundaries of a plurality of alignment spaces. For example, the first to fourth markers may be arranged in a square shape at the beginning of the operation of step S420, and the first to fourth markers may be moved in the up, down, left, and right directions, respectively, until the first condition is satisfied.

[0079] Polygons formed by extending the first to fourth markers can be calculated as a plurality of first interactable areas (step S430). For example, the positions of the first to fourth markers at the beginning of the operation in step S420 may be different from the positions after movement. For example, when extending four of the first to fourth markers, a square can be formed. Therefore, when using the first to fourth markers, the plurality of first interactable areas can be calculated in the shape of squares.

[0080] FIGS. 19, 20, 21, 22, 23 and 24 are drawings for explaining the setting of a plurality of first interactable areas in the operating method of the mixed reality telepresence system according to embodiments of the present invention.

[0081] Referring to FIGS. 19, 20, 21, 22, 23 and 24, a process of acquiring a plurality of first interactable areas (ISS1_1, ISS1_2, ISS1_3, ISS1_4) using first to fourth markers (M1, M2, M3, M4) can be illustrated.

[0082] Hereinafter, the description will be made on the assumption that the plurality of clients include first to fourth clients, the plurality of alignment spaces include first to fourth alignment spaces, and the plurality of first interactable areas include a first-first interactable area (ISS1_1), a first-second interactable area (ISS1_2), a first-third interactable area (ISS1_3), and a first-fourth interactable area (ISS1_4). In addition, although the case of the first alignment space (AS1) is described below as a representative example, the interactable areas can be produced in the second to fourth alignment spaces as well, as will be described below with reference to FIGS. 19 to 21.

[0083] As illustrated in FIG. 19, the first alignment space (AS1) may represent a space that overlaps the first subspace of the host with the second subspace of the first client. For example, the first to fourth markers (M1, M2, M3, and M4) may be arranged to surround the first location (PL1_1) of the first client. For example, the first marker (M1) may move in the first direction (D1), the second marker (M2) may move in the second direction (D2), the third marker (M3) may move in the third direction (D3), and the fourth marker (M4) may move in the fourth direction (D4).

[0084] As illustrated in FIG. 20, the first to fourth markers (M1, M2, M3, M4) can move in the first to fourth directions (D1, D2, D3, D4) respectively until the first condition is satisfied. For example, the first marker (M1) can move until it contacts the target object. For example, the second to fourth markers (M2, M3, M4) can move until it contacts the floor boundary of the first alignment space (AS1).

[0085] As illustrated in Fig. 21, a polygon formed by extending the first to fourth markers (M1, M2, M3, M4) can be acquired as the first-first interaction possible area (ISS1_1).

[0086] As illustrated in FIG. 22, the 1st-2nd interactable area (ISS1_2), the 1st-3rd interactable area (ISS1_3), and the 1st-4th interactable area (ISS1_4) can be obtained as described above with reference to FIGS. 19 to 21.

[0087] As illustrated in FIG. 23, according to an embodiment, each of the plurality of first interactable areas (ISS1_1, ISS1_2, ISS1_3, ISS1_4) may further include the first overlapping areas described above with reference to FIG. 14.

[0088] As illustrated in FIG. 24, a plurality of first interactable areas (ISS1_1, ISS1_2, ISS1_3, ISS1_4) may be finally set in the first virtual space (VS1) of the host. For example, the first virtual space (VS1) in which a plurality of first interactable areas (ISS1_1, ISS1_2, ISS1_3, ISS1_4) are set may be used for the operation of the mixed reality telepresence system. For example, the first virtual space may include all data related to the plurality of first interactable areas (ISS1_1, ISS1_2, ISS1_3, ISS1_4). For example, data related to the first-first interactable area (ISS1_1) of the first client may be provided only to the first client among the plurality of clients. Similarly, data related to the 1st-2nd, 1st-3rd and 1st-4th interactable areas (ISS1_2, ISS1_3, ISS1_4) may be provided only to the 2nd, 3rd and 4th clients, respectively.

[0089] Additionally, when a client is added, steps S100 to S400 of FIG. 1 may be performed again so that a plurality of new first interactive areas can be set in the first virtual space (VS1).

[0090] Embodiments of the present invention can be usefully utilized in any electronic device and system including a storage device. For example, embodiments of the present invention can be more usefully applied to electronic systems such as PCs (Personal Computers), server computers, data centers, workstations, laptops, cellular phones, smart phones, MP3 players, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), digital TVs, digital cameras, portable game consoles, navigation devices, wearable devices, IoT (Internet of Things) devices, IoE (Internet of Everything) devices, e-books, VR (Virtual Reality) devices, AR (Augmented Reality) devices, drones, automotives, etc. Although the present invention has been described above with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

Claims

1. A step of extracting a first subspace in which a target object is set and a plurality of second subspaces in which the target object is set from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients; A step of generating a plurality of alignment spaces overlapping the plurality of second subspaces with the first subspace based on the first subspace, the plurality of second subspaces, and the optimization function; a step of determining a plurality of first positions on the plurality of alignment spaces; and A method of operating a mixed reality telepresence system, comprising the step of setting a plurality of first interactable areas in the first virtual space based on the plurality of first locations.

2. In the first paragraph, the step of extracting the first subspace in which the target object is set and the plurality of second subspaces in which the target object is set is as follows: A step of obtaining initial subspaces from the first virtual space and the plurality of second virtual spaces; a step of generating the target object from the initial subspaces; and A method of operating a mixed reality telepresence system, characterized by comprising a step of obtaining the first subspace and the plurality of second subspaces based on the initial subspaces and the target object.

3. In paragraph 2, A method of operating a mixed reality telepresence system, wherein the initial subspaces include information about a floor, first objects that can be interacted with, second objects on which the host and the plurality of clients can sit, and third objects representing obstacles.

4. In paragraph 3, The above initial subspaces include a first initial subspace corresponding to the first virtual space and a plurality of second initial subspaces corresponding to the plurality of second virtual spaces, The step of generating the target object from the above initial subspaces is: A step of selecting the target object among the first objects in the first initial subspace; and A method of operating a mixed reality telepresence system, characterized by comprising a step of determining a target object among the first objects on the plurality of second initial subspaces by using the target object and scene graph on the first initial subspace.

5. In paragraph 1, The step of generating the above multiple alignment spaces is: A step of determining an optimization function based on the collaboration context; A step of setting a second position and a second angle for each of the plurality of second subspaces based on the first subspace, the plurality of second subspaces, and the optimization function; and A method of operating a mixed reality telepresence system, characterized by comprising a step of obtaining the plurality of alignment spaces based on the second position and the second angle.

6. In paragraph 5, A method of operating a mixed reality telepresence system, characterized in that the above optimization function is set such that an area in which the target object of the first subspace and the target objects of the plurality of second subspaces overlap is maximized.

7. In paragraph 1, A method of operating a mixed reality telepresence system, characterized in that the plurality of alignment spaces include a first overlap area indicating an area where the target object of the first subspace and the target objects of the plurality of second subspaces overlap.

8. In paragraph 7, The step of determining the plurality of first positions is: A step of setting a plurality of first sample locations of the plurality of clients to be spaced apart from the first overlapping area by a first distance; A step of selecting a plurality of second sample locations among the plurality of first sample locations based on obstacle information, body information of the plurality of clients, etc.; and A method of operating a mixed reality telepresence system, characterized by comprising the step of obtaining the plurality of first locations based on proxemics between the second sample locations and the plurality of clients.

9. In paragraph 1, The step of setting the above plurality of first interactive areas is: A step of setting different first to fourth markers in the form of lines at the plurality of first locations; and A method of operating a mixed reality telepresence system, characterized by comprising a step of acquiring a plurality of first interactable areas based on the first to fourth markers.

10. In paragraph 9, The step of acquiring the above plurality of first interactable areas is: A step of moving the first to fourth markers in the first to fourth directions respectively until the first condition is satisfied; and A method of operating a mixed reality telepresence system, characterized by comprising a step of generating polygons formed by extending the first to fourth markers as the plurality of first interactable areas.

11. A subspace extraction unit that extracts a first subspace in which a target object is set and a plurality of second subspaces in which the target object is set from a first virtual space of the host and a plurality of second virtual spaces of a plurality of clients; A subspace alignment unit that receives the first subspace and the plurality of second subspaces from the subspace extraction unit, and generates a plurality of alignment spaces overlapping the plurality of second subspaces with the first subspace based on the first subspace, the plurality of second subspaces, and an optimization function; and A mixed reality telepresence system comprising a subspace setting unit which receives a plurality of alignment spaces from the subspace alignment unit, determines a plurality of first positions on the plurality of alignment spaces, and sets a plurality of first interactable areas in the first virtual space based on the plurality of first positions.

12. A step of obtaining initial subspaces from a first virtual space of the host and a plurality of second virtual spaces of a plurality of clients; A step of generating target objects from the above initial subspaces; A step of obtaining a first subspace corresponding to the first virtual space and a plurality of second subspaces corresponding to the second virtual space based on the initial subspaces and the target object; A step of determining an optimization function based on the collaboration context; A step of setting a second position and a second angle for each of the plurality of second subspaces based on the first subspace, the plurality of second subspaces, and the optimization function; A step of obtaining a plurality of alignment spaces overlapping the plurality of second sub-spaces with the first sub-space based on the second position and the second angle; A step of determining a plurality of first positions on the plurality of alignment spaces; A step of setting different first to fourth markers in the form of lines at the plurality of first locations; A step of moving the first to fourth markers in the first to fourth directions respectively until the first condition is satisfied; and A method of operating a mixed reality telepresence system, comprising the step of generating polygons formed by extending the first to fourth markers as the plurality of first interactable areas.

Images