XR content control method and system in XR CAVE environment

Abstract

The invention relates to an XR content control method in an XR CAVE environment. The method comprises the following steps: projecting XR content into XR CAVE; detecting a plurality of pedestrians through a camera configured in the XR CAVE, and calculating an average value and directions of moving speeds of the plurality of pedestrians; correcting the projection position and speed of the XR content on the basis of the calculated average value and direction of the moving speeds of the plurality of pedestrians; and reflecting and projecting the corrected XR content on the wall surface of the XR CAVE.

Description

Technical Field

[0001] The present invention relates to an XR content control method and system in an XR CAVE environment. Without wearing an HMD, it shares the field of view among users in the XR CAVE environment, enabling long-term immersion in extended reality (XR) with higher resolution and a wider field of view, and allowing direct viewing of the real bodies of users and others, thereby enabling an XR content control method and system in the XR CAVE environment with a high sense of presence. Background Art

[0002] Extended Reality (XR) is a general term covering various immersive and interactive technology fields, and is a collective term for Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR). XR has been used in daily life for a quite long time and has gone through various evolutions from the early 1990s to the present. With the increasing attention to the 3D evolution of the Internet, i.e., the metaverse, the application of XR is rapidly expanding into new industries.

[0003] In a general Virtual Reality (VR) environment, to determine the positions of individual users in the real space, infrared (IR) tracking of a Head Mounted Display (HMD) is used to track the head position and direction of the user. However, if an HMD is worn, not only does the field of view become narrow, but also the actual body of the user cannot be directly seen, so the sense of presence is also reduced. In addition, when the movements of multiple users are detected in an XR environment, a technology capable of changing the viewpoints in XR content according to multiple pedestrians is required. As a prior art, there is Korean Patent Publication No. 10-2019-0105532 (XR content providing method and XR device), but it only includes the following steps: to display XR content according to the movement of the user, generate user movement prediction information based on one or more coordinate systems and the coordinate system of the XR device; obtain an image in front of the XR device; separate the obtained image into a first image and a second image; generate conversion movement information based on at least any one of the first image and the second image; correct the user movement prediction information based on the generated conversion movement information; and display the XR content at the position and in the direction shown by the corrected user movement prediction information.

[0004] Detailed Description of the Invention

[0005] Technical Problem

[0006] The problem to be solved by the present invention is to provide an XR content control method and system in an XR CAVE environment, which can share the field of view among users in the XR CAVE environment without wearing an HMD, so that users can immerse themselves in extended reality (XR) for a long time with higher resolution and wider field of view (FOV), and can directly see the real bodies of the user and others, thus being able to experience a high sense of presence.

[0007] Effects of the Invention

[0008] Through the present invention, it is possible to share the field of view among users in the XR CAVE environment without wearing an HMD, so that users can immerse themselves in extended reality (XR) for a long time with higher resolution and wider field of view (FOV).

[0009] In addition, since the HMD is not worn, it is possible to directly see the real bodies of the user and others, thus being able to experience a high sense of presence.

[0010] Brief Description of the Drawings

[0011] Figure 1 is a flowchart for describing an XR content control method in an XR CAVE environment according to an embodiment of the present invention.

[0012] Figure 2 is a flowchart for describing a method of detecting multiple pedestrians according to an embodiment of the present invention.

[0013] Figure 3 is a configuration diagram of an XR content control system in an XR CAVE environment according to an embodiment of the present invention.

[0014] Figure 4 is a configuration diagram for describing the structure of the wall surface of an XR CAVE according to an embodiment of the present invention.

[0015] Figure 5 is an example diagram showing the wall surface of an XR CAVE on which XR content is projected according to an embodiment of the present invention.

[0016] Embodiments of the Invention

[0017] Regarding the embodiments of the concepts according to the present invention disclosed in this specification, the specific structural or functional descriptions are only for describing the embodiments of the concepts according to the present invention, are exemplary, and the embodiments of the concepts according to the present invention can be implemented in various forms and are not limited to the embodiments described in this specification.

[0018] Various changes can be applied to embodiments according to the inventive concept, and they can have various forms. Therefore, embodiments will be shown in the drawings and described in detail in this specification. However, this does not limit the embodiments according to the inventive concept to a specific disclosed form, but includes all changes, equivalents, or alternatives included within the spirit and technical scope of the present invention.

[0019] The terms used in this specification are only for describing specific embodiments and are not intended to limit the present invention. Expressions in the singular include expressions in the plural unless clearly different in context. In this specification, terms such as "including" or "having" should be understood to mean that there are the features, numbers, steps, operations, components, parts, or combinations thereof described in this specification, and do not preclude the existence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof in advance.

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings in this specification.

[0021] Figure 1 It is a flowchart for describing an XR content control method in an XR CAVE environment according to an embodiment of the present invention.

[0022] Referring to Figure 1 , in the XR content control method in the XR CAVE environment, a plurality of beam projectors 220 scan the XR content in the XR CAVE 100 (S101). At this time, the scanned XR content may be the original uncorrected XR content before detecting the movement of multiple pedestrians. The original XR content may be an XR content generated and stored based on a database of spatial information, user information, etc. that can be collected within the XR CAVE 200. In addition, the original XR content may be an XR content generated and stored by studying the technical aspect characteristics of the XR content that can be run within the XR CAVE 200.

[0023] Detect multiple pedestrians through the camera 210 disposed within the XR CAVE, and calculate the average value and direction of the movement speeds of the multiple pedestrians (S103). The camera 210 is composed of a single camera and can capture moving images of multiple pedestrians. The camera 210 does not require devices such as an infrared camera or a depth camera, and can capture moving images of multiple pedestrians with only one camera, and the images can infer the movement paths and walking directions of individual pedestrians. The camera 210 can be disposed in the upper part of the environment of the XR CAVE 200 and can capture moving images of multiple pedestrians, and the images are used for collecting walking data of multiple pedestrians and machine learning.

[0024] Based on the calculated average value and direction of the moving speeds of multiple pedestrians, correct the projection position and speed of the XR content S105, and reflect and project the corrected XR content onto the wall surface 230 of the XR CAVE S107. At this time, the wall surface 230 can be one or more, and can be composed of one wall surface 230, so that the XR CAVE 200 is formed in a strip shape, or can be composed of four wall surfaces 230, so that the XR CAVE 200 is formed in a quadrilateral structure, but is not necessarily limited to this. In addition, one or more wall surfaces 230 can be connected to the touch panel using conductive ink, and when multiple pedestrians touch the corresponding position, interaction can be performed.

[0025] Figure 2 It is a flowchart for describing a method of detecting multiple pedestrians according to an embodiment of the present invention.

[0026] Refer to Figure 2 , in the method of detecting multiple pedestrians, the camera 210 captures moving images of multiple pedestrians S201, and the detection unit 110 tracks two-dimensional coordinates and detects direction vectors from the moving images of multiple pedestrians received from the camera 210 S203. At this time, the detection unit 110 takes the center of the moving image screen of multiple pedestrians received from the camera 210 as a reference, and converts the change in the position of the pedestrians mapped on the two-dimensional image plane into an angle value, so that the two-dimensional coordinates of multiple pedestrians can be tracked. In addition, when tracking the two-dimensional coordinates, a special area can be selected for tracking instead of finding the two-dimensional coordinates of all areas of multiple pedestrians.

[0027] The detection unit 110 detects the moving speeds of multiple pedestrians from the moving images of multiple pedestrians received from the camera 210 S205. The detection unit 110 can detect the direction vector and the moving speed based on the two-dimensional coordinates tracked from the moving images of multiple pedestrians received from the camera 210.

[0028] Using the two-dimensional coordinates tracked by the detection unit 110, the detected direction vector, and the moving speed, the calculation unit 120 calculates the average value and direction of the speeds of multiple pedestrians S207. Since there are multiple pedestrians moving in real time, the speed of the XR content to be corrected can be calculated by obtaining the average value of the individual moving speeds of multiple pedestrians detected by the detection unit 110.

[0029] Figure 3 It is a configuration diagram for describing an XR content control system in an XR CAVE environment according to an embodiment of the present invention.

[0030] Refer to Figure 3 , the XR content control system 10 in the XR CAVE environment is composed of an XR content providing server 100 and an XR CAVE 200.

[0031] The XR content providing server 100 is composed of a detection unit 110, a calculation unit 120, an XR content correction unit 130, a touch control unit 140, a storage unit 150, and a communication unit 160.

[0032] The detection unit 110 can receive moving images of multiple pedestrians received from the camera 210 of the XR CAVE 100. After tracking the two-dimensional coordinates of multiple pedestrians from the images received from the camera 210, the detection unit 110 can detect the direction vector and the moving speed. Since the XR content control system 10 in the XR CAVE environment of the present invention does not use a head-mounted display (HMD) that tracks the head position and direction of the user, it is necessary to know the position of the user in the real space in order to identify the walking direction of pedestrians in the XR CAVE environment. The detection unit 110 can use vision-based tracking to achieve general tracking.

[0033] The detection unit 110 can track the two-dimensional coordinates of multiple pedestrians that change in real time as the multiple pedestrians move from the moving images of multiple pedestrians received from the camera 210. The detection unit 110 takes the center of the moving image frame of multiple pedestrians received from the camera 210 as a reference, and converts the change in the position of the pedestrians mapped on the two-dimensional image plane into an angular value, so as to track the two-dimensional coordinates of multiple pedestrians. In addition, when tracking the two-dimensional coordinates, the detection unit 110 can select a special area for tracking instead of finding the two-dimensional coordinates of all areas of multiple pedestrians. The special area may be an object in the image received from the camera 210, that is, the edge part of the multiple pedestrian areas or points with large changes in pixel values, but is not necessarily limited thereto.

[0034] Based on the two-dimensional coordinates tracked from the moving images of multiple pedestrians received from the camera 210, the detection unit 110 can detect the direction vector and the moving speed. At this time, the three-dimensional coordinates in the actual space XR CAVE environment can be inferred from the moving images, so as to detect the direction vector and the moving speed.

[0035] The calculation unit 120 can use the two-dimensional coordinates tracked by the detection unit 110, the detected direction vector, and the moving speed to calculate the average value of the speeds of multiple pedestrians. Since there are multiple pedestrians moving in real time, the speed of the XR content to be corrected can be calculated by obtaining the average value of the individual moving speeds of multiple pedestrians detected by the detection unit 110. If the speed of the corrected XR content is faster or slower than the average value of the moving speeds of multiple pedestrians, the number of pedestrians who feel dizzy will increase, and the immersion will be eliminated. Therefore, the calculation unit 120 needs to calculate and adjust the error rate so that the speed of the modified XR content does not deviate significantly from the average value of the moving speeds. At this time, the error rate can be adjusted to less than 5%, but is not necessarily limited thereto.

[0036] In addition, the calculation unit 120 can calculate the directions of multiple pedestrians by reflecting the average speed of the multiple pedestrians. Based on the two-dimensional coordinates tracked by the detection unit 110 and the detected direction vectors, the positions of the multiple pedestrians can be grasped, and it can be calculated in which direction to run. At this time, the more the number of pedestrians increases, the greater the deviation of the direction vectors of each pedestrian, so the direction can be calculated within a specified error rate. At this time, the error rate can be adjusted to less than 10%, but it is not necessarily limited to this.

[0037] The XR content correction unit 130 can correct the projection position and speed of the XR content based on the average value and direction of the moving speeds of the multiple pedestrians calculated by the calculation unit 120. As an example, if the average moving speed of the pedestrians calculated by the calculation unit 120 is 8 km / h, it is determined that the pedestrians are jogging, so the running speed of the XR content can be corrected according to the jogging speed of ordinary people. As another example, if the average moving speed of the pedestrians calculated by the calculation unit 120 is less than 0.2 km / h, it is determined that the pedestrians are hardly moving, so the running speed of the XR content can be corrected very slowly. In addition, the XR content correction unit 130 receives a correction request for the XR content from the touch control unit 140, and can correct the projection position and speed of the XR content based on the corresponding request.

[0038] The touch control unit 140 can receive an interaction from the wall surface 230 of the XR CAVE 200. At this time, the interaction can refer to the interaction between the pedestrians touching the wall surface 230 and the content of the XR content. The touch control unit 140 can request the XR content information unit 130 to modify the XR content based on the received interaction.

[0039] The storage unit 150 can store the original XR content that has not been corrected before detecting the movement of multiple pedestrians. The original XR content can be generated and stored based on a database such as spatial information and user information that can be collected within the XR CAVE 200. In addition, the original XR content can be generated and stored by studying the technical aspect characteristics of the XR content that can be run within the XR CAVE 200. In addition, the original XR content includes a sound mark, so the immersion can be increased through the auditoryization of data. The sound mark refers to a technology that expresses the inherent characteristics of a space, a product, etc. by generating a short sound clip. When multiple pedestrians pass near a specific space or product with a sound mark applied, as long as there are pedestrians within a certain distance radius, the sound mark can be played. Pedestrians who hear the corresponding sound mark can associate a specific space or product just by the sound, so they can feel a higher sense of immersion.

[0040] The communication unit 160 can communicate with the components of the XR CAVE 200 via a network. The network device can include at least one of the following networks: a mobile communication network based on CDMA (or based on HSDPA); and / or an ultra-high-speed wireless network based on IEEE802.16x; and / or a wireless local area network based on IEEE 802.11x, but is not necessarily limited thereto.

[0041] The XR CAVE 200 is composed of a camera 210, a beam projector 220, a wall surface 230, and a communication unit 240.

[0042] The camera 210 is composed of a single camera and can capture moving images of multiple pedestrians. The camera 210 does not require devices such as an infrared camera or a depth camera. With just one camera, it can capture moving images of multiple pedestrians, and the images can be used to infer the movement paths and walking directions of individual pedestrians. The camera 210 can be set at the upper part of the environment of the XR CAVE 200 to capture moving images of multiple pedestrians, which are used for collecting walking data of multiple pedestrians and machine learning. In addition, the moving images of multiple pedestrians captured by the camera 210 are transmitted to the XR content providing server 100.

[0043] The beam projector 220 can be composed of one or more beam projectors 220. As an embodiment, when multiple beam projectors 220 are used, interactive media can be provided in a large space through multi-surface projection mapping technology, and a service that multiple pedestrians can enjoy content simultaneously can be provided. In addition, the beam projector 220 projects XR content onto multiple wall surfaces 230, enabling multiple pedestrians to have an active immersive experience in the immersive space.

[0044] The wall surface 230 can be composed of one or more wall surfaces 230. As an embodiment, it can be composed of one wall surface 230 to form the XR CAVE 200 in a strip shape, or it can be composed of four wall surfaces 230 to form the XR CAVE 200 in a quadrilateral structure, but is not necessarily limited thereto. In addition, one or more wall surfaces 230 can be connected to a touch panel using conductive ink, and when multiple pedestrians touch the corresponding position, interaction can be performed and transmitted to the touch control unit 140. The wall surface 230 can use conductive ink that conducts electricity by itself to achieve continuous power use in non-powered areas. In addition, the wall surface 230 can be implemented such that the conductive touch panel and Unity perform interaction according to a serial value through serial communication. In addition, the wall surface 230 can set a sensitivity range so that touch can be performed on thicker walls through variable setting.

[0045] The communication unit 240 can communicate with each component of the XR content providing server 100 via a network. The network device may include at least one of the following networks: a mobile communication network based on CDMA (or HSDPA); and / or an ultra-high speed wireless network based on IEEE 802.16x; and / or a wireless local area network based on IEEE 802.11x, but is not necessarily limited thereto.

[0046] Figure 4 It is a configuration diagram for describing the structure of the wall surface of the XR CAVE according to an embodiment of the present invention.

[0047] Figure 4 (a) thereof shows that it is constituted by one wall surface 230 and the XR CAVE 200 is constituted in a strip shape. Figure 4 (b) thereof shows that it is constituted by four wall surfaces 230 and the XR CAVE 200 is constituted in a quadrilateral structure. As another embodiment, the wall surface 230 of the XR CAVE may be constituted by more wall surfaces 230, and thus the wall surface 230 of the XR CAVE can be constituted according to the content of the XR content projected into the XR CAVE by the beam projector.

[0048] Figure 5 It is an example diagram showing the wall surface of the XR CAVE onto which the XR content is projected according to an embodiment of the present invention.

[0049] Figure 5 (a) thereof shows an example diagram in which the XR content is projected onto the wall surface 230 of the XR CAVE 200 constituted in a strip shape. Figure 5 (b) thereof shows an example diagram in which the XR content is projected onto the wall surface 230 constituted by four wall surfaces 230 so as to constitute the XR CAVE 200 in a quadrilateral structure. Referring to Figure 5 , a pedestrian or multiple pedestrians can experience the XR content while walking in the XR CAVE 200 environment, and the XR content is projected onto the wall surface 230 of the XR CAVE 200 by multiple beam projectors 220.

[0050] The present invention has been described with reference to the embodiments shown in the drawings, but this is only exemplary, and it can be understood that those with ordinary knowledge in the technical field can make various deformations and equivalent other embodiments therefrom. Therefore, the true technical protection scope of the present invention should be determined according to the technical idea of the appended registered claims.

Claims

1. An XR content control method in an XR CAVE environment, characterized in that, it includes the following steps: Project XR content into the XR CAVE; Detect multiple pedestrians through cameras configured in the XR CAVE, and calculate the average value and direction of the moving speeds of the multiple pedestrians; Based on the calculated average value and direction of the moving speeds of the multiple pedestrians, correct the projection position and speed of the XR content; and Reflect and project the corrected XR content onto the wall surface of the XR CAVE, The step of detecting multiple pedestrians through cameras configured in the XR CAVE and calculating the average value and direction of the moving speeds of the multiple pedestrians includes the following steps: The camera captures moving images of multiple pedestrians; The detection unit tracks two-dimensional coordinates and detects direction vectors from the moving images of multiple pedestrians received from the camera; The detection unit detects the moving speeds of multiple pedestrians from the moving images of multiple pedestrians received from the camera; and Using the two-dimensional coordinates tracked by the detection unit, the detected direction vectors, and the moving speeds, the calculation unit tracks the average value and direction of the moving speeds of multiple pedestrians.

2. The XR content control method in an XR CAVE environment according to claim 1, characterized in that, The camera for capturing the moving images of the multiple pedestrians is a single camera.

3. The XR content control method in the XR CAVE environment according to claim 1, characterized in that, The wall surface is the wall surface constituting the XR CAVE, is linked with a touch panel, and controls XR content through touch.

4. An XR content control system method in an XR CAVE environment, characterized in that, it includes: An XR content providing server, which receives the images captured by the camera, thereby detecting multiple pedestrians, calculating the average value and direction of the moving speeds of the multiple pedestrians, and based on the calculated average value and direction of the moving speeds of the multiple pedestrians, correcting the projection position and speed of the XR content, and reflecting and projecting the corrected XR content onto the wall surface of the XR CAVE; and An XR CAVE, which captures moving images of multiple pedestrians and projects XR content onto the wall surface of the XR CAVE, The XR content providing server includes a detection unit, The detection unit is that the detection unit tracks two-dimensional coordinates and detects direction vectors from the moving images of multiple pedestrians received from the camera, and detects the moving speeds of multiple pedestrians from the moving images of multiple pedestrians received from the camera. Using the tracked two-dimensional coordinates, the detected direction vectors, and the moving speeds, the calculation unit tracks the average value and direction of the moving speeds of multiple pedestrians.

Images