XR content control method and system in an XR cave environment

Abstract

The method for controlling XR content in an XR cave environment includes the steps of projecting XR content into the XR cave, detecting a large number of pedestrians using a camera arranged in the XR cave and calculating the average moving speed and direction of the large number of pedestrians, correcting the projection position and speed of the XR content based on the calculated average moving speed and direction of the large number of pedestrians, and reflecting and projecting the corrected XR content onto the wall surface of the XR cave.

Description

[Technical Field]

[0001] The present invention relates to a method and system for controlling XR content in an XR cave environment, and more particularly to a method and system for controlling XR content in an XR cave environment that allows users to share a field of view in an XR cave environment without wearing an HMD, allowing for extended immersion in augmented reality (XR) through higher resolution and a wider field of view, and allowing users to see the real bodies of users and others as they are, thereby allowing users to experience a high sense of presence. [Background technology]

[0002] Extended Reality (XR) is an umbrella term that encompasses a variety of immersive and interactive technology areas, collectively referring to Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR). XR has been used in everyday life for a long time and has undergone various evolutions since the early 1990s. As interest in the metaverse, a 3D evolution of the internet, grows, the use of XR is rapidly expanding into new industries.

[0003] In a typical Virtual Reality (VR) environment, the position and direction of a user's head are tracked using Infrared (IR) tracking in a Head Mounted Display (HMD) to determine the location of each individual user in real space. However, wearing an HMD not only narrows the field of view, but also reduces the sense of presence because the user's actual body cannot be seen. Furthermore, when the movements of multiple users are detected in an XR environment, a technology is needed that can change the time point in the XR content provided to match the multiple moving pedestrians.

[0004] Prior art includes Korean Patent Publication No. 10-2019-0105532 (XR content providing method and XR device), which only includes the steps of generating user motion estimation information to display XR content according to user movement based on one or more coordinate systems and the coordinate system of the XR device, acquiring an image of the front of the XR device, separating the acquired image into a first image and a second image, generating transformation motion information based on at least one of the first image and the second image, correcting the user motion estimation information based on the generated transformation motion information, and displaying the XR content at a position and direction indicated by the corrected user motion estimation information. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Korean Patent Publication No. 10-2019-0105532 Summary of the Invention [Problem to be solved by the invention]

[0006] The problem to be solved by the present invention is to provide a method and system for controlling XR content in an XR cave environment, which allows users to share a field of view in an XR cave environment without wearing an HMD, allowing for prolonged immersion in XR through higher resolution and a wider field of view, and allows users to experience a high sense of presence by being able to see the real bodies of users and others as they are. [Effects of the Invention]

[0007] According to the present invention, users can share their field of view in an XR cave environment without wearing an HMD, allowing them to be immersed in augmented reality (XR) for a long period of time through higher resolution and a wider field of view.

[0008] In addition, since no HMD is required, users can see the real bodies of others and the users themselves, allowing them to experience a high sense of presence. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a flowchart illustrating a method for controlling XR content in an XR CAVE environment according to an embodiment of the present invention. [Figure 2] 3 is a flowchart illustrating a method for detecting multiple pedestrians according to an embodiment of the present invention. [Figure 3] FIG. 1 is a configuration diagram illustrating an XR content control system in an XR CAVE environment according to an embodiment of the present invention. [Figure 4] FIG. 1 is a configuration diagram illustrating the structure of the wall of an XR CAVE according to an embodiment of the present invention. [Figure 5] 1 is an exemplary diagram showing a wall surface of an XR cave on which XR content is projected according to an embodiment of the present invention. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] Specific structural or functional descriptions for embodiments in accordance with the inventive concepts disclosed herein are merely exemplary for purposes of describing embodiments in accordance with the inventive concepts, which may be embodied in various forms and are not limited to the embodiments described herein.

[0011] Because embodiments according to the inventive concept may be variously modified and may have various forms, the embodiments are illustrated in the drawings and described in detail herein, but it is not intended to limit the embodiments according to the inventive concept to the particular disclosed form, and all modifications, equivalents, or alternatives falling within the spirit and scope of the present invention are included.

[0012] The terms used in this specification are merely used to describe specific embodiments and are not intended to limit the present invention. The singular expressions include the plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprise" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preclude the presence or possibility of addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0014] FIG. 1 is a flowchart illustrating a method for controlling XR content in an XR CAVE environment according to an embodiment of the present invention.

[0015] As shown in FIG. 1, in the method for controlling XR content in an XR cave environment, a plurality of beam projectors 220 project XR content into the XR cave 200 (S101). The projected XR content may be uncorrected initial XR content before the movement of a plurality of pedestrians is detected. The initial XR content may be XR content generated and stored based on a database of spatial information and user information that can be collected within the XR cave 200. The initial XR content may also be XR content generated and stored after studying the technical aspects and characteristics of XR content that can be played within the XR cave 200.

[0016] A camera 210 installed in the XR CAVE detects a large number of pedestrians and calculates the average moving speed and direction of the large number of pedestrians (S103). The camera 210 is configured as a single camera and can capture moving images of a large number of pedestrians. The camera 210 does not have equipment such as an infrared camera or a depth camera, and can capture moving images of a large number of pedestrians, from which the moving path and walking direction of each pedestrian can be estimated. The camera 210 may be installed above the environment of the XR CAVE 200 and can capture moving images of a large number of pedestrians, which can be used to collect walking data of a large number of pedestrians and for machine learning.

[0017] The projection position and speed of the XR content are corrected based on the calculated average moving speed and direction of the multiple pedestrians (S105), and the corrected XR content is reflected and projected onto the walls 230 of the XR cave (S107). At this time, the wall 230 may be one or multiple, and the XR cave 200 may be configured in a strip shape with one wall 230, or in a square structure with four walls 230, but is not limited thereto. In addition, one or multiple walls 230 may be connected to a touch board using conductive ink, and interactions may be performed when multiple pedestrians touch the corresponding positions.

[0018] FIG. 2 is a flowchart illustrating a method for detecting multiple pedestrians according to an embodiment of the present invention.

[0019] As shown in Fig. 2, the method of detecting a large number of pedestrians involves the camera 210 capturing an image of the movement of a large number of pedestrians (S201), and the detection unit 110 tracking two-dimensional coordinates and detecting direction vectors from the image of the movement of the large number of pedestrians received from the camera 210 (S203). At this time, the detection unit 110 can track the two-dimensional coordinates of the large number of pedestrians by converting the positional displacement of the pedestrians mapped onto the two-dimensional image plane into an angle value based on the center of the image of the movement of the large number of pedestrians received from the camera 210. In addition, when tracking the two-dimensional coordinates, it is possible to select and track characteristic areas rather than searching for two-dimensional coordinates of all areas of the large number of pedestrians.

[0020] The detection unit 110 detects the movement speeds of the multiple pedestrians from the moving images of the multiple pedestrians received from the camera 210 (S205). The detection unit 110 can detect the direction vectors and movement speeds based on the two-dimensional coordinates tracked from the moving images of the multiple pedestrians received from the camera 210.

[0021] The calculation unit 120 calculates the average speed and direction of multiple pedestrians using the 2D coordinates tracked by the detection unit 110, the detected direction vector, and the moving speed (S207). Since there are multiple pedestrians moving in real time, the speed of the XR content to be corrected can be calculated by averaging the individual moving speeds of the multiple pedestrians detected by the detection unit 110.

[0022] FIG. 3 is a configuration diagram illustrating an XR content control system in an XR cave environment according to an embodiment of the present invention.

[0023] As shown in FIG. 3, the XR content control system 10 in the XR cave environment is composed of an XR content providing server 100 and an XR cave 200.

[0024] The XR content providing server 100 is composed of a detection unit 110, a calculation unit 120, an XR content correction unit 130, a touch control unit 140, a storage unit 150, and a communication unit 160.

[0025] The detection unit 110 can receive moving images of multiple pedestrians from the camera 210 of the XR cave 200. The detection unit 110 can track the two-dimensional coordinates of multiple pedestrians from the images received from the camera 210 and then detect their direction vectors and movement speeds. The XR content control system 10 in the XR cave environment of the present invention does not use a head-mounted display (HMD) to track the position and direction of the user's head, so it is necessary to grasp the user's position in real space to recognize the walking direction of pedestrians in the XR cave environment. The detection unit 110 can perform versatile tracking using vision-based tracking.

[0026] The detection unit 110 can track the 2D coordinates of a number of pedestrians, which change in real time as the pedestrians move, from the image of the pedestrians' movements received from the camera 210. The detection unit 110 can track the 2D coordinates of a number of pedestrians by converting the positional displacement of the pedestrians mapped onto the 2D image plane into an angle value based on the center of the image of the pedestrians' movements received from the camera 210. Furthermore, when tracking the 2D coordinates, the detection unit 110 can select and track characteristic areas rather than searching for 2D coordinates of all areas of the pedestrians. The characteristic areas can be objects in the image received from the camera 210, i.e., corners of areas of a number of pedestrians or points with large changes in pixel values, but are not necessarily limited thereto.

[0027] The detection unit 110 can detect a direction vector and a movement speed based on two-dimensional coordinates tracked from a moving image of a number of pedestrians received from the camera 210. At this time, the direction vector and the movement speed can be detected by estimating three-dimensional coordinates in the XR CAVE environment, which is a real space, from the moving image.

[0028] The calculation unit 120 can calculate the average speed of multiple pedestrians using the 2D coordinates tracked by the detection unit 110 and the detected direction vector and movement speed. Since there are multiple pedestrians moving in real time, the speed of the XR content to be corrected can be calculated by averaging the individual movement speeds of the multiple pedestrians detected by the detection unit 110. If the speed of the corrected XR content is faster or slower than the average movement speed of the multiple pedestrians, the number of pedestrians who feel dizzy may increase, which may impair the immersive experience. Therefore, the calculation unit 120 needs to calculate and adjust the error rate so that the speed of the corrected XR content does not deviate significantly from the average movement speed. In this case, the error rate can be adjusted to less than 5%, but is not necessarily limited to this.

[0029] In addition, the calculation unit 120 can calculate the direction of multiple pedestrians by reflecting the average speed of multiple pedestrians. The detection unit 110 can grasp the positions of multiple pedestrians and calculate the direction in which they should move based on the two-dimensional coordinates tracked and the direction vectors detected. At this time, as the number of pedestrians increases, the deviation of the direction vectors of each pedestrian may become more severe, so the direction can be calculated within a specified error rate. At this time, the error rate can be adjusted to less than 10%, but is not necessarily limited to this.

[0030] The XR content correction unit 130 may correct the projection position and speed of the XR content based on the average value and direction of the movement speeds of multiple pedestrians calculated by the calculation unit 120. In one embodiment, if the average movement speed of pedestrians calculated by the calculation unit 120 is 8 km / h, it may be determined that the pedestrians are jogging, and the progression speed of the XR content may be corrected to match the jogging speed of ordinary people. In another embodiment, if the average movement speed of pedestrians calculated by the calculation unit 120 is less than 0.2 km / h, it may be determined that the pedestrians are barely moving, and the progression speed of the XR content may be corrected to be very slow. Furthermore, when a request for correction of the XR content is received from the touch control unit 140, the XR content correction unit 130 may correct the projection position and speed of the XR content based on the corresponding request.

[0031] The touch controller 140 may receive an interaction from the wall 230 of the XR CAVE 200. Here, the interaction refers to an interaction between a pedestrian who touches the wall 230 and the XR content. The touch controller 140 may request the XR content corrector 130 to correct the XR content based on the received interaction.

[0032] The storage unit 150 can store uncorrected initial XR content before the movement of a large number of pedestrians is detected. The initial XR content can be generated and stored based on a database of spatial information and user information that can be collected within the XR CAVE 200. The initial XR content can also be generated and stored by studying the technical characteristics of XR content that can be played within the XR CAVE 200. The initial XR content can also enhance immersion by auralizing data, including a sound logo. A sound logo is a technique that generates short sound clips to express the unique characteristics of a space, product, etc. When a large number of pedestrians pass near a specific space or product to which a sound logo is applied, the sound logo is played if pedestrians are within a certain radius. Pedestrians who hear the sound logo can associate the specific space or product with the sound alone, thereby experiencing a greater sense of immersion.

[0033] The communication unit 160 can communicate with each component of the XR CAVE 200 via a network. The network means may be configured to include at least one of the CDMA-based (or HSDPA-based) mobile communication network, and / or an IEEE802.16x-based ultra-high speed wireless Internet, and / or an IEEE802.11x-based wireless LAN communication network, but is not necessarily limited thereto.

[0034] The XR CAVE 200 is composed of a camera 210, a beam projector 220, a wall surface 230, and a communication unit 240.

[0035] The camera 210 is configured with a single camera and can capture moving images of multiple pedestrians. The camera 210 does not have equipment such as an infrared camera or a depth camera, but can capture moving images of multiple pedestrians, allowing the movement path and walking direction of each pedestrian to be estimated with just one camera. The camera 210 may be installed above the environment of the XR CAVE 200 and can capture moving images of multiple pedestrians that can be used to collect walking data of multiple pedestrians and for machine learning. In addition, the camera 210 can transmit the captured moving images of multiple pedestrians to the XR content providing server 100.

[0036] The beam projector 220 may be configured with one or multiple beam projectors 220. In one embodiment, when multiple beam projectors 220 are used, interactive media can be provided in a large space using multi-surface projection mapping technology, allowing multiple pedestrians to simultaneously view content. In addition, the beam projector 220 can project XR content onto multiple wall surfaces 230, allowing multiple pedestrians to actively experience the immersive space.

[0037] The wall 230 may be composed of one or more walls 230. In one embodiment, the XR CAVE 200 may be configured in a strip shape with one wall 230, or in a square shape with four walls 230, but this is not necessarily limited to this. One or more walls 230 may be connected to a touch board using conductive ink, and when multiple pedestrians touch corresponding positions, an interaction is performed and transmitted to the touch control unit 140. The wall 230 can be used by connecting electricity to places where electricity is not available through the conductive ink itself. In addition, the wall 230 can be realized so that interaction is performed using serial values ​​by using serial communication between the conductive touch board and Unity. In addition, the wall 230 can be configured to have a sensitivity range, allowing touches to be performed even on thick walls by setting variables.

[0038] The communication unit 240 can communicate with each component of the XR content providing server 100 via a network. The network means may be configured to include at least one of the CDMA-based (or HSDPA-based) mobile communication network, and / or IEEE802.16x-based ultra-high speed wireless internet, and / or IEEE802.11x-based wireless LAN communication network, but is not necessarily limited to this.

[0039] FIG. 4 is a configuration diagram illustrating the structure of the wall of the XR CAVE according to the embodiment of the present invention.

[0040] 4(a) shows an XR cave 200 configured in a strip shape with one wall 230, and FIG. 4(b) shows an XR cave 200 configured in a square structure with four walls 230. In another embodiment, the XR cave may have a greater number of walls 230, and the walls 230 of the XR cave may be configured according to the XR content projected by a beam projector within the XR cave.

[0041] FIG. 5 is an exemplary view showing a wall surface of an XR cave on which XR content is projected according to an embodiment of the present invention.

[0042] Fig. 5(a) is an exemplary diagram showing XR content projected onto the wall surfaces 230 of an XR cave 200 configured in a strip shape. Fig. 5(b) is an exemplary diagram showing XR content projected onto the wall surfaces 230 of an XR cave 200 configured in a quadrangular structure with four wall surfaces 230. As shown in Fig. 5, a pedestrian or multiple pedestrians can experience XR content projected onto the wall surfaces 230 of the XR cave 200 by multiple beam projectors 220 while walking within the XR cave 200 environment.

[0043] Although the present invention has been described with reference to the embodiments shown in the drawings, these are merely illustrative, and those skilled in the art will recognize that various modifications and equivalent embodiments are possible. Therefore, the true technical scope of the present invention should be determined by the technical ideas of the appended claims. [Explanation of symbols]

[0044] 10 XR Content Control System 100 XR content provider servers 110 Detector 120 Calculation Department 130 XR Content Correction Unit 140 Touch control section 150 Storage Unit 160 Communications Department 210 Camera 220 Beam Projector 230 Wall 240 Communications Department

Claims

1. A method for controlling XR content in an XR CAVE environment, projecting XR content into the XR CAVE; Detecting a large number of pedestrians using a camera installed in the XR CAVE and calculating the average moving speed and direction of the large number of pedestrians; Correcting the projection position and speed of the XR content based on the calculated average moving speed and direction of multiple pedestrians; and reflecting and projecting the corrected XR content onto a wall surface of the XR CAVE; The step of detecting a large number of pedestrians by a camera arranged in the XR CAVE and calculating an average value of the speed and direction of the large number of pedestrians includes: A step in which a camera captures moving images of a large number of pedestrians; a step of tracking two-dimensional coordinates and detecting direction vectors from the moving images of a large number of pedestrians received from the camera by the detection unit; detecting the movement speeds of the multiple pedestrians from the moving images of the multiple pedestrians received from the camera by the detection unit; and A method for controlling XR content in an XR CAVE environment, comprising a step in which a calculation unit tracks the average value and direction of movement speeds of a large number of pedestrians using the two-dimensional coordinates tracked by the detection unit, the detected direction vector, and the movement speed.

2. The method of claim 1 , wherein the camera capturing the moving images of the multiple pedestrians is a single camera.

3. The method of claim 1 , further comprising controlling the XR content by touching the wall of the XR cave, the wall being a part of the XR cave, in conjunction with a touch board.

4. An XR content providing server that receives images captured by the camera, detects a large number of pedestrians, calculates the average moving speed and direction of the large number of pedestrians, corrects the projection position and speed of the XR content based on the calculated average moving speed and direction of the large number of pedestrians, and reflects and projects the corrected XR content onto the wall of the XR CAVE; and The system includes an XR CAVE that captures moving images of many pedestrians and projects the XR content onto the walls of the XR CAVE. The XR content providing server includes a detection unit, The detection unit tracks two-dimensional coordinates and detects direction vectors from the moving images of the multiple pedestrians received from the camera, detects the moving speeds of the multiple pedestrians from the moving images of the multiple pedestrians received from the camera, and the calculation unit tracks the average value and direction of the moving speeds of the multiple pedestrians using the tracked two-dimensional coordinates, the detected direction vectors, and the moving speeds.

Images