XR Content Control Method and System in an XR CAVE Environment

Abstract

The method for controlling XR content in an XR cave environment includes the steps of projecting XR content into the XR cave, detecting a large number of pedestrians using a camera arranged in the XR cave and calculating the average moving speed and direction of the large number of pedestrians, correcting the projection position and speed of the XR content based on the calculated average moving speed and direction of the large number of pedestrians, and reflecting and projecting the corrected XR content onto the wall surface of the XR cave.

Description

Technical Field

[0001] The present invention relates to an XR content control method and system in an XR CAVE environment. In particular, without wearing an HMD, it is possible to share the field of view among users in an XR CAVE environment and immerse in extended reality (XR) for a long time through higher resolution and a wider field of view (FOV), and it is possible to directly see the real bodies of the user and others, so the present invention relates to an XR content control method and system in an XR CAVE environment that can experience a high sense of presence (presence).

Background Art

[0002] Extended Reality (XR) is a comprehensive term covering various immersive and interactive technology fields, and generally refers to Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR). XR has been used in daily life for a long time and has evolved in various ways from the early 1990s until now. As the interest in the metaverse, which is the 3D evolutionary form of the Internet, increases, the utilization of XR is rapidly expanding into new industries.

[0003] In a general VR (Virtual Reality) environment, in order to grasp the position of an individual user in the real space, the position and direction of the user's head are tracked using IR (Infra-Red) tracking of an HMD (Head Mounted Display). However, wearing an HMD not only narrows the field of view but also makes it impossible to directly see the actual body of the user, resulting in a decrease in the sense of presence (presence). Also, when the movement of a large number of users is detected within the XR environment, there is a situation where a technology is needed to change the time point within the XR content provided according to the large number of moving pedestrians.

[0004] Prior art includes Korean Published Patent No. 10-2019-0105532 (XR Content Provisioning Method and XR Device), but it only includes the steps of generating user motion estimation information to display XR content in accordance with user movement based on one or more coordinate systems and the coordinate system of the XR device, acquiring an image of the front of the XR device, separating the acquired image into a first image and a second image, generating transformed motion information based on at least one of the first image and the second image, correcting user motion estimation information based on the generated transformed motion information, and displaying the XR content at the position and direction indicated by the corrected user motion estimation information. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Korean Published Patent No. 10-2019-0105532 [Overview of the project] [Problems that the invention aims to solve]

[0006] The problem that this invention aims to solve is to provide an XR content control method and system in an XR CAVE environment that allows users to share their field of view with each other in an XR CAVE environment without wearing an HMD, enabling them to immerse themselves in augmented reality (XR) for extended periods through higher resolution and a wider field of view, and allowing them to see the real bodies of themselves and others as they are, thereby providing a high sense of presence. [Effects of the Invention]

[0007] According to the present invention, users can share their field of view in an XR CAVE environment without wearing an HMD, allowing for extended immersion in augmented reality (XR) through higher resolution and a wider field of view.

[0008] Furthermore, because users do not wear an HMD, they can see the real bodies of themselves and others, allowing them to experience a high level of presence. [Brief explanation of the drawing]

[0009] [Figure 1] This is a flowchart illustrating an XR content control method in an XR CAVE environment according to an embodiment of the present invention. [Figure 2] This is a flowchart illustrating a method for detecting multiple pedestrians according to an embodiment of the present invention. [Figure 3] This is a diagram illustrating an XR content control system in an XR CAVE environment according to an embodiment of the present invention. [Figure 4] This is a diagram illustrating the structure of the wall surface of an XR CAVE according to an embodiment of the present invention. [Figure 5] This is an illustrative diagram showing the wall surface of an XR CAVE illuminated with XR content according to an embodiment of the present invention. [Modes for carrying out the invention]

[0010] The specific structural or functional descriptions of embodiments of the concept of the present invention disclosed herein are merely illustrative for the purpose of illustrating embodiments of the concept of the present invention, and embodiments of the concept of the present invention can be carried out in a variety of forms and are not limited to the embodiments described herein.

[0011] Embodiments of the concept of the present invention can be modified in various ways and may take on various forms; therefore, embodiments are described in detail herein with reference to the drawings. However, this is not intended to limit embodiments of the concept of the present invention to any particular disclosure, but rather to include all modifications, equivalents, or substitutions that fall within the spirit and technical scope of the present invention.

[0012] The terms used herein are used solely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. Terms such as “includes” or “having” herein are intended to specify the existence of features, figures, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood not to preemptively exclude the existence or possibility of adding one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0013] Embodiments of the present invention will be described in detail below with reference to the drawings attached to this specification.

[0014] Figure 1 is a flowchart illustrating an XR content control method in an XR CAVE environment according to an embodiment of the present invention.

[0015] As shown in Figure 1, the XR content control method in the XR CAVE environment involves multiple beam projectors 220 illuminating the XR CAVE 200 with XR content (S101). The XR content illuminated at this time may be the initial, uncorrected XR content before the movement of multiple pedestrians is detected. This initial XR content may be XR content generated and stored based on a database of spatial information and user information that can be collected within the XR CAVE 200. Alternatively, the initial XR content may be XR content generated and stored after studying the technical characteristics of XR content that can be processed within the XR CAVE 200.

[0016] A camera 210 placed inside the XR CAVE detects a large number of pedestrians and calculates the average speed and direction of movement of the large number of pedestrians (S103). The camera 210 consists of a single camera and can capture video of the movement of a large number of pedestrians. The camera 210 can capture video of the movement of a large number of pedestrians with just one camera, without the need for equipment such as an infrared camera or a depth camera, and can estimate the movement path and direction of individual pedestrians. The camera 210 may also be installed above the environment of the XR CAVE 200 and can capture video of the movement of a large number of pedestrians for collection of walking data and use in machine learning.

[0017] Based on the calculated average speed and direction of numerous pedestrians, the projection position and speed of the XR content are corrected (S105), and the corrected XR content is projected onto the wall surface 230 of the XR CAVE (S107). At this time, the wall surface 230 may be one or more, and the XR CAVE 200 may be composed of one wall surface 230 in the shape of a strip, or it may be composed of four wall surfaces 230 in the shape of a rectangle, but is not necessarily limited to these. In addition, one or more wall surfaces 230 can be connected to a touch board using conductive ink, and interaction can be performed when multiple pedestrians touch the corresponding location.

[0018] Figure 2 is a flowchart illustrating a method for detecting multiple pedestrians according to an embodiment of the present invention.

[0019] As shown in FIG. 2, the method for detecting a large number of pedestrians is as follows: The camera 210 captures the moving images of a large number of pedestrians (S201), and the detection unit 110 tracks the two-dimensional coordinates and detects the direction vectors from the moving images of a large number of pedestrians received from the camera 210 (S203). At this time, the detection unit 110 can track the two-dimensional coordinates of a large number of pedestrians by converting the position displacement of the pedestrians mapped on the two-dimensional video plane into an angular value based on the center of the moving image screen of a large number of pedestrians received from the camera 210. Also, when tracking the two-dimensional coordinates, instead of searching for the two-dimensional coordinates of all regions of a large number of pedestrians, characteristic regions can be selected and tracked.

[0020] The detection unit 110 detects the moving speeds of a large number of pedestrians from the moving images of a large number of pedestrians received from the camera 210 (S205). The detection unit 110 can detect the direction vectors and moving speeds based on the two-dimensional coordinates tracked from the moving images of a large number of pedestrians received from the camera 210.

[0021] The calculation unit 120 calculates the average value and direction of the speeds of a large number of pedestrians by using the two-dimensional coordinates tracked by the detection unit 110, the detected direction vectors, and the moving speeds (S207). Since there are a large number of pedestrians moving in real time, the speed of the XR content can be calculated by obtaining and correcting the average of the individual moving speeds of a large number of pedestrians detected by the detection unit 110.

[0022] FIG. 3 is a configuration diagram for explaining an XR content control system in an XR CAVE environment according to an embodiment of the present invention.

[0023] As shown in FIG. 3, the XR content control system 10 in the XR CAVE environment is composed of an XR content providing server 100 and an XR CAVE 200.

[0024] The XR content providing server 100 is composed of a detection unit 110, a calculation unit 120, an XR content correction unit 130, a touch control unit 140, a storage unit 150, and a communication unit 160.

[0025] The detection unit 110 can receive video footage of multiple pedestrians moving from the camera 210 of the XR CAVE 200. After tracking the two-dimensional coordinates of multiple pedestrians from the video footage received from the camera 210, the detection unit 110 can detect the direction vector and movement speed. The XR content control system 10 in the XR CAVE environment of the present invention does not use an HMD (Head Mounted Display) to track the position and direction of the user's head; therefore, in order to recognize the direction of pedestrians' walking in the XR CAVE environment, it is necessary to know the user's position in real space. The detection unit 110 can perform versatile tracking using vision-based tracking.

[0026] The detection unit 110 can track the 2D coordinates of numerous pedestrians, which change in real time due to their movement, from the moving images of numerous pedestrians received from the camera 210. The detection unit 110 can track the 2D coordinates of numerous pedestrians by converting the positional changes of the pedestrians, which are mapped onto a 2D image plane, into angular values, using the center of the moving image screen of numerous pedestrians received from the camera 210 as a reference. Furthermore, when tracking 2D coordinates, the detection unit 110 can select characteristic areas to track, rather than searching for the 2D coordinates of all areas of the numerous pedestrians. These characteristic areas may be, but are not necessarily limited to, objects in the image received from the camera 210, i.e., corners of the areas of the numerous pedestrians or points where the pixel value changes significantly.

[0027] The detection unit 110 can detect direction vectors and movement speeds based on two-dimensional coordinates tracked from the movement images of numerous pedestrians received from the camera 210. At the same time, it can estimate three-dimensional coordinates within the XR CAVE environment, which is the actual space, from the movement images and detect direction vectors and movement speeds.

[0028] The calculation unit 120 can calculate the average speed of a large number of pedestrians using the 2D coordinates tracked by the detection unit 110, the detected direction vector, and the movement speed. Since there are many pedestrians moving in real time, the calculation unit 120 can calculate the speed of the corrected XR content by calculating the average of the individual movement speeds of the many pedestrians detected by the detection unit 110. If the speed of the corrected XR content is faster or slower than the average movement speed of the many pedestrians, the number of pedestrians who feel dizzy may increase, potentially impairing immersion. Therefore, the calculation unit 120 needs to calculate and adjust the error rate so that the speed of the corrected XR content does not deviate significantly from the average movement speed. In this case, the error rate can be adjusted to less than 5%, but is not necessarily limited to this.

[0029] Furthermore, the calculation unit 120 can calculate the direction of multiple pedestrians by reflecting the average speed of multiple pedestrians. Based on the two-dimensional coordinates tracked by the detection unit 110 and the detected direction vector, it can determine the position of multiple pedestrians and calculate which direction they must proceed in. At this time, as the number of pedestrians increases, the deviation of the direction vector for each pedestrian may become more pronounced, so the direction can be calculated within a specified error rate. In this case, the error rate can be adjusted to less than 10%, but is not necessarily limited to this.

[0030] The XR content correction unit 130 can correct the illumination position and speed of the XR content based on the average value and direction of the movement speed of a large number of pedestrians calculated by the calculation unit 120. In one embodiment, if the average movement speed of pedestrians calculated by the calculation unit 120 is 8 km / h, the unit can determine that the pedestrians are jogging and correct the progression speed of the XR content to match the jogging speed of ordinary people. In another embodiment, if the average movement speed of pedestrians calculated by the calculation unit 120 is less than 0.2 km / h, the unit can determine that the pedestrians are hardly moving and correct the progression speed of the XR content to be very slow. Furthermore, when the touch control unit 140 requests correction of the XR content, the XR content correction unit 130 can correct the illumination position and speed of the XR content based on the relevant request.

[0031] The touch control unit 140 can receive interaction signals from the wall surface 230 of the XR CAVE 200. In this case, interaction refers to the interaction between a pedestrian who touches the wall surface 230 and the XR content. Based on the transmitted interaction, the touch control unit 140 can request the XR content correction unit 130 to correct the XR content.

[0032] The storage unit 150 can store the initial, uncorrected XR content before the movement of a large number of pedestrians is detected. The initial XR content can be generated and stored based on a database of spatial information and user information that can be collected within the XR CAVE 200. Alternatively, the initial XR content can be generated and stored after studying the technical aspects of XR content that can be processed within the XR CAVE 200. Furthermore, the initial XR content can enhance immersion by making the data audible, including a sound logo. The sound logo refers to a technique that generates short sound clips to express the unique characteristics of a space, product, etc. When a large number of pedestrians pass through a specific space or near a product to which the sound logo has been applied, the sound logo is played if pedestrians are within a certain radius. Pedestrians who hear the sound logo can associate it with the specific space or product by sound alone, thus experiencing a higher level of immersion.

[0033] The communication unit 160 can communicate with each component of the XR CAVE200 via the network. The network means may include, but is not limited to, at least one of the following: a CDMA-based (or HSDPA-based) mobile communication network, and / or an IEEE 802.16x-based ultra-high-speed wireless internet, and / or an IEEE 802.11x-based wireless LAN communication network.

[0034] The XR CAVE200 consists of a camera 210, a beam projector 220, a wall unit 230, and a communication unit 240.

[0035] Furthermore, the calculation unit 120 can calculate the direction of multiple pedestrians by reflecting the average speed of multiple pedestrians. Based on the two-dimensional coordinates tracked by the detection unit 110 and the detected direction vector, it can determine the position of multiple pedestrians and calculate which direction they must proceed in. At this time, the deviation of the direction vector for each pedestrian may become more pronounced as the number of pedestrians increases. Therefore, the calculation unit 120 can determine the overall direction vector of movement of the large number of pedestrians based on the deviation of the direction vector, within a predetermined error rate, for example, within a range of less than 10%. The calculation unit 120 can then correct the illumination position and speed of the XR content by reflecting the determined direction vector and the average value of the movement speeds of the large number of pedestrians.

[0036] The beam projector 220 may consist of one or more beam projectors 220. In one embodiment, when multiple beam projectors 220 are used, interactive media can be provided in a wide space using multi-face projection mapping technology, and a service can be provided in which many pedestrians can view content simultaneously. In addition, by projecting XR content onto multiple wall surfaces 230, the beam projectors 220 can enable many pedestrians to have an active, immersive experience within the space.

[0037] The wall surface 230 may consist of one or more wall surfaces 230. In one embodiment, the XR CAVE 200 may be composed of one wall surface 230 in the shape of a strip, or it may be composed of four wall surfaces 230 in the shape of a rectangle, but is not necessarily limited to these. One or more wall surfaces 230 can be connected to a touch board using conductive ink, and when multiple pedestrians touch the corresponding location, an interaction can be performed and transmitted to the touch control unit 140. The wall surface 230 can be used to connect electricity even in places where electricity does not normally pass through conductive ink, which conducts electricity in the ink itself. Furthermore, the wall surface 230 can be implemented so that interaction is performed by serial values ​​using serial communication with the conductive touch board Unity. In addition, the wall surface 230 can have a sensitivity range set and can be configured to allow touch to be performed even on thick walls by setting a variable.

[0038] The communication unit 240 can communicate with each component of the XR content provision server 100 via the network. The network means may include, but is not limited to, at least one of the following: a CDMA-based (or HSDPA-based) mobile communication network, and / or an IEEE 802.16x-based ultra-high-speed wireless internet, and / or an IEEE 802.11x-based wireless LAN communication network.

[0039] Figure 4 is a diagram illustrating the structure of the wall surface of an XR CAVE according to an embodiment of the present invention.

[0040] Figure 4(a) shows that the XR CAVE 200 is composed of one wall surface 230 in a strip shape, and Figure 4(b) shows that the XR CAVE 200 is composed of four wall surfaces 230 in a rectangular structure. In another embodiment, the XR CAVE's wall surfaces 230 may be composed of a larger number of wall surfaces 230, and the XR CAVE's wall surfaces 230 can be configured according to the content of the XR content projected into the XR CAVE by the beam projector.

[0041] Figure 5 is an illustrative diagram showing the wall surface of an XR CAVE illuminated with XR content according to an embodiment of the present invention.

[0042] Figure 5(a) is an illustrative diagram showing XR content being projected onto the wall surface 230 of an XR CAVE200, which is composed of a strip shape. Figure 5(b) is an illustrative diagram showing XR content being projected onto the wall surface 230 of an XR CAVE200, which is composed of four wall surfaces 230 to form a rectangular structure. As shown in Figure 5, pedestrians or a large number of pedestrians can experience XR content projected onto the wall surface 230 of the XR CAVE200 by multiple beam projectors 220 while walking within the XR CAVE200 environment.

[0043] Although the present invention has been described with reference to the embodiments illustrated in the drawings, these are merely illustrative, and a person with ordinary skill in the art will understand that a variety of modifications and equivalent other embodiments are possible therefrom. Therefore, the true scope of technical protection of the present invention must be determined by the technical idea of ​​the appended claims. [Explanation of Symbols]

[0044] 10 XR Content Control System 100 XR content delivery servers 110 Detection unit 120 Calculation Department 130 XR Content Correction Unit 140 Touch Control Unit 150 Storage Unit 160 Communications Department 210 Cameras 220 Beam Projector 230 Wall surface 240 Communications Department

Claims

1. In an XR (Extended Reality) CAVE (CAVE Automatic Virtual Environment) environment, the method for controlling XR content is as follows: Steps include illuminating the XR CAVE with XR content; A step of detecting multiple pedestrians using cameras placed inside the XR CAVE and calculating the average speed and direction of movement of the multiple pedestrians; A step of correcting the illumination position and speed of the XR content based on the average value and direction of the calculated movement speed of a number of pedestrians; and The process includes the step of reflecting and illuminating the corrected XR content onto the wall surface of the XR CAVE, The steps of detecting a large number of pedestrians using cameras placed within the XR CAVE and calculating the average speed and direction of the large number of pedestrians are as follows: The camera captures video footage of numerous pedestrians in motion; The detection unit tracks the two-dimensional coordinates and detects the direction vector from the video footage of numerous pedestrians moving, received from the camera; A step in which the detection unit detects the movement speed of a number of pedestrians from video footage of the movement of a number of pedestrians received from the camera; and An XR content control method in an XR CAVE environment, further comprising the steps of: using the two-dimensional coordinates tracked by the detection unit, the detected direction vector and movement speed, the calculation unit determines the overall movement direction vector of the numerous pedestrians within a predetermined error rate range based on the deviation of the direction vector of each of the numerous pedestrians, and calculates the average value of the movement speed of the numerous pedestrians.

2. The XR content control method in an XR CAVE environment according to claim 1, characterized in that the camera that captures the movement images of the numerous pedestrians is a single camera.

3. The method for controlling XR content in an XR CAVE environment according to claim 1, further comprising controlling XR content by touch in conjunction with a touch board, wherein the wall surface of the XR CAVE is a wall surface constituting the XR CAVE.

4. An XR content provider server that receives video footage captured by a camera, detects a large number of pedestrians, calculates the average speed and direction of the movement of the large number of pedestrians, corrects the projection position and speed of the XR content based on the calculated average speed and direction of the movement of the large number of pedestrians, and projects the corrected XR content onto the wall surface of the XR CAVE; and This includes an XR CAVE that captures video footage of numerous pedestrians in motion and projects XR content onto the wall of the XR CAVE. The aforementioned XR content provision server is An XR content control system in an XR CAVE environment, characterized by tracking two-dimensional coordinates and detecting direction vectors from video footage of numerous pedestrians moving received from a camera, detecting the movement speed of numerous pedestrians from the video footage of numerous pedestrians received from the camera, and using the tracked two-dimensional coordinates, detected direction vectors, and movement speeds, determining the overall movement direction vector of the numerous pedestrians within a predetermined error rate based on the deviation of the direction vectors of each of the numerous pedestrians, and calculating the average value of the movement speeds of the numerous pedestrians.

Images