Digital treatment environment provision system using metaverse and polyhedral display

Abstract

The present invention relates to a digital therapeutic environment providing system utilizing a metaverse and a polyhedral display, which configures an entire room as a display screen to alleviate patient discomfort and provides various content for use in treatment, while monitoring the user's condition in real time and controlling the content to minimize dizziness, nausea, headache, eye strain, etc., and can reduce confusion, unreality, and emptiness during and after exposure to an immersive VR / AR environment. The details of the project that supported this invention are as follows. [Assignment No.] 202300330001 [Supporting Organization] JIT Co., Ltd. [Research Project Name] Industry-Industry Joint Research Project Support Program [Research Project Title] Establishment of a Digital Therapeutic Environment Utilizing the Metaverse and Polyhedral Displays [Contribution Rate] 1 / 1 [Organizing Institution] Konyang University Industry-Academic Cooperation Foundation [Research Period] February 13, 2023 – May 13, 2023

Description

Technology Field

[0001] The present invention relates to providing a digital therapeutic environment, and more specifically, to a system for providing a digital therapeutic environment utilizing a metaverse and a polyhedral display that configures an entire room as a display screen to alleviate patient discomfort and provides various content to be used for treatment, while monitoring the user's condition in real time and controlling the content to minimize dizziness, nausea, headache, eye strain, etc., and can reduce confusion, unreality, and emptiness during and after exposure to an immersive VR / AR environment. Background Technology

[0002] The metaverse refers to a three-dimensional virtual world where social, economic, and cultural activities similar to those in the real world take place. It has begun to attract increasing attention due to the development of information and communication technology following the commercialization of 5G and the acceleration of the non-face-to-face trend caused by the COVID-19 pandemic. Recently, along with various industrial sectors, there has been a continuing trend of utilizing the metaverse in the medical field as well.

[0003] Examples of this include, basically, creating an environment using virtual reality or augmented reality to alleviate phobias against specific targets; representative examples include providing realistic sound exposure identical to a spider habitat by implementing an environment where various types, shapes, forms, and movements of spiders are complexly exposed to alleviate arachnophobia (fear of spiders); providing exposure to physical environments such as building shapes at various altitudes using a map that combines the user's gaze and environment so that altitude can be felt at any location within a 360-degree range to treat acrophobia; and providing exposure through the implementation of an environment and internal item configurations in a virtual space where virtual scripts and audiences are simulated to treat social anxiety disorders such as public speaking anxiety and stage fright.

[0004] In addition, there are cases utilizing virtual or augmented reality in the treatment of anxiety disorders, which enhance freedom of movement by allowing patients to select treatment locations based on their preferences. This is achieved by providing diverse visual, auditory, and olfactory experiences derived from natural environments within a virtual space, offering various environments such as mountains, plains, seas, and beaches to reduce mental and physical anxiety, and enabling physical location changes at any time. Another example involves the treatment of visual impairments caused by brain injury, which exposes patients to visual stimuli through videos in various orientations (horizontal, vertical, and rotation) for a set period and displays motion images of visual stimuli to facilitate task performance or learning. Furthermore, there are instances where effectiveness is enhanced through content designed to provide immersive olfactory experiences using scent machines to deliver situation-related fragrances, and to promote the movement and exposure of sensory organs within specific eye regions by strictly excluding elements that distract from visual focus other than the induced motion images, thereby encouraging visual exploration.

[0005] While utilizing the metaverse to construct environments similar to the real world can significantly reduce time and costs, currently developed technologies cause users to experience dizziness, nausea, headaches, and eye strain. Furthermore, feelings of unreality, emptiness, or issues with self-perception arise both during and after exposure to the metaverse environment, and there is a possibility of inducing depersonalization; therefore, a method to address these issues is required. Prior art literature

[0006] Republic of Korea Published Patent No. 10-2022-0131451 (September 28, 2022) The problem to be solved

[0007] The present invention was created in response to the above-mentioned needs, and the objective of the present invention is to provide a digital therapeutic environment system utilizing a metaverse and a polyhedral display that configures an entire room as a display screen and provides various content to be used for treatment, while monitoring the user's condition in real time, controlling the content to minimize dizziness, nausea, headache, eye strain, etc., and reducing confusion, unreality, and emptiness during and after exposure to an immersive VR / AR environment. means of solving the problem

[0008] For the above purposes, the present invention provides a digital therapeutic environment providing system comprising: a display module formed to the front, left, and right of a user located in a space and outputting an image; an audio output module that outputs sound into the space; a database in which video and audio content corresponding to the user's therapeutic purpose is stored; an interface module having a measurement unit that collects the user's biosignals, a motion detection unit that detects the user's movements, an eye tracking unit that tracks the user's gaze, and an input unit that receives control commands and status information from the user; an image control module having an image output unit that generates virtual reality graphics with video content according to a control command selected by the user and outputs them to the display module, and an image control unit that modifies the virtual reality graphics generated in correspondence with the biosignals, movements, gaze, and status information collected from the user; and an audio control module that outputs audio content according to a control command selected by the user through the audio output module.

[0009] At this time, it may further include a tactile control module comprising a seat portion on which a user can sit, a blower fan that generates airflow within the space, a blower control portion that controls the blower fan's blowing intensity and direction in correspondence with the virtual reality graphics, and a scent spraying portion that sprays a scent set toward the discharge side of the blower fan.

[0010] In addition, the virtual reality graphics are composed of stereoscopic images that can be viewed by a user wearing 3D glasses, and the sound output module is composed of a plurality of speakers installed at set locations, and the sound control module can individually control the sound content output to the speakers and provide stereoscopic sound.

[0011] Additionally, it may further include a sensory control module comprising a state analysis unit that analyzes biosignals and state information collected from a user to generate sensory information including the user's dizziness and fatigue, and a first linkage control unit that monitors the sensory information in real time and links the image control unit to control the virtual reality graphics so that the intensity of movement increases stepwise from a static image.

[0012] Additionally, it may further include a position control module comprising a moving station that moves the user's position including the interface module in the forward and backward directions, and a second linked control unit that links and controls the image control unit to sequentially change the size of the virtual reality graphics output through the display module in correspondence with the position of the moving station. Effects of the invention

[0013] The present invention can be utilized for the treatment of patients with mental disabilities using virtual / augmented reality, the provision of a non-face-to-face collaborative environment for medical staff, the provision of a practice environment for prospective medical professionals, the provision of a medical education environment for students, and the explanation of various examination and treatment methods to patients. In this process, the user's condition is monitored in real time, and the content is controlled to minimize dizziness, nausea, headache, and eye strain. It can also reduce confusion, unreality, and emptiness during and after exposure to an immersive VR / AR environment.

[0014] In particular, a polyhedral display that makes up an entire room can provide a high degree of realism without the need to wear a device, and in an immersive environment, it can help improve self-perception and alleviate mental disorders such as insomnia and phobias while reducing dizziness, nausea, headaches, and eye strain, or provide symptom relief and pain reduction through the provision of a virtual environment for therapeutic purposes.

[0015] Furthermore, the barrier to entry for self-treatment can be lowered through a UI that enhances patients' understanding of and expected outcomes of treatment, and an easy-to-use treatment sequence. Additionally, interactions within the content that facilitate deep relaxation, attention switching, and abdominal breathing for rest and meditation, along with beautiful video content, can lead to a shift in the patient's perception of their pain, and patients can consult with experts regarding treatment information and receive feedback within the content. Brief explanation of the drawing

[0016] FIG. 1 is a conceptual diagram of the present invention, FIG. 2 is a hardware configuration diagram according to an embodiment, FIG. 3 is a block diagram showing the configuration and connection relationships according to an embodiment of the present invention. Specific details for implementing the invention

[0017] The configuration of the digital therapeutic environment providing system utilizing the metaverse and polyhedral display of the present invention will be explained in detail below with reference to the attached drawings.

[0018] FIG. 1 is a conceptual diagram of the present invention, and FIG. 2 is a hardware configuration diagram according to an embodiment. The present invention configures the entire room as a polyhedral display screen, provides various VR / AR content, and enables its use in treatment. To this end, a series of operations are performed based on a polyhedral display in the form of an LED WALL installed in an indoor space, as well as a KIOSK serving as an interface for measuring the condition of the patient, i.e., the user, and controlling content.

[0019] FIG. 3 is a block diagram showing the configuration and connection relationships according to an embodiment of the present invention. To this end, the present invention is provided with a display module (110), an audio output module (120), a database (130), an interface module (140), an image control module (150), and an audio control module (160) as main components, and a touch control module (170) as a component to enhance immersion, and a sensory control module (180) and a position control module (190) as components to reduce problems such as dizziness, nausea, headache, and eye strain that arise in conventional immersive environments.

[0020] The above display module (110) is configured to output images by being formed on the front, left, and right sides of the user located in the space. That is, as a polyhedral display in the form of the aforementioned LED WALL, it is configured to surround the user within the entire space to enhance realism when providing video content to the user, and the LED WALL is configured to consist of at least three sides: the front, left, and right. Of course, it is obvious that the realism can be further enhanced by installing display modules in the same way on the ceiling as well as behind the user.

[0021] The above sound output module (120) is configured to output sound into a space, and the sound output module is configured to output stereo sound by being composed of a plurality of speakers (121) each installed at a set location.

[0022] The above database (130) is configured to store video and audio content corresponding to the user's treatment purpose. Various video and audio content can be utilized, and for the purpose of treating specific phobias, the database (130) is constructed by pre-producing content including video and audio content related to the object of fear or the situation to be overcome, as well as content including various natural landscapes or natural sounds for mental and physical stability.

[0023] The above interface module (140) is positioned directly in front of the user and is configured in the form of a kiosk for monitoring the user and controlling content, and is equipped with a measurement unit (141), a motion detection unit (142), an eye tracking unit (143), and an input unit (144) as detailed components.

[0024] The above-mentioned measuring unit (141) is configured to collect the user's biosignals and is configured using commercial sensors for collecting various biosignals. For example, it may be equipped with an electrocardiogram sensor to measure the user's electrocardiogram and a brainwave sensor to measure brainwaves. These biosignals are configured to monitor the user's psychological state during content playback. As previously mentioned, for the purpose of treating specific phobias, the user's anxiety / stability state can be monitored through changes in electrocardiograms or brainwaves when playing video and audio content related to the object of fear or the situation to be overcome, or content for mental and physical stability.

[0025] For electrocardiogram measurement, electrodes are formed that the user can touch with their hands, and for measuring brain waves, etc., a wearable device is provided that can be worn on the head and contact the scalp.

[0026] The motion detection unit (142) is configured to detect the user's motion and may be composed of a camera capable of video analysis through a motion analysis algorithm. By utilizing this, in addition to providing content that interacts according to the user's motion recognition, the user's state may be recognized through specific actions that respond to fear or aversion, such as lowering the head to avoid or sitting down.

[0027] The above-mentioned eye tracking unit (143) is configured to track the user's gaze and performs eye tracking through an image analysis technique that captures the user's face. That is, it recognizes and magnifies the eye area in the user's facial image for analysis, tracks the movement of the pupil, and calculates the approximate gaze position and coordinates. By linking this with the movement of the user's head obtained through the motion detection unit (142) mentioned earlier, it is possible to determine the approximate coordinates of which part of the display module (110) the user is looking at.

[0028] The above input unit (144) is configured to receive control commands and status information from a user and is configured in the form of a touchscreen, keyboard, or mouse, so that detailed adjustments of video or audio, as well as the selection, playback, and pause of desired content, can be made.

[0029] The above image control module (150) controls the image content selected by the user and outputs it through the display module (110), and is equipped with an image output unit (151) and an image control unit (152) as detailed components.

[0030] The above video output unit (151) generates virtual reality graphics with video content according to a control command selected by the user and outputs them to the display module (110). Basically, the entire video content is adjusted to fit the size and aspect ratio of the display module (110) before being output, and as described later, the size and brightness of the output video, as well as the playback speed, can be adjusted according to the user's state.

[0031] At this time, the virtual reality graphics may be configured as stereoscopic images that can be viewed by the user wearing 3D glasses, so that the user can view them while wearing 3D glasses.

[0032] The above image control unit (152) modifies the virtual reality graphics generated in response to biosignals, movements, gaze, and state information collected from the user. Basically, in the case of interactive content, it controls the interaction within the image in response to the user's movements detected through the motion detection unit (142).

[0033] In the case of video content intended for the treatment of phobias or anxiety disorders, the intensity can be varied based on the monitoring results of biosignals collected from the user. For example, in the case of arachnophobia, virtual reality graphics are implemented to provide complex exposure to various types, shapes, forms, and movements of spiders, and realistic sound identical to a spider habitat is provided as audio content. In this scenario, if the user's heart rate or other biosignals related to anxiety rise sharply, the virtual reality graphics are modified by reducing or removing the number of spiders within them.

[0034] In addition, for such video content, it is possible to detect the user's intentional avoidance of specific graphic elements through eye tracking, and configure the system so that screen brightness is adjusted or modifications such as the increase or decrease of graphic elements occur in response to the results of biosignal monitoring.

[0035] As another example, in video content designed to address acrophobia, public speaking anxiety, or stage fright, if an indicator related to anxiety among heart rate or vital signs rises rapidly, virtual reality graphics can be modified to lower the altitude within the video closer to the ground and reduce or eliminate the number of audience members, respectively.

[0036] The above sound control module (160) outputs sound content according to a control command selected by the user through the sound output module (120), and as mentioned above, since a plurality of speakers (121) are provided, the sound content output to the speakers (121) can be individually controlled and stereoscopic sound can be provided.

[0037] The above tactile control module (170) is intended to add elements that can be felt by touch in addition to sight and hearing, and is equipped with a seat part (171), a blower fan (172), a blower control part (173), and a scent spray part (174) as detailed components.

[0038] The above-mentioned seat portion (171) is a chair on which a user can sit, and corresponds to the height of the above-mentioned interface module (140) so that the user can access the interface module (140) and be exposed to video and audio content while sitting.

[0039] The above-mentioned blower fan (172) is configured to generate airflow within the space. That is, it is configured to generate wind of a set intensity and is positioned at the upper front so that the user receives wind from the front, or is provided on both the left and right sides, or is equipped with a rotating mechanism that can rotate the blower fan (172) in the left and right directions to control the wind direction from the left and right.

[0040] The above-mentioned blower control unit (173) is configured to control the blower intensity and direction of the blower fan in response to the above-mentioned virtual reality graphics. When a situation is implemented in which a user is exposed to wind outdoors through video and audio content, the blower fan controls the intensity and direction of the person, thereby enhancing realism and immersion.

[0041] The above fragrance spraying unit (174) is configured to spray a fragrance set on the discharge side of the blower fan (172), and by installing a spray that can spray a fragrance associated with video content such as a forest or sea or crops on the discharge side of the blower fan (172), the spraying is performed according to the amount and cycle set corresponding to the video content, thereby enhancing realism and immersion.

[0042] The sensory control module (180) is designed to reduce dizziness, nausea, headache, eye strain, etc., that occur in conventional immersive environments, and is equipped with a state analysis unit (181) and a first interlocking control unit (182) as detailed components.

[0043] The above-mentioned state analysis unit (181) analyzes biosignals and state information collected from the user to generate sensory information including the user's dizziness and fatigue. To this end, it utilizes biosignals collected in advance from the user or a number of sample individuals in a state of feeling dizziness and fatigue as reference signals to analyze biosignals collected from the user in real time, or periodically asks the user about dizziness, nausea, headache, and eye fatigue through the interface module (140) to receive responses, or allows the user to input state information through relevant responses at any time, thereby generating sensory information related to the sensations the user is currently feeling.

[0044] The first interlocking control unit (182) monitors the sensory information in real time and interlocks with the image control unit (152) so that the movement intensity of the virtual reality graphic increases stepwise from a static image.

[0045] In other words, in the case of still images, there is practically no dizziness, nausea, headache, or eye strain; however, since these problems occur with moving images, if biosignal analysis is performed indicating that the user feels dizziness, nausea, headache, or eye strain, or if the user directly inputs status information, the image is maintained in a practically static state until these problems are resolved, and then movement is applied to the image according to the set stages.

[0046] In addition to the method of resolving dizziness, nausea, headaches, and eye strain through the movement of the image as described above, the present invention can also resolve these problems through the size of the output image by using an LED WALL-type display module. In other words, when the image size is small, dizziness, nausea, headaches, and eye strain are inevitably relatively less severe compared to the full screen.

[0047] The above position control module (190) is intended to adjust the position of a user within the space in correspondence with the size of the image, and is equipped with a mobile station (191) and a second interlocking control unit (192) as detailed components.

[0048] The above-mentioned mobile station (191) is configured to move the position of a user, including the interface module (140), in the forward and backward directions. That is, it moves the position of a user standing and looking at the interface module (140). The interface module (140) and the footrest part where the user stands are integrated so that they can move back and forth in the forward and backward directions, and if the aforementioned seat part (171) is provided, the seat part (171) can also move integrally with it.

[0049] To this end, a rail section in the front direction can be configured on the bottom part to provide a driving means so that the moving station (191) can move back and forth along it, or wheels and a driving means can be provided on the lower side to enable positional movement in the front and back directions.

[0050] The second interlocking control unit (192) interlocks with the image control unit (152) to sequentially change the size of the virtual reality graphics output through the display module (110) in correspondence with the position of the mobile station (191). That is, as previously mentioned, when the size of the image is small, dizziness, nausea, headache, and eye strain are relatively reduced compared to the full screen, but since it becomes difficult to view the image content when the distance between the display module and the user increases, the user's position is moved forward to allow viewing the content of the small screen image.

[0051] Similarly, if the user experiences dizziness, nausea, headache, or eye strain, the biosignal analysis is performed or the user directly inputs the status information, the user is made to move forward through the moving station (191) while reducing the size of the virtual reality graphics by a set ratio until these problems are resolved, and then the user is made to move backward through the moving station (191) while increasing the size of the virtual reality graphics according to a set step, thereby reducing dizziness, nausea, headache, and eye strain.

[0052] The present invention classifies video content into exposure therapy (ET) and cognitive behavioral therapy (CBT), and can provide it by subdividing it into real-life exposure, imagery exposure, and sensory organ exposure methods. Furthermore, to provide the necessary stimulation for each therapy, items and environments are configured within the virtual reality content, and synesthetic sounds are used to repeatedly provide a realistic exposure environment that appears like a real situation.

[0053] This real-world exposure method is suitable for short-term treatment and can provide patients with the opportunity to select content during treatment, offer information on treatment effects and prior explanations, and provide realistic virtual reality video and sound to offer patients an optimal and stable visual and auditory experience and an environment similar to traveling to another place, thereby fostering a sense of presence and enabling mental exposure within a virtual natural space.

[0054] In sensory organ exposure therapy, efficient treatment is possible by restricting the operation of other sensory organs that interfere with the training of a specific sensory organ, thereby increasing concentration on that single organ to enhance the function of that organ.

[0055] Cognitive behavioral therapy allows patients to select their treatment items at their current level of consciousness to verify their own state of awareness, and enables them to visually experience and learn the causes of pain and physiological phenomena within a virtual space. Subsequently, it provides guidance and narration to aid in understanding the pain, thereby maximizing assistance in identifying and understanding the patient's problem.

[0056] The rights of the present invention are not limited to the embodiments described above but are defined by the claims, and it is obvious that a person skilled in the art may make various modifications and adaptations within the scope of the rights described in the claims. Explanation of the symbols

[0057] 110: Display module 120: Audio output module 121: Speaker 130: Database 140: Interface module 141: Measurement unit 142: Motion detection unit 143: Eye tracking unit 144: Input section 150: Video control module 151: Video Output Unit 152: Video Control Unit 160: Acoustic control module 170: Tactile control module 171: Seating section 172: Blower fan 173: Blower control unit 174: Fragrance spray unit 180: Sensory Control Module 181: State Analysis Unit 182: 1st Interlocking Control Unit 190: Position Control Module 191: Mobile Station 192: Second Interlocking Control Unit

Claims

Claim 1 In a digital treatment environment providing system, a display module (110) composed of three or more LED WALLs that surround the user and are formed to output images on the front, left, and right sides of the user located in the space; an audio output module (120) that outputs sound into the space; a database (130) in which video and audio content corresponding to the user's treatment purpose is stored; an interface module (140) equipped with a measurement unit (141) that collects the user's biosignals, a motion detection unit (142) that detects the user's movements, an eye tracking unit (143) that tracks the user's gaze, and an input unit (144) that receives control commands and status information from the user; an image control module (150) equipped with an image output unit (151) that generates virtual reality graphics with video content according to a control command selected by the user and outputs them to the display module (110), and an image control unit (152) that modifies the virtual reality graphics generated in correspondence with the biosignals, movements, gaze, and status information collected from the user; and audio content according to a control command selected by the user A digital treatment environment providing system characterized by comprising: a sound control module (160) that outputs through a sound output module (120); a state analysis unit (181) that analyzes biosignals and state information collected from a user to generate sensory information including the user's dizziness and fatigue, and a first linkage control unit (182) that monitors the sensory information in real time and links the image control unit (152) to control the virtual reality graphic so that the intensity of movement increases stepwise from a static image; a position control module (190) that includes a movement station (191) that moves the user's position including the interface module (140) in the forward and backward directions, and a second linkage control unit (192) that links the image control unit (152) to sequentially change the size of the virtual reality graphic output through the display module (110) in correspondence with the forward and backward movement position of the movement station (191) to synchronize the sense of perspective. Claim 2 A digital therapeutic environment providing system further comprising: a tactile control module (170) having, in claim 1, a seat portion (171) on which a user can sit, a blower fan (172) that generates airflow within the space, a blower control portion (173) that controls the blower intensity and direction of the blower fan (172) in correspondence with the virtual reality graphic, and a scent spray portion (174) that sprays a scent set to the discharge side of the blower fan (172). Claim 3 A digital therapeutic environment providing system according to claim 1, wherein the virtual reality graphics are composed of stereoscopic images that can be viewed by a user wearing 3D glasses, the sound output module (120) is composed of a plurality of speakers (121) each installed at a set location, and the sound control module (160) individually controls the sound content output to the speakers (121) and provides stereoscopic sound. Claim 4 delete Claim 5 delete

Images